I want to extract limited data(say first 1000 rows...
# announcementsb
boopathy_raja_thirugnanasigha
03/21/2021, 6:59 AMI want to extract limited data(say first 1000 rows) from the database in the ELT. Is there a way to do it??
v
visch
03/21/2021, 2:22 PMThe tap would have to support that, I'm not sure how common or uncommon it is.
t
taylor
03/22/2021, 2:42 PM@boopathy_raja_thirugnanasigha are you trying to limit the amount for testing or for another purpose?
b
boopathy_raja_thirugnanasigha
03/22/2021, 2:43 PM@taylor testing only!
boopathy_raja_thirugnanasigha
03/22/2021, 2:44 PMAlso is there any possible, I can filter out data of certain period of time alone
t
taylor
03/22/2021, 2:47 PMCheckout https://gitlab.com/meltano/meltano/-/issues/2640 and some of the linked issues. We don’t have a `--test`flag in yet but it’s on our radar. To your 2nd point, we generally make the assumption that you want to get most all of the data in. You can set a backfill date on some taps but there’s currently no filter option like you’re asking for. We are thinking about some generic plugin that could be used in between the extract and load steps
b
boopathy_raja_thirugnanasigha
03/22/2021, 2:49 PMThank U @taylor
a
arnis_dizdari
04/02/2021, 3:58 PM@boopathy_raja_thirugnanasigha Hey I ran into this same issue, if your tap supports a --state, generate a state.json file and pass it thru. Then your tap will only pull from a sub set that you defined in the state.
4 Views