Is there a way to limit number of rows to be extracted from the taps? I don't need get all the data, just want to run a test on a sample.

Theres several open issues in the SDK discussing this feature https://github.com/meltano/sdk/issues/1333 and https://github.com/meltano/sdk/issues/1366. I want this feature too 😁! cc @ken_payne @edgar_ramirez_mondragon @taylor

When I'm in a pinch and just testing I might do

meltano invoke tap_name > out

wait a few seconds and Ctrl+C, then manually pass the data to the target with

cat out | meltano invoke target-name

Oh yeah also many taps support a start date setting so you can use that to limit the records, its just not very precise. If you set a start date of yesterday you might get a large volume for some sources. I do that in our CI pipeline tests and I ended up using a bunch of different start dates to match the tap volume as best I can

@visch to be honest, I do that too 😄

@pat_nadolny @visch SDK-based taps have the one record test, right? so users could run

meltano invoke tap-<> --test

. But the question is a configurable number of rows - seems like a nice feature for the SDK!

--test --50

I forgot about the 1 record test, yeah good point! So yeah basically that feature but in a configurable way, AJ clarified in https://github.com/meltano/sdk/issues/1333#issuecomment-1387458098 that its really X records per stream too