Hi Meltano team 👋 Is it possible to pre-filter datasets in the

meltano.yml

while defining column selection already (equivalent of using a

WHERE

clause in the select definition)? And for JSON columns, is it possible to only select some elements of it as columns and not the whole body already in the select definition?

Is it possible to pre-filter datasets in the

meltano.yml

while defining column selection already (equivalent of using a

WHERE

clause in the select definition)?

Pre-filtering meaning that you pull only the selected fields (i.e. "push down" filtering) vs pulling everything then filter out fields in memory? That's totally doable with the singer SDK by leveraging the Stream.mask mapping.

And for JSON columns, is it possible to only select some elements of it as columns and not the whole body already in the select definition?

Yes. This thread might help (discuss.meltano.com).

Yes, push down filtering is what I meant @edgar_ramirez_mondragon - thanks for the quick and detailed response, will give these a read 👌 thankyou