Meltano

image.png

For example, I need to extract data from a table with a filter by block_id = 1

<@U06CEA1SNSY> theres no way that I'm aware of to inject a filter into the query used for extracting the data from the database but there are a few ways to do this downstream once the data is extracted from the table:
1. let the full dataset replicate to your destination then use something like dbt to create a model i.e. view on top of that raw data that only selects your desired block_id. This is a common pattern when loading into warehouses
2. use meltano <https://docs.meltano.com/guide/mappers|inline data mapping> functionality to filter for block_id = 1 before loading into the destination

In both cases all the data in that table will be extracted so its not improving performance of the extraction. You just get to chose whether you're ok with loading the rest of the data into the warehouse then filtering or if you want it to never arrive in the warehouse at all.

Thank you! I used __filter__ function for this