Hello, just learned about Meltano and very excited...
# getting-started
n
Hello, just learned about Meltano and very excited to dive in! I know this might conflict slightly with the ELT approach, but is there a way to filter data by an attribute in the Meltano extractors? Some of our sources have data we know we will never need and would prefer to just filter it out during the extract/load process rather than storing it for no reason. In some cases we really just need a small subset of the source data.
a
Hi, @nicholas_van_kuren! And welcome! We've piloted a new "stream maps" capability for Singer taps, which allows lightweight inline transformations and filtering. This is available out of box for newer SDK based taps and is coming soon as a Meltano capability for legacy taps as well. More info on the capabilities here: https://sdk.meltano.com/en/latest/stream_maps.html
n
Thanks so much AJ! I will take a look at the docs for this.