Hi everyone 👋 I am building my first project with Meltano, and it is being great so far. Just with a some configuration and a couple of transformers I was able to extract and load data into my warehouse. Absolutely awesome! The biggest challenge I am facing so far is about testing. I have seen that I can have tests for DBT, but I wonder how to implement integration tests. Is there a way to do that through Meltano? Thank you,

Great to hear!! The fact that meltano is all code enables you to configure integration tests but right now its not necessarily something thats built into meltano itself. I chose to set up CI tests by leveraging meltano environments to configuring start dates for all of my taps using now - 2 hrs, -4 hrs, -24 hrs, etc. depending on the tap and the data volume, then in my github actions CI job I run each el job into a branch prefixed schema in my snowflake db. Theoretically once all of these run successfully I have a small sample of real data in my CI test environment, I then run my full dbt project and tests against it. If the data volume gets too big then the tests will take too long to run so theres a chance ill need to refactor it as our project grows but it works great right now.

Its essentially asserting: • all EL jobs connect and work • all dbt model succeed with the fresh EL data. If I change my EL schemas in a PR it could break my dbt queries, these tests would catch that before prod. • all dbt tests pass with the new data

One of the integration tests cases that I think would be ideal for meltano is something light in a ‘dev’ meltano project definition that is the real project but could execute in the internal product teams CI. Basically a set of expectations that the data team maintains, but the product runs and respects any failures. I.e. ‘we need this field, if this expectation is broken please fix it or tell us the impact’

Are you talking about something like this? https://github.com/calogica/dbt-expectations

Yeah. Although the dream setup requires no service dependencies to run. So that with duck db possibly.

Thank you all for the feedback. I will look into everything and will keep you posted.