Hi everyone! Trying to move meltano - airflow - k8s to production and I was wondering about building an strong CI/CD around it. In that part I would like to know your ideas about what kind of tests do you run before pushing to prod Also if you've any example projects to share on how to perform any type of tests (integration, unit...) that would be awesome! thank you !

@javi_cortes did you already check out our Meltano internal deployment? https://github.com/meltano/squared • the deployment process (in github actions): https://github.com/meltano/squared/blob/main/.github/workflows/prod_deploy.yml • testing on code changes: https://github.com/meltano/squared/blob/main/.github/workflows/test.yml

My comments assume youre doing the common EL + dbt: We decided to do end to end testing (EL + dbt + reverse ETL) on small subsets of each EL source stream. This means that the data in the warehouse during tests is a fraction of what it is in prod, which has pros and cons, its faster but could miss some assertions because the data volume/variety isnt large enough. I could see an alternative approach where you test EL and staging, to assert EL changes dont break your assumptions in your dbt models (i.e. dropping columns or renames that break dbt staging logic), then you could run dbt using the slim CI approach with defer/state. For me the challenge is always that CI runtimes with dbt grow as data grows so our "subset" approach has been pretty good at limiting that.

Hey @Sven Balnojan - thanks for your direction to the Meltano internal deployment CI/CD in github, very useful. A question about the success criteria for the tests: as far as I can tell the Extract-Load test is a success as long as you get to this line and it runs (basically the

run

command does not fail), is that correct? I.e. there is no test on the output like checking the expected records are loaded or the replication method works as expected? https://github.com/meltano/squared/blob/1366021ae1ca824856bf682cd2f1216af9be033a/.github/workflows/test.yml#L119 Cheers!

@jamie_bamforth jup you're right. And I recommend that way of setting it up. Basically, you want to have tests covering different areas. These ones are for what you asked for, checking that you didn't mess anything up on deployment, upgrading some Python version, the k8n cluster, a tap version etc etc. etc. They are not about the data. But the next step is 🙂 The next step runs dbt, and then dbt tests. (You could also use great expectations or something like that to test that the data replication etc works) For the data,

Ah, nice! I just assumed the dbt tests were intended to test some form of post-load aggregation and not the load itself. Thanks for clarifying. Great workflow for isolating the two different sources of failure.

Yeah no the idea is that you can simply use dbt tests on your raw staged data. + You then run the downstream models to make sure no weird things sneaked into (like a string that's too long, a new category in something categorical etc.etc.) . But all of that just runs on one days worth of data (and you could even make that time window smaller).

To chime in here again, that matrix run step does load EL output to snowflake where it’s used by the next set of dbt tests. The dbt tests assert that the data looks as expected and if EL didn’t work then dbt will fail because the source tables are missing. So although we’re not explicitly “unit” testing some of the things you mentioned we do have “integration” test coverage