Has anyone ever see or used https://github.com/sodadata/soda-sql?

I got a intro from the team (soda.io) a couple months ago. To me it felt like a SQL-first and enterprise focused version of Great Expectations. Especially with being able to write results to db and reporting.

Love me some sql-first stuff. I'll have to check it out. I'm flagging it for potential integration with meltano 🙂

I liked it because I've been a part of too many custom sql testing frameworks where that are built from scratch all the time. If not for dbt test and dbt-expectations I might lean towards that. The other benefit is that it can test against (maybe not across, not sure) different databases. I appreciate that they use the various domains of data quality.

I took some time to try it out and really enjoyed using it! I hear they have automated profiling in the works, which I’m psyched about. Some positives from my experience: • The SQL first approach is great • Your project is almost entirely YAML and it feels very lightweight. I had to do almost nothing to get up and running • It generates one query where possible for all the metrics being measured against a table instead of one per metric, making it really fast

Haven't thought about it too much, but the simple things we could do is: make it easy to integrate into any meltano project (just the files or more), make it installable w/in Meltano, and then I think eventually pull out / help with the metrics and metadata it generates.

One challenge is how can you sort of couple it within the development flow with the pipeline and ELT. The 'swivel-chair' approach to test automation sometimes leads to breakdowns. Could it somehow be a feature or extension of the other code?

It could be. I'd need to understand more about how it works. It seems like some of it is redundant to dbt tests (but obviously not all of it).

For what it's worth... https://www.dataengineeringpodcast.com/soda-data-quality-management-episode-178/

Just to start this thread again, I got a message from a member of the soda team. It seems soda core is now an open-source product which can be integrated to DBT easily @taylor I know meltano is moving in the direction of Great Expectations; as a best practice, would it be better to integrate GE directly, or does meltano accommodate soda as well?

Hey! Taylor A. Murphy (Meltano) is OOO from 2022-05-20 to 2022-05-20. Here are the folks covering while Taylor is away:

Hey @Stéphane Burwash! I definitely want us to support soda core! The integration with Great Expectations is through the “utilities” interface, which is relatively lightweight. With our 2.0 launch I expect many more utilities to be able to be added the Hub including Soda and many more. As users find value with Meltano and other tools we can build more relevant abstractions for data quality tools generally. Does that answer your question?

Yes absolutely, thank you for your answer!