Meltano

Hello meltano, is it possible to use transformers that are non-sql? One of our primary goals is relying less on sql to do transforms and doing it more in code. Having custom transformers as Python code for example would be very useful.

<@U06C8S3DZ6H> Hey Cody :slightly_smiling_face:

Are you planning to transform data between extracting and loading (ETL), or after it's been loaded into the data warehouse (ELT)?

Meltano only currently supports dbt transformations, but you could of course easily put `meltano elt`  in an Airflow DAG (for example), followed by another step that connects to the same database and performs some modifications using Python.

Meltano could certainly support arbitrary Python transformations out of the box though, and they would get the benefit of pipeline environment variables (<https://meltano.com/docs/command-line-interface.html#pipeline-environment-variables>) to know what database to connect with. Would you like to create an issue to discuss that some more? :slightly_smiling_face:

Note also that dbt prides itself on being more like code than traditional SQL: (from <https://www.getdbt.com/>):
&gt; *Built for data analysts*
&gt; dbt applies the principles of software engineering to analytics code, an approach that dramatically increases your leverage as a data analyst. <https://www.getdbt.com/about|The dbt viewpoint>
&gt;  Version Control &amp; Code Review
&gt;  Automated Testing
&gt;  Sandboxing &amp; Environments
&gt;  Documentation
&gt;  Modularity
&gt;  Package Management

<@U06C8S3DZ6H> we have an orchestrator similar to airflow, and we manage our transformations separately from the `meltano elt` command. It works great the way Douwa explained it

Also, dbt is a huge step forward for SQL transformations