Hi all,
I am trying to build an ETL pipeline using...
# randomc
charan_tej
11/06/2023, 4:27 AMHi all,
I am trying to build an ETL pipeline using meltano, but i dont want to use DBT, i have a custom python script which i want to use as a transformer, is there any guide on how to achieve this ?
e
edgar_ramirez_mondragon
11/07/2023, 12:44 AMHi!
This guide might be useful: https://docs.meltano.com/guide/advanced-topics/#running-custom-scripts
c
charan_tej
11/07/2023, 6:52 AMThanks @edgar_ramirez_mondragon for sharing the doc 👍
charan_tej
11/07/2023, 9:17 AMYeah this is working when i am running meltano run my_custom_utility properly
But can you let me know how to use it in between my extraction and loading phase
Like
meltano run my_custom_extractor my_custom_utility:run_script target-csv(loader)
e
edgar_ramirez_mondragon
11/07/2023, 5:40 PMYou'd have to, for example fork meltano-map-transform to implement the functionality you desire. The difficulty with custom implementations is you have to be more or less familiar with the Singer specification cause you need to parse Singer messages, run your transformation, and then emit new Singer messages.
c
charan_tej
11/07/2023, 5:59 PMThanks @edgar_ramirez_mondragon for the info, will try out of the above works for me
e
edgar_ramirez_mondragon
11/07/2023, 6:15 PMCool. Let me know how it goes.