Promised article about evaluating Meltano in a CIC...
# show-us-what-you-gotj
jan_soubusta
02/09/2023, 4:23 PMPromised article about evaluating Meltano in a CICD data pipeline.
"The whole world of data is now open, eagerly awaiting me to design new analytics use cases" 😉
https://medium.com/gooddata-developers/extending-ci-cd-data-pipeline-with-meltano-7de3bce74f57
s
Sven Balnojan
02/10/2023, 8:41 AMLove the comparison chart
Sven Balnojan
02/10/2023, 8:41 AMBtw. do you mind commenting on the performance comparison? I'm just curious, not a heavy Airbyte/ Fivetran user myself 😄
Sven Balnojan
02/10/2023, 8:47 AM• @jan_soubusta btw. you can actually get rid of the external state store dependency and have it all inside your CICD pipeline using trhe SQLlite database and simply passing on the file as an artifact. You'd do it like this:
https://github.com/brooklyn-data/meltano-on-github-actions/blob/main/%7B%7B%20cook[…]ect_folder_name%20%7D%7D/.github/workflows/example-workflow.yml
And fwiw, I'm working on including this in our more prominent demos as well (complete deployment on github actions).
j
jan_soubusta
02/10/2023, 8:51 AMRegarding performance - Airbyte is very heavy consisting of multiple JVM containers, nothing you would like to run in a Gitlab workers. Of course I could deploy it persistently somewhere and just deliver a new config there. This is the next step (production-like deployment), where this cons of Airbyte won't be that important. But, from cost perspective, running Airbyte is more expensive.
Personally, I do not have much experience with Fivetran. It is basically too proprietary 😉
jan_soubusta
02/10/2023, 8:53 AMRegarding the state backend - you are correct. It is possible to pass artifacts on both Gitlab and Github between jobs as well as (at least on Gitlab) it is possible to cache such artifacts to reuse it in multiple pipeline runs.
However, I curios to test the S3 state backend anyway 😉
jan_soubusta
02/10/2023, 8:56 AMAnyone having any follow-up questions, please, attend the meetup(on Monday 13th), announced in this community as well, The discussion in broader audience can uncover other interesting ideas 😉
3 Views