Hello everyone, I am still evaluating open source ELT platform and meltano is one of them. For the architecture, I am using kubernetes and already using dagster for processing. I have few API (we are a small company... yet 🙂 ) with openapi schema and want to extract data from it and push them back some times. I am also looking Airbyte which is quite interesting but 7 services to run on k8s is too much for things like this. However it was easier to me to build my connector (thanks to Low-Code connector developpment with yaml config). https://docs.airbyte.com/connector-development/tutorials/cdk-tutorial-python-http/read-data#adding-incremental-sync Meltano feels way more light and seems to fit more my use case. There is a dagster plugin also which does the job. Maybe some minimal help for this API would help me GET /api/v2/invoices with API key https://axonaut.com/api/v2/doc I will dig a bit more the documentation and waiting for your insights 🙂

As of very recently, Meltano can run Airbyte connectors, so you should be able to run the one you already built with tap-airbyte-wrapper! https://meltano.com/blog/meltano-community-contribution-enables-running-over-200-airbyte-connectors/

Yes. I saw this and explain why I am here 🙂 but it is not clear if it is possible for custom component and how to do it ? Do I just need to create an Airbyte Image containing the required connector ? Also I am not sure how it can work in dagster if dagster will create the job.

this slackbot is a bit annoying 👀

@thomas_pedot Yes, you need to create the connector image and then reference it in your

meltano.yml

. Have a look at this part of the docs. 🙂

OK ! Thanks for your insight. It feels like, it will not work easily since dagster will be the pipeline manager (triggering the job). So with dagster meltano wrapper run inside docker zill try to trigger another one. Maybe if tap-airbyte could run directly as python. At the end, it seems that writing directly the meltano.yaml with tap-rest api is more straightforward and smaller / cheaper. I look forward to an UI ;)