hey am i supposed to use meltano with airbyte? are...
# getting-startedj
jake
06/08/2023, 4:07 PMhey am i supposed to use meltano with airbyte? are they mutually exclusive, are the complimentary? like how does this fit in
t
taylor
06/08/2023, 4:14 PMUp to you! Meltano can run Airbyte connectors thanks to a community contribution (and how Meltano is built to be flexible). It requires docker though. Checkout https://hub.meltano.com/extractors/tap-adjust as an example of the warnings. We also link to an FAQ about the feature.
j
jake
06/08/2023, 4:17 PMcool, so im looking at the docs and seems the ui is deprecated? how do you guys visually track the pipelines
t
taylor
06/08/2023, 4:25 PMYes, the UI is deprecated. You can use Airflow to orchestrate the pipelines if you want https://hub.meltano.com/utilities/airflow . We’re also building Cloud to help with that - but we’re focused on a CLI-first experience
j
jake
06/08/2023, 4:28 PMok im trying to understand the overlap on some of things and where the strengths are, for example i use dagster for orchestration, but i like that airbyte has a ui for people to see connectors. and i use gitlab so thats how i found meltano
jake
06/08/2023, 4:28 PMif my goal is to drop data into an iceberg format in s3 is this a tool thats good for that
j
jan_soubusta
06/08/2023, 4:35 PMMy 2c: I decided to use Meltano over Airbyte, I described why in this article:
https://medium.com/gooddata-developers/extending-ci-cd-data-pipeline-with-meltano-7de3bce74f57 (chapter "Picking the Right Tool for the Use Case")
t
taylor
06/08/2023, 4:36 PMthe dagster folks wrote a nice article about dagster + meltano too https://dagster.io/blog/dagster-meltano-integration-tutorial
taylor
06/08/2023, 4:38 PMif my goal is to drop data into an iceberg format in s3 is this a tool thats good for thatI’m not actually aware of a target that writes in the Iceberg format. there’s a few S3 targets that do differen file formats, but that’s usually CSV, JSON, or Parquet. We’re working on rebuilding S3 target support based on our SDK. I don’t think it’d be too hard to add support for Iceberg cc @visch
j
jake
06/08/2023, 4:39 PM• @jan_soubusta "Simple fast CLI (Meltano) over quite heavy docker-compose consisting of multiple JVM services"
love this, one of my biggest problems with airbyte has been that it's impossible to deploy with podman
j
jan_soubusta
06/08/2023, 4:52 PMRegarding Iceberg - because Meltano adopted Airbyte connectors, could it we solved by:
https://docs.airbyte.com/integrations/destinations/iceberg/
?
j
jake
06/08/2023, 4:54 PM@jan_soubusta i havent tried it yet, im basically shopping around right now
t
taylor
06/08/2023, 4:55 PM@jan_soubusta no support for airbyte targets yet. just sources
p
pat_nadolny
06/08/2023, 5:28 PMthis target might be relevant too https://hub.meltano.com/loaders/target-iomete they write data to iceberg for their platform. I wonder if this is basically an iceberg target. I'm sure it could be tweaked too if not, although its not on the SDK so it might be better to rebuild
a
aaron_phethean
06/08/2023, 7:34 PMWe thought API and UI for meltano was a good idea so we made ours free to use https://medium.com/@danielpdwalker/a-user-interface-for-meltano-397089465cd9