We want your input! What other OS ELT (!) tools ha...
# best-practicesu
user
03/23/2023, 2:32 PMWe want your input! What other OS ELT (!) tools have you tried besides Meltano?
s
Sven Balnojan
03/23/2023, 2:36 PMFeel free to add whatever you experimented with 😉 @pat_nadolny @ken_payne @visch @Henning Holgersen all the other people...
Sven Balnojan
03/23/2023, 2:39 PMRudderStack: Nice to data movement (not so much move from many to one); has some pretty decent delivery guarantees (if server dies, message gets stuck etc.). Not so nice is the OS model (only parts of it truly OS).
NiFi: GUI based, pretty logical that's nice, OS in java (jikes). Model is meant for real time but really works in mini batches. That's cool for most applications, but not for anything that requires a persistent connection (e.g. event stores, etc.)
Kettle: Pretty advanced GUI including with generic REST connectors that actually work (including authentification, inheritance,...) - but Java just scares me (yes we had custom built connectors even sending emails)
p
pat_nadolny
03/23/2023, 2:47 PMI added Airflow even though its considered an orchestrator. I used it in the past for basically custom ELT pipelines. Its very similar to the DIY option but I think its different enough to deserve its own option
m
Matt Menzenski
03/23/2023, 3:08 PMWe’ve done a ton with Apache NiFi for loading client-provided data files into our platform - I hate it
Matt Menzenski
03/23/2023, 3:10 PMthat is, for our use cases it’s been way overkill, it’s hard to define custom processor groups (although we’ve done it)
s
Sven Balnojan
03/23/2023, 3:11 PMTry versioning it all and deploying it as IaC 😉 I think we needed at least three services to get it deployed, the "registry", some ldap component and NiFi itself... Oh and some bash magic to deploy our git repository to the registry to then deploy it to nifi... (and restart the flows or something like that).
h
Henning Holgersen
03/23/2023, 3:14 PMDon’t know if debezium and Kafka fits in here as E/L? Only time I have really tried Kafka was with Confluent so hardly open-source… and Debezium is Java so I’m not even touching it.
m
Matt Menzenski
03/23/2023, 3:16 PMwe have been running NiFi since mid-2016. We do now have some Terraform IaC set up, and a proper registry, but getting to this point has been such a slog
p
pat_nadolny
03/23/2023, 3:20 PM@Henning Holgersen thats a good one, I'd say yes Debezium fits.
s
Stéphane Burwash
03/27/2023, 3:29 PMDo we include stitch in TalenD?