Dear all, in this Medium article I put together my experience in evaluating the open source Meltano as ELT framework (in which exercise many of you helped and I am truly grateful). Please feel free to comment, find flaws in my writing, and propose solutions to problems if my solutions are imperfect. Especially @Edgar Ramírez (Arch.dev) and @Steve Clarke may be interested. https://medium.com/@pp_85623/meltano-in-action-hands-on-evaluation-of-an-open-source-elt-framework-5a1d5b93b483

However, I later got into numerous dependencies conflicts. The conflicts were not caused by Meltano itself, but by taps and targets, some of which required different versions of dependent libraries. I had to reinstall several time and finally found that if I wanted both Postgre and Oracle taps and targets, the problems will be gone if using Python 3.9. Here is the final, modified installation sequence that worked.

We should probably do a better job advertising https://docs.meltano.com/reference/settings/#python and https://docs.meltano.com/reference/settings/#project-python.

Copy code

.meltano/loaders/target-postgres/venv/bin/pip install psycopg

Hmm this really shouldn't have been necessary. I'll see if I can repro it. target-oracle has a small bug: https://github.com/MeltanoLabs/target-oracle/pull/2

Unrecognized replication_method None

the tap should probably default to

FULL_TABLE

to avoid this error

I think an ELT process should be resilient to potential source data errors, which means it should somehow handle errors and run even if source data was faulty, however I did not find an easy way to achieve it.

Noted. Meltano (loaders) should support some kind of dead letter queue for problematic messages. I'll start a GH discussion and maybe spin off a few concrete issues. Thanks for sharing!