aloof-twilight-6754109/24/2020, 8:26 PM
I can complete the full sync ELT if I break it into 3 steps:
.venv/meltano/bin/meltano elt --job_id job_directsource tap-postgres target-postgres --transform run
.venv/meltano/bin/meltano invoke tap-postgres > singer.jsonl
cat singer.jsonl | .venv/meltano/bin/meltano invoke target-postgres
I was wondering if there's something I can do to complete the ELT without having to increase the memory in the container or break it into 3 steps. Ideally I want to have log-based replication, but I am not sure if that will work with meltano-invoke.
meltano invoke dbt run --models my_meltano_project --profiles-dir $(pwd)/transform/profile/ --profile meltano
ripe-musician-5993309/24/2020, 8:50 PM
file? How much memory do the tap and target invocations use when running by themselves? Meltano keeps at most 1MB worth of Singer messages in its buffer, so I think we're either looking at a memory leak, or an inefficiently implemented tap or target.
aloof-twilight-6754109/24/2020, 9:29 PM
ripe-musician-5993309/24/2020, 9:31 PM
setting may make a difference: https://github.com/transferwise/pipelinewise-target-postgres#configuration-settings For what it's worth, I'm currently working on a feature that will let us start defaulting to the pipelinewise version of target-postgres instead of ours, for new users at least: https://gitlab.com/meltano/meltano/-/issues/2134
aloof-twilight-6754110/29/2020, 4:26 PM
ripe-musician-5993310/29/2020, 4:46 PM
aloof-twilight-6754110/30/2020, 11:53 PM
ripe-musician-5993311/02/2020, 3:52 PM