Hello World! I am new to Meltano - it seems awesome so far! But I am trying to do something I don't know how. I am using

tap-klaviyo

to fetch data and load into

target-postgres

. It works fine, but we have many millions of rows in the "event" table, so I would like to run meltano in batches. Only fetching few thousands of records per run. How can a approach this problem? Thank you!!

Hi @Filip Vesely! I'm curious what's your use case. Many taps do support incremental replication, and the

events

stream is sorted, so replication can be safely interrupted and it will start from the stored bookmark. Some related issues and PRs: • https://github.com/meltano/meltano/pull/8364 • https://github.com/meltano/meltano/issues/8716

Hi @Edgar Ramírez (Arch.dev), thanks for getting back. The use-case with Klaviyo data source is that they have API limits, and we have 100+ millions of rows in one of their table. So it would take days (or weeks) of constant meltano process to run to fetch all the data into our postgrtes DB. So I want to run it in chunks, in case soemething happens. When I tryied to terminate meltano process after a while of running, it simply didn't loaded any data into our DB and started from scratch next time I started it again. I will check the links you provided, thanks.

When I tryied to terminate meltano process after a while of running, it simply didn't loaded any data into our DB and started from scratch next time I started it again.

Hmm. Might be worth setting https://hub.meltano.com/loaders/target-postgres/#batch_size_rows-setting to a small value to confirm that data is flowing. Also, which command did you run?

meltano run

handles incremental replication automatically, but

meltano el

requires you to specify a

--state-id

.

OK batch_size_rows settings is feeding data into target. But when I terminate the

meltano run

(or even

meltano el

), it doesn't save any state and next time it runs from the beginning.

can you check the state contents with

meltano state list

and

meltano state get <state-id>

? If it's empty, then it's probably a bug.

Yeah, no state is created when I terminate process (by

Ctrl + C

). However, I am able to provide a custom state to the

el

command, where I set the most recent replication key value. So I am kinda able to do what I want. But it's a bit hacky.