Hi, guys! What is the best approach for running --full-refresh for multiple scheduled connectors that already have DAG runs.

@bulky-park-65916 You have a few options: 1. Delete all rows for these schedules (with the same

job_id

) from the

job

table in the system database, so that the next time the DAG runs, there will be no previous state, and they'll automatically start from scratch 2. Manually SSH into the machine and run

meltano schedule run <name> --full-refresh

for each schedule, but first pause the DAGs in the Airflow web UI to prevent them from accidentally running while the full refresh is running, which could result in duplicate records. Once we have https://gitlab.com/meltano/meltano/-/issues/2356, pausing shouldn't be necessary anymore

I've considered option 2, but if there are 20-30 connector schedules (or more) it becomes a hassle. As for option (1), where can I find more information about how to do it? 🙂

The system database is documented here: https://meltano.com/docs/project.html#system-database In your production setup, you're probably using a separate Postgres database. If you connect to that database using

psql

or any graphical Postgres UI, you'll see a

job

table with a row for every schedule run, with a

job_id

corresponding to the schedule name, a unique

run_id

, and a

payload

containing (among other things)

singer_state

. If you delete all records for a given

job_id

there, the next schedule run will not be able to find any old singer state and will start from scratch. If you'd like to fully refresh all schedules, you can delete all rows in that table

Thanks, I'll try now 🙂

@bulky-park-65916 Are you sure you're looking at the same system database?

Well, it's the same DB. I just changed the names of the connectors and they started to fill up in the same table.

😬

By changing the name of the schedule connector works, but still that's weird 😄

No! That looks like one of Airflow's tables

😄

Meltano's should be named

job

and have

job_id

and

run_id

columns: https://meltano.slack.com/archives/CFG3C3C66/p1611935913067200?thread_ts=1611934620.063300&cid=CFG3C3C66

image.png

Those are all Airflow's

Right, let me find how to connect to Meltano's then

Look at the database_uri you've configured, probably with the

MELTANO_DATABASE_URI

env var

Not defined, perhaps it uses a default configuration?

If so, that would be the default SQLite database inside your project directory at

.meltano/meltano.db

this should be it, right?

Yeah

oh, right, sorry for the small heart attack.. 😄

Haha no worries

I think all should be fine now

Glad you figured it out

Small thing, it appears that

start_date

on the schedules level is overwritten by the extractor's default config

start_date

. As shown on picture - this date is not taken into consideration. Is this to be expected as behavior?

Yeah, that

start_date

doesn't actually do anything, it just has to be in the past: https://gitlab.com/meltano/meltano/-/issues/2529

Right, so an additional extractor would be required to setup different start_date, correct?

Each extractors should have its own

start_date

configured (under the plugin definition's

config

), but the

start_date

s under

schedules

really don't matter, they can be the same for every schedule, as long as it's in the past, because otherwise the pipeline won't run until that day

Got it 🙂