https://linen.dev logo
Join Slack
Powered by
I have set up a Postgres backing DB and environmen...
# announcements
a
I have set up a Postgres backing DB and environment var to point to it, but it looks pretty empty. It did get created. Does state from extractors not get stored there?
d
@adam_roderick Can you show me the exact command you're running (including env vars and flags) that you'd expect to populate the Postgres DB but doesn't?
a
meltano elt <tap> <target>
let me grab the env var
MELTANO_DATABASE_URI=postgresql://<user>:<pass>@<host>:5432/meltano
(redacted sensitive valuees)
The meltano database does get created
And I can see that it is recording jobs in the jobs table
But I'm not sure where state is stored or how it is applied to subsequent jobs
d
@adam_roderick Do you see anything in the 
payload
column of the job table?
a
Yes--looks like state!
The ending state of the job?
d
a
Oh I think I see. I was understanding a job to be an individual run. But it looks like the state mechanics intend a job to be persistent, and to be run multiple times. Is that right?
d
The naming is a little inconsistent, I'm afraid. The Job ID is the same thing as the pipeline/schedule name, so there can be multiple runs for a job (pipeline), and what the "job" table really stores are individual runs, not individual jobs
So "it looks like the state mechanics intend a job to be persistent, and to be run multiple times" is correct, but it's easier to think about "a job" as "a pipeline"
a
So the intended/target labeling is a pipeline is a setup of a tap and target, with specific configuration and a unique ID. Pipelines can have multiple jobs, which are point-in-time runs of the pipeline?
But current labeling stores things in the job table. So job.job_id is conceptually the pipeline ID, and job.run_id is the point-in-time run ID.
d
Correct, but with the actual terminology used in the code today, "Pipelines can have multiple jobs" would be "Jobs can have multiple runs"
I think of it as "Pipelines have multiple runs" so that we can ignore the ambiguous "job" term entirely
a
I like that
d
Let me file an issue to refactor this in the future... Not a priority right now, but I want to make sure we track it
a
So the state gets applied on a new run by querying the job table for the state of the most recent run of the same pipeline, based on job.job_id?
d
Exactly
a
Thanks! Definitely not a priority right now
And the pipeline/run terminology is very clear
d
@adam_roderick To make matters worse, "pipelines" are actually stored as "schedules" in 
meltano.yml
, since they're scheduled using 
meltano schedule
. But I think "a pipeline", "a scheduled pipeline", and "a schedule" are less confusingly interchangeable than "pipelines" and "jobs" and "runs" 🙂
9 Views
Open in Slack
PreviousNext
Powered by