Linen
Join Slack
Another issue I am having is that my remote state doesnt seem to be getting the correct ids.
# troubleshooting
r
Another issue I am having is that my remote state doesnt seem to be getting the correct ids.
I have defined a couple of schedules like the following:
Copy code
schedules:
- name: postgres-to-parquet
  interval: '@daily'

  extractor: tap-postgres-s002
  loader: target-parquet
  transform: skip
  start_date: 2024-02-01 00:00:00
  env:
    OUTPUT_FOLDER: s002

- name: salesforce-to-parquet
  interval: '@daily'
  extractor: tap-salesforce
  loader: target-parquet
  transform: skip
  start_date: 2024-02-01 00:00:00
  env:
    OUTPUT_FOLDER: salesforce

- name: coc-to-parquet
  interval: '@daily'
  extractor: tap-postgres-coc
  loader: target-parquet
  transform: skip
  start_date: 2024-02-01 00:00:00
  env:
    OUTPUT_FOLDER: coc


- name: dh-to-parquet
  interval: '@daily'
  extractor: tap-postgres-dataharbor
  loader: target-parquet
  transform: skip
  start_date: 2024-02-01 00:00:00
  env:
    OUTPUT_FOLDER: dataharbor
and I call these with the command: 
meltano --environment={staging|production} schedule run <schedule_name>
however, if I check the values that are actually being written to my remote state in azure blob storage I only see the following:
e
Hi Roberto 👋 The schedule names are used as state IDs because you're passing a schedule name to 
meltano run
. Which values did you expect?
r
i would have expected there to be a prefix with the enviornment name like: 
staging:<schedule-name>
👀 1
I might be misunderstanding these docs but it seems as though the state_id should be: 
<environment_name>:<tap_name>-to-<target_name>:<state_id_suffix>
https://docs.meltano.com/concepts/environments#state-id-suffix
Hi, not sure if this was in the documentation and I missed it, but defining my schedules like above does not save the environment prefix in the state_id, however if I define a job first like
Copy code
- name: salesforce-to-parquet
  tasks:
  - tap-salesforce target-parquet
and then reference that in the schedule block, then I do get the environment prefix.
e
Yeah, that's the conclusion I came up with just now. Referencing a 
job
does indeed result in using the 
<environment_name>:<tap_name>-to-<target_name>:<state_id_suffix>
scheme because it defers to `meltano run`: https://github.com/meltano/meltano/blob/effdf0bf6f346002951022ef60e6e98eb8b29869/src/meltano/core/schedule_service.py#L355-L361 Referencing a extractor-loader pair results in the schedule name being used as the state ID: https://github.com/meltano/meltano/blob/effdf0bf6f346002951022ef60e6e98eb8b29869/src/meltano/core/schedule.py#L111-L116 This should probably be explained in https://docs.meltano.com/reference/command-line-interface/#schedule. PRs welcome if you'd like to contribute. https://github.com/meltano/meltano/issues/8565
Got a PR to document the difference i behavior: https://github.com/meltano/meltano/pull/8566
2 Views
Open in Slack
PreviousNext
Powered by

Meltano

Meltano community

Powered by