Another issue I am having is that my remote state doesnt see Meltano #troubleshooting

Join Slack

Another issue I am having is that my remote state ...

# troubleshooting

Roberto Noorduijn Londono

05/31/2024, 11:25 AM

Another issue I am having is that my remote state doesnt seem to be getting the correct ids.

Roberto Noorduijn Londono

05/31/2024, 11:27 AM

I have defined a couple of schedules like the following:

Copy code

schedules:
- name: postgres-to-parquet
  interval: '@daily'

  extractor: tap-postgres-s002
  loader: target-parquet
  transform: skip
  start_date: 2024-02-01 00:00:00
  env:
    OUTPUT_FOLDER: s002

- name: salesforce-to-parquet
  interval: '@daily'
  extractor: tap-salesforce
  loader: target-parquet
  transform: skip
  start_date: 2024-02-01 00:00:00
  env:
    OUTPUT_FOLDER: salesforce

- name: coc-to-parquet
  interval: '@daily'
  extractor: tap-postgres-coc
  loader: target-parquet
  transform: skip
  start_date: 2024-02-01 00:00:00
  env:
    OUTPUT_FOLDER: coc


- name: dh-to-parquet
  interval: '@daily'
  extractor: tap-postgres-dataharbor
  loader: target-parquet
  transform: skip
  start_date: 2024-02-01 00:00:00
  env:
    OUTPUT_FOLDER: dataharbor

and I call these with the command:

meltano --environment={staging|production} schedule run <schedule_name>

however, if I check the values that are actually being written to my remote state in azure blob storage I only see the following:

Edgar Ramírez (Arch.dev)

05/31/2024, 12:58 PM

Hi Roberto 👋 The schedule names are used as state IDs because you're passing a schedule name to

meltano run

. Which values did you expect?

Roberto Noorduijn Londono

05/31/2024, 12:59 PM

i would have expected there to be a prefix with the enviornment name like:

staging:<schedule-name>

👀 1

Roberto Noorduijn Londono

05/31/2024, 1:02 PM

I might be misunderstanding these docs but it seems as though the state_id should be:

<environment_name>:<tap_name>-to-<target_name>:<state_id_suffix>

https://docs.meltano.com/concepts/environments#state-id-suffix

Roberto Noorduijn Londono

05/31/2024, 2:30 PM

Hi, not sure if this was in the documentation and I missed it, but defining my schedules like above does not save the environment prefix in the state_id, however if I define a job first like

Copy code

- name: salesforce-to-parquet
  tasks:
  - tap-salesforce target-parquet

and then reference that in the schedule block, then I do get the environment prefix.

Edgar Ramírez (Arch.dev)

05/31/2024, 2:53 PM

Yeah, that's the conclusion I came up with just now. Referencing a

job

does indeed result in using the

<environment_name>:<tap_name>-to-<target_name>:<state_id_suffix>

scheme because it defers to `meltano run`: https://github.com/meltano/meltano/blob/effdf0bf6f346002951022ef60e6e98eb8b29869/src/meltano/core/schedule_service.py#L355-L361 Referencing a extractor-loader pair results in the schedule name being used as the state ID: https://github.com/meltano/meltano/blob/effdf0bf6f346002951022ef60e6e98eb8b29869/src/meltano/core/schedule.py#L111-L116 This should probably be explained in https://docs.meltano.com/reference/command-line-interface/#schedule. PRs welcome if you'd like to contribute. https://github.com/meltano/meltano/issues/8565

Edgar Ramírez (Arch.dev)

05/31/2024, 6:24 PM

Got a PR to document the difference i behavior: https://github.com/meltano/meltano/pull/8566

3 Views

Open in Slack

Previous Next