Hi, I am using Meltano locally, does anyone know what could cause taps not to save states from each run? I can see the same tap is able to save its state on production, but on my machine using default system db and paths, state's are not being saved

Hi @Meisam Ebrahimi!

state's are not being saved

Maybe you're using a different state backend locally and states are not in the expected place. Otherwise how are you validating that states are saved in prod but not in your local environment?

@Edgar Ramírez (Arch.dev) The original problem I was facing has been resolved. The local environment is using the default sqlite db and the production environment is using postgresql. I validated by checking the logs of the runs and managed to get the same results locally, so that ruled out problems with my local installation. However, the state not saving/retrieving is actually affecting a custom tap I am developing. When I have this tap set up like below which is sitting locally under

extract/

directory. I can't seem to get the states to be picked up on the re-run:

plugins:
  extractors:
  - name: tap-example
    namespace: tap_example
    pip_url: -e extract/tap-example
    executable: tap-example
    config:
      accounts: x
      start_date: '2024-11-11T00:00:00Z'

When I run

meltano state get dev:tap-example-to-target-example

I get the below results:

2024-11-27 17:30:52 | INFO     | meltano.core.project | {'event': "Environment 'dev' is active", 'level': 'info', 'timestamp': '2024-11-27T17:30:52.267938Z'}
2024-11-27 17:30:52 | INFO     | meltano.core.project | {'event': "Environment 'dev' is active", 'level': 'info', 'timestamp': '2024-11-27T17:30:52.783381Z'}
2024-11-27 17:30:53 | WARNING  | meltano.cli.state | {'event': 'No plugins found for provided state_id.', 'level': 'warning', 'timestamp': '2024-11-27T17:30:53.050213Z'}
2024-11-27 17:30:53 | WARNING  | meltano.cli.state | {'event': 'No plugins found for provided state_id.', 'level': 'warning', 'timestamp': '2024-11-27T17:30:53.050213Z'}
{"singer_state": {"bookmarks": {"example_table": {"replication_key": "start_date", "replication_key_value": "2024-11-25T00:00:00+00:00"}}}}

The warnings that it cannot find the plugin is something I'm not quite getting. Are locally installed plugins problematic when it comes to states?

can you add a

capabilities

field to your custom tap:

plugins:
  extractors:
  - name: tap-example
    namespace: tap_example
    pip_url: -e extract/tap-example
    executable: tap-example
    capabilities
    - discover
    - catalog
    - state
    config:
      accounts: x
      start_date: '2024-11-11T00:00:00Z'

If that works, we have a few issues for alternative ways we can solve for the need to define these: • https://github.com/meltano/meltano/issues/7156 • https://github.com/meltano/meltano/issues/7265 • https://github.com/meltano/meltano/issues/2986 and feel free to 👍 any that would improve your experience developing a custom tap.

Ah, thank you! That solved the problem. I have been a little confused with the set up. I had already added capabilities in the meltano.yml (plugin definition one), but it seems we also have to apply it here too in the main meltano.yml where we are referencing the plugin... Will look at those issues. Thanks so much.