Hi- this is my first question to this community. Please redirect me if a better place is available. We’re having issues of using

state.json

to incremental load tap-github. The issues I’m encounterings are: 1. The state file is stored in a timestamped folder so every time when the tap runs, there’s no state found. e.g.

2024-09-13T175856--tap-github--target-jsonl/state.json

2. Even if I provide

--state state.json

to override the default setting, I can see that all history data are being fetched, instead of the net-new

pull-requests

that I’m expecting. Maybe I’m understanding it wrong. Please let me know. Thanks!

Can you provide your meltano.yml, and what commands you're running to make your issue come up?

version: 1
default_environment: dev
project_id: 8d46d142-ee51-42fd-90fd-0f1b3a8bba5d
environments:
- name: dev
- name: staging
- name: prod
state_backend:
  uri: file:///${MELTANO_SYS_DIR_ROOT}/state/
plugins:
  extractors:
  - name: tap-github
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-github.git>
    config:
      # organizations: 
      #   - 'ThirtyMadison'
      repositories:
      - ThirtyMadison/data-eng-admin
    select:
    - pull_requests.*

command I ran: 1.

meltano el tap-github target-jsonl

2.

meltano el tap-github target-jsonl --state state/03/state.json

That helps 1. Why not use

meltano run

2. Why are you passing in a state file? Meltano handles state for y ou

1. ok I’ve just been using

meltano el

. Let me try the other. 2. the reason is what said in 1) , the state file is stored with a timestamp in it

I see now you aren't passing a state-id. iirc the logs have a message about this if you don't provide a state id. Could you post the logs from a run?

ok using run is the same thing:

First run yes, next run I'd think you'd have state?

oh I see. Using

meltano run

does change the folder name to something different now.

What folder? Can you

ls

or picture?

the first 4 are from `meltano el`:

zoom out please, tree the diretory. You're looking at the

.meltano

directory so it seems?

no, that’s my project root folder

tree

your project root please

that’s quite lot of things. I can’t send all of them.

ok how about go to your root directory and type

ls -l

The 1) issue is now resolved. The 2) issue is- even the second run, it still fetches data that I don’t expect to be net-new. e.g. logs:

2024-09-13T19:57:52.565226Z [info     ] 2024-09-13 15:57:52,564 | INFO     | tap-github           | Beginning incremental sync of 'pull_requests' with context: {'org': 'ThirtyMadison', 'repo': 'data-eng-admin', 'repo_id': 396928701}

2024-09-13T19:57:54.325915Z [info     ] 2024-09-13 15:57:54,324 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "record_count", "value": 100, "tags": {"stream": "pull_requests", "context": {"org": "ThirtyMadison", "repo": "data-eng-admin", "repo_id": 396928701}}} cmd_type=elb consumer=False job_name=dev:tap-github-to-target-jsonl name=tap-github producer=True run_id=7cc53685-c32e-49b5-838c-4af92a6faafc stdio=stderr string_id=tap-github

I'm just confused why

state

is in your main root directory is the only thing I"m left with

oh because I specified it in `meltano.yml`:

state_backend:
  uri: file:///${MELTANO_SYS_DIR_ROOT}/state/

There we go, I missed that! You could also provide no state_backend and it'd auto use the sqllite DB in

.meltano

got it. Yeah I merely just want to see it easierly.

Glad we have you working 🙂

Yes, thank you for that! Didn’t know such a big difference between

run

and

el

run

just handles the state-id for you

is that the only difference I should bear in mind?

I'd read https://docs.meltano.com/reference/command-line-interface/#run 🙂

ok fair enough. Thank you!

Hey @Tony Yun 👋, welcome to the community! I'm glad you got things sorted out. I'm curious if you were following any guides or examples of

meltano el

that were missing an explanation of how to enable incremental replication (ie

--state-id

). All fine if you found it, for example, by just exploring the CLI. > The 2) issue is- even the second run, it still fetches data that I don’t expect to be net-new. > That may be because of the At-Least-Once nature of the replication that Meltano extractors use. Happy to answer other questions.