State will be stored in your project's system database, which lives at

.meltano/meltano.db

by default (https://meltano.com/docs/production.html#storing-metadata), so if that is getting wiped in between runs for some reason, you would also not see state picked up correctly.

Following up here @douwe_maan

id            | 3
job_id        | gitlab-to-postgres
state         | FAIL
started_at    | 2020-07-06 22:02:00.864956
ended_at      | 2020-07-06 22:37:59.551985
payload       | {"error": ""}
payload_flags | 0
run_id        | 1ba5fbe0-da8b-4ea6-b360-3a1841584dc7
trigger       | cli

This is a row that I see in the db

```2020-07-06 183759,667] {bash_operator.py:123} INFO - INFO METRIC: {"type": "counter", "metric": "record_count", "value": 103, "tags": {"endpoint": "reviews"}} [2020-07-06 183759,668] {bash_operator.py:123} INFO - INFO METRIC: {"type": "counter", "metric": "record_count", "value": 39, "tags": {"endpoint": "pull_requests"}} [2020-07-06 183759,668] {bash_operator.py:123} INFO - CRITICAL {"message":"API rate limit exceeded for user ID 2761999.","documentation_url":"https://developer.github.com/v3/#rate-limiting"} [2020-07-06 183759,668] {bash_operator.py:123} INFO - Traceback (most recent call last): [2020-07-06 183759,668] {bash_operator.py:123} INFO - File "/Users/nehiljain/snaptravel/universal-elt-pipeline/airflow-elt/.meltano/extractors/tap-github/venv/bin/tap-github", line 11, in <module> [2020-07-06 183759,668] {bash_operator.py:123} INFO - load_entry_point('tap-github==1.8.0', 'console_scripts', 'tap-github')() [2020-07-06 183759,668] {bash_operator.py:123} INFO - File "/Users/nehiljain/snaptravel/universal-elt-pipeline/airflow-elt/.meltano/extractors/tap-github/venv/lib/python3.7/site-packages/singer/utils.py", line 225, in wrapped [2020-07-06 183759,668] {bash_operator.py:123} INFO - return fnc(*args, **kwargs) [2020-07-06 183759,668] {bash_operator.py:123} INFO - File "/Users/nehiljain/snaptravel/universal-elt-pipeline/airflow-elt/.meltano/extractors/tap-github/venv/lib/python3.7/site-packages/tap_github.py", line 946, in main [2020-07-06 183759,668] {bash_operator.py:123} INFO - do_sync(args.config, args.state, catalog) [2020-07-06 183759,668] {bash_operator.py:123} INFO - File "/Users/nehiljain/snaptravel/universal-elt-pipeline/airflow-elt/.meltano/extractors/tap-github/venv/lib/python3.7/site-packages/tap_github.py", line 934, in do_sync [2020-07-06 183759,668] {bash_operator.py:123} INFO - state = sync_func(stream_schemas, repo, state, mdata) [2020-07-06 183759,668] {bash_operator.py:123} INFO - File "/Users/nehiljain/snaptravel/universal-elt-pipeline/airflow-elt/.meltano/extractors/tap-github/venv/lib/python3.7/site-packages/tap_github.py", line 602, in get_all_pull_requests [2020-07-06 183759,669] {bash_operator.py:123} INFO - for review_rec in get_reviews_for_pr(pr_num, schemas['reviews'], repo_path, state, mdata): [2020-07-06 183759,669] {bash_operator.py:123} INFO - File "/Users/nehiljain/snaptravel/universal-elt-pipeline/airflow-elt/.meltano/extractors/tap-github/venv/lib/python3.7/site-packages/tap_github.py", line 631, in get_reviews_for_pr [2020-07-06 183759,669] {bash_operator.py:123} INFO - 'https://api.github.com/repos/{}/pulls/{}/reviews'.format(repo_path,pr_number) [2020-07-06 183759,669] {bash_operator.py:123} INFO - File "/Users/nehiljain/snaptravel/universal-elt-pipeline/airflow-elt/.meltano/extractors/tap-github/venv/lib/python3.7/site-packages/tap_github.py", line 114, in authed_get_all_pages [2020-07-06 183759,669] {bash_operator.py:123} INFO - r = authed_get(source, url, headers) [2020-07-06 183759,669] {bash_operator.py:123} INFO - File "/Users/nehiljain/snaptravel/universal-elt-pipeline/airflow-elt/.meltano/extractors/tap-github/venv/lib/python3.7/site-packages/tap_github.py", line 105, in authed_get [2020-07-06 183759,669] {bash_operator.py:123} INFO - raise AuthException(resp.text) [2020-07-06 183759,669] {bash_operator.py:123} INFO - tap_github.AuthException: {"message":"API rate limit exceeded for user ID 2761999.","documentation_url":"https://developer.github.com/v3/#rate-limiting"} [2020-07-06 183759,669] {bash_operator.py:123} INFO - INFO Loading 41 rows into 'assignees' [2020-07-06 183759,669] {bash_operator.py:123} INFO - INFO COPY assignees_temp ("_sdc_repository", "id", "login", "type", "url") FROM STDIN WITH (FORMAT CSV, ESCAPE '\') [2020-07-06 183759,669] {bash_operator.py:123} INFO - INFO UPDATE 41 [2020-07-06 183759,669] {bash_operator.py:123} INFO - INFO INSERT 0 0 [2…

because the tap is throwing API limit exception the state is not being stored and hence no incremental right?

Which they clearly didn't here:

Exception: Subprocesses didn't exit cleanly: tap(1), target(0)

Is it possible for meltano to aid in this process? What I mean is, may be we can make meltano allow to add acceptable exceptions or something. The development cycle of taps is brutal so I dont know if that is the only option to fix issues like this.

You can consider using this fork: https://github.com/singer-io/tap-github/pull/63

how do I reinstall

tap-github

extractors:
  - name: tap-github
    namespace: tap_github
    pip_url: git+<https://github.com/nehiljain/tap-github.git>
    executable: tap-github
    capabilities:
    - discover
    - properties
    settings:
    - name: access_token
      env: TAP_GITHUB_ACCESS_TOKEN
    - name: repository
      env: TAP_GITHUB_REPOSITORY

I wanna change to my version of the tap

(universal-elt-pipeline) nehiljain@ip-192-168-2-75 ~/s/u/airflow-elt (master)> meltano install extractor tap-github                                                                                                                                                    (base)
Installing 1 plugins...

Installing extractor 'tap-github'...
Installed extractor 'tap-github'

this makes me feel it is not installing the tap again from my repo

also the command exits in a couple of seconds which is suspicious

meltano install extractor tap-github

will recreate it 🙂

cool