Hello everyone, I just have a small question about the jsonl loader. I created my first extraction process but, even without any error, just this warning: "No state was found, complete import." It runs properly but doesn't create the jsonl file after the end of the block run I may have forgot something but don't think so and even with debug logs, nothing seems so out of the ordinary

What Meltano version are you using, what commands are you running and can you provide some logs?

meltano, version 2.19.0 meltano --log-level=debug run tap-github target-jsonl What part of the logs are useful because the log file is ~9700 lines long?

Yeah, good question... 😅 SInce we are just interested in the target, you can split the extract and load processes out like this:

# redirect tap stdout to file
meltano invoke tap-github > tap.out

# pipe tap stdout from file to target
cat tap.out | meltano invoke target-jsonl

See if you see anything from the second command?

the second command returns this:

[info     ] Environment 'dev' is active
{"bookmarks": {"tempStream": {}, "repositories": {"partitions": [{"context": {"org": "sbalnojan", "repo": "meltano-lightdash", "repo_id": 536571932}}]}, "commits": {"partitions": [{"context": {"org": "sbalnojan", "repo": "meltano-lightdash"}}]}, "anonymous_contributors": {}, "assignees": {}, "collaborators": {}, "commit_comments": {}, "community_profile": {}, "contributors": {}, "dependencies": {}, "dependents": {}, "events": {}, "issue_comments": {}, "issue_events": {}, "issues": {}, "labels": {}, "languages": {}, "milestones": {}, "project_cards": {}, "project_columns": {}, "projects": {}, "pull_request_commits": {}, "reviews": {}, "pull_requests": {}, "readme_html": {}, "readme": {}, "releases": {}, "extra_metrics": {}, "review_comments": {}, "stargazers": {}, "stargazers_rest": {}, "stats_contributors": {}, "traffic_clones": {}, "traffic_pageviews": {}, "traffic_referral_paths": {}, "traffic_referrers": {}, "workflow_run_jobs": {}, "workflow_runs": {}, "workflows": {}}}

So, if the stdout works properly, that means that the problem exist within the jsonl loader part?

I just tried and have

tap-github

working with

target-jsonl

fine. What does you

tap-github

config look like (excluding your auth token)?

This is why it looks like (with my auth token anonymous)

{
  "auth_token": "Shhhhhhhh, Secret",
  "repositories": [
    "sbalnojan/meltano-lightdash"
  ],
  "start_date": "2023-06-01"
}

Where are you providing your config from? I'm using environment variables in a

.env

file:

TAP_GITHUB_REPOSITORIES=["sbalnojan/meltano-lightdash"]
TAP_GITHUB_AUTH_TOKEN=xxxxx
TAP_GITHUB_START_DATE=2023-06-01

I just provided them from this command:

meltano config tap-github

Because I'm using

meltano config tap-github set --interactive

to make my config (so my auth token is in my .env and the others in my meltano.yml file

Hmm... Not sure I have an answer for this. Last thing I would check would be directory permissions maybe? If it helps, I am using

tap-github

version

1.1.1

and

target-jsonl

version

0.1.2

.

The target folder permission are 755 and is owned by me I'm gonna try to check versions next week if it can help, Sorry for the trouble I hope I will come back with a crazy idea to fix this to not bother more next week 😅 Thanks for everything

No problem, sorry I couldn't help further though. Maybe someone else will jump in with some suggestions. 🙂 Good luck!

@lucas_gambier when you followed Reuben's suggestion did you actually see any

RECORD

messages in the tap.out file? I think your start_date is too recent, the last commit for that sample repo was in 2022 https://github.com/sbalnojan/meltano-lightdash/commits/main. I see the getting started guide uses a start_date of

2022-01-01

, try setting that and see what happens

FWIW, I set the same start date (in

.env

) as @lucas_gambier and got a record for the

repositories

stream:

{
  "type": "RECORD",
  "stream": "repositories",
  "record": {
    "id": 536571932,
    "node_id": "R_kgDOH_twHA",
    "name": "meltano-lightdash",
    "full_name": "sbalnojan/meltano-lightdash",
    "private": false,
    "owner": {
      "login": "sbalnojan",
      "id": 43072233,
      "node_id": "MDQ6VXNlcjQzMDcyMjMz",
      "avatar_url": "<https://avatars.githubusercontent.com/u/43072233?v=4>",
      "gravatar_id": "",
      "html_url": "<https://github.com/sbalnojan>",
      "type": "User",
      "site_admin": false
    },
    "html_url": "<https://github.com/sbalnojan/meltano-lightdash>",
    "description": null,
    "fork": false,
    "created_at": "2022-09-14T12:41:20Z",
    "updated_at": "2022-12-13T14:55:00Z",
    "pushed_at": "2022-09-14T12:41:21Z",
    "git_url": "<git://github.com/sbalnojan/meltano-lightdash.git>",
    "ssh_url": "git@github.com:sbalnojan/meltano-lightdash.git",
    "clone_url": "<https://github.com/sbalnojan/meltano-lightdash.git>",
    "homepage": null,
    "size": 2,
    "stargazers_count": 0,
    "watchers_count": 0,
    "language": null,
    "forks_count": 0,
    "archived": false,
    "disabled": false,
    "open_issues_count": 0,
    "license": {
      "key": "mit",
      "name": "MIT License",
      "spdx_id": "MIT",
      "url": "<https://api.github.com/licenses/mit>"
    },
    "topics": [],
    "visibility": "public",
    "forks": 0,
    "open_issues": 0,
    "watchers": 0,
    "default_branch": "main",
    "network_count": 0,
    "subscribers_count": 2,
    "org": "sbalnojan",
    "repo": "meltano-lightdash"
  },
  "time_extracted": "2023-06-02T16:08:49.787882+00:00"
}

meltano.yml

version: 1
default_environment: dev
project_id: 7a3fae5c-7f28-4cc1-a976-10347fd956cb
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-github
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-github.git>
    select:
    - repositories.*
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl

Oh thats true - @lucas_gambier are you using any select criteria?

I'm not using anything more than what I posted higher

The select criteria doesnt show up in the config though, it might help if you shared your full meltano.yml with anything sensitive removed

Sorry I can't send it today anymore I'm not at work I'm gonna send it monday early to update this conv.

@lucas_gambier I think my theory about your start date is correct, can you follow my suggestion from https://meltano.slack.com/archives/C01TCRBBJD7/p1685721834994939?thread_ts=1685716862.775899&cid=C01TCRBBJD7?

@pat_nadolny Thanks, that gave me an output with the file in the correct folder, I didn't thought that could mean that it was too recent to show any commit and, because of that, wouldn't create the output file