Hi everyone first time using meltano My first use case is to Meltano #getting-started

Hi everyone, first time using meltano. My first us...

miguel_angel_lobato

10/29/2021, 5:03 PM

Hi everyone, first time using meltano. My first use case is to extract data from github and load into a json file. Running on docker the following commands:

Copy code

export TAP_GITHUB_ACCESS_TOKEN='XXXX'
export TAP_GITHUB_REPOSITORY='XXXX'
export TAP_GITHUB_START_DATE='2000-01-01'

meltano init github-json
cd github-json
meltano --log-level=debug add extractor tap-github
meltano --log-level=debug add loader target-jsonl
meltano --log-level=debug elt tap-github target-jsonl

Some files are created in output by process crash by an error

miguel_angel_lobato

10/29/2021, 5:04 PM

Copy code

jsonschema.exceptions.ValidationError: {'name': 'XXXX', 'id': 3432713, 'node_id': 'XXXX=', 'slug': 'eng-leaders', 'description': '', 'privacy': 'closed', 'url': 'https
://api.github.com/organizations/XXXX/team/XXXX', 'html_url': '<https://github.com/orgs/XXXX/teams/eng-leaders>', 'members_url': '<https://api.github.com/organizations/5XXXX/team/3XXX/members{/member}>',
'repositories_url': '<https://api.github.com/organizations/5XXX/team/343XXX/repos>', 'permission': 'pull'} is not of type 'null', 'string'

miguel_angel_lobato

10/29/2021, 5:06 PM

output is as following:

Copy code

root@bbfd8fc42107:/project/github-csv# ls -la output/
total 64
drwxr-xr-x  2 root root  4096 Oct 29 17:03 .
drwxr-xr-x 11 root root  4096 Oct 29 17:02 ..
-rw-r--r--  1 root root 43025 Oct 29 17:03 team_members.jsonl
-rw-r--r--  1 root root  7315 Oct 29 17:03 team_memberships.jsonl
-rw-r--r--  1 root root  1107 Oct 29 17:03 teams.jsonl

miguel_angel_lobato

10/29/2021, 5:06 PM

Thanks!

josh_lloyd

10/29/2021, 8:23 PM

looks like your config is looking for a json string not a raw json object or dictionary

miguel_angel_lobato

10/29/2021, 8:59 PM

its a clear docker with no previous config, no volumes mounted...

josh_lloyd

10/29/2021, 9:14 PM

It’s clearly outputting some files. So I assume this error is produced in the midst of the replication of one of the streams?

josh_lloyd

10/29/2021, 9:36 PM

this would indicate that the data being extracted doesn’t match the schema provided for the stream. I don’t have much working knowledge about tap-github. But you can look at the taps’s source code and see if the data output in the error matches what exists in the schema. Might want <!subteam^S02BCD9FFEF> to take a closer look.

miguel_angel_lobato

10/30/2021, 5:26 PM

Thanks! Hi <!subteam^S02BCD9FFEF>, should I open a issue?

edgar_ramirez_mondragon

10/31/2021, 1:45 AM

Hi @miguel_angel_lobato! This looks like a bug in tap-github. In particular, it looks like somewhere a

team

object schema is incorrectly declared as

string

, so json validation from

target-jsonl

is catching it. You should be able to override that schema definition with https://meltano.com/docs/plugins.html#schema-extra to make it work, and also let the singer-io/tap-github folks know of this bug simple smile. Let me know if that solves the problem.

miguel_angel_lobato

10/31/2021, 8:01 AM

thanks!

Open in Slack

Previous Next