https://meltano.com/ logo
Channels
Powered by
#announcements
Title
# announcements
p

powerful-businessperson-37571

03/30/2021, 8:24 PM
Hey there! I am using docker to run ELT’s and I think I have everything set up correctly. I am using tap-github and target-sqlite. I run 
docker-compose run meltano elt tap-github target-sqlite --job_id=github-to-sqlite
and it looks like everything went fine, but it runs super fast and no data shows up in the sqlite db.
p

proud-pillow-55935

03/30/2021, 8:34 PM
Do you have those taps and targets configured?
r

ripe-musician-59933

03/30/2021, 8:36 PM
@powerful-businessperson-37571 Can you share any output you see?
p

powerful-businessperson-37571

03/30/2021, 8:37 PM
Copy code
➜  testing_initative git:(master) ✗ docker-compose run meltano elt tap-github target-sqlite --job_id=github-to-sqlite
Creating meltano-projects_meltano_run ... done
meltano       | Running extract & load...
meltano       | Found state from 2021-03-30 20:01:45.535658.
meltano       | No state was found, complete import.
tap-github    | INFO Starting sync of repository: tactivos/murally
meltano       | Incremental state has been updated at 2021-03-30 20:20:14.065573.
meltano       | Extract & load complete!
meltano       | Transformation skipped.
Copy code
version: 1
send_anonymous_usage_stats: true
project_id: bcba3a22-6342-48fd-9647-c73b7a598a18
plugins:
  extractors:
  - name: tap-github
    namespace: tap_github
    pip_url: tap-github
    executable: tap-github
    capabilities:
    - catalog
    - discover
    - properties
    - state
    settings:
    - name: access_token
    - name: repository
    config:
      access_token: TAP_GITHUB_ACCESS_TOKEN
      repository: tactivos/murally
    select:
    - pull_requests.id
    - pull_request_reviews.state
    - pull_request_reviews.commit_id
    - projects.id
    - projects.url
    - review_comments.base
    - review_comments.pull_request_review_id
    - review_comments.user
    - reviews.submitted_at
    - reviews.user
  loaders:
  - name: target-sqlite
    variant: meltano
    pip_url: git+<https://gitlab.com/meltano/target-sqlite.git>
I believe I have the taps and targets configured because I can 
invoke
them per the tutorial.
I used the CLI to do the whole thing 🙂
r

ripe-musician-59933

03/30/2021, 8:39 PM
Can you run again in debug mode (https://meltano.com/docs/command-line-interface.html#debugging) with 
meltano --log-level=debug elt
to see if any RECORD messages actually make it out of the tap?
p

powerful-businessperson-37571

03/30/2021, 8:39 PM
Sure
Holy crap. That’s a ton of information. One second
r

ripe-musician-59933

03/30/2021, 8:40 PM
Not all of it will be relevant
Look for lines starting with 
tap-github (out)
p

powerful-businessperson-37571

03/30/2021, 8:42 PM
r

ripe-musician-59933

03/30/2021, 8:42 PM
OK, so the tap only creates this output:
Copy code
tap-github (out)    | [0m{"type": "STATE", "value": {}}
And not any schemas or records
p

powerful-businessperson-37571

03/30/2021, 8:42 PM
So here’s the thing
My state is recent because I’ve run this like 5 times
Where is the state stored?
r

ripe-musician-59933

03/30/2021, 8:43 PM
This part is wrong, by the way:
Copy code
capabilities:
    - catalog
    - discover
    - properties
    - state
You should have either catalog or properties, not both 🙂 It looks like tap-github uses properties: https://github.com/singer-io/tap-github/blob/master/tap_github/__init__.py#L982
Fixing that may just fix the whole thing
Where is the state stored?
In the system DB: https://meltano.com/docs/project.html#system-database
p

powerful-businessperson-37571

03/30/2021, 8:44 PM
Hold on deleting that bc I saw some data in there will recreate
Ok
Oh that’s a better error
Copy code
meltano       | Incremental state has been updated at 2021-03-30 20:45:14.831306.
meltano       | Extraction failed (1): requests.exceptions.HTTPError: 410 Client Error: Gone for url: <https://api.github.com/repos/tactivos/murally/projects?sort=created_at&direction=desc>
meltano       | ELT could not be completed: Extractor failed
ELT could not be completed: Extractor failed
ERROR: 1
r

ripe-musician-59933

03/30/2021, 8:45 PM
All right we're getting somewhere
But I have no idea why the GH API would be returning 410
p

powerful-businessperson-37571

03/30/2021, 8:47 PM
hmmm
OK just cycled my access token
Those logs are DANGEROUS
😆 😢
r

ripe-musician-59933

03/30/2021, 8:48 PM
Heh, you can turn of debug mode now
Any errors will be logged anyway
p

powerful-businessperson-37571

03/30/2021, 8:48 PM
Yes
Maybe that URL is an old API
but projects doesn’t make sense in that context
Oh now it’s working
I removed that
r

ripe-musician-59933

03/30/2021, 8:50 PM
yay!
p

powerful-businessperson-37571

03/30/2021, 8:50 PM
Sooooo @ripe-musician-59933 what is this going to download? Everything from the beginning of time?
r

ripe-musician-59933

03/30/2021, 8:51 PM
Hmm, taps typically have a start_date setting but I don't see one
p

powerful-businessperson-37571

03/30/2021, 8:51 PM
uh oh lol 🙂
Yeah it’s still going and we have a gazillion PR’s and commits in our repo
r

ripe-musician-59933

03/30/2021, 8:52 PM
Yeah I think it's going all the way back by default
I wonder if one of the forks supports a start_date: https://github.com/singer-io/tap-github/network
p

powerful-businessperson-37571

03/30/2021, 8:52 PM
There’s no other option probably
r

ripe-musician-59933

03/30/2021, 8:53 PM
With an entry for every repo and stream
And pass it into 
meltano elt
using 
--state state.json
p

powerful-businessperson-37571

03/30/2021, 8:54 PM
aha
Well that worked lol
Copy code
drwxr-xr-x 3 root root      96 Mar 30 20:55 .
drwxr-xr-x 1 root root    4096 Mar 30 20:55 ..
-rw-r--r-- 1 root root 2600960 Mar 30 20:54 testing_initiative.db
r

ripe-musician-59933

03/30/2021, 8:55 PM
Nice
p

powerful-businessperson-37571

03/30/2021, 8:55 PM
The initial one. I cancelled it
Thanks @ripe-musician-59933 this looks like it’s going to work swimmingly
r

ripe-musician-59933

03/30/2021, 9:02 PM
Awesome!
Powered by