https://linen.dev logo
Join Slack
Powered by
I had use this command ``` meltano elt tap-gitlab ...
# getting-started
r
I had use this command
Copy code
meltano elt tap-gitlab target-postgres --transform run
following the getting started https://meltano.com/docs/getting-started.html Error Message:
Copy code
Completed with 15 errors and 0 warnings:Database Error in model gitlab_group_labels (models/base/gitlab_group_labels.sql)
t
Did you have 
--transform=run
? the 
=
is important
but maybe not since it seems like dbt actually ran
r
the command is straight away from the docs
i just pasted it
t
w/in the /transform folder do you see a /logs folder? It should show what was run
oh wow - that should be updated! the one on the homepage has 
=
r
So yeah a line from the log file
Copy code
dbt.exceptions.DatabaseException: Database Error in model gitlab_branches (models/base/gitlab_branches.sql)
  relation "none.branches" does not exist
  LINE 8:     select * from None.branches
So dbt trying to query for tables that does not exist?
Because i just got the some of the data like in the tutorial
t
the 
none
is coming from the 
schema
environment variable. That may not be getting set appropriately
The extract and load definitely got run though? You see data in postgres?
r
message has been deleted
yes i ran the EL part alone and it worked fine
these are the tables that were tapped
t
So I think running the full transforms assumes you extracted everything, which it doesn’t seem you did. So maybe that’s why schema isn’t getting set. You can run just a single dbt model by doing 
meltano invoke dbt run --models gitlab_commits
r
Copy code
Running with dbt=0.16.1
Encountered an error:
Compilation Error
  Env var required but not provided: 'DBT_TARGET'
oops
t
do you have a meltano.yml setup or are you just running the command line arguments?
r
I didnt explicitly change anything
Copy code
meltano config tap-gitlab set start_date 2020-05-01T00:00:00Z
i just ran command like those and yml just got updated
t
ah dang, b/c I guided you to use 
invoke
we ran into this bug https://gitlab.com/meltano/meltano/-/issues/2546
there’s a workaround in there but that’s pretty suboptimal
r
with the workaround i ran the single model transformation
but it still throws the None error
Copy code
Database Error in model gitlab_commits (models/base/gitlab_commits.sql)
  relation "none.commits" does not exist
  LINE 8:     select * from None.commits
t
I think if you update the 
dbt_project.yml
file to specify the schema it will work. I’m not sure why that’s not already set though. @douwe_maan?
r
yes i just updated the profiles.yml inside the transformation/
and dbt_project.yml as well
t
nice - does that work? the sql files are reading from dbt_project.yml, so that variable should be passed in
r
Copy code
meltano invoke dbt run --models gitlab_commits
running this above
Copy code
Running with dbt=0.16.1
Found 23 models, 0 tests, 0 snapshots, 0 analyses, 127 macros, 0 operations, 0 seed files, 0 sources

19:47:27 | Concurrency: 2 threads (target='postgres')
19:47:27 | 
19:47:27 | 1 of 1 START view model public.gitlab_commits........................ [RUN]
19:47:27 | 1 of 1 ERROR creating view model public.gitlab_commits............... [ERROR in 0.09s]
19:47:27 | 
19:47:27 | Finished running 1 view model in 0.51s.

Completed with 1 error and 0 warnings:

Database Error in model gitlab_commits (models/base/gitlab_commits.sql)
  relation "none.commits" does not exist
  LINE 6:     select * from None.commits
                            ^
  compiled SQL at ../.meltano/transformers/dbt/target/run/tap_gitlab/base/gitlab_commits.sql

Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
error
t
do you want to hop on a quick screenshare?
r
Oh gosh it finally woked
t
oh yay!
r
So i had to hardcode the credentials for the profiles.yml file
Do you think the core reason would be this?
Copy code
schema:   public
under
Copy code
`models:
  tap_gitlab:
    base:
      materialized: table
    transform:
      materialized: table

    vars:
      schema:   public
t
that variable should be passed into the models - is that where your tables landed in postgres?
r
inside transform/dbt_modules/dbt_project.yml
I left out the schema field blank
Yes i think you are right
t
right, the 
var('schema')
jinja call in https://gitlab.com/meltano/dbt-tap-gitlab/-/blob/master/models/base/gitlab_commits.sql#L3 will be read from dbt_project.yml. that should match the schema where your data landed in postgres
r
Oh crap, Sorry my bad
that fixed it
btw what does
Copy code
raw Carbon Intensity data
mean
t
where are you seeing that?
r
Copy code
....the tap-gitlab dbt package requires three variables, which are used for finding the tables where the raw Carbon Intensity data have been loaded during the Extract-Load phase....
t
that seems like a hold over from when the demo page used the default example of tap-carbon-intensity
r
The docs discuss about tap-gitlab
so i really didnt know what they were talking about
t
right - those docs are definitely a bit out of date (see the alert at the top of the page).
r
Yeah i get it. I should have done better
t
I don’t mean to imply you did anything wrong! We recognize we definitely need to improve our dbt integration and documentation https://gitlab.com/groups/meltano/-/epics/82
r
Thank you so much for the help.
t
My pleasure! Glad you got it sorted out
r
Its only been a week i started knowing about ELT stuff
t
Nice work 🙂
Open in Slack
PreviousNext
Powered by