Hi there I m just trying to get a super simple pipeline done meltano #announcements

meltano | Running transformation...
dbt | Running with dbt=0.16.1
dbt | Encountered an error while reading profiles:
dbt | ERROR Runtime Error
dbt | The profile 'meltano' does not have a target named 'target_jsonl'. The valid target names for this profile are:
dbt | - snowflake
dbt | - postgres
dbt | - bigquery
dbt | Defined profiles:
dbt | - meltano
dbt |
dbt | For more information on configuring profiles, please consult the dbt docs:
dbt |
dbt | <https://docs.getdbt.com/docs/configure-your-profile>
dbt |
dbt | Encountered an error:
dbt | Runtime Error
dbt | Could not run dbt
meltano | Transformation failed (2): Could not run dbt
meltano | ELT could not be completed: `dbt clean` failed

GitLab to postgres

Copy code

dbt |
dbt | Completed with 8 errors and 0 warnings:
dbt |
dbt | Database Error in model gitlab_group_labels (models/base/gitlab_group_labels.sql)
dbt | relation "public.group_labels" does not exist
dbt | LINE 8: select * from public.group_labels
dbt | ^
dbt | compiled SQL at ../.meltano/transformers/dbt/target/run/tap_gitlab/base/gitlab_group_labels.sql
dbt |
dbt | Database Error in model gitlab_merge_requests (models/base/gitlab_merge_requests.sql)
dbt | column "assignees" does not exist
dbt | LINE 26: assignees as assignees,
dbt | ^
dbt | HINT: Perhaps you meant to reference the column "source.assignee_id".
dbt | compiled SQL at ../.meltano/transformers/dbt/target/run/tap_gitlab/base/gitlab_merge_requests.sql
dbt |
dbt | Database Error in model gitlab_group_milestones (models/base/gitlab_group_milestones.sql)
dbt | relation "public.group_milestones" does not exist
dbt | LINE 8: select * from public.group_milestones
dbt | ^
dbt | compiled SQL at ../.meltano/transformers/dbt/target/run/tap_gitlab/base/gitlab_group_milestones.sql
dbt |
dbt | Database Error in model gitlab_issues (models/base/gitlab_issues.sql)
dbt | relation "public.issues" does not exist
dbt | LINE 8: select * from public.issues
dbt | ^
dbt | compiled SQL at ../.meltano/transformers/dbt/target/run/tap_gitlab/base/gitlab_issues.sql
dbt |
dbt | Database Error in model gitlab_project_milestones (models/base/gitlab_project_milestones.sql)
dbt | relation "public.project_milestones" does not exist
dbt | LINE 8: select * from public.project_milestones
dbt | ^
dbt | compiled SQL at ../.meltano/transformers/dbt/target/run/tap_gitlab/base/gitlab_project_milestones.sql
dbt |
dbt | Database Error in model gitlab_projects (models/base/gitlab_projects.sql)
dbt | column "shared_with_groups" does not exist
dbt | LINE 64: shared_with_groups as shared_with_groups,
dbt | ^
dbt | compiled SQL at ../.meltano/transformers/dbt/target/run/tap_gitlab/base/gitlab_projects.sql
dbt |
dbt | Database Error in model gitlab_commits (models/base/gitlab_commits.sql)
dbt | column "parent_ids" does not exist
dbt | LINE 20: parent_ids as parent_ids,
dbt | ^
dbt | compiled SQL at ../.meltano/transformers/dbt/target/run/tap_gitlab/base/gitlab_commits.sql
dbt |
dbt | Database Error in model gitlab_releases (models/base/gitlab_releases.sql)
dbt | relation "public.releases" does not exist
dbt | LINE 8: select * from public.releases
dbt | ^
dbt | compiled SQL at ../.meltano/transformers/dbt/target/run/tap_gitlab/base/gitlab_releases.sql
dbt |
dbt | Done. PASS=15 WARN=0 ERROR=8 SKIP=0 TOTAL=23
meltano | Transformation failed (1): Done. PASS=15 WARN=0 ERROR=8 SKIP=0 TOTAL=23
meltano | ELT could not be completed: `dbt run` failed

Beside that in Analysis I just get:

Copy code

The dialect 'postgres' is not yet supported by Meltano Analyze.

I guess I'm doing just some basic things wrong... Do you have hints for me? Thanks in advanced

ripe-musician-59933

02/12/2021, 4:03 PM

Hey @most-jelly-62979! Since dbt only supports data warehouses, not JSONL files, you cannot run transformations when you're using the JSONL target. If you run

meltano elt

without

--transform=run

(or with

--transform=skip

), the

tap-gitlab target-jsonl

pipeline should succeed.

👍 2

As for Postgres, the default dbt models for tap-gitlab expect all streams and properties to be selected on the tap, but it looks like you've selected a subset that doesn't include the

group_labels

and

issues

streams, and is also missing some columns on other streams. Could that be? If you're just trying to load a subset of GitLab data into Postgres, you may not need the transformations at all, or you could write your own for just the streams you actually need.

As for Analyze, I'm not sure why it's claiming

postgres

isn't supported yet, because because it should be. If you're interested in using Meltano's Analyze functionality, I'll gladly help you debug this, but be aware that the Analyze functionality (and much of the UI) is experimental, and we recommend sticking with the CLI and ELT functionality: https://meltano.com/docs/ui.html#current-status

most-jelly-62979

02/12/2021, 5:07 PM

Thanks for your detailed response! I think this will help me to fiddle out the details. I f not I'll may come back here. :)

👍 1