Hi all I am trying to add MotherDuck to my data pipeline I s Meltano #troubleshooting

Hi all, I am trying to add MotherDuck to my data p...

jan_soubusta

09/07/2023, 5:41 PM

Hi all, I am trying to add MotherDuck to my data pipeline. I set:

Copy code

- name: target-duckdb
    variant: jwills
    pip_url: target-duckdb~=0.6
    config:
      path: "md:${MOTHERDUCK_DBNAME}"
      token: ${MOTHERDUCK_TOKEN}
      default_target_schema: ${TARGET_SCHEMA}
      add_metadata_columns: true

When I execute:

Copy code

TARGET_SCHEMA=$INPUT_SCHEMA_GITHUB meltano --environment cicd_dev_local run tap-github-repo target-duckdb --full-refresh

It fails with:

Copy code

duckdb.CatalogException: Catalog Error: Schema with name github_input_stage does not exist!

dbt-duckdb

capable of connecting to correct database and auto-create/set the default_target_schema?

edgar_ramirez_mondragon

09/07/2023, 5:56 PM

You're probably running into https://github.com/jwills/target-duckdb/issues/25. It's fixed on main, but a new release hasn't been published to PyPI, so I'm using a git ref (I should update it to point to main).

jan_soubusta

09/07/2023, 6:29 PM

You rock, thanks!

jan_soubusta

09/07/2023, 6:52 PM

Next step - dbt. Models, which contain e.g.:

Copy code

{{ config(
  schema=var('input_schema_github'),

finishes OK. Without that, the schema definition in dbt_project.yml should be used, but such models fail:

Copy code

18:46:21  Runtime Error in model aircraft (models/faa/aircraft/aircraft.sql)
18:46:21    Catalog Error: Table with name aircraft__dbt_tmp does not exist!
18:46:21    Did you mean "cicd_output_stage.aircraft__dbt_tmp"?

Similar root cause? Any thoughts?

jan_soubusta

09/07/2023, 8:52 PM

I cross-posted it to MotherDuck community, will see. It's 11pm here, will continue tomorrow.

edgar_ramirez_mondragon

09/07/2023, 8:58 PM

👀 https://motherduckcommunity.slack.com/archives/C058PJL4R0S/p1694118680451929

edgar_ramirez_mondragon

09/07/2023, 8:59 PM

My dbt setup for MD is much simpler so maybe that's why I have not come across that issue 😅

jan_soubusta

09/08/2023, 6:17 AM

Maybe. Let's wait for answer. Worst case scenario is that I add schema config to all models 😉

Open in Slack

Previous Next