Hello All fairly new to meltano Trying to load parquet files Meltano #troubleshooting

Hello All, fairly new to meltano. Trying to load p...

santosh_desai

07/04/2023, 4:16 PM

Hello All, fairly new to meltano. Trying to load parquet files to duckdb via utility dbt-duckdb.

Copy code

[0m16:05:30.440438 [error] [MainThread]: [33mRuntime Error in model state (models/state.sql)[0m
[0m16:05:30.443905 [error] [MainThread]:   Binder Error: Catalog "main" does not exist!

profiles.yml

Copy code

target: "{{ env_var('MELTANO_ENVIRONMENT', 'dev') }}"
  outputs:
    dev:
      type: duckdb
      path: ../output/duckdb/analytics.duckdb
      threads: 1
      extensions:
        - httpfs
        - parquet

sources.yml

Copy code

version: 2
sources:
  - name: parquet-source
    schema: main
    meta: 
      external_location: "{{ env_var('MELTANO_PROJECT_ROOT') }}/output/parquet/{identifier}-*.parquet"
    tables:
      - name: state
        identifier: public-address_state

models/state.sql

Copy code

{{
  config(
    materialized = "table"
  )
}}
with stg as (
SELECT * FROM {{source('parquet-source', 'state')}}
)
SELECT * from stg

Looks pretty basic, any guidance in troubleshooting what is wrong? Target compiled file runs fine on duckcli. Thanks in advance 🙏

santosh_desai

07/04/2023, 4:19 PM

The run folder contains the following state.sql

Copy code

create  table
      "main"."state__dbt_tmp"
    as (
      
with stg as (
SELECT * FROM '/Users/ganaka/bh-data/output/parquet/public-address_state-*.parquet'
)
SELECT * from stg
    );

Open in Slack

Previous Next