Is there a way in which I can run two dbt transforms after a Meltano #best-practices

Is there a way in which I can run two dbt transfor...

david_tout

07/27/2021, 5:30 AM

Is there a way in which I can run two dbt transforms after a pipeline? e.g. a tap specific one, and then another generic one that I want to run after every pipeline?

david_tout

07/27/2021, 7:42 AM

More accurately, I suppose I want to combine two sets of models. Some from an external package, and some from my

transform/

folder.

david_tout

07/27/2021, 7:44 AM

The scenario is I have some custom taps, and everything goes to postgres so I have

tap_A -> target_postgres

and there is a

dbt-tap-a

custom transform I also have

tap_B -> target_postgres

(on a different schedule) and a corresponding

dbt-tap-b

custom transform. After both pipelines, I want to run a custom merge step, but I'm not sure how to get Meltano/DBT to see ALL my models, and not just the ones from say

dbt-tap-a

david_tout

07/27/2021, 9:02 AM

Copy code

dbt     | Running with dbt=0.19.1
dbt     | [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
dbt     | There are 1 unused configuration paths:
dbt     | - models.my_meltano_project.ods
dbt     |
dbt     | Found 21 models, 0 tests, 0 snapshots, 0 analyses, 326 macros, 0 operations, 0 seed files, 0 sources, 0 exposures
dbt     | The selector '' does not match any nodes and will be ignored
dbt     | The selector 'my_meltano_project' does not match any nodes and will be ignored

david_tout

07/27/2021, 9:18 AM

Here's my

dbt_project.yml

Copy code

name: my_meltano_project
version: 0.0.1

profile: meltano
config-version: 2

source-paths:
  - sources
analysis-paths:
  - analysis
test-paths:
  - tests
data-paths:
  - data
macro-paths:
  - macros
snapshot-paths:
  - snapshots

clean-targets:
- ../.meltano/transformers/dbt/target
- dbt_modules
- logs

target-path: ../.meltano/transformers/dbt/target
modules-path: dbt_modules
log-path: logs

models:
  my_meltano_project:
    ods:
      +schema: ods
      materialized: merge

david_tout

07/27/2021, 9:19 AM

and

packages.yml

Copy code

packages:
  - local: dbt-tap-a
  - local: dbt-tap-b

david_tout

07/27/2021, 9:20 AM

and my folder structure is roughly:

Copy code

transform/
    dbt-tap-a/          # git submodule
    dbt-tap-b/          # git submodule
    models/
        ods/
            files.sql
            files.sql
            files.sql

edward_ryan

07/29/2021, 7:38 PM

@gunnar

Open in Slack

Previous Next