DBT stuff from office hours :thread:
# plugins-dbt
v
DBT stuff from office hours ๐Ÿงต
meltano.yml
Copy code
transformers:
  - name: dbt
    pip_url: dbt-postgres==1.0.0
    namespace: dbt_postgres #Needed in order to override discovery.yml to add password
    config:
      target: postgres
      host: localhost
      port: 5432
      user: postgres
      dbname: postgres
      project_dir: '$MELTANO_PROJECT_ROOT/transform'
      PROFILES_DIR: '$MELTANO_PROJECT_ROOT/transform/profile'
      source_schema: '$MELTANO_LOAD__TARGET_SCHEMA'
      target_schema: 'autoidm'
      models: 'my_meltano_project'
    commands:
      clean:
        args: clean
        description: Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)
      compile:
        args: compile --models $DBT_MODELS
        description: Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory.
      deps:
        args: deps
        description: Pull the most recent version of the dependencies listed in packages.yml
      run:
        args: run --models $DBT_MODELS
        description: Compile SQL and execute against the current target database.
      seed:
        args: seed
        description: Load data from csv files into your data warehouse.
      snapshot:
        args: snapshot
        description: Execute snapshots defined in your project.
      test:
        args: test
        description: Runs tests on data in deployed models.
    settings:
    - name: password
      kind: password
  files:
  - name: dbt
    pip_url: git+<https://gitlab.com/meltano/files-dbt.git@config-version-2>
profiles.yml
Copy code
config:
  send_anonymous_usage_stats: False
  use_colors: True
meltano:
  target: "{{ env_var('DBT_TARGET') }}"
  outputs:
    postgres:
      type: postgres
      threads: 2
      host: "{{ env_var('DBT_HOST') }}"
      port: "{{ env_var('DBT_PORT') | int }}"
      user: "{{ env_var('DBT_USER') }}"
      pass: "{{ env_var('DBT_PASSWORD') }}"
      dbname: "{{ env_var('DBT_DBNAME') }}"
      schema: "{{ env_var('DBT_TARGET_SCHEMA') }}"
a
Awesome - thanks, @visch! Yeah, this is super helpful.
I think we should aim to make the new experience at least as smooth + explicit as you have here.
v
awesome, Great part is that Meltano already supports all of it! Just a shuffling up of where the mappings are at made things a lot easier for me, I didn't really understand how discovery.yml played into everything until I dove in here so some of this is just a lack of understanding on my end.
a
Do you have any strong feelings about calling those settings entries
user
vs
postgres_user
or
postgres_username
? I started without prefixes and then added prefixes like
postgres_
to adapter-specific settings. Then standardized on
username
and
password
since different adapters call them something different. Another option would be to just "call them what dbt calls them" as you've done here, and I think there's some merit to just passsing along the input exactly as dbt calls it.
Oh, I see you used 'password' in place of 'pass'. If we go with the exact dbt naming, that would move that one to
pass
as the setting and config entry.
p
@aaronsteers also to clarify my comments about having to open up discovery.yml to see settings is not dbt specific. I think we can improve the way that developers experience the "hidden"/"magical" settings without adding them to meltano.yml and exploding its size ๐Ÿ˜„
a
Totally.
@pat_nadolny - On the setting discovery/entry point, wdyt of us extending our JSON Schema so those could be on autocomplete?
Perhaps our JSON schema config options could also know what settings are valid for common plugin types. Although I think that actually might be difficult to setup and maintain, now that I think on it more.
v
I echo Edgar's thinking that you really only have one database target with DBT anyways so username / password alone makes sense. Although right now I duplicate the postgres pw / hostname 3 times in this meltano project (1 for tap-postgres, 1 for target-postgres, 1 for dbt) To me I don't care if it said one or the other too much (postgres_pass or dbt_pass), but it should have something like this at a minimum
Copy code
settings:
    - name: password
      kind: password
      env: tap_postgres_password
Or similar so you don't have to replicate it all over the place. The nice part is that it removed a bunch of boilerplate from the profiles.yml file. To @pat_nadolnyโ€™s point ,
meltano config dbt --list
does show all of these mappings so it's pretty good, but when I first started with Meltano and DBT I didn't' understand any of the DBT mappings (Profile, Project Dir, target, etc) or what they meant so I just let the "black magic" go. Maybe something could point folks to the list functionality? Or maybe something more crazy like
meltano config --remove_discovery_magic
which would just plop all the discovery config mappings into your meltano.yml akin to
pip freeze
I don't know throwing out more crazy ideas
p
@aaronsteers yeah I was just seeing how that would work, that could make it better but I agree our schema might start getting hard to maintain
@visch good point
config --list
might be what I'm asking for haha