visch
02/09/2022, 5:55 PMvisch
02/09/2022, 5:55 PMtransformers:
- name: dbt
pip_url: dbt-postgres==1.0.0
namespace: dbt_postgres #Needed in order to override discovery.yml to add password
config:
target: postgres
host: localhost
port: 5432
user: postgres
dbname: postgres
project_dir: '$MELTANO_PROJECT_ROOT/transform'
PROFILES_DIR: '$MELTANO_PROJECT_ROOT/transform/profile'
source_schema: '$MELTANO_LOAD__TARGET_SCHEMA'
target_schema: 'autoidm'
models: 'my_meltano_project'
commands:
clean:
args: clean
description: Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)
compile:
args: compile --models $DBT_MODELS
description: Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory.
deps:
args: deps
description: Pull the most recent version of the dependencies listed in packages.yml
run:
args: run --models $DBT_MODELS
description: Compile SQL and execute against the current target database.
seed:
args: seed
description: Load data from csv files into your data warehouse.
snapshot:
args: snapshot
description: Execute snapshots defined in your project.
test:
args: test
description: Runs tests on data in deployed models.
settings:
- name: password
kind: password
files:
- name: dbt
pip_url: git+<https://gitlab.com/meltano/files-dbt.git@config-version-2>visch
02/09/2022, 5:55 PMvisch
02/09/2022, 5:55 PMconfig:
send_anonymous_usage_stats: False
use_colors: True
meltano:
target: "{{ env_var('DBT_TARGET') }}"
outputs:
postgres:
type: postgres
threads: 2
host: "{{ env_var('DBT_HOST') }}"
port: "{{ env_var('DBT_PORT') | int }}"
user: "{{ env_var('DBT_USER') }}"
pass: "{{ env_var('DBT_PASSWORD') }}"
dbname: "{{ env_var('DBT_DBNAME') }}"
schema: "{{ env_var('DBT_TARGET_SCHEMA') }}"aaronsteers
02/09/2022, 6:00 PMaaronsteers
02/09/2022, 6:01 PMvisch
02/09/2022, 6:03 PMaaronsteers
02/09/2022, 6:03 PMuser vs postgres_user or postgres_username ? I started without prefixes and then added prefixes like postgres_ to adapter-specific settings. Then standardized on username and password since different adapters call them something different. Another option would be to just "call them what dbt calls them" as you've done here, and I think there's some merit to just passsing along the input exactly as dbt calls it.aaronsteers
02/09/2022, 6:05 PMpass as the setting and config entry.pat_nadolny
02/09/2022, 6:05 PMaaronsteers
02/09/2022, 6:05 PMaaronsteers
02/09/2022, 6:06 PMaaronsteers
02/09/2022, 6:07 PMvisch
02/09/2022, 6:09 PMsettings:
- name: password
kind: password
env: tap_postgres_password
Or similar so you don't have to replicate it all over the place.
The nice part is that it removed a bunch of boilerplate from the profiles.yml file.
To @pat_nadolnyโs point , meltano config dbt --list does show all of these mappings so it's pretty good, but when I first started with Meltano and DBT I didn't' understand any of the DBT mappings (Profile, Project Dir, target, etc) or what they meant so I just let the "black magic" go. Maybe something could point folks to the list functionality? Or maybe something more crazy like meltano config --remove_discovery_magic which would just plop all the discovery config mappings into your meltano.yml akin to pip freeze I don't know throwing out more crazy ideaspat_nadolny
02/09/2022, 6:10 PMpat_nadolny
02/09/2022, 6:11 PMconfig --list might be what I'm asking for haha