And another best practice request: Is it possible...
# best-practices
a
And another best practice request: Is it possible to use meltano inhert/override mechanics to add arbitrary env variables to airflow? Use case: i want to provide to DAG different db connect credentials through custom variable in
dev
and
prod
environments.
a
As of now, Meltano does accept an env: {} mapping in each environment. We are also working on having a plugin-level env: {} config that also be overrideable per environment but would send those env configs only to airflow and not to other plugins in the environment.
a
Ideally, DAGs would be first class citizens in meltano, because not everything can be done with DBT or tap/target pair.
(just a thought)
@aaronsteers it seems that there is some sort of inconsistency in interpolation of
env
variables. For example this interpolation would work in
config
, but not in
env
This works:
Copy code
config:
          default_target_schema: dev_${USER}_${MELTANO_EXTRACT__LOAD_SCHEMA}
This does not (variable is substituted to empty value):
Copy code
env:
    DAG_PAYPAL_BQ_TABLE_ID: project-xxx.dev_${USER}_raw.paypal_invoices
USER
is defined in
.env
file
a
@andrey_tatarinov - Yes, that sounds correct for where we are at as of now. This is something we're working on actively right now. We'd like to allow interpolation at any level.
cc @cody_hanson
c
👋 @andrey_tatarinov just to echo @aaronsteers, this is in my current list of issues for this iteration. Here's the relevant issue with info on the desired behavior, feel free to chime in there with any additional details about your use case.
a
Nice, thanks!