https://linen.dev logo
Join Slack
Powered by
Does anyone have experience using Azure key vault ...
# best-practices
i
Does anyone have experience using Azure key vault to pass secrets (in my case, snowflake credentials) to a Meltano target profile? I'm not sure how to do this with my dbt profile either. Also, say you're deploying this to a container, do you have those same secrets configured in your local environment during development? How does this work? I also just have some other general questions related to a Dagster/Meltano/Dbt project setup
👀 1
h
We use Prefect for orchestration and I run Meltano from there, it is all dockerized, and I pick up secrets from AKV in python and pass them in as env variables when I execute the command. When developing locally, I create a temporary .env file. I know there has been talk of extending Meltano to natively support AKV or similar, I will let you know if I find something on it.
🙌 1
e
How is Meltano running in your infra? I think if you don't need to change the settings of a deployed Meltano instance, you can Link secrets from an Azure key vault.
a
do you have those same secrets configured in your local environment during development?
Yes, I keep a local .env file that has the secrets in plaintext. That is `git/Dockerignore`'d obvs, and then I make the secrets available as environmental variables in my Container App.
When I deploy to ACA I put those variables directly as secret env variables for the container. So if/when the secret gets rotated, I need to deploy a new revision in the container. I think there's a way to manage the container secrets in ACA so when they change in KV then the new value gets assiged to the container quickly. That would negate the need to deploy a new revision of the container https://learn.microsoft.com/en-us/azure/container-apps/manage-secrets?tabs=azure-portal#key-vault-secret-uri-and-secret-rotation
i
Okay awesome thank you all. A few other questions, Andy, since I've heard you mention using dagster and dbt as well - How do you get Dagster to interact with Meltano? Do you load your extractors/loaders as dagster assets similar to how dagster-dbt loads dbt models? When creating a custom tap, do you store the project for your tap in your meltano repo as well? Essentailly - how does meltano/dagster/dbt work together? with this do you just copy the project into your dockerfile and then just expose the dagster webserver? Lastly - I'm building/developing/testing most of this locally through the CLI in a venv and then I'm pip freezing my requirements to a .txt file and installing those when I build my image - does that sound standard? Thank you. @Andy Carter
a
@Ian 1. Loading dagster assets - I've got an example here https://github.com/quantile-development/dagster-meltano/issues/28#issuecomment-1655283987 2. Custom taps - I create completely separate repos and specify them via 
pip_url
3. Dockerfile - My dbt project is in my full meltano repo. With the stock dockerfile this causes issues as a single line change to a sql file prompts a full docker build to redeploy. I've found some optimisations with layers I will share below. 4. Exposing dagster webserver: On ACA my container command is as below
Copy code
{
          image: '${containerRegistry.name}.<http://azurecr.io/myorg_elt:latest|azurecr.io/myorg_elt:latest>'
          name: 'myorg_elt'
          env: dagsterEnvSettings
          command: ['meltano', 'invoke', 'dagster:dev']
          resources:{
            cpu: 2
            memory: '4Gi'
          }
And in 
meltano.yml
I have that as a custom command
Copy code
- name: dagster
    variant: quantile-development
    pip_url: dagster-ext dagster-postgres dagster-dbt dbt-postgres dagster-azure pendulum==2.1.2
    settings:
    - name: dagster_home
      env: DAGSTER_HOME
      value: $MELTANO_PROJECT_ROOT/orchestrate/dagster
    commands:
      dev:
        args: dev -f $REPOSITORY_DIR/repository.py --dagit-host 0.0.0.0 -d $REPOSITORY_DIR
        executable: dagster_invoker
Here's how I ended up with my dockerfile: https://meltano.slack.com/archives/C069CSV7NHY/p1698245332199959?thread_ts=1698241653.084859&cid=C069CSV7NHY my dbt project goes into 
meltano\orchestrate\dagster\myproj
This means small changes to .sql files don't prompt a full 
meltano install
which can take a long time.
5 Views
Open in Slack
PreviousNext
Powered by