Hi @Stéphane Burwash I was reading up about elementary and discovered your meltano extension! I actually use the

dagster

ext, and install dbt as part of that

dagster

env to run meltano and then dbt in dagster. To put it another way, I don't have dbt as a top-level plugin in

meltano.yml

. Would I still be able to use your extension or not?

@Andy Carter I think so ... all you would need to specify in your pip url is the name of the "flavour" of elementary package you would like to download:

pip_url: elementary-data[bigquery] git+<https://github.com/potloc/elementary-ext>

If you add it in your dbt-project and download the dbt package dependencies, you should be good 😄

Thanks, I'll give it a whirl. I spotted what I think are a few typos in your blog post maybe:

Yes, version has change since I wrote the article, and thank you for the heads up 😄 I'll go correct that.

Where can I find the latest version numbers that cooperate? Elementary 0.6.6 has some issues with postgres apparently 🙂

I am getting an error:

2023-10-27 17:05:45 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Encountered an error:
Runtime Error
  Credentials in profile "elementary", target "default" invalid: Runtime Error
    Could not find adapter type *****!
Please reach out to our community for help with this issue.

In my profiles.yml

elementary:
  outputs:
    default:
      type: postgres
      host: prod-dwh-...
      user: postgres
      password: VRx7cn...
      port: 5432
      database: dwh
      schema: andy_elementary

- name: elementary
      variant: elementary
      pip_url: elementary-data[bigquery] git+<https://github.com/potloc/elementary-ext>
      executable: elementary_invoker
      settings:
        - name: project_dir
          kind: string
          value: ${MELTANO_PROJECT_ROOT}/transform/
        - name: profiles_dir
          kind: string
          value: ${MELTANO_PROJECT_ROOT}/transform/profiles/bigquery/
        - name: file_path
          kind: string
          value: ${MELTANO_PROJECT_ROOT}/utilities/elementary/report.html
        - name: skip_pre_invoke
          env: ELEMENTARY_EXT_SKIP_PRE_INVOKE
          kind: boolean
          value: true
          description:
            Whether to skip pre-invoke hooks which automatically run dbt clean
            and deps
        - name: slack-token
          kind: password
        - name: slack-channel-name
          kind: string
          value: elementary-notifs
        - name: google-service-account-path
          kind: string
        - name: gcs-bucket-name
          kind: string
        - name: days-back
          kind: string
        - name: env
          kind: string
        - name: config-dir-path
          kind: string
      commands:
        initialize:
          args: initialize
          executable: elementary_extension
        describe:
          args: describe
          executable: elementary_extension
        monitor-report:
          args: monitor-report
          executable: elementary_extension
        send-report-gcs: send-report
          --google-service-account-path ${ELEMENTARY_GOOGLE_SERVICE_ACCOUNT_PATH}
          --gcs-bucket-name ${ELEMENTARY_GCS_BUCKET_NAME}
          --update-bucket-website true
          --executions-limit 5
        send-report-slack: send-report
          --slack-token ${ELEMENTARY_SLACK_TOKEN}
          --slack-channel-name ${ELEMENTARY_SLACK_CHANNEL_NAME}
        monitor-send-report-slack: monitor send-report
          --slack-token ${ELEMENTARY_SLACK_TOKEN}
          --slack-channel-name ${ELEMENTARY_SLACK_CHANNEL_NAME}
        monitor-slack: monitor
          --slack-token ${ELEMENTARY_SLACK_TOKEN}
          --slack-channel-name ${ELEMENTARY_SLACK_CHANNEL_NAME}

      config:
        profiles-dir: ${MELTANO_PROJECT_ROOT}/transform/profiles/bigquery/
        file-path: ${MELTANO_PROJECT_ROOT}/utilities/elementary/report.html
        slack-channel-name: team-data-engineering-notifications
        google-service-account-path: ${MELTANO_PROJECT_ROOT}/.secrets/elementary-gcs.json
        gcs-bucket-name: potloc-elementary-data-storage
        skip_pre_invoke: true
        env: prod
        days-back: 1

This is an example config that I currently use

Thanks. By the way, it looks like you can pass

force

as an arg here, and the default is

False

, but project gets created either way. There's no check against

force

if that's what you intended: https://github.com/potloc/elementary-ext/blob/e62244435786a0ba74a51c0fbbe47d95ff6b1abd/elementary_ext/extension.py#L142

Have to admint I cannot even remember my intent 😅 it's been a while