https://linen.dev logo
Join Slack
Powered by
Hello All, For some reason my environment configur...
# troubleshooting
d
Hello All, For some reason my environment configurations are not being picked up in my meltano.yml file and was wondering if someone could help me debug as to why. My meltano configuration for environments look like this:
Copy code
# dev environment
      loaders:
        - name: target-snowflake
          config:
            database: RAW_DEV
# prod environment
        loaders:
          - name: target-snowflake
            config:
              database: RAW
My Meltano environment variable is set to prod as you can see below:
Copy code
root@dep-meltano-689ffcc4b8-d99pt:/project# echo $MELTANO_ENVIRONMENT
prod
However when I run 
meltano config target-snowflake
I get the following output:
Copy code
[2021-11-24 17:05:35,056] [43|MainThread|meltano.cli.cli] [INFO] Environment 'prod' is active
{
  "account": "*****",
  "username": "******",
  "password": "******",
  "role": "LOADER",
  "database": "RAW_DEV",
  "warehouse": "LOADING",
  "batch_size": 5000,
  "timestamp_column": "__loaded_at"
}
Is the target database for loaders stored in the system database rather than in the meltano.yml? I’m wondering why it is not referencing the prod target database.
p
hey @drew_ipson - would you mind sharing your full meltano.yml? Also I think environment variables would overwrite your meltano.yml environment configs if theyre set, do you have anything environment variables set in your shell?
d
Sure thing @pat_nadolny
```version: 1 send_anonymous_usage_stats: true project_id: 9a1b585d-349d-4004-836e-b14504a82a49 plugins: extractors: - name: tap-salesforce variant: meltano pip_url: git+https://gitlab.com/meltano/tap-salesforce.git config: api_type: BULK username: integration@halosight.com capabilities: - catalog - discover - state select: - Task.* - Account.* - User.* - Contact.* - Lead.* - Opportunity.* - OpportunityHistory.* - ForecastingQuota.* - Event.* - Case.* - Moneyball_Target__c.* - RecordType.* - Group.* - name: tap-gitlab variant: meltano pip_url: pipelinewise-tap-postgres - name: tap-metadata variant: transferwise namespace: tap_metadata pip_url: git+https://github.com/drewipson/pipelinewise-tap-postgres.git executable: tap-postgres capabilities: - catalog - discover - state settings: - name: host - name: port - name: user - name: password - name: dbname - name: ssl - name: filter_schemas - name: default_replication_method - name: tap-job-status namespace: tap_job_status pip_url: git+https://github.com/drewipson/pipelinewise-tap-postgres.git executable: tap-postgres capabilities: - catalog - discover - state settings: - name: host - name: port - name: user - name: password - name: dbname - name: ssl - name: filter_schemas - name: default_replication_method loaders: - name: target-snowflake variant: meltano pip_url: git+https://gitlab.com/meltano/target-snowflake.git config: account: ****** role: LOADER username: ***** warehouse: LOADING environments: - name: prod config: extractors: - name: tap-metadata config: dbname: ******* default_replication_method: FULL_TABLE filter_schemas: public host: ******** port: "5432" ssl: "false" user: ****** start_date: 2020-12-21 020839.212793 select: - public-job_descriptor.* - public-job_field_descriptor.* - public-job_query_descriptor.* - public-organization.* - public-parent_account.* - public-parent_account_user.* metadata: public-job_descriptor: replication-method: INCREMENTAL replication-key: updated_at updated_at: is-replication-key: true public-job_field_descriptor: replication-method: INCREMENTAL replication-key: updated_at updated_at: is-replication-key: true public-job_query_descriptor: replication-method: INCREMENTAL replication-key: updated_at updated_at: is-replication-key: true public-oauth_info: replication-method: INCREMENTAL replication-key: updated_at updated_at: is-replication-key: true public-organization: replication-method: INCREMENTAL replication-key: updated_at updated_at: is-replication-key: true public-parent_account: replication-method: INCREMENTAL replication-key: updated_at updated_at: is-replication-key: true publ…
I also have MELTANO_PROJECT_READONLY set to TRUE
p
The only thing I noticed is that you have
Copy code
environments:
  - name: prod
    config:
        extractors:
versus the expected with `plugins`:
Copy code
environments:
- name: prod
  config:
    plugins:
      extractors:
that might do it although im not sure where it would be getting the database from then. When you run this you dont get anything, right?
Copy code
echo $TARGET_SNOWFLAKE_DATABASE
d
@pat_nadolny It is blank
p
did adding 
plugins
fix it?
d
@pat_nadolny That did it — thank you for your help!!
2 Views
Open in Slack
PreviousNext
Powered by