I m running into an issue with the `tap spreadsheets anywher Meltano #troubleshooting

I'm running into an issue with the `tap-spreadshee...

jacob_mulligan

05/01/2023, 2:40 PM

I'm running into an issue with the

tap-spreadsheets-anywhere

tap. Here's how I have it configured in meltano.yml:

Copy code

- name: tap-s3-ccda-documents
    inherit_from: tap-spreadsheets-anywhere
    variant: ets
    pip_url: git+<https://github.com/ets/tap-spreadsheets-anywhere.git>

    env:
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION}

    config:
      tables:
      - path: "<s3://medex-prod>"
        name: "ccda_documents"
        pattern: "BulkDataDownload/json/.*json"
        start_date: "2023-01-01T00:00:00Z"
        key_properties: []
        format: json

When I run

meltano config tap-s3-ccda-documents test

I get the following error:

Copy code

2023-05-01T14:39:09.302564Z [info     ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Plugin configuration is invalid
No RECORD message received

jacob_mulligan

05/01/2023, 2:41 PM

I have exported the following env variables:

Copy code

export AWS_ACCESS_KEY_ID=***redacted***
export AWS_SECRET_ACCESS_KEY=***redacted***
export AWS_DEFAULT_REGION=us-west-2

jacob_mulligan

05/01/2023, 2:42 PM

And here's what I get when running

invoke

to a JSON file 🤔

jacob_mulligan

05/01/2023, 2:50 PM

What's the right way to provide these env variables to

tap-spreadsheets-anywhere

, thanks!

jacob_mulligan

05/01/2023, 3:21 PM

ok, after a lot of trial and error i am past this error. Setting the env variables like this was working for me:

Copy code

environments:
- name: dev
  env:
    AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
    AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
    AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION}

Setting

env: ...

at the root wasn't working for me. The

--print-var

flag was very useful for debugging!

pat_nadolny

05/01/2023, 4:05 PM

It looks like you got it resolved already but one additional note that since you're directly mapping the names 1-1 you dont actually need to define them in the meltano.yml file. If the tap is looking for AWS_ACCESS_KEY_ID and you have it in your .env file then it will find it without having to redefine the mapping in

env

. Although its sometimes nice to be explicit like that.

jacob_mulligan

05/02/2023, 3:12 PM

i'm not using the .env file method right now.. the variables are being exported from

.zshrc

for local dev and exported by github actions for CI and our prod daily job.

jacob_mulligan

05/02/2023, 3:13 PM

should the variables be available if they're "only" exported by github actions / .zshrc? I noticed that without adding the env vars in meltano.yml that the variables were not working for me.

Open in Slack

Previous Next