Hello Team,
Got stuck while trying to use a tap th...
# plugins-generalh
harsha_aleti
09/28/2020, 10:32 PMHello Team,
Got stuck while trying to use a tap that will fetch CSV files from AWS S3. I'm using tap-spreadsheets-anywhere which is part of Meltano in-built taps. I have added it using
meltano add extractor tap-spreadsheets-anywhere$ meltano --log-level=debug invoke tap-spreadsheets-anywhere -c tap-spreadsheets-anywhere-config.jsond
douwe_maan
09/28/2020, 10:40 PM@harsha_aleti In your
meltano --log-level=debug invoke--config-cdouwe_maan
09/28/2020, 10:40 PMSince YAML is a superset of JSON, you can put the content of your manually generated config file right after
config:meltano.ymlh
harsha_aleti
09/29/2020, 12:31 AMI have added the following to `meltano.yml`using `meltano config tap-spreadsheets-anywhere set tables <name> <value>`commands. Running into an error shown in attached screenshot
harsha_aleti
09/29/2020, 12:32 AMmessage has been deleted
d
douwe_maan
09/29/2020, 1:12 AM@harsha_aleti The value of the “tables” setting needs to be an array of dictionaries, while it currently just holds a dictionary. To fix, add a “-“ just ahead of “key_properties”, like on the 3rd line in your screenshot
h
harsha_aleti
09/29/2020, 1:15 AM@douwe_maan I just figured it out by going back and forth with JSON to YAML and YAML to JSON conversion scenarios 😥
harsha_aleti
09/29/2020, 1:15 AMmessage has been deleted
harsha_aleti
09/29/2020, 1:19 AMLooks like it read through config parameters! But now it is throwing some weird error again
File "/usr/local/lib/python3.6/json/encoder.py", line 180, in default
o.class.name)
TypeError: Object of type 'datetime' is not JSON serializable
During handling of the above exception, another exception occurred:
File "/usr/local/lib/python3.6/pathlib.py", line 387, in wrapped
return strfunc(str(pathobj), *args)
FileNotFoundError: [Errno 2] No such file or directory: '/project/.meltano/run/tap-spreadsheets-anywhere/tap.config.json
d
douwe_maan
09/29/2020, 1:44 AM@harsha_aleti Wrap the start_date value in quotes, the YAML parser is currently picking it up as a datetime rather than a string.
We can make Meltano handle this better, would you like to file a bug report?
douwe_maan
09/29/2020, 1:45 AMNote also that YAML is a superset of JSON, so you can just paste the JSON in, and converting back and forth is not actually necessary :)
h
harsha_aleti
09/29/2020, 1:46 AMThanks @douwe_maan for the responses. How does the credentials get attached during this process?
harsha_aleti
09/29/2020, 1:47 AMmeltano.core.plugin.error.PluginExecutionError: Catalog discovery failed: command ['/project/.meltano/extractors/tap-spreadsheets-anywhere/venv/bin/tap-spreadsheets-anywhere', '--config', '/project/.meltano/run/tap-spreadsheets-anywhere/tap.config.json', '--discover'] returned 1: CRITICAL Unable to locate credentials
Traceback (most recent call last):
File "/project/.meltano/extractors/tap-spreadsheets-anywhere/venv/lib/python3.6/site-packages/botocore/auth.py", line 357, in add_auth
raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials
d
douwe_maan
09/29/2020, 1:56 AM@harsha_aleti That’s coming from inside tap source code, so I’m not sure! It likely indicates that you’re missing some required settings when using S3. You can review the current JSON config with “meltano config tap-spreadsheets-everywhere”, and I’d suggest double checking the README or source code with regard to AWS creds settings
douwe_maan
09/29/2020, 2:08 AM@harsha_aleti It sounds like you should add your S3 creds to the S3 URL like in the example here: https://github.com/RaRe-Technologies/smart_open#s3-credentials
h
harsha_aleti
09/29/2020, 2:24 AMThanks @douwe_maan I will look further into it.