https://linen.dev logo
Join Slack
Powered by
Hello everyone! G’Day! Thanks for your help yester...
# troubleshooting
l
Hello everyone! G’Day! Thanks for your help yesterday, I made some progress. Just met another error that I am not able to trace down to find the root cause when using 
tap-dynamodb
: 
Run invocation could not be completed as block failed: Cannot start plugin tap-dynamodb: Catalog discovery failed: command ['..../meltano-pipeline/.meltano/extractors/tap-dynamodb/venv/bin/tap-dynamodb', '--config', '..../meltano-pipeline/.meltano/run/tap-dynamodb/tap.02c4052f-bdb0-49d4-a8c3-bd19f19b0f04.config.json', '--discover'] returned 1 with stderr:
INFO Starting discover
INFO Found credentials in shared credentials file: ~/.aws/credentials
CRITICAL No streams found
Traceback (most recent call last):
File "..../meltano-pipeline/.meltano/extractors/tap-dynamodb/venv/bin/tap-dynamodb", line 8, in <module>
sys.exit(main())
File "..../meltano-pipeline/.meltano/extractors/tap-dynamodb/venv/lib/python3.8/site-packages/singer/utils.py", line 229, in wrapped
return fnc(*args, **kwargs)
File "..../meltano-pipeline/.meltano/extractors/tap-dynamodb/venv/lib/python3.8/site-packages/tap_dynamodb/__init__.py", line 103, in main
do_discover(args.config)
File "..../meltano-pipeline/.meltano/extractors/tap-dynamodb/venv/lib/python3.8/site-packages/tap_dynamodb/__init__.py", line 24, in do_discover
raise Exception("No streams found")
Exception: No streams found
Just wonder when using tap-dynamodb, is it similar to the case of tap-mongodb, regarding the configuration of config.json file (https://github.com/singer-io/tap-mongodb) ? and Is there any detailed instructions for tap-dynamodb setting up? I guess I may miss some correct configs. Many Thanks in advance! 🙏
much appreciated if anyone can provide some guidance
c
tap-dynamodb
indeed looks like one of those taps where finding the right documentation is challenging. The starting point for the configuration settings is this source code line. https://github.com/singer-io/tap-dynamodb/blob/e02d8bc02097917940d9ff23cca5a15251473e65/tap_dynamodb/__init__.py#L15
And AJ did put some notes into the issue log on the tap a while back ... https://github.com/singer-io/tap-dynamodb/issues/15
I did a quick test, and this is the minimal config:
Copy code
plugins:
  extractors:
  - name: tap-dynamodb
    variant: singer-io
    pip_url: tap-dynamodb
    config:
      account_id: "<YOURAWSACCOUNTID>"
      external_id: "NOT_REQUIRED_BUT_NEEDS_TO_HAVE_MORE_THAN_2_CHARACTERS"
      role_name: "<NameOfAnIamRoleWithSufficientAccess"
      region_name: "ap-southeast-2"
    metadata:
      '*':
        replication-method: FULL_TABLE
The tap uses boto and will pick up your personal creds from 
~/.aws/{config,credentials}
and log you in automatically.
l
Thanks mate! Just wonder can i give these settings some fake values if I only wanna test it on local dynamodb? Thanks
c
Oh. Yeah. It seems that the tap has support for local dev with a localhost endpoint. I haven't tested it with local DynamoDB, but it sounds like it should work if you add 
use_local_dynamo: True
to the config keys. https://github.com/singer-io/tap-dynamodb/blob/e02d8bc02097917940d9ff23cca5a15251473e65/tap_dynamodb/dynamodb.py#L97
l
emmmm, no luck 😅 , same err.
a
@lin_liu This should match your local dynamodb endpoint
message has been deleted
localhost:8000 It should also have tables in it
l
Thanks mate! I used this default url as well. 🤔
a
Its failing here which means it is executing 
discover_streams
to completion I suspect the 
client.list_tables(...)
is not returning anything TBH
Are you sure you have tables in your local db? or is it empty right now?
l
emmm, let me double check. Thanks heaps! quite new to this area. 🙏
updates: Thanks guys! Found the root-cause, forgot to put sharedDb opt into docker-compose file! Now I can get data from local dynamo 👍
p
I took the work you guys did and added it to the hub so its easier for the next person https://github.com/meltano/hub/pull/741. Feel free to leave comments if you see anything that I got wrong
I havent used external IDs much but I'm slightly confused by the way theyre using it, I would expect that to not be required but it is 🤷. Right?
c
@pat_nadolny The External ID config parameter value must be non-null and longer than 2 characters but does not have to be valid in any way. I couldn't see the backstory of why the tap has this external id setting in it, I suspect it was a requirement from someone who is using the tap and was just implemented in this particular way it is now, which indeed looks a bit odd.
Open in Slack
PreviousNext
Powered by