https://linen.dev logo
Join Slack
Powered by
Can someone please help me with this `CRITICAL dat...
# troubleshooting
a
Can someone please help me with this 
CRITICAL datetime must be pegged at UTC tzone
error? Got this couple of times, doing a full refresh seems to be solution but I can’t do a full refresh for large tables every time this happens. ``` 2022-11-30T055412.805948Z [info ] CRITICAL datetime must be pegged at UTC tzoneinfo cmd_type=extractor name=tap-dynamodb run_id=3224b6eb-46a8-4227-a738-03a90509d9d1 state_id=ddb stdio=stderr 2022-11-30T055412.806970Z [info ] Traceback (most recent call last): cmd_type=extractor name=tap-dynamodb run_id=3224b6eb-46a8-4227-a738-03a90509d9d1 state_id=ddb stdio=stderr 2022-11-30T055412.807265Z [info ] File "/home/ubuntu/dynamodb-to-bigquery/.meltano/extractors/tap-dynamodb/venv/bin/tap-dynamodb", line 8, in <module> cmd_type=extractor name=tap-dynamodb run_id=3224b6eb-46a8-4227-a738-03a90509d9d1 state_id=ddb stdio=stderr 2022-11-30T055412.807490Z [info ] sys.exit(main()) cmd_type=extractor name=tap-dynamodb run_id=3224b6eb-46a8-4227-a738-03a90509d9d1 state_id=ddb stdio=stderr 2022-11-30T055412.807692Z [info ] File "/home/ubuntu/dynamodb-to-bigquery/.meltano/extractors/tap-dynamodb/venv/lib/python3.8/site-packages/singer/utils.py", line 229, in wrapped cmd_type=extractor name=tap-dynamodb run_id=3224b6eb-46a8-4227-a738-03a90509d9d1 state_id=ddb stdio=stderr 2022-11-30T055412.807887Z [info ] return fnc(*args, **kwargs) cmd_type=extractor name=tap-dynamodb run_id=3224b6eb-46a8-4227-a738-03a90509d9d1 state_id=ddb stdio=stderr 2022-11-30T055412.808053Z [info ] File "/home/ubuntu/dynamodb-to-bigquery/.meltano/extractors/tap-dynamodb/venv/lib/python3.8/site-packages/tap_dynamodb/__init__.py", line 105, in main cmd_type=extractor name=tap-dynamodb run_id=3224b6eb-46a8-4227-a738-03a90509d9d1 state_id=ddb stdio=stderr 2022-11-30T055412.808206Z [info ] do_sync(config, args.catalog.to_dict(), args.state) cmd_type=extractor name=tap-dynamodb run_id=3224b6eb-46a8-4227-a738-03a90509d9d1 state_id=ddb stdio=stderr 2022-11-30T055412.808358Z [info ] File "/home/ubuntu/dynamodb-to-bigquery/.meltano/extractors/tap-dynamodb/venv/lib/python3.8/site-packages/tap_dynamodb/__init__.py", line 56, in do_sync cmd_type=extractor name=tap-dynamodb run_id=3224b6eb-46a8-4227-a738-03a90509d9d1 state_id=ddb stdio=stderr 2022-11-30T055412.808540Z [info ] counts[stream_name] = sync_stream(config, state, stream) cmd_type=extractor name=tap-dynamodb run_id=3224b6eb-46a8-4227-a738-03a90509d9d1 state_id=ddb stdio=stderr 2022-11-30T055412.808690Z [info ] File "/home/ubuntu/dynamodb-to-bigquery/.meltano/extractors/tap-dynamodb/venv/lib/python3.8/site-packages/tap_dynamodb/sync.py", line 62, in sync_stream cmd_type=extractor name=tap-dynamodb run_id=3224b6eb-46a8-4227-a738-03a90509d9d1 state_id=ddb stdio=stderr 2022-11-30T055412.808837Z [info ] rows_saved += log_based.sync(config, state, stream) cmd_type=extractor name=tap-dynamodb run_id=3224b6eb-46a8-4227-a738-03a90509d9d1 state_id=ddb stdio=stderr 2022-11-30T055412.808996Z [info ] File "/home/ubuntu/dynamodb-to-bigquery/.meltano/extractors/tap-dynamodb/venv/lib/python3.8/site-packages/backoff/_sync.py", line 94, in retry cmd_type=extractor name=tap-dynamodb run_id=3224b6eb-46a8-4227-a738-03a90509d9d1 state_id=ddb stdio=stderr 2022-11-30T055412.809261Z [info ] ret = target(*args, **kwargs) cmd_type=extractor name=tap-dynamodb run_id=3224b6eb-46a8-4227-a738-03a90509d9d1 state_id=ddb stdio=stderr 2022-11-30T055412.809526Z [info ] File "/home/ubuntu/dynamodb-to-bigquery/.meltano/extractors/tap-dynamodb/venv/lib/python3.8/site-packages/tap_dynamodb/sync_strategies/log_based.py", line 208, in sync cmd_type=extractor name=tap-dynamodb run_id=3224b6eb-46a8-4227-a738-03a90509d9d1 state_id=ddb stdio=stderr 2022-11-30T055412.809782Z [info ] rows_synced += sync_shard(shard, seq_number_bookmarks, cmd_type=extractor name=tap-dynamodb run_id=3224b6eb-…
s
Hm is that a singer error message @aaronsteers? It looks like https://github.com/singer-io/tap-wootric/issues/13 had the issue as well and fixed it by fixing the sync timezone to utc. If that's the case this sounds like a tap-related bug. @ken_payne
a
I've not seen this before. Agreed that those two could be related though - perhaps a library in common between the two. The fix there on 
tap-wootric
seems to have been pretty simple. Probably this requires a new issue open on 
tap-dynamodb
.
a
What confuses me most is how could it work sometimes but not every time?! 🤦
a
Is it some machines and not others? Or sometimes success or fail on the same machine?
(If some machines and not others, it could be related to timezone settings on the local OS.)
a
I have only one EC2 instance with Meltano. 😕
One thing I have noticed is that the tap syncs one or two tables correctly, and then any table with a DELETE sync, it fails. That doesn’t make sense but might be when timezone error comes. ``` 2022-12-01T114035.558155Z [info ] INFO Syncing log based for stream: PreOrder cmd_type=extractor name=tap-dynamodb run_id=1fba293b-f252-4ffc-8699-9f231e1c52d3 state_id=ddb stdio=stderr 2022-12-01T114035.751629Z [info ] CRITICAL datetime must be pegged at UTC tzoneinfo cmd_type=extractor name=tap-dynamodb run_id=1fba293b-f252-4ffc-8699-9f231e1c52d3 state_id=ddb stdio=stderr 2022-12-01T114035.752558Z [info ] Traceback (most recent call last): cmd_type=extractor name=tap-dynamodb run_id=1fba293b-f252-4ffc-8699-9f231e1c52d3 state_id=ddb stdio=stderr 2022-12-01T114035.752849Z [info ] File "/home/ubuntu/dynamodb-to-bigquery/.meltano/extractors/tap-dynamodb/venv/bin/tap-dynamodb", line 8, in <module> cmd_type=extractor name=tap-dynamodb run_id=1fba293b-f252-4ffc-8699-9f231e1c52d3 state_id=ddb stdio=stderr 2022-12-01T114035.753067Z [info ] sys.exit(main()) cmd_type=extractor name=tap-dynamodb run_id=1fba293b-f252-4ffc-8699-9f231e1c52d3 state_id=ddb stdio=stderr 2022-12-01T114035.753283Z [info ] File "/home/ubuntu/dynamodb-to-bigquery/.meltano/extractors/tap-dynamodb/venv/lib/python3.8/site-packages/singer/utils.py", line 229, in wrapped cmd_type=extractor name=tap-dynamodb run_id=1fba293b-f252-4ffc-8699-9f231e1c52d3 state_id=ddb stdio=stderr 2022-12-01T114035.753477Z [info ] return fnc(*args, **kwargs) cmd_type=extractor name=tap-dynamodb run_id=1fba293b-f252-4ffc-8699-9f231e1c52d3 state_id=ddb stdio=stderr 2022-12-01T114035.753677Z [info ] File "/home/ubuntu/dynamodb-to-bigquery/.meltano/extractors/tap-dynamodb/venv/lib/python3.8/site-packages/tap_dynamodb/__init__.py", line 105, in main cmd_type=extractor name=tap-dynamodb run_id=1fba293b-f252-4ffc-8699-9f231e1c52d3 state_id=ddb stdio=stderr 2022-12-01T114035.753900Z [info ] do_sync(config, args.catalog.to_dict(), args.state) cmd_type=extractor name=tap-dynamodb run_id=1fba293b-f252-4ffc-8699-9f231e1c52d3 state_id=ddb stdio=stderr 2022-12-01T114035.754111Z [info ] File "/home/ubuntu/dynamodb-to-bigquery/.meltano/extractors/tap-dynamodb/venv/lib/python3.8/site-packages/tap_dynamodb/__init__.py", line 56, in do_sync cmd_type=extractor name=tap-dynamodb run_id=1fba293b-f252-4ffc-8699-9f231e1c52d3 state_id=ddb stdio=stderr 2022-12-01T114035.754301Z [info ] counts[stream_name] = sync_stream(config, state, stream) cmd_type=extractor name=tap-dynamodb run_id=1fba293b-f252-4ffc-8699-9f231e1c52d3 state_id=ddb stdio=stderr 2022-12-01T114035.754491Z [info ] File "/home/ubuntu/dynamodb-to-bigquery/.meltano/extractors/tap-dynamodb/venv/lib/python3.8/site-packages/tap_dynamodb/sync.py", line 62, in sync_stream cmd_type=extractor name=tap-dynamodb run_id=1fba293b-f252-4ffc-8699-9f231e1c52d3 state_id=ddb stdio=stderr 2022-12-01T114035.754672Z [info ] rows_saved += log_based.sync(config, state, stream) cmd_type=extractor name=tap-dynamodb run_id=1fba293b-f252-4ffc-8699-9f231e1c52d3 state_id=ddb stdio=stderr 2022-12-01T114035.754843Z [info ] File "/home/ubuntu/dynamodb-to-bigquery/.meltano/extractors/tap-dynamodb/venv/lib/python3.8/site-packages/backoff/_sync.py", line 94, in retry cmd_type=extractor name=tap-dynamodb run_id=1fba293b-f252-4ffc-8699-9f231e1c52d3 state_id=ddb stdio=stderr 2022-12-01T114035.755017Z [info ] ret = target(*args, **kwargs) cmd_type=extractor name=tap-dynamodb run_id=1fba293b-f252-4ffc-8699-9f231e1c52d3 state_id=ddb stdio=stderr 2022-12-01T114035.755189Z [info ] File "/home/ubuntu/dynamodb-to-bigquery/.meltano/extractors/tap-dynamodb/venv/lib/python3.8/site-packages/tap_dynamodb/sync_strategies/log_based.py", line 208, in sync cmd_type=extractor name=tap-dynamodb run_id=1fba293b-f252-4ffc-8699-9f231e1c52d3 state_id=ddb stdio=stderr 2022…
a
Yeah, this makes a lot of sense! The calculation of 
_SDC_DELETED_AT
may be a unique codepath for the tap - and may be the source of the timezone casting bug.
a
Thanks @aaronsteers! Any idea what I should do to make it work then?
I verified that the dynamodb tap is using 
singer.utils.strftime
only for deletion; And that function is throwing the error when the DynamoDB record’s 
ApproximateCreationDateTime
(UNIX epoch time) has a mismatching 
.utcoffset()
But I just know the cause now, have no idea what to do to correct it. 🥲
k
Hey @abhishek_ajmera well done for getting to the bottom of this 👏 Its a difficult edge-case for sure! In terms of what to do next, I would recommend raising an issue on the 
tap-dynamodb
repo if you haven't already, which alerts the maintainers and the wider community of users of that tap to the bug. After that you can either fork the tap, fix the bug and rasie a Pull Request to contribute the fix, or you can try another variant of 
tap-dynamodb
which will hopefully not contain the same bug. If you go down the route of submitting a PR, you can override the pip URL of your currently install 
tap-dynamodb
to point at your fork in git, temporarily whilst your PR is reviewed upstream. If you go down the variant route, there is a list of available variants on the hub page (a good selection for DynamoDB 🚀). We have docs on adding or switching variants here.
Open in Slack
PreviousNext
Powered by