https://linen.dev logo
Join Slack
Powered by
Hi all, I got this error when trying to parse a un...
# troubleshooting
a
Hi all, I got this error when trying to parse a unix timestamp (stored like a number) and convert it to a readable date-time format.
Copy code
2023-05-08T10:38:33.273344Z [info     ]   File "/home/ubuntu/meltano-v1/.meltano/loaders/target-bigquery/venv/lib/python3.10/site-packages/singer_sdk/helpers/_simpleeval.py", line 516, in _eval_name cmd_type=loader name=target-bigquery-ddb run_id=5cc9aaf0-5f69-4f0e-9954-d9666f5c8ce5 state_id=ddb stdio=stderr
2023-05-08T10:38:33.273544Z [info     ]     raise NameNotDefined(node.id, self.expr) cmd_type=loader name=target-bigquery-ddb run_id=5cc9aaf0-5f69-4f0e-9954-d9666f5c8ce5 state_id=ddb stdio=stderr
2023-05-08T10:38:33.273794Z [info     ] singer_sdk.helpers._simpleeval.NameNotDefined: 'datetime' is not defined for expression 'datetime.datetime.strptime(str(createdAt), "%s")' cmd_type=loader name=target-bigquery-ddb run_id=5cc9aaf0-5f69-4f0e-9954-d9666f5c8ce5 state_id=ddb stdio=stderr
The docs for Stream Maps say that these are in-built functions, but I’m unable to call them using the syntax, am I doing something wrong? 
created: datetime.datetime.from_timestamp(createdAt)
e
Hi @abhishek_ajmera! The SDK version of that target-bigquery may be outdated. Can you run 
meltano invoke target-bigquery --about --format json
to confirm the versions?
a
Sure, here it is!
Untitled.json
Looks like the SDK is a couple versions behind. https://github.com/meltano/sdk/releases/tag/v0.14.0 Any suggestion on what I can do?
e
Can you try 
meltano install loader target-bigquery
? If that won't do it, maybe look at the 
pip_url
for the plugin in 
meltano.yml
, maybe the reference is outdated. Last option is to send a PR to the plugin repo to bump the sdk dependency.
a
Okay I didn’t wanna change the bigquery target (tried switching to the default variant but it wasn’t working well with dynamodb) so I figured I can use 
meltano-map-transformer
utility. But now that the conversion is happening properly, it is still uploading it to bigquery as 
string
. Just need to figure out a way to tell bigquery that the integer converted to a datetime, not a string. 🤔
e
Oh, you can wrap it in 
int()
i.e. 
created: int(datetime.datetime.from_timestamp(createdAt))
a
TypeError: int() argument must be a string, a bytes-like object or a real number, not 'datetime.datetime'
e
oh, I see. The mapper is not doing the appropriate transformation to the data type: https://github.com/meltano/sdk/blob/33d64c8dc94d039368bee16da6ab0037e6672ff6/singer_sdk/mapper.py#L372-L379 Logging an issue…
Open in Slack
PreviousNext
Powered by