Hi, I use tap-mysql and target-athena.
Tap send me...
# troubleshootinga
alexandre_brehelin
03/07/2022, 1:55 PMHi, I use tap-mysql and target-athena.
Tap send message like this json:
{"type": "RECORD", "stream": "schema-table_name", "record": {"id": 667, "active": true}, "version": 1, "time_extracted": "2022-03-07T093801.198348Z"}
Then, during target step I get an error because athena don't success create table with name format like schema-table_name.
Do you know a workaround to create table inside athena with only the table_name from my tap-mysql ?
Thank you.
p
pat_nadolny
03/07/2022, 2:36 PMHey @alexandre_brehelin - If I understand correctly, the problem is that the target isnt cleaning (i.e. converting the hyphen to an underscore) the stream name to something thats acceptable by Athena. In my view thats the responsibility of the target and would be considered a bug. A work around is to use stream maps to alias the stream like I did in this example https://gitlab.com/meltano/squared/-/blob/36b1ad4066e635ba814916a186baebbbc8f5e723/data/meltano.yml#L168. Or you could use the new mapper plugins feature that was just released which has similar stream manipulation features
a
alexandre_brehelin
03/07/2022, 3:47 PMHi Pat, you well undertstand our issue. Stream maps seems to be a good workaround except I have to fill mapping for the whole tap source table. Do you think I have to open an issue in the target-athena repository ?
p
pat_nadolny
03/07/2022, 3:51 PM@alexandre_brehelin Yes definitely open a bug issue in the repo! You're right, if you have a long list of tables that will be an annoying solution but at least it should get you unblocked for the time being
pat_nadolny
03/07/2022, 3:52 PMOh actually it looks like an issue already exists https://github.com/MeltanoLabs/target-athena/issues/33
a
alexandre_brehelin
03/07/2022, 4:14 PMThank you @pat_nadolny, we run new pipeline with your workaroud, it well avoided the issue.
alexandre_brehelin
03/07/2022, 4:48 PMI wrote to fast. The workaround not able us to complete the pipeline. Athena target seems to run two create external table when we fill the stream_maps arg. One without apply the mapping and second with the mapping. So the pipeline fail because the first name isn't acceptable
p
pat_nadolny
03/07/2022, 5:06 PMAhh I ran into that else where also. There was a bug in the SDK that has since been fixed in 0.4.2 but it looks like the target hasnt been bumped to that version yet. Feel free to submit and issue or fork it and create an MR
a
alexandre_brehelin
03/07/2022, 5:27 PMEven if, it would be important to use the last sdk version, I think it's better to use the PR#42 solution to resolve the issue. Thank you @pat_nadolny