Hi I m trying to write tap facebook data into our S3 Data la

Question

Hi I m trying to write tap facebook data into our S3 Data lake on AWS by using the pipelinewise target s3 csv target I m running into the following type of error Loading failed 1 TypeError Object of type Decimal is not JSON serializable With trial and error I m able to find some columns fields causing the problem but that s not really productive debugging Since I m new to Meltano I would like to know how to tackle such issues best Many thanks for any input

ripe-musician-59933 · Answer

< clever football 96790> The first step would be to enable debug logging on `meltano elt` so that you get to see each message output by the tap which will help you determine which specific records are triggering the problem in the target

ripe-musician-59933 · Answer

Beyond that it ll be a matter of diving into the `pipelinewise target s3 csv` source code you should see paths printed in the error stack trace that you can open and modify at will and continuing debugging there with print statements etc

ripe-musician-59933 · Answer

Since the error originates in the target s own source code there s not much more we can do from Meltano s perspective

clever-football-96790 · Answer

< ripe musician 59933> Thanks for the feedback I will investigate and maybe come back to you with questions regarding my findings or the interpretation of the log information

clever-football-96790 · Answer

< ripe musician 59933> I was not able to identify the problem any further since it seems to be a problem of the JSON serializing function I then switched to the `pipelinewise target redshift` and that worked like a charm and it also handles missing values much better which would cause a shift in the csv file using the `pipelinewise target s3 csv` Are you planning to add a Redshift Target as a standard component Let me know if I can help there in any way Thx

ripe-musician-59933 · Answer

< clever football 96790> I m glad to hear your got it working I d love to make target redshift discoverable and supported out of the box Would you mind contributing the custom plugin definition you ended up with in `meltano yml` as described under

clever-football-96790 · Answer

Sure I will try to put that together in the next week or so

ripe-musician-59933 · Answer

< clever football 96790> Thanks

clever-football-96790 · Answer

< ripe musician 59933> I sent a merge request this morning I hope it is up to your standard slightly smiling face

ripe-musician-59933 · Answer

< clever football 96790> Great thanks I ll check check it out shortly slightly smiling face