Hi I m trying to write tap facebook data into our S3 Data la Meltano #announcements

Hi, I'm trying to write tap-facebook data into our...

ingo_klose

01/15/2021, 7:49 AM

Hi, I'm trying to write tap-facebook data into our S3 Data lake on AWS by using the pipelinewise-target-s3-csv target. I'm running into the following type of error: Loading failed (1): TypeError: Object of type 'Decimal' is not JSON serializable. With trial and error I'm able to find some columns/fields causing the problem but that's not really productive debugging. Since I'm new to Meltano I would like to know how to tackle such issues best. Many thanks for any input!

douwe_maan

01/15/2021, 4:17 PM

@ingo_klose The first step would be to enable debug logging (https://meltano.com/docs/command-line-interface.html#debugging) on

meltano elt

so that you get to see each message output by the tap, which will help you determine which specific records are triggering the problem in the target.

douwe_maan

01/15/2021, 4:19 PM

Beyond that, it'll be a matter of diving into the

pipelinewise-target-s3-csv

source code (you should see paths printed in the error stack trace that you can open and modify at will) and continuing debugging there with print statements etc

douwe_maan

01/15/2021, 4:19 PM

Since the error originates in the target's own source code, there's not much more we can do from Meltano's perspective

ingo_klose

01/18/2021, 7:31 AM

@douwe_maan Thanks for the feedback...I will investigate and maybe come back to you with questions regarding my findings or the interpretation of the log information.

ingo_klose

01/27/2021, 9:10 AM

@douwe_maan I was not able to identify the problem any further, since it seems to be a problem of the JSON serializing function. I then switched to the

pipelinewise-target-redshift

and that worked like a charm and it also handles missing values much better, which would cause a shift in the csv file, using the

pipelinewise-target-s3-csv

. Are you planning to add a Redshift Target as a standard component? Let me know, if I can help there in any way. Thx!

douwe_maan

01/27/2021, 3:41 PM

@ingo_klose I'm glad to hear your got it working! I'd love to make target-redshift discoverable and supported out of the box. Would you mind contributing the custom plugin definition you ended up with in

meltano.yml

as described under https://meltano.com/docs/contributor-guide.html#discoverable-plugins?

ingo_klose

01/27/2021, 3:52 PM

Sure, I will try to put that together in the next week or so.

douwe_maan

01/27/2021, 3:59 PM

@ingo_klose Thanks!

ingo_klose

02/22/2021, 2:39 PM

@douwe_maan I sent a merge request this morning...I hope it is up to your standard. 🙂

douwe_maan

02/22/2021, 3:55 PM

@ingo_klose Great, thanks! I'll check check it out shortly 🙂

8 Views

Open in Slack

Previous Next