https://linen.dev logo
Join Slack
Powered by
Hello again <@U06CBNR7GM9> - looks like I'm having...
# plugins-general
r
Hello again @eric_simmerman - looks like I'm having issues loading in a particular CSV now. It's failing on 
psycopg2.errors.NotNullViolation: null value in column "number" violates not-null constraint
but I think it might be getting screwed up because this particular CSV has some new line characters in some of the field values. How does your tap work with newline characters in csvs? Is this expected? Curious how you would handle this. I see that there is a 
universal_newlines
option but I assume that is 
true
by default, so I don't think I would change anything there. Thank you!
e
yes universal_newlines is true by default.
The spreadsheets-anywhere tap doesn’t use/import/require psycopg2
So my first reactions is that error must be coming from a postgres loader you’ve attached to the meltano pipe?
Possible your CSV has nulls where your DB column is non-null?
Of course - like you suggest - it could be a parsing issue that’s resulting in the null value. You can use meltano to declare a schema that will be enforced by the tap. The README of the tap shows an example of this.
If you did that - you’d at least see the null error before it reached the loader/target.
BTW I implemented the universal_newlines boolean when I needed to parse some dirty pipe delimited CSVs. For those setting to False solved my parsing woes. I recommend trying a run set to false - that more or less reduces the tap sensitivity to newlines so that it only honors explicit ‘\n’
r
Good to know. We ended up cleaning up the incoming CSV's so there are no new line characters. But I will definitely reference this thread in the future
Open in Slack
PreviousNext
Powered by