I m using tap outbrain to send data to target postgres But w

Question

I m using tap outbrain to send data to target postgres But when loading data to Postgres i get some NotNullViolation error has anyone experienced this berfore I ve tried to send the same data using Stitch and it creates the schema correctly and loads the data same postgres server

ripe-musician-59933 · Answer

< thousands library 17999> Can you share some more details of the error like the stacktrace and the specific SCHEMA or RECORD message that s causing the issue If you enable debug mode you should see each message output by the tap printed with a `tap outbrain out ` prefix The one s printed just before the error are the likely culprits slightly smiling face

thousands-library-17999 · Answer

Actually the problem comes from target postgres

ripe-musician-59933 · Answer

Right but presumably it s caused by a specific message coming from the tap that the target is failing to process

thousands-library-17999 · Answer

```target postgres | ERROR Exception writing records target postgres | Traceback most recent call last target postgres | File home ubuntu meltano projects taboola2bigquery meltano loaders target postgres venv lib python3 8 site packages target postgres postgres py line 295 in write batch target postgres | written batches details = self write batch helper cur target postgres | File home ubuntu meltano projects taboola2bigquery meltano loaders target postgres venv lib python3 8 site packages target postgres sql base py line 834 in write batch helper target postgres | batch rows persisted = self write table batch target postgres | File home ubuntu meltano projects taboola2bigquery meltano loaders target postgres venv lib python3 8 site packages target postgres postgres py line 609 in write table batch target postgres | self persist csv rows cur target postgres | File home ubuntu meltano projects taboola2bigquery meltano loaders target postgres venv lib python3 8 site packages target postgres postgres py line 563 in persist csv rows target postgres | cur copy expert copy csv rows target postgres | psycopg2 errors NotNullViolation null value in column campaignonair violates not null constraint target postgres | DETAIL Failing row contains 00e5cac90872acdff6e41d66fc63cd4ed6 ES BMG BLOG ART2 null null f 006d1f4f4dc46dfd9b02934416355f6883 ES BMG BLOG ART2 f 30 EUR null null 2020 04 22 09 46 01+00 2020 05 06 11 48 20+00 2020 04 22 2021 05 31 f DAILY SPEND ASAP null null 0 1 2020 12 17 00 13 32 7786+00 1608165459 null 2020 12 17 00 37 39 9168+00 target postgres | CONTEXT COPY tmp 0c7ed48b b33b 4ad1 8f70 80812500d2a8 line 1 00e5cac90872acdff6e41d66fc63cd4ed6 ES BMG BLOG ART2 NULL NULL False 006d1f4f4dc46dfd9b02934416355f68 target postgres | target postgres | CRITICAL Exception writing records NotNullViolation null value in column campaignonair violates not null constraint nDETAIL Failing row contains 00e5cac90872acdff6e41d66fc63cd4ed6 ES BMG BLOG ART2 null null f 006d1f4f4dc46dfd9b02934416355f6883 ES BMG BLOG ART2 f 30 EUR null null 2020 04 22 09 46 01+00 2020 05 06 11 48 20+00 2020 04 22 2021 05 31 f DAILY SPEND ASAP null null 0 1 2020 12 17 00 13 32 7786+00 1608165459 null 2020 12 17 00 37 39 9168+00 nCONTEXT COPY tmp 0c7ed48b b33b 4ad1 8f70 80812500d2a8 line 1 00e5cac90872acdff6e41d66fc63cd4ed6 ES BMG BLOG ART2 NULL NULL False 006d1f4f4dc46dfd9b02934416355f68 n ```

thousands-library-17999 · Answer

Sorry for the long copy

thousands-library-17999 · Answer

Tap outbrain runs correctly but i ll try enabling debug

ripe-musician-59933 · Answer

All right it looks like the `campaignOnAir` column should hold a non null boolean value <https github com singer io tap outbrain blob a111e43b64047219eacd70d8920f3d431c0b62b2 tap outbrain schemas py L98> but the target is receiving a record from the tap where it is null anyway

ripe-musician-59933 · Answer

Since the target created the table column not expecting nulls it s raising NotNullViolation now that a null is trying to make its way into the table anyway

thousands-library-17999 · Answer

thousands-library-17999 · Answer

I m trying this But not sure if I m doing this correctly

ripe-musician-59933 · Answer

Ah you fixed that already smile That looks right

ripe-musician-59933 · Answer

Did you drop the table and let it be recreated after making those changes to make sure the new type is used

ripe-musician-59933 · Answer

Also try using debug mode to confirm that the tap actually uses the correct types in its SCHEMA messages now

thousands-library-17999 · Answer

Not yet i ve forgot to merge on my master I m on it now

thousands-library-17999 · Answer

But i though Stitch where using the same tap version

ripe-musician-59933 · Answer

It s possible that their own target for postgres assumes every column can be `null` anyway

ripe-musician-59933 · Answer

They re using their own proprietary version not one of the open source targets

ripe-musician-59933 · Answer

So tap outbrain may work as is with theirs but not with the open source version s of target postgres that take the advertised type and its lack of null literally

thousands-library-17999 · Answer

That makes sense

ripe-musician-59933 · Answer

Once you ve updated the `pip url` for `tap outbrain` in your project don t forget to reinstall it using `meltano install extractor tap outbrain`