https://linen.dev logo
Join Slack
Powered by
ELT : tap-postgres to target-postgres (testing wit...
# troubleshooting
a
ELT : tap-postgres to target-postgres (testing with single table) singer_sdk.exceptions.InvalidRecord: Record Message Validation Error: 1 is not of type 'string', 'null' cmd_type=loader name=target-postgres run_id=d987a7dc-2e24-4170-bb48-da5a2d9b8fd3 state_id=2024-06-10T030056--tap-postgres--target-postgres stdio=stderr Any idea what causing this error when I run meltano --log-level=debug elt tap-postgres target-postgres
v
Full stack trace is always helpful, but this smells like https://github.com/MeltanoLabs/tap-postgres/issues/437?thread_ts=1717954350.178639&cid=C069CQNHDNF which is just like the message from @Gustavo Silva
➕ 1
g
If I deselect the information-schema, the problem continues in some tables that has column with real numbers. Anothers tables doesn’t happen the error.
r
Good info to add to the issue if you have details on the column data types/record values @Gustavo Silva OP states issues with 
bpchar
, but that may have been a red herring https://meltano.slack.com/archives/C06A1MD6A6L/p1717605228789809
g
@Reuben (Matatika) When i arrive in my home, i will test change the type from the column then i atualize the information
👍 1
r
Thanks!
g
@Reuben (Matatika) i changed the column to numeric and didn't work, but when i change to varchar worked
j
I am getting 
singer_sdk.exceptions.InvalidRecord: Record Message Validation Error: '[]' is not of type 'number', 'null'
with target-snowflake. It started to happen while I was on vacation - no changes to the code base. I tried to upgrade target-snowflake to newest meltanolabs version, did not help. It's happening only when tapping data with tap-jira - other taps work OK.
I am getting the same error when using target-postgres
Relevant log (would be really helpful if I can exclude custom field warnings, they are huge and are hiding the real cause):
Copy code
2024-07-14T12:24:26.517418Z [info     ] Traceback (most recent call last): cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2024-07-14T12:24:26.517611Z [info     ]   File "/home/jacek/work/src/gooddata-data-pipeline/data_pipeline/.meltano/loaders/target-postgres/venv/lib/python3.11/site-packages/singer_sdk/sinks/core.py", line 121, in validate cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2024-07-14T12:24:26.517767Z [info     ]     self.validator.validate(record) cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2024-07-14T12:24:26.517917Z [info     ]   File "/home/jacek/work/src/gooddata-data-pipeline/data_pipeline/.meltano/loaders/target-postgres/venv/lib/python3.11/site-packages/jsonschema/validators.py", line 451, in validate cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2024-07-14T12:24:26.518068Z [info     ]     raise error                cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2024-07-14T12:24:26.518267Z [info     ] jsonschema.exceptions.ValidationError: '[]' is not of type 'number', 'null' cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2024-07-14T12:24:26.518425Z [info     ]                                cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2024-07-14T12:24:26.518574Z [info     ] Failed validating 'type' in schema['properties']['fields__customfield_10002']: cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2024-07-14T12:24:26.518716Z [info     ]     {'type': ['number', 'null']} cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2024-07-14T12:24:26.518848Z [info     ]                                cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2024-07-14T12:24:26.518973Z [info     ] On instance['fields__customfield_10002']: cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2024-07-14T12:24:26.519135Z [info     ]     '[]'                       cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2024-07-14T12:24:26.519296Z [info     ]                                cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2024-07-14T12:24:26.519424Z [info     ] The above exception was the direct cause of the following exception: cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
Seems like I need to exclude these f*cking custom fields from jira issues stream. I already tried it but without success.
I tried to PoC how to include only certain columns:
Copy code
select:
        - issues.fields__aggregateprogress
# OR
      select:
        - issues.fields.aggregateprogress
Does not work, the target table contains only ID column (PK). Any help would be more than welcome 😉
v
Interesting, I don't know if that's related or not. This smells like a change on the jira side which is why it would suddenly start happening. The schema does look incorrect for that data right? 
fields__customfield_10002
should have list as a type from jira?
j
Yes, but the point is that I don't want to tap these custom fields at all. I need to exclude them but can't find a way to do it.
v
ok so this is completely different than this thread we should make a different one
j
Ok. I already did one, but already expired I am afraid
v
and to answer you I would try 
!*.*fields__customfield_10002
but I'm not sure how jira structures stuff
j
Thanks, will try and open new thread if it doesn't work
v
try this too 
!*.fields__customfield_10002
if you look at select list it should show you what it's listed as 🤞
27 Views
Open in Slack
PreviousNext
Powered by