https://linen.dev logo
Join Slack
Powered by
Hi all! I am trying to setup a project with tap-r...
# troubleshooting
s
Hi all! I am trying to setup a project with tap-redshift(datadotworld) and target-postgres(transferwise), with py 3.7.15, meltano 2.9.0. I am getting this error in load part -> 2023-06-08T215330.340291Z [info ] time=2023-06-09 032330 name=target_postgres level=CRITICAL message=Primary key is set to mandatory but not defined in the [my_table] stream cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres. However, my redshift table does have primary key constraint defined. I even tried with a different table. Not sure what’s going wrong here. Can someone help ? PS: I have already checked the documentation. It says “primary_key_required” which is true by default. I can make it work by setting it to false but my use case is handling upserts. Thanks!
t
I would run 
meltano invoke tap-redshift > someFile
and inspect the schema it's providing for the stream in question. Honestly, odds are, it's not specifying the PK.
s
Looks like the PK’s are coming empty. It appears that my 
information_schema.table_constraints
and 
information_schema.key_column_usage
tables are empty, which is weird.
This query to extract primary keys seems incorrect. https://github.com/datadotworld/tap-redshift/blob/master/tap_redshift/__init__.py#L94 Redshift doesn’t maintain these tables. I am able to get the PKeys via this query instead.
Copy code
SELECT
  c.relname AS table_name,
  a.attname AS column_name,
  conname AS constraint_name,
  contype AS constraint_type
FROM
  pg_catalog.pg_constraint AS con
  JOIN pg_catalog.pg_class AS c ON c.oid = con.conrelid
  JOIN pg_catalog.pg_attribute AS a ON a.attrelid = c.oid AND a.attnum = ANY(con.conkey)
WHERE contype IN ('p', 'u', 'f');
So should I create a PR and then build the tap from there ?
I raised a PR to fix these issues and was able to unblock myself but this connector doesn’t look in a good shape, I am facing many more misc issues. https://github.com/datadotworld/tap-redshift/pull/42
u
I dont see any SDK based tap-redshift variants in the community yet if someone wanted to attempt to build it on the SDK. This tap-postgres https://github.com/MeltanoLabs/tap-postgres implementation is probably very similar to what is needed for redshift, could be a good place to start
u
I also created a singer mosted wanted issue for it https://github.com/MeltanoLabs/Singer-Most-Wanted/issues/82
s
@pat_nadolny I have started working on this, hitting a small road blocker here, https://meltano.slack.com/archives/C01TCRBBJD7/p1687032291570149 in case you happen to stumble across this 🙂
Open in Slack
PreviousNext
Powered by