https://linen.dev logo
Join Slack
Powered by
Hi! Any specific reason why target-redshift creat...
# plugins-general
s
Hi! Any specific reason why target-redshift creates table and column names in upper case ? https://github.com/transferwise/pipelinewise-target-redshift/blob/7e9e9e114452d2b33e02a0a911ee961c49f5ec5a/target_redshift/db_sync.py#L356 This requires putting double quotes while querying the table name and is not a standard. I can’t find a way to override this without modifying the code. Also, every time my sync runs, I notice that it tries to create table again and again, even though it exists. The check for existing table is not working for me. Am I doing something wrong ?
Copy code
│ =2023-01-16 06:02:04 name=target_redshift level=INFO message=Table 'lever' exists cmd_type=elb consumer=True name=target-redshift producer=False stdio=std │
c
The PR that introduced the change does not explain why they changed the table names to upper case. To me, it looks like a bug. https://github.com/transferwise/pipelinewise-target-redshift/pull/49
Reading through that PR, it says that this redshift PR was created as a "copy" or "mirror" of a corresponding target-snowflake PR ... https://github.com/transferwise/pipelinewise-target-snowflake/pull/65 At least in that snowflake PR, there is mention of the fact the tablenames will get changed to uppercase, but no reasoning as to why:
Wrap the table name in double quotes and capitalize it to make it safe to be in a Snowflake sql query.
s
hmm, I see. So should I install the plugin from a previous commit ?
I am currently using this commit to build the target-redshift loader. https://github.com/transferwise/pipelinewise-target-redshift/pull/202 Since the latest version has some incompatibilities with psycopg2 Ref - https://meltano.slack.com/archives/C01TCRBBJD7/p1671686394821779?thread_ts=1670449431.189059&cid=C01TCRBBJD7
or the other option is to try another connector perhaps https://github.com/datamill-co/target-redshift
Open in Slack
PreviousNext
Powered by