Hello, I am working to set up <target-redshift>, p...
# troubleshooting
p
Hello, I am working to set up target-redshift, passing an AWS IAM Role ARN to use as authentication for the COPY statement. Data is loaded from my source to S3, but I am getting a Psycopg2 syntax error when the COPY statement is executed by Psycopg2 (doesn't actually hit the cluster):
Copy code
psycopg2.errors.SyntaxError: syntax error at or near "iam_role"
Has anyone seen this? Using the transferwise variant: https://github.com/transferwise/pipelinewise-target-redshift Here is where the statement is formed: https://github.com/transferwise/pipelinewise-target-redshift/blob/master/target_redshift/db_sync.py#L427 Is there a good way to set debug log level for this loader? Setting Meltano log level does not affect loader log level in my testing
p
Hey @patrick_leahey - did you confirm that your config looks as expected?
meltano config target-redshift
I wonder if theres unexpected characters in there or something i.e. a quote
p
Everything seems in order when I run what you suggested. I am running Meltano in a container deployed on ECS via Airflow, so that is part of the reason I am having trouble debugging. I likely need to get this going locally again. I had success running locally, and only thing I have added since is the IAM Role ARN
No odd characters in the ARN
Thanks for the suggestion
p
I wonder if the iam_role is a red herring, sometimes the sql errors arent very precise
m
I have been running this target-redshift plugin using AWS access key and secret environment variables to configure it, and have had no issues.
are you setting the S3 bucket in the configuration?
Copy code
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
AWS_DEFAULT_REGION=...
TARGET_REDSHIFT_HOST=...
TARGET_REDSHIFT_PORT=...
TARGET_REDSHIFT_DBNAME=...
TARGET_REDSHIFT_USER=...
TARGET_REDSHIFT_PASSWORD=...
TARGET_REDSHIFT_S3_BUCKET=...
TARGET_REDSHIFT_DEFAULT_TARGET_SCHEMA=...
TARGET_REDSHIFT_BATCH_SIZE_ROWS=...
TARGET_REDSHIFT_ADD_METADATA_COLUMNS="true"
TARGET_REDSHIFT_PRIMARY_KEY_REQUIRED="false"
I am setting all of these environment variables
p
I am - let me check my config against yours. I appreciate your comment
@pat_nadolny I am starting to get suspicious of the same
m
also just in case it makes a difference, I am explicitly unsetting these environment variables to ensure that the target-redshift uses the default AWS vars:
Copy code
TARGET_REDSHIFT_AWS_ACCESS_KEY_ID=
TARGET_REDSHIFT_AWS_SECRET_ACCESS_KEY=
p
Let me try that
I might have just figured something completely unrelated out. I believe I have one of my postgres db hostnames specified in my target-redshift config...
That is 100% wrong, but surprising it threw that error!
p
I've had that happen before too, error message from data warehouses are always tricky to interpret. Never sure if I can trust them 😆 . Glad you figured it out!