Hi All, We have configured

target-redshift

to access an S3 bucket. When running locally with

$ meltano invoke target-redshift

the process only gets as far as this then stalls:

time=2023-06-20 15:43:10 name=target_redshift level=INFO message=Getting catalog objects from table cache...
time=2023-06-20 15:43:10 name=botocore.credentials level=INFO message=Found credentials in shared credentials file: ~/.aws/credentials

Any suggestions for solving this?

Hi @ian_lewis! A target needs to be paired with a tap to move data. If you intend to move data from S3 to Redshift you might wanna take a look at tap-s3 and the data replication guide.

Hi @edgar_ramirez_mondragon

target-redshift

is capable of invoking the Redshift

COPY

command. That suggests to me that if data is already present then it can load that into Redshift.

Yeah,

target-redshift

can probably run a

COPY

command, but I’m sure it’s only used to load staged files created during the sync process. I’m sure the target batches a number of records it got from the tap, creates a CSV or similar file type in S3 with those records, then runs COPY on the file. I don’t think the target has a way of running the COPY command in isolation from the rest of the process.

OK, thanks @edgar_ramirez_mondragon that makes sense. I still need to understand why

target-redshift

hangs with no further output.

time=2023-06-20 15:43:10 name=target_redshift level=INFO message=Getting catalog objects from table cache...
time=2023-06-20 15:43:10 name=botocore.credentials level=INFO message=Found credentials in shared credentials file: ~/.aws/credentials

It’s waiting for data from

stdin

. The normal way to run with Meltano is to use

run

. For example:

meltano run tap-gitlab target-redshift

Thank you, that makes perfect sense. I misunderstood the purpose of

invoke

.