Hi all, i'm setting up a pipeline to move data bet...
# troubleshootingj
jose
04/04/2023, 4:42 PMHi all, i'm setting up a pipeline to move data between a mysql db and bigquery. Given that we are already using GCP, i was trying to set the state to be managed on GCS and have set state_backend.uri and state_backend.gcs.application_credentials. When i try to run the pipeline i get the following message:
RunnerError('Cannot start plugin tap-mysql: google-cloud-storage required but not installed. Install meltano[gcs] to use GCS as a state backend.')t
taylor
04/04/2023, 4:58 PMHave you installed meltano with the
[gcs]w
Will Da Silva (Arch)
04/04/2023, 5:22 PM@taylor They're using the official Meltano Docker image which has Meltano pre-installed. I've logged this issue to fix the Docker image: https://github.com/meltano/meltano/issues/7489
Will Da Silva (Arch)
04/04/2023, 9:07 PM@jose This will be resolve by our next release, which will probably be next week. At that point we'll publish a new Docker image which has the
[gcs]j
jose
04/04/2023, 11:34 PMHi @Will Da Silva (Arch) thanks for the feedback and also to add this to next release. I will move forward for now with a postgres db and once this feature is released i will get back to it
jose
04/06/2023, 12:18 PMHi @Will Da Silva (Arch), in fact i may need some help here.
I'm trying to setup the state to be written to an external Postgres db, using the commands below:
docker run -v $(pwd):/project -w /project meltano/meltano config meltano set state_backend.uri <postgresql://user:pass@server12345.aws.com:5432/dbtest?options=-csearch_path%3Dmelty>docker run -v $(pwd):/project -w /project meltano/meltano config meltano set database_uri <postgresql://user:pass@server12345.aws.com:5432/dbtest?options=-csearch_path%3Dmelty>Cannot start plugin tap-mysql: 'postgresql' is not a valid StateBackendw
Will Da Silva (Arch)
04/06/2023, 12:44 PM@jose Sorry, but I'm not sure. I'd suggest making a new post in #C01TCRBBJD7 to see if someone else can help.