This is a longshot but has anyone tried getting Meltano up and running on Google Cloud Run using Google Cloud SQL as Meltano's datastore? I'm using Postgres as it doesn't make sense to use sqlite in a production setting

You’re further in the rabbit hole than I am, I am planning to implement the same cloud run + cloud sql in prod too. Did you add the connection on the cloud run front too? https://cloud.google.com/sql/docs/mysql/connect-run I am curious about how you are implementing your main.py for cloud run

Thanks for that link, that helped me get it running!

Wait, you’re in OZ ? That’s the time on my clock too LOL

haha yup, in Sydney. Whereabouts are you?

Melbourne, just closed the border to prevent y’all from infecting us tho 😛

haha yep, been hearing about the stage 3 stuff...this lockdown has given me more time to look at Meltano 😛

I just got it running at https://meltano-conhde2ava-ts.a.run.app but getting errors when trying to set up the extractors

@al_whatmough What errors? 🙂

@douwe_maan in the logs I'm seeing this error:

sqlalchemy.exc.ProgrammingError: (pg8000.core.ProgrammingError) {'S': 'FATAL', 'V': 'FATAL', 'C': '53300', 'M': 'remaining connection slots are reserved for non-replication superuser connections', 'F': 'postinit.c', 'L': '828', 'R': 'InitPostgres'} (Background on this error at: <http://sqlalche.me/e/f405>)

Hmm, where are you seeing this error exactly? In the ELT output?

I see it in the Cloud SQL logs and it seems to be happening at various points, even just loading the Extractors page in the UI. I created a pipeline via the UI and I'm currently trying to run it, but this also triggers the error

"remaining connection slots are reserved for non-replication superuser connections" looks like a legit Postgres error to me, and Google suggests the same

yeah and the FATAL bit doesn't fill me with any reassurance, so I guess I'll need to have a look at how my Cloud SQL instance is configured

Have you given up on this ? what’s the learning ?

Hi @al_whatmough, I'd love to hear if this has worked and if there was any learning.

Hey @keat and @murilo_tavolaro_de_nigris - sorry only just saw these, I sometimes have Slack threads fail and miss seeing replies to a thread

It’s okay, Melbourne is still in lockdown and I can wait ❤️

unfortunately I never managed to get Meltano running on Google Cloud Run using cloud SQL. I actually never figured out the issue with the Postgres connections running out, but it's possible that because I was trying to get this running on a very small instance it just didn't have enough resources and the connection pool was being used up. It was just for a proof of concept so I didn't actually want to spend much money, but in a real environment I'd set up a more powerful instance

Thank you for the reply @al_whatmough (I miss threads sometimes too haha) I was just curious, but I decided to deploy my first project on a normal instance for now. If you do and document, it would be great, but no rush on this 🙂

Hey everyone! Following up on this thread to see if you ever got this working. Considering going this route in GCP

Hi @ricky_renner! I'd also be interested to hear if anyone got this working. I never pursued it in the end. I've seen someone in the introductions channel mention that they're deploying Meltano on GKE and using Composer rather than the Airflow which is included in Meltano. This is also an interesting approach, but I'm guessing that the costs could be a bit higher than running the whole thing on GCP

(The issue Al is referring to: https://gitlab.com/meltano/meltano/-/issues/2635)