Hi All, Is there a possibility of using a cloud database like Snowflake as the system db for Meltano? If yes, Can someone please share any link.

We used cloud sql for that currently. In case you a curious I can try to help you out with specifics on how we set things up currently 🙂

Hi @thomas_schmidt, Yes please, it would be really helpful.

So let me try to recap @sebastian_rehm build that part recently but I hope I understood it properly 😄 So basically we run our Meltano setup using gitlab pipelines. In the

.gitlab-ci.yml

file we take the following approach: • On each merge request and merge into master we build an image for Meltano runs (including the Meltano project + setup AND the cloud sql proxy installed) • We have scheduled jobs defined which basically use the latest image, connect to the CloudSQL instance using the proxy and then run the elt job

yaml
# Generic setup for el jobs
.el-setup
  stage: el
  image:
    name: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA}
    entrypoint: [""]
  before_script:
    - cp -Rn /project/. .
    - echo $SA_FILE > ${GOOGLE_APPLICATION_CREDENTIALS}
    - ./cloud_sql_proxy -instances="<instance-name>"=tcp:0.0.0.0:1234 > cloudsql.log 2>&1 &
  artifacts:
    paths:
      - cloudsql.log
    expire_in: 1 week
  only:
    - schedules

# Single EL job
el-gitlab:
  extends:
    .el-setup
  script:
    - export TARGET_BIGQUERY_DATASET_ID=meltano_warehouse
    - meltano elt tap-gitlab target-bigquery --job_id=gitlab-bigquery
  retry: 2

Note: We ingest our service account credentials using the Gitlab environment variables

echo $SA_FILE > ${GOOGLE_APPLICATION_CREDENTIALS}

which allows us to connect to CloudSQL then The database URI is then set in the environment variables with

MELTANO_DATABASE_URI=postgresql://<username>:<password>@<host>:<port>/<database>

Let me know whether this helps you or you have additional questions 🙂

Thanks @thomas_schmidt 🙂 We are actually looking out for setting Snowflake as the system db.