Hey everyone! Back from a long break and catching up! waves ❓ Tried Metlano Speedun but got an error on the following command:

meltano elt tap-gitlab target-postgres

Logs in thread 🧵

Create the tap_gitlab schema in postgres manually and you'll be good to go! Honestly this happens so much there should be a feature request for auto adding it!

Hey @visch! How could I do this? Shall I connect to psql before running metlano elt?

Yes, however you would normally connect! Command line works as well

I think you can also try with

meltano config target-postgres set before_run_sql "CREATE SCHEMA IF NOT EXISTS ${MELTANO_EXTRACT__LOAD_SCHEMA};"

Thanks @visch and @edgar_ramirez_mondragon! Tried before_run_sql and still getting an error when running meltano elt. ❓ question: What is MELTANO_EXTRACT__LOAD_SCHEMA? Is this an existing variable?

@gtsiolis

MELTANO_EXTRACT__LOAD_SCHEMA

is an environment variable that Meltano sets at runtime. You're right, in the command I shared bash tries to interpolate it so it will be empty 🤦‍♂️. So you can edit it in `meltano.yml`:

plugins:
  loaders:
  - name: tap-postgres
    ...
    config:
      before_run_sql: "CREATE SCHEMA IF NOT EXISTS ${MELTANO_EXTRACT__LOAD_SCHEMA};"

@edgar_ramirez_mondragon could I use it inline from the terminal using meltano config? Trying to figure out how to make the speedrun[1] to work. I don’t have a meltano.yml yet.

@gtsiolis maybe you can escape the

$

:

meltano config target-postgres set before_run_sql "CREATE SCHEMA IF NOT EXISTS \${MELTANO_EXTRACT__LOAD_SCHEMA};"

@edgar_ramirez_mondragon It worked like a charm! 🔮

@gtsiolis Awesome! Thanks for trying it out 😄

Got it running!

https://twitter.com/gtsiolis/status/1461518754249154567

Tried testing this out @gtsiolis Ran into a few issues 1. Have to logon with a github / gitlab / bitbucket account (makes sense from the product side, but was a slight annoyance), when trying to log in with my gitlab account gitpod wanted full access to all of my repos read/write. No thank you! 2. Used github instead with a test account, then tried to load the project you posted and I get this

Thanks for trying this @visch! 1. In general, we aim to request as few permissions as possible initially and to gradually ask for additional permissions as we need them for specific user actions. In this case, this is currently a limitation[1][2] of the GitLab API. Good news is that this is currently planned for this release, see https://gitlab.com/gitlab-org/gitlab/-/issues/321359#note_735612186. 🗺️ 2. Yes, you’ll need to authorize with GitLab since this is a GitLab repository and Gitpod is using your account to clone the repository, etc. You could try cloning the repository on GitHub and using GitHub instead. FWIW, this is not an error page but a prompt to authorize the corresponding provider. There’s definitely room for improving this page! 🐙

I get it, thanks just wanted to share that with you!

@edgar_ramirez_mondragon - Nice work on the settings-based config option! @pat_nadolny, @taylor - I created an MR to set this as the default value; for your consideration: https://gitlab.com/meltano/hub/-/merge_requests/171

issue: Running the same script[1] for the speedrun that used to work, currently fails on the elt step with the following error:

https://pastebin.com/hvsJsMrT

For a reproducible example, click https://gitpod.io/#https://gitlab.com/gtsiolis/meltano-speedrun-with-gitpod/. question: Did we change anything on the postgres loader? Could this be related to https://gitlab.com/meltano/hub/-/merge_requests/175? Cc @pat_nadolny

Yes, this is most likely related to the swapping out of the variant for postgres. Looks like you’ll need to update the config