guys, regarding my post above. How do you like this idea? That is, we can create 10-100-1000 or more files with schedules, they will automatically appear in dagster. The only problem is that I can't import the contents of the 'schedules' folder into meltano.yml/schedules (but as far as I know it's not hard to make such an import possible) I need your opinion on this idea. In this case, it seems that there is no need to run many instances with the meltano project and the dagster will automatically notice new schedules. @jules_huisman maybe you can give some feedback about this idea

I wouldn't use meltano's concept of schedules and do your own thing outside of meltano

Do you mean it's better to pack this into a docker container and run the docker container as many times as needed with different env environments? Like this

docker run \
  --mount type=bind,src=/absolute/path/to/bigquery/credentials/client_secrets.json,dst=/var/client_secrets.json \
  -e TAP_CONNECTWISE_AUTOMATE_DOMAIN='<http://domain.com|domain.com>' \
  -e TAP_CONNECTWISE_AUTOMATE_USERNAME='username' \
  -e TAP_CONNECTWISE_AUTOMATE_PASSWORD='password' \
  -e TAP_CONNECTWISE_AUTOMATE_HEADER_CLIENT_ID='client_id' \
  -e TARGET_BIGQUERY_PROJECT_ID=$GCP_PROJECT \
  -e GOOGLE_APPLICATION_CREDENTIALS=/var/client_secrets.json \
  meltano_cw_demo \
  elt tap-connectwise-automate target-bigquery --job_id=cw_automate_to_bigquery

Docker isn't needed to do that but it's a very good option! Yes run the meltano process with the variables you want when you want them to run

I haven't done something like this myself. But there is a concept of dynamic jobs in Dagster. This might be applicable for this use case. https://docs.dagster.io/concepts/ops-jobs-graphs/dynamic-graphs#a-dynamic-job In the

dagster-meltano

part of the

dagster-ext

there is already a way to create your own ops.

from dagster_meltano import meltano_run_op

. Which allows you to dynamically create your own Meltano run commands. Soon I will add a functionality to also inject Meltano config in a better way. Then you will be able to do something like this (pseudo code):

from dagster_meltano import meltano_run_op

@job
def ingest_for_clients:
  for client in clients:
    meltano_run_op(
      command="tap-connectwise-automate target-bigquery",
      config={
        domain: client.domain,
        username: client.username,
        ...
      }
    )()

Hmmm, looks great, when do you think it will be ready?

You will have one

op

for each client. They will all be in one

job

it seems like it will work. Do you know anything about when you can add it?

I might be able to add it next week. But depends a bit on the whole vacation period. I will let you know!

Great, thank you