Hi team our team will deploy and mange Meltano environment b Meltano #best-practices

Hi team, our team will deploy and mange Meltano e...

lidong_zhao

06/21/2022, 4:51 AM

Hi team, our team will deploy and mange Meltano environment, but we want the owner of each database from other teams to configure data replication using meltano. What would be the best practice for this scenario? here is what I am thinking: 1. We provide a tool to help the database owner to config extractor and select tables/columns. 2. That tool will translate the input into meltano.yml on the fly 3. Trigger deployment, deploy meltano.yml to production. Any suggestions? Thanks.

lidong_zhao

06/21/2022, 5:00 AM

Another option would be that, before each run, generate the meltano.yml file dynamically from a central database. I am not sure how I can add a step before each run.

edgar_ramirez_mondragon

06/21/2022, 3:27 PM

Are those teams going to replicate to the same target? If so, you can use multiple yaml files, one for each team's plugins

lidong_zhao

06/21/2022, 3:44 PM

All data go to snowflake, but not the same database or schema.

lidong_zhao

06/21/2022, 4:23 PM

we prefer not let each team update yaml file for 2 reasons: 1. ymal file is stored in git repo, update to this file has to go through deployment process (code review, deploy), which is slow and manual. 2. we don't want the connection string stored in yaml file.

edgar_ramirez_mondragon

06/21/2022, 5:33 PM

Most if not all of the actual plugin configuration can come from environment variables of the form

TAP_EXAMPLE_TOKEN

and

TAP_EXAMPLE__SELECT

, so you can have a different application write those values to a store like aws ssm, and shim those at deploy time in the Meltano server/container with something like chamber. The plugin definitions would still be stored in meltano.yml, but nothing else. Wdyt?

lidong_zhao

06/21/2022, 7:01 PM

Thanks, good to know, I'll try that.

Open in Slack

Previous Next