I am new to Meltano and I have a question is it possible to Meltano #getting-started

I am new to Meltano, and I have a question, is it ...

sean_han

02/10/2023, 1:38 AM

I am new to Meltano, and I have a question, is it possible to add multiple postgres-dbt to one meltano project, each of them for a different database?

sean_han

02/10/2023, 1:40 AM

When I tried to add a new postgres-dbt transformer using inherit-from, I got the following error: Lockfile already exists: /Users/.../Workspaces/data_platform/meltano/plugins/files/files-dbt-postgres--meltano.lock

Will Da Silva (Arch)

02/10/2023, 1:41 AM

Hi @sean_han. This is possible using plugin inheritance: https://docs.meltano.com/concepts/plugins#plugin-inheritance

Will Da Silva (Arch)

02/10/2023, 1:43 AM

Please let us know if you run into any problems, or simply find something confusing.

sean_han

02/10/2023, 1:44 AM

here is my command, but I got an error

Copy code

meltano add transformer dbt-postgres-new --inherit-from dbt-postgres

sean_han

02/10/2023, 1:44 AM

here is the error:

Copy code

Lockfile already exists: /Users/.../Workspaces/data_platform/meltano/plugins/files/files-dbt-postgres--meltano.lock

Will Da Silva (Arch)

02/10/2023, 1:45 AM

Ah sorry, I missed "using inherit-from" in your message from earlier.

Will Da Silva (Arch)

02/10/2023, 1:46 AM

@edgar_ramirez_mondragon Have you seen this error before?

sean_han

02/10/2023, 1:47 AM

when I added dbt-postgres, files-dbt-postgres was also added, I am not sure what it is for. could you please explain what files-dbt-postgres is for?

sean_han

02/10/2023, 1:48 AM

I can't find any docs for it.

Will Da Silva (Arch)

02/10/2023, 1:52 AM

That's a file bundle: https://docs.meltano.com/concepts/plugins#file-bundles

Will Da Silva (Arch)

02/10/2023, 1:52 AM

Specifically, this one: https://github.com/meltano/files-dbt-postgres

Will Da Silva (Arch)

02/10/2023, 1:53 AM

The readme in its GitHub repository details the files it adds

sean_han

02/10/2023, 1:56 AM

thanks, let me review them

sean_han

02/10/2023, 1:59 AM

please let me know if you guys know how to solve my error above. thx

aaronsteers

02/10/2023, 2:14 AM

Hi, @sean_han - Your error message above looks like it may be a bug in

meltano add

. The lock file for

dbt-postgres

shouldn't need to be in conflict since you are adding

dbt-postgres-new

. I'll see if I can reproduce the issue on my side.

aaronsteers

02/10/2023, 2:14 AM

If it is a bug, the workaround would be to edit your yaml file directly. I don't have a code sample for that handy, but we can provide one for you if helpful.

aaronsteers

02/10/2023, 2:15 AM

It's after hours here in the US, so please pardon delays in our messages. 🙂

sean_han

02/10/2023, 2:22 AM

no worries, thx,

edgar_ramirez_mondragon

02/10/2023, 3:48 AM

Have you seen this error before?

Nope, it’s definitely a bug, in transitive deps plugins it seems

aaron_phethean

02/10/2023, 11:47 AM

I wonder, as an alternative to plugin inheritance, is possible for you to set the env for each postgres before

meltano run

? (This is what we do to point to different databases https://www.matatika.com/docs/how-to-guides/import-data/running-your-data-import-locally) Not sure if your goal is to manage separate dbt project configurations or just point to a different database with everything else the same.

sean_han

02/10/2023, 6:40 PM

just want to point to a different databases, one is app database, one is data warehouse, the source data is the same.

aaronsteers

02/10/2023, 8:07 PM

Circling back here on this comment... I wasn't able to reproduce the lock file conflict issue on my side.

Copy code

$ meltano add extractor tap-github-2 --inherit-from=tap-github
Added extractor 'tap-github-2' to your Meltano project
Inherit from:   tap-github
Repository:     <https://github.com/MeltanoLabs/tap-github>
Documentation:  <https://hub.meltano.com/extractors/tap-github--meltanolabs>

Installing extractor 'tap-github-2'...
Installed extractor 'tap-github-2'

To learn more about extractor 'tap-github-2', visit <https://hub.meltano.com/extractors/tap-github--meltanolabs>

edgar_ramirez_mondragon

02/10/2023, 8:08 PM

@aaronsteers try it with dbt-postgres instead of tap-github. It’s an issue with the files plugin’s lockfile that dbt-postgres depends on

aaronsteers

02/10/2023, 8:09 PM

I see. Will retry...

aaronsteers

02/10/2023, 8:16 PM

Okay, now I am able to repro. Thanks, @edgar_ramirez_mondragon. There are two easy workarounds I found... 1. Simply delete the offending lock file (

cd meltano && rm ./plugins/files/files-dbt-postgres--meltano.lock

). This file will never be needed for files bundles, and is safe to delete. 2. Add fresh as a utility (

meltano add utility dbt-postgres

) and then build the second

dbt-postgres

instance as inheriting from this one.

aaronsteers

02/10/2023, 8:19 PM

Our latest 'utility' versions of the transformers do not rely on files bundles, and instead have an

init

initialize

command that handles project scaffolding, installing files, etc. https://docs.meltano.com/guide/advanced-topics#meltano-plugin-types

sean_han

02/10/2023, 8:56 PM

thanks, AJ.

Open in Slack

Previous Next