Hey all- is there a way to specify a shared depend...
# troubleshootingj
josh_wills
11/29/2022, 5:17 PMHey all- is there a way to specify a shared dependency between a loader and a dbt plugin? I got this issue on dbt-duckdb b/c two different builds of DuckDB are getting installed for
target-duckdbdbt-duckdbj
jacob_matson
11/29/2022, 5:21 PMI ended up using target parquet for reasons like this.
jacob_matson
11/29/2022, 5:22 PMwas running into just weird issues.
j
josh_wills
11/29/2022, 5:23 PMgotcha, makes sense-- I don't think there's an obvious way to make Meltano do what I want here, it's just not a use case that would normally apply
v
visch
11/29/2022, 6:33 PMThis seems a bit cheeky, but I wonder if you could use a constraints file. Like specify duckdb=0.6.0 by using PIP_CONSTRAINT=constraints.txt (in your meltano project). Would force pip to use a certain duckdb version when it's needed. Since duckdb is still new your surface area is low that it could break other things
The most appropriate way I can think is to just point to dbt-duckdb@duckdb0.6.0 and target-duckdb@duckdb0.6.0 using tags or something that you know work 🤷
j
josh_wills
11/29/2022, 6:41 PMdoes that work?
josh_wills
11/29/2022, 6:41 PMwasn't familiar with meltano's support for pip constraints, but if that is a way to do it, I am game to try
v
visch
11/29/2022, 6:42 PMWell it's not really meltano, it's PIP, if you can get that env variable set while meltano install is running it should work. It's cheeky though 🤷
visch
11/29/2022, 6:43 PMProbably easiest to just set PIP_CONSTRAINT yourself before running meltano (in your shell), but a global
envvisch
11/29/2022, 6:44 PMhand waving here, it sounds good I haven't done it exactly but it gets used in places like https://github.com/meltano/sdk/blob/main/.github/workflows/test.yml#L52 (not for meltano specifically but theoretically this should work)
j
josh_wills
11/29/2022, 7:12 PMnice, thanks!
v
visch
12/06/2022, 8:38 PM@josh_wills Found an easier way
Still not the best but
pip_url: target-duckdb~=0.4 duckdb==0.6.1visch
12/06/2022, 8:38 PMat least locks wherever you use duckdb to the same version, so you can add that to dbt, and target-duckdb 🤷
j
jacob_matson
12/06/2022, 8:47 PMgood fix tbh, not optimal but makes sense.
v
visch
12/06/2022, 10:08 PMalso just to slip this in https://github.com/MeltanoLabs/tap-duckdb 😉