https://linen.dev logo
Join Slack
Powered by
I’m writing a tap that relies on a local package. ...
# best-practices
r
I’m writing a tap that relies on a local package. So I ran 
poetry add -e ss-common
. The problem is that running 
meltano install
doesn’t keep the editable install. So I’m fine editing the tap locally, but if I make any changes inside 
ss-common
, those are not reflected in the meltano install. What’s the best way to do this? This is from the tap’s `pyproject.toml`:
Copy code
[tool.poetry.dependencies]
python = ">=3.9 <3.12"
# ...
ss-common = { path = "../ss-common", develop = true }
➕ 1
Use case: writing lots of taps with a shared utilities library. Guess I could also put everything in one massive monorepo, but I’d rather not
I was stuck at this for hours, then posted, then immediately found a workaround… would still love to hear whether this is recommended:
Copy code
source .meltano/extractors/tap-mytap/venv/bin/activate
pip install -e ../ss-common
r
Agreed - I've ran into this before when making changes to my fork of the SDK and trying it out on some taps locally. I've always just 
meltano install --clean <type> <name>
after I've made a significant enough change, which is a pretty slow way to develop...
💯 1
e
I'd be curious what the metadata for the ss-common package looks like in the venv managed by Meltano 🤔. i.e. whether there's a 
.meltano/extractors/my-tap/venv/lib/python3.11/site-packages/ss_common.dist-info/direct_url.json
file in there with the right metadata.
should look something like this
Copy code
{
  "dir_info": {
    "editable": true
  },
  "url": "file:///path/to/ss-common"
}
r
This is what a local SDK dependency looks like (no idea where 
0.6.1.post924.dev0+48949c6
is coming from):
e
yeah, a quick experiment yielded similar results
Copy code
{
  "dir_info": {},
  "url": "file:///Users/edgarramirez/Code/edgarrmondragon/poc/editable-inception/sub-dep"
}
r
Should pip respect nested editable dependencies?
e
I don't think it does. The packaging METADATA spec doesn't seem to support editable direct references.
😔 1
The alternative is not to use 
-e .
but instead point the 
executable
to the poetry-managed venv
r
With the 
executable
route you mean inside 
meltano.yml
? If setting that to the local tap, will the editable dependency of that tap be respected? Happy to experiment
e
Yes to both. By removing or commenting out the 
pip_url
and declaring the 
executable
pointing to the tap-x in the venv managed by poetry, the editable dependency should be respected. 
meltano install
will have no effect.
14 Views
Open in Slack
PreviousNext
Powered by