Hello :slightly_smiling_face: I'm currently trying...
# singer-tap-developmentt
tomas_jevocin
12/04/2023, 5:12 PMHello 🙂 I'm currently trying to dockerize a meltano project which contains: 1) custom extractor (singer-sdk) 2) meltano project making use of the custom extractor.
When building the image, we get the following error upon
meltano installmeltano 3.1.0 requires python-dotenv<2.0.0,>=1.0.0, but you have python-dotenv 0.21.1 which is incompatible. Copy code
ARG MELTANO_IMAGE=meltano/meltano:latest
FROM $MELTANO_IMAGE
# Copy over Meltano project directory
COPY custom-rest-api/ .
COPY my-el-project/ .
WORKDIR /project/my-el-project
# Install any additional requirements
COPY requirements.txt .
RUN pip install -r requirements.txt
# RUN meltano install
RUN meltano lock --update --all
RUN meltano install
ENV PYTHONUNBUFFERED=1
COPY run-el.py .
ENTRYPOINT ["python", "run-el.py"]e
edgar_ramirez_mondragon
12/04/2023, 5:16 PMHi @tomas_jevocin! What's in your
requirements.txtt
tomas_jevocin
12/04/2023, 5:17 PMHi @edgar_ramirez_mondragon, only
snowflake-connector-pythone
edgar_ramirez_mondragon
12/04/2023, 5:18 PMOk, so the stage that's failing is
RUN pip install -r requirements.txtt
tomas_jevocin
12/04/2023, 5:19 PMthe failing stage is the
RUN meltano installtomas_jevocin
12/04/2023, 5:19 PMThe installation of the custom plugin fails, specfically
e
edgar_ramirez_mondragon
12/04/2023, 5:22 PMHmm, what's weird is that the error you mentioned seems to imply that you're trying to install the plugin in the same virtualenv as Meltano 🤔
t
tomas_jevocin
12/04/2023, 5:23 PMYeah, I assumed something simillar, but so far I wasn't able find out how did it happen or how to resolve it 😕
e
edgar_ramirez_mondragon
12/04/2023, 5:23 PMDoes your custom plugin depend on
meltanoedgar_ramirez_mondragon
12/04/2023, 5:24 PMOr better, can you share the list of deps for your custom plugin(s)?
t
tomas_jevocin
12/04/2023, 5:25 PMThe custom plugin deps should be located within the pyproject.toml, right?
tomas_jevocin
12/04/2023, 5:25 PMThe requirements.txt shown in the docker image is for purposes of different script ran before and after the meltano run
tomas_jevocin
12/04/2023, 5:25 PM(metada table entries in snowflake)
e
edgar_ramirez_mondragon
12/04/2023, 5:27 PMThe custom plugin deps should be located within the pyproject.toml, right?Correct 👍
The requirements.txt shown in the docker image is for purposes of different script ran before and after the meltano runGotcha, and the failing stage is
RUN meltano installt
tomas_jevocin
12/04/2023, 5:28 PM Copy code
...
[tool.poetry.dependencies]
python = "<3.12,>=3.7.1"
singer-sdk = { version="^0.31.1" }
fs-s3fs = { version = "^1.1.1", optional = true }
requests = "^2.31.0"
genson = "^1.2.2"
[tool.poetry.group.dev.dependencies]
pytest = "^7.4.0"
singer-sdk = { version="^0.31.1", extras = ["testing"] }
[tool.poetry.extras]
s3 = ["fs-s3fs"]
[tool.mypy]
python_version = "3.9"
warn_unused_configs = true
[tool.ruff]
ignore = [
"ANN101", # missing-type-self
"ANN102", # missing-type-cls
"TD003", # missing issue link in TODO
"FBT001", # bool-typed pos argument
"FBT002", # bool-default pos argument
]
select = ["ALL"]
src = ["mews_rest_api"]
target-version = "py37"
[tool.ruff.flake8-annotations]
allow-star-arg-any = true
[tool.ruff.isort]
known-first-party = ["mews_rest_api"]
[tool.ruff.pydocstyle]
convention = "google"
[build-system]
requires = ["poetry-core>=1.0.8"]
build-backend = "poetry.core.masonry.api"
...tomas_jevocin
12/04/2023, 5:54 PMThe meltano image we pull from has meltano installed in its own virtualenv, right?
e
edgar_ramirez_mondragon
12/04/2023, 5:56 PMt
tomas_jevocin
12/05/2023, 10:03 AMHi again @edgar_ramirez_mondragon, we managed to resolve the issue, there was an incorrect version of requirements.txt (which contained singer-sdk dependency) passed to the docker build. Sorry about the hassle and thanks for all the help! 🙂
e
edgar_ramirez_mondragon
12/05/2023, 4:02 PMNo worries! Glad that you figured it out.