I am trying to build my project with meltano and dbt dependencies Following is my

pyproject.toml

file

[tool.poetry.dependencies]
python = ">=3.9,<3.10"


[tool.poetry.group.base.dependencies]
dagster = "^1.0.7"
meltano = "<=2.5.0"

(venv) ➜  python-buildsystem-poetry poetry add dbt-core -G base
Using version ^1.2.1 for dbt-core

Updating dependencies
Resolving dependencies... (0.9s)

Because no versions of hologram match >0.0.14,<0.0.15
 and hologram (0.0.14) depends on jsonschema (>=3.0,<3.2), hologram (>=0.0.14,<0.0.15) requires jsonschema (>=3.0,<3.2).
And because hologram (0.0.15) depends on jsonschema (>=3.0,<4.0), hologram (>=0.0.14,<=0.0.15) requires jsonschema (>=3.0,<4.0).
Because no versions of dbt-core match >1.2.1,<2.0.0
 and dbt-core (1.2.1) depends on hologram (>=0.0.14,<=0.0.15), dbt-core (>=1.2.1,<2.0.0) requires hologram (>=0.0.14,<=0.0.15).
Thus, dbt-core (>=1.2.1,<2.0.0) requires jsonschema (>=3.0,<4.0).
And because meltano (2.5.0) depends on jsonschema (>=4.9,<5.0)
 and no versions of meltano match >2.5.0,<3.0.0, dbt-core (>=1.2.1,<2.0.0) is incompatible with meltano (>=2.5.0,<3.0.0).
So, because acadia depends on both meltano (^2.5.0) and dbt-core (^1.2.1), version solving failed.

Is it possible to use dbt-core and meltano together at all ? Has anybody made it work ? I understand recommendation of

pipx

but I need to use the meltano as package instead of CLI

I need to use the meltano as package instead of CLI

Mind explaining why? The Python library interface is mostly internal-only - breaking changes to the core code does not result in major semantic version increments.

I am trying to build an integration with Dagster Orchestrator, where, I am trying to execute ELT jobs programmatically.

Doesn't solve for your question, but our recommendation is generally to use the public CLI instead of calling the Python modules directly. What our Airflow implementation does is to call

meltano schedule list --format=json

.

I am in discussion with @jules_huisman too, his current implementation (

development

) version is using Meltano libraries underneath, with future of possibly switching to cli. So is it final that Meltano will always be CLI first ? I see your EDK proposals too @aaronsteers is that something that can become a limited programmatic access into Meltano ?

So is it final that Meltano will always be CLI first ?

Not forever. But for the immediate future, the CLI interface is still our focus. We don't prohibit developers from consuming Meltano as a library but we admittedly have not tuned it for that usage: 1. We don't have a distinction as of yet between public + stable python modules and endpoints, versus internal ones. 2. We sometimes have CLI libraries doing "more than they should". (This is mostly in legacy code but there are a few places where this is known to be true.) 3. We don't have a separate "pure" module that is exclusive of things like

Click

(CLI) and

Gunicorn

(web UI). That can mean unnecessary conflicts and install footprint versus what we'd ideally have in a proper Python-based interface. We have an issue on this, but it is not actively being prioritized: Publish Meltano core as an importable Python library · Issue #2414 · meltano/meltano (github.com).

Thank you for the explanation, Yes, I had come across that issue on the core library request, but assumed that it had died down due to inactivity for a year. I’ll try to wrap the functionality with cli.

Thanks, @binoy_shah. The feedback is helpful, as is knowing that there is interest in the pure Python API.

A minimal and stable python interface with Meltano would be great, but I understand the use cases are a bit more limited, making it less of a priority. But especially with the edk, it would be nice to have some sort of 'introspection' into the Meltano project it is being run on. For example, an easy way to list all other installed plugins, access config or run a job etc. For the dagster-meltano plugin I am also relying on the Meltano python package, as some things are not possible to do using the cli (and is also easier to work with). However, I did run into a lot of dependency conflicts, especially with dbt-core (which is required by dagster-dbt).

@jules_huisman you kind of hinted that the dependency conflicts can be resolved, do you have further thoughts on how ? I tried building with dependencies but the highest meltano version I could successfully install with latest dagster and dbt-core was meltano

2.1.0

anything above that failed. your implementation of

extract_load

utilizes

Job

from meltano which has parameter

job_name

only available in later versions, So does your version of dagster_dbt not use anything from dbt-core ?