Hello everyone. Please help me setup dagster_meltano. So, i have run 1.

meltano add utility dagster

2.

meltano invoke dagster:initialize

3.

pip install dagster_meltano

then I wanted to understand how everything works, but when I went to meltano_project/orchestrate/dagster/repository.py I see an import error there. Someone knows what's the matter? Does anyone have a working example of the correct usage. To be honest, now I don't understand what meltano invoke dagster:describe [args...] is used for. Is there some more detailed documentation about this somewhere? I will be glad if you can answer and give a couple of recommendations on how to properly start using it. I thought maybe some magic was going on and I just need to run meltano invoke dagster:start, but as I expected, I got an empty dagster

3.

pip install dagster_meltano

I don't think that this

pip install

would do any good.

but i run it because i found it on readme. And it's still weird to me that it's trying to extract methods that don't exist.

but i run it because i found it on readme.

Ah. I see. I think it would be better if the README said

pipx install meltano

(which is probably superfluous because anyone reading the README already should have run that command) And THEN, what really needs to happen is probably to inject the dagster binaries into the same meltano pipx environment with

pipx inject --include-apps meltano dagster

ah, now I'll try to create one task. But I still don't understand why I see an import error in /orchestrate/dagster/repository.py or it's normall?

But I still don't understand why I see an import error

When you run

meltano add utility dagster

, the

dagster_meltano

python module gets installed into a virtualenv that is managed by meltano in the

.meltano

folder. It's not installed into any PYTHONPATH that would be available to your editor. If you want to do development on the dagster extension itself, or even just do development on the

dagster_meltano

package alone without the extension, you would need a slightly different development set up compared to how a "user" would install the dagster extension.

ahh, i understand. Okay. It seems to have worked for me after I created the job

Is I need create dagster.yml?

You can ignore the warning for now. When it comes to the point where you think you will need a dagster.yml file, you can always create one at that point.

That is, So it is normal if I really see 1 task and all my existing streams are in it. Do I understand correctly that I can’t divide my 3 streams into 3 separate tasks in any way (for example, so that you can specify different targets or different intervals)?

Do I understand correctly that I can’t divide my 3 streams into 3 separate tasks in any way

Regarding your other question in the #CMN8HELB0 channel. The common practice of configuring different

select

extras to match your business requirements is to create multiple inherited plugins from your tap. And then configure different

select

extras for each of the inherited child taps. Then you create different jobs, using the child taps in the job definitions. And these jobs can have different schedules.

but at the same time, each of them will again receive authorization data (in the form of a token)

There will be some merging happening of config settings from the parent plugin, so yes, you can configure authentication settings in the parent plugin definition and all children will inherit that authentication config from the parent (unless you decide to override the authentication config settings in one of the children for some reason)

Thanks Christoph for answering the questions! The

dagster-meltano

repo is an older project which is separate from the

dagster_meltano

the extension installs (this is a bit confusing, and I will change this in the future). The previous screenshot you saw was indeed a proof of concept and currently depends on a feature the meltano team is developing. At that point I will be able to create Dagster assets for the individual streams.

The previous screenshot you saw was indeed a proof of concept and currently depends on a feature the meltano team is developing. At that point I will be able to create Dagster assets for the individual streams.

Can't wait for that to become available 😁 @jules_huisman: What's your preferred way for collecting feedback for the

dagster_meltano

functionality? Should I open GH Issues in the

dagster-ext

repo for any feature ideas or questions that I have?

@christoph Yes, that makes sense for now!

you can configure authentication settings in the parent plugin definition and all children will inherit that authentication config from the parent

@christoph Is I need do something for it? Or will accessToken be automatically inherited from the parent?