@jacob_matson - First of all, congrats on your MDS-in-a-box project and the associated blog post. 🎉 I wanted to reach out on regarding your invitation to work together on a Docker example. We would be excited to work with you on this. Not just dockerization though, I think this could make a compelling devcontainer experience as well! Starting this thread 🧵 to discuss.

yes!

💯

I think I can cut a version of this next week, the template is all in David's repo.

Exciting! I personally don't have any experience with Rill and I'm interested to learn more about their solution.

same, superset was "fine"

Logged some info here which might be helpful: Adding Rill Data BI tool to MeltanoHub · Issue #932 · meltano/hub (github.com)

Thanks AJ, chatting with them in a bit to see where this takes us. Btw have a dumb question specific to tap-spreadsheet-anywhere and grabbing data from the web (no timestamp is persisted, so data is duplicated) - where is the best place to post it?

Depending on the nature of the question, perhaps #C013EKWA2Q1 maybe/or #C01PKLU5D1R and/or a new issue in the tap's repo?

I think its more of a general question, because its really about "how do I do an upsert for a given key" instead of a full load.

Sounds like a great plan 👍

This BI part will be the hold up I think. Both have bits to clear before I can ship v2.

@Sven Balnojan -Just now got to watch your demo! Looks fantastic. A couple things... 1. I'd love to see if we could get a VSCode-based database browser in here. Two options that come to mind are (1) the excellent SQLTools extension and (2) @alexander_butler's fork of DBT Power User, which includes some of his unique DBT-Osmosis features. 2. Not sure of the best place/manner to share this with the community but I'd love to get a reference dev container out based on your learnings here. Cc @edgar_ramirez_mondragon, @Sven Balnojan 3. We should host you as a guest of honor sometime soon in our weekly #C01QS0RV78D. I know folks would be interested in seeing the demo of this and discussing with you.

for the db browser....hear me out on this - is there a way to pass the dbt-osmosis query to the duckdb cli in the terminal? the duckdb cli is excellent

and I think I'm on the books for a demo on the 26th during office hours...

Fantastic! And also, I probably should have known this already... 😅 🤦‍♂️

dbt osmosis is fine. I think I have to add it to the container for dbt power user to work right anyway

https://docs.meltano.com/reference/plugin-definition-syntax#commands Anything declared in this way is accessible via

meltano run ... <some stuff here> ... my-tool:my-command <more-commands-here>

and also

meltano invoke my-tool:my-command <extra-args-if-you-need-them>

Getting warmer with dbt-osmosis / dbt-power-user. When I try to run queries, I'm getting the following issue:

{
  "message": "Runtime Error\n  The profile 'meltano' does not have a target named '{{ env_var('MELTANO_ENVIRONMENT', 'dev') }}'. The valid target names for this profile are:\n   - dev\n   - staging\n   - prod\n   - parquet",
  "data": "Traceback (most recent call last):\n  File \"<string>\", line 22, in <module>\n  File \"/home/vscode/.local/lib/python3.9/site-packages/dbt_osmosis/core/osmosis.py\", line 244, in _init_\n    self.parse_project(init=True)\n  File \"/home/vscode/.local/lib/python3.9/site-packages/dbt_osmosis/core/osmosis.py\", line 291, in parse_project\n    self.config = RuntimeConfig.from_args(self.args)\n  File \"/home/vscode/.local/lib/python3.9/site-packages/dbt/config/runtime.py\", line 228, in from_args\n    project, profile = cls.collect_parts(args)\n  File \"/home/vscode/.local/lib/python3.9/site-packages/dbt/config/runtime.py\", line 204, in collect_parts\n    profile = cls._get_rendered_profile(args, profile_renderer, profile_name)\n  File \"/home/vscode/.local/lib/python3.9/site-packages/dbt/config/runtime.py\", line 189, in _get_rendered_profile\n    return Profile.render_from_args(args, profile_renderer, profile_name)\n  File \"/home/vscode/.local/lib/python3.9/site-packages/dbt/config/profile.py\", line 426, in render_from_args\n    return cls.from_raw_profiles(\n  File \"/home/vscode/.local/lib/python3.9/site-packages/dbt/config/profile.py\", line 392, in from_raw_profiles\n    return cls.from_raw_profile_info(\n  File \"/home/vscode/.local/lib/python3.9/site-packages/dbt/config/profile.py\", line 336, in from_raw_profile_info\n    target_name, profile_data = cls.render_profile(\n  File \"/home/vscode/.local/lib/python3.9/site-packages/dbt/config/profile.py\", line 296, in render_profile\n    raw_profile_data = cls._get_profile_data(raw_profile, profile_name, target_name)\n  File \"/home/vscode/.local/lib/python3.9/site-packages/dbt/config/profile.py\", line 221, in _get_profile_data\n    raise DbtProfileError(msg, result_type=\"invalid_target\")\ndbt.exceptions.DbtProfileError: Runtime Error\n  The profile 'meltano' does not have a target named '{{ env_var('MELTANO_ENVIRONMENT', 'dev') }}'. The valid target names for this profile are:\n   - dev\n   - staging\n   - prod\n   - parquet\n"
}

So it’s not that simple. The code written by Michiel which is actually deviated from how I originally authored the integration is pulling the profile string from raw profile yml via typescript code which is obviously not rendered and then passing that into subprocess calls.

yeah makes sense.

I think I might take a swing at a dev container template that shows an example of how you could integrate it with the forks of the extension that work the way I designed it (datacoves fork and mine). It’s been a slow drag to get parity on the main repo as we are trying to align on some design decisions around sub processes vs server. You could then use that as a reference caveated by whether a dev container can specify an extension from a VSIX to use. Need to confirm :) or make a bash script at worst that automated it. The issue would be on power user side but I think an issue is already open for it.

Sounds good to me, thanks for the help. I'll eagerly await the (published) fork.

It's published. I think ideally we work this stuff back into the original and as a maintainer on the original codebase, I am working with the creator to hash out details but its a slow burn. So in the meantime, I have published my fork which enables faster iteration and more fun stuff on my end that I can work with before it circles round to the main repo ( / if it does) 🙂 Before that ideal state of main repo parity, I would 100% advocate this version to anyone I talk to. Its super fast and easy. You can search for this in vscode extension panel or use this in your dev container extension specifications

z3z1ma.vscode-dbt-power-user-osmosis

dbt-osmosis should be version

0.9.7

https://marketplace.visualstudio.com/items?itemName=z3z1ma.vscode-dbt-power-user-osmosis vscode:extension/z3z1ma.vscode-dbt-power-user-osmosis

Awesome, I dropped it into my devcontainer, but queries are not resolving (no error, but don't ever return results). Not sure how to turn on logging etc so I can give better feedback.

Interesting. They definitely have a timeout and error handling. Can you

curl <http://localhost:8581/health>

? Also I can clone this (the nba repo) and verify locally right?

yes you can clone it a verify locally!

Ok cool. I’ll give it a shot too. Basically as long as dbt-osmosis is installed and available on startup, the extension should spin up a server. If it was not installed or available on start up (based on python extension venv) then the server won’t kick on at the extension will use sub processes instead. Dev containers are sweet!

thanks 🙏 I'm sure its some config somewhere but its a bit tough to suss out sometimes.

@jacob_matson sounds like there is a registration (aka project parse issue). Try adding this flag to the invocation which will exit early if project parse fails

--exit-on-error

or it might be more elucidating to run this while the server is up

dbt-osmosis server register-project --project-dir /workspaces/my_project/transform --profiles_dir /workspaces/my_project/transform/profiles/duckdb

It should look like this screenshot on server init (notice the INFO log message pointing out project registration success)

Totally get it.

split terminal and ensure server is running in one terminal, then run the command in the other -- thanks for being a rockstar with all this

roger doing that now

So the solution is (probably), start the server the exact same way but put

meltano invoke

in front of it 🔥

meltano invoke dbt-osmosis server serve ...

oh

I got my docker setup so I can help now 🙂 https://hub.meltano.com/utilities/dbt-osmosis This actually outlines theres an extra 2 steps

so i missed the namespace step

sweet, it should be fast AF

it is crazy fast

so break it in

dude this rules

you can probably add the server invocation here to the dev container (as a

&

backgrounded task) alternatively technically dbt power user (my version you are using) will start the server automatically but there is just a tiny bit more config required that I plan on simplifying for meltano users

for real time invocation, how does that work

Cmd+'

on Mac or

Control+'

on windows/linux with a model open. It compiles jinja ~as you type

ok so one issue that is duckdb specific

is it transient at least? meaning that everything extension wise still works but you intermittently get the above

anything that is dbt build/run/compile etc will cause the issue

If you add this to the extension setting (which you can add to devcontainer.json

hmmm

Can confirm server management via extension works (see message in bottom right, server was started by extension automatically)

alright thank you SO MUCH for stepping me through this

Adding these two to your devcontainer should make the whole shebang work automagically