Hey all, I'd like to define a job like this:

jobs:
  - name: data-pipeline
    tasks:
      - some-tap some-target
      - dbt:test --select sources:*
      - dbt:run
      - dbt:test --exclude sources:*

but it seems that command arguments (like -s sources:*) are interpreted as additional blocks. Is there any way to allow command args in jobs?

Hey @benjamin_mitzkus! I think @Stéphane Burwash opened a similar issue yesterday, is that what your looking for too?

@Will Da Silva (Arch) quoting sadly doesn't help 😞 @pat_nadolny Thanks, that looks exactly like what I'm trying to achieve

Ah, sorry. I thought this was just a yaml parsing issue.

Glad to hear I'm not the only one with this issue 😉 @pat_nadolny do you think this could be a quick fix PR, or more of a long term restructure?

@Stéphane Burwash I'm not familiar enough with the code to know but it feels like a larger restructuring to me

@Stéphane Burwash this is a bit of a longer term restructuring, but maybe I’m not thinking creatively enough. I could actually imagine the pattern shown here creating ephemeral commands on the plugin so that you wouldn’t have to create them manually. @florian.hines you wrote a lot of that - what are your thoughts? 😄

The biggest issue is that meltano run itself doesn't support block level args. Adding those is definitely a non-trivial change (both run and jobs would require changes to support those). I'm just spitballing/trying to think outside of the box but, wonder if the new extension could help with this @taylor? It feels like we could support allowing users to configure some shortcuts so that

selects

automatically get included when something like

dbt-bigquery test

is called... when certain conditions are present during invocation. Since we're wrapping dbt in another layer and doing things like pre-invoking things - this seems like a pretty natural fit for that. Start making the extension smarter ?

@florian.hines it doesn’t support block level args, but if you define it as a command then it should work right? My quick thought was basically having

dbt:test --select sources:*

act as an “ephemeral” plugin command. but maybe I’m being silly. but yeah, making the extension smarter could work too

act as an “ephemeral” plugin command

As in having a meltano yaml commands entry - only jobs creates that entry on the fly for the invocation of that task ?

yes - that’s how I was thinking about it

that would require some weird hooks behind the scenes but seems do able for jobs. Does seem weird though that you can't use the same arguments though when directly calling

meltano run

yeah, lots to ponder here 🤔

@taylor honestly both having support in jobs for ephemeral commands AND making the dbt extension smarter seems like things that would be nice to have.

I’ll make a note to pop an issue on the ephemeral commands later today

If I can help in any way for this PR please let me know

@Stéphane Burwash I don’t see this getting prioritized in the next few months, but we’re always willing to accept PRs (though this would probably be pretty involved…). Happy to talk more about the use case at the next Office hours though

On the main comment?

yep! I think that’s what shows up on the list view when sorted by popularity

@benjamin_mitzkus you can drop a like as well if you want 😉

iirc you can run

meltano invoke dbt:run -m test_models*

but yeah doesn't work with run