Hello friends wave been a while I was wondering if anyone ha Meltano #singer-tap-development

Hello friends :wave: been a while. I was wonderin...

Stéphane Burwash

09/05/2024, 11:24 AM

Hello friends 👋 been a while. I was wondering if anyone had any insights on how to more easily explode a tap run into multiple multiple different DAGS? Example: I have an extractor with 10 streams (stream_a, stream_b, ...) Currently, I'm running everything as 1 task:

metlano run tap target

I would like to explode this into multiple tasks inside 1 job:

meltano run tap-stream_a target; meltano run tap-stream_b target; ...

Or even inside different jobs. The easiest way in my mind to do this would be to create 1 child extractor per stream, but this would be a quite manual and tedious process. I was wondering if there was a way in the CLI to select streams inside an extractor. Example:

meltano run tap.stream_a target; meltano run tap-stream_b target; ...

Please let me know if this question requires additional clarification. Thanks 😄

👋 1

✅ 1

visch

09/05/2024, 12:54 PM

https://docs.meltano.com/concepts/plugins/#select_filter-extra is what you're looking for 🙂

👍 1

Stéphane Burwash

09/05/2024, 1:55 PM

Thank you sir, as always 🫡

🫡 1

np 1

Stéphane Burwash

09/05/2024, 5:49 PM

@visch @Edgar Ramírez (Arch.dev) would you know if the

select_filter-extra

feature is functional for the

run

command? Here is the output I am getting:

Copy code

root@ee93c6fc5718:/project/meltano# meltano run tap-hibob target-jsonl --select employees_search
Usage: meltano run [OPTIONS] [BLOCKS]...
Try 'meltano run --help' for help.

Error: No such option: --select
root@ee93c6fc5718:/project/meltano#

Following the https://docs.meltano.com/concepts/plugins/?meltano-tabs=terminal#how-to-use-5 instructions. I'm using

meltano, version 3.5.0

Stéphane Burwash

09/05/2024, 5:52 PM

I can see it supported in the

elt

command

Copy code

-s, --select TEXT               Select only these specific entities for
                                  extraction.
  -e, --exclude TEXT              Exclude these specific entities from
                                  extraction.

Edgar Ramírez (Arch.dev)

09/05/2024, 5:53 PM

Not as a CLI option for

meltano run

, but the env vars should work, i.e.

TAP_HIBOB__SELECT_FILTER=employees_search meltano run ...

ty 1

🎯 1

🙌 1

2 Views

Open in Slack

Previous Next