Anyone with experience adding <https://github.com/...
# plugins-dbtk
k_s
02/16/2022, 10:04 AMAnyone with experience adding https://github.com/fal-ai/fal to their meltano project as a custom utility?
t
taylor
02/16/2022, 4:50 PMI’d be curious to see how it goes for folks. with the
meltano runj
jacob_matson
07/12/2023, 2:05 AM@taylor is there anywhere in the docs that docs about how to run arbitrary python with
meltano runjacob_matson
07/12/2023, 2:14 AMAll i see in the docs is
Copy code
meltano run tap-gitlab target-postgres simple-jobt
taylor
07/12/2023, 1:46 PMyou would add it as a custom utility https://docs.meltano.com/concepts/plugins#custom-utilities
taylor
07/12/2023, 1:46 PMbut even looking at that, it’s not a great example
taylor
07/12/2023, 1:48 PM@pat_nadolny do we have a good example in Squared of just running a random python script?
p
pat_nadolny
07/12/2023, 2:25 PMI dont have one in squared but I posted this a while ago. The example uses bash but it should work all the same with a python script too https://meltano.slack.com/archives/C01TCRBBJD7/p1685628417315209?thread_ts=1685613727.607619&cid=C01TCRBBJD7
j
jacob_matson
07/12/2023, 4:52 PMoh sweet will try this
jacob_matson
07/12/2023, 4:52 PMthank you @pat_nadolny!
jacob_matson
07/13/2023, 12:35 AMok i can get it to run with
meltano invoke custom_util:commandjacob_matson
07/13/2023, 12:35 AMi'm cheating and hooking it into another util but the syntax works.
jacob_matson
07/13/2023, 12:40 AMis the job syntax for this
- invoke custom_util:commandp
pat_nadolny
07/13/2023, 2:07 PMI think you just need the plugin + command that comes after your
meltano runmeltano run custom_util:commandmeltano run tap-x target-y dbt-postgres:runj
jacob_matson
07/13/2023, 2:30 PMah ok. i guess i will just add it as a shim to dbt for now.
t
taylor
07/13/2023, 3:04 PMadd it as a shim to dbt for now.I don’t think it needs to be a shim for dbt specifically. you can do
meltano run <tap> <target> custom_util:commandmeltano run custom_util:commandj
jacob_matson
07/13/2023, 3:04 PMwell alternatively i have to make a custom util, right? that seems harder?
t
taylor
07/13/2023, 3:09 PMmaybe slightly. I’m sure we can bang it out real quick if you want to hop on a call cc @pat_nadolny 😄
p
pat_nadolny
07/13/2023, 3:18 PMcan you share more about the use case? Yeah I'm understanding shim as a command off of the dbt plugin so you'd run it like
meltano run dbt-snowflake:my_custom_py_script Copy code
plugins:
utilities:
- name: dbt-snowflake
variant: dbt-labs
pip_url: dbt-core~=1.3.0 dbt-snowflake~=1.3.0 git+<https://github.com/meltano/dbt-ext.git@c6a30cac299184bae9424ccf9db8f41b30b27314>
commands:
my_python_script:
executable: python
args: transform/my_python_script.py Copy code
transform/my_python_script.pyprint("Hello!!!")meltano run dbt-snowflake:my_python_script Copy code
(meltano) Patricks-MBP:data pnadolny$ meltano run dbt-snowflake:my_python_script
2023-07-13T15:14:37.699304Z [info ] Environment 'userdev' is active
2023-07-13T15:14:38.442054Z [info ] Hello!!! cmd_type=command name=dbt-snowflake stdio=stderr
2023-07-13T15:14:38.445345Z [info ] Block run completed. block_type=InvokerCommand err=None set_number=0 success=Truepat_nadolny
07/13/2023, 3:20 PMBut yes the alternative is to build a custom utility, heres an example that I built custom for squared https://github.com/meltano/squared/tree/main/data/utilities/snowflake-cloner. It was useful to use a standalone utility in this case because the code was more complex and I needed to install dependencies
j
jacob_matson
07/13/2023, 3:21 PMyes i probably need to consider the same
jacob_matson
07/13/2023, 3:24 PMmy use case is writing parquet files to azure blob storage
3 Views