https://linen.dev logo
Join Slack
Powered by
I'm trying to add a custom transform and it doesn'...
# announcements
r
I'm trying to add a custom transform and it doesn't seem like 
--custom
is supported there. I found an example by looking at the discovery.yml for the format so I wrote it up by hand. The issue I'm facing is that the repo the 
pip_url
setting points to is private so I see a 404 error. I tried a few different things like prefixing 'git+ssh' and was able to avoid the 404 but then I get a message 
Command "python setup.py egg_info" failed with error code 1
.
So the usage of 
pip_url
in the configuration seems odd but I'm guessing the underlying code knows not to treat it as a python package if it's not prefixed with the 
git+
.
I tried using an entry in 
.netrc
as it seems pip will honor that in other contexts but it didn't work either. I'm out of ideas so thought I'd ask.
m
As far as transformation goes, you could create them directly in your 
transform/
directory, which is a dbt project of its own.
You don't need 
meltano add --custom
for this, unless you want to bundle them as a package.
r
OK, thanks. I'm unclear if I need the my_meltano_project if I go with that approach? Can I just create the structure under transform like I would any other dbt project?
It seems like 
my_meltano_project
is meant more for when you have a transformation bundled with a tap and you then want to customize that?
r
Please let me know if this doesn't make sense. Right now the docs are kind of light on implementing your own transformations in the absence of existing or provided ones.
m
You should do your transforms inside 
my_meltano_project
, and yes you are 100% right that our documentation is unclear about this as it is an advanced use-case.
@yannis_roussos could you please share some insights here, you've build quite a lot of transforms now.
y
At the moment when transforms run for a specific extractor, Meltano runs: 1. The bundled with Meltano transforms for that extractor (for example, the dbt transforms for Zuora if you've run 
meltano elt tap-zuora ...
) 2. Whatever the user has put under the 
transform/models/my_meltano_project/
directory
The reason we are doing that is cause a user may have added 10 different dbt packages to her project, so we are running dbt, explicitly selecting which models to run. Using the 
my_meltano_project
directory (as a default model to be used always) was a quick and easy solution to that problem. Going forward, when we are going to add orchestration, we should make available more options for which transforms (dbt models) are run
I hope that most of the issues discussed in this thread are addressed in the "Custom Transformations and Models Tutorial": https://www.meltano.com/docs/tutorial.html#advanced-adding-custom-transformations-and-models A quick note: Custom transforms work like a charm, but there is an issue at the moment with custom Models not showing up in Meltano UI. This problem is addressed in the following issue and, hopefully, it will be very soon fixed: https://gitlab.com/meltano/meltano/issues/580
r
Thanks to you both @micael_bergeron and @yannis_roussos. I've spent a lot of time with the docs and appreciate what you've provided online. We're not using the UI at the moment so that bug isn't an issue currently.
I'm seeing some odd behavior I'm hoping you can help with. I had my dbt transformations in the directory above 
my_meltano_project
previously and everything was working. After our discussion this morning, I moved them under 
my_meltano_project
and now I'm seeing this error when trying to run 
meltano elt tap-postgres target-postgres --transform run
Copy code
(meltano) rob@rob-Precision-5520:~/git/intelligo-elt$ ./elt-wrapper.sh
Transform 'tap-postgres' is missing, trying to install it...
Traceback (most recent call last):
  File "/home/rob/python-virtual-env/meltano/lib/python3.6/site-packages/meltano/core/config_service.py", line 24, in get_plugin
    for plugin in self.plugins()
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rob/python-virtual-env/meltano/lib/python3.6/site-packages/meltano/cli/elt.py", line 115, in install_missing_plugins
    plugin = config_service.get_plugin(PluginType.TRANSFORMS, extractor)
  File "/home/rob/python-virtual-env/meltano/lib/python3.6/site-packages/meltano/core/config_service.py", line 28, in get_plugin
    raise PluginMissingError(plugin_name)
meltano.core.plugin.error.PluginMissingError: tap-postgres
nm. I wasn't at 0.18.1. Once I upgraded all is well.
One thing I did run into with this change was now the tranform is complaining about a refernece in my 
dbt_project.yml
. It references the directory 
staging
but thinks it's non-existent. If I move the staging dir up a level then it works fine.
Specifically, this config doesn't work:
Copy code
models:
    my_meltano_project:
      enabled: true
      materialized: table
      staging:
        materialized: view
v
(cc: @danielle_morrill)
@rob_moore - thanks so much for trying to use the tool, do you mind creating an issue/s, so that we can ensure your bugs are taken care of if you see any or in general if you have proposals as to how to make this portion more intuitive? @yannis_roussos, would you be ok with trying to debug with @rob_moore tomorrow, please?
r
Thanks, @virjinia_alexieva. I'm happy to create tickets if that's preferable. I realize you have other work to get done so don't mean to barrage you with questions here. : )
I had a convo with @micael_bergeron and @jacob_schatz on the dbt Slack about a variation on the issue I'm seeing. It, too, has to do with the set up/location of 
dbt_project.yml
. I think micael understands the problem with running 
invoke
with dbt and has something in mind to address it.
Open in Slack
PreviousNext
Powered by