https://linen.dev logo
Join Slack
Powered by
Hi, I want to extend the existing csv tap extracto...
# plugins-general
h
Hi, I want to extend the existing csv tap extractor. i.e. to handle the data types, errors ..etc. Tried the cookie cutter example too, but have various issues when trying to install it. Any good video to refer about the custom extractor writing and installation?
t
h
Oh, I see. I tried a relative path for the executable. Let me try with an absolute path. thanks.
d
@aaronsteers Relative path should work, right?
h
It seems relative path also might work. Cos I got the same error when I tried with a relative path. This is the error I'm getting. "ModuleOrPackageNotFound". I saw few youtube videos related to taps. I could not find a proper one for a tap installation. Any idea to do one for us?
Untitled
d
@hirantha_bandara Can you share your 
meltano.yml
definition for 
tap-mycsvtap
?
@hirantha_bandara What Meltano version are you using? 
meltano --version
I'm surprised that 
meltano install
is still trying to actually install the plugin using pip instead of skipping that step, since there's no 
pip_url
h
meltano, version 1.75.0
d
All right, that is the latest version. And with the 
meltano.yml
you just shared over DM, you're still getting that same 
failed to install plugin 'tap-mycsvtap'.
error message from 
meltano install
?
h
yes
d
Very weird, the error output is also talking about poetry and 
Discarding file:///home/hirantha/Experiments/Meltano/meltano-projects/myproject3/extract/tap-mycsvtap
, but I don't know why that would come into play at 
meltano install
instead of `meltano elt`/`meltano invoke`
@aaronsteers Any ideas?
h
I actually don't understand how to build the tap first locally and then install it to meltano project. Assume tap project is in a different folder (cookie cutter) and the meltano project is in different folder.
I assume if I build and install the tap, then i can add the extractor to meltano project as a custom extractor.
d
Yeah, that should work
@aaronsteers should be able to provide some more insight
h
so the pip url also should refer to an absolute path?
d
No, if you have a local installation, you just need 
executable
, not 
pip_url
Do you have a 
pip_url
set currently?
h
okay.
no I removed it.
a
@hirantha_bandara - It looks like there was a pip_url set previously, since this is pointing to a meltano-managed virtual environment: 
/home/hirantha/Experiments/Meltano/meltano-projects/myproject3/.meltano/extractors/tap-mycsvtap/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py
You might try removing that plugin first with 
meltano remove
. You could also just bring back any valid pip_url to get through the 
install
step, but still use your custom shell script path, as you have it.
Warning though: meltano remove will also undo your meltano.yml config.) You can comment it out before running meltano remove if you don’t want to have it removed.
h
got it, I'll give it a try.
d
@aaronsteers Even if a venv already exists, why would 
meltano install
try to reinstall in the venv when there's no 
pip_url
?
a
I’m not sure. The only thing I can think of is a cached pip_url somewhere, which would be a bug if so.
I actually don’t understand how to build the tap first locally and then install it to meltano project. Assume tap project is in a different folder (cookie cutter) and the meltano project is in different folder.
@hirantha_bandara - Just to touch back on these points: 1. The cookicutter and meltano projects in different folders is expected, yes. 2. There’s not really a ‘build’ step, per se, but the analagous step is the 
poetry install
command being run from your custom tap’s project folder. That command installs all the tap dependencies in its own virtual environment, and it’s run automatically by the shell script (tap-mycsvtap.sh) so you can just “run” the shell script and everything gets updated automatically whenever you call it. 3. Meltano has it’s own virtual-environment management features to prevent version conflicts between plugins, but in this case we’re bypassing those so we can just call the shell-script executable directly.
Is that helpful?
h
@aaronsteers, @douwe_maan - Thanks for your guidance. I need to jump into another call. It is midnight to me. I'll update you both about the progress tomorrow. Thanks again,
a
👍 Thanks for the update! Hopefully this gets you through. We will also look into the issue you were seeing with the virtual environment being used after removing pip_url. Cheers!
h
@aaronsteers, @douwe_maan I managed to write a custom tap for my requirement. When installing I gave absolute paths to both executable and pip url. With this I could overcome my installation issue. (If I don't give the pip url then it skips my tap installation) Then when I try to invoke my tap, I had to remove the executable path from the meltano.yml file. May be this is not the correct approach. But for my requirement this is enough for now. Anyway, thanks for the help.
Open in Slack
PreviousNext
Powered by