Hello dears, Hope you're having a good start of the week. When installing my custom extractor, the python tap file extractor>tap>venv>bin is initialized to use the cli.

#!/project/.meltano/extractors/tap-whise/venv/bin/python

# -*- coding: utf-8 -*-

import re

import sys

from tap_whise.tap import Tapwhise

if __name__ == '__main__':

sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])

sys.exit(Tapwhise.cli())

However, I want it to use the main. Such as:

from tap_whise import main

[...]

sys.exit(main())

Can you guide me please?

@charley_guillaume - If not using the poetry/shell approach from the SDK defaults, you'll probably need to pip-install with editable mode during testing. To do this, you can specify your pip_url prefixed with

-e

. The two options for local dev/test of taps using meltano are: (1) specify an executable and no

pip_url

, and (2) specify an 'editable mode' pip_url with the

-e

flag.

This seems complicated tbh. Tap is working correctly when I am using the

meltano invoke

! I'll check your method in a few minutes

This seems complicated tbh.

Appreciate your feedback. And others have shared similar feedback so it's definitely something we are thinking about. The SDK was built with the smoothest onramp being via pyproject.toml and poetry. We have more info here: https://sdk.meltano.com/en/latest/python_tips.html#tip-1-intro-to-virtual-environments-poetry-and-pipx Others have noted that they are used to using pip and pre-PEP-517/518 install methods, and we may add additional guidance in the future on how to get this working without poetry and pyproject.toml. (Although it's generally harder and more steps to do so.)

At first I tried to use the SDK as it seemed to be easier. But I found it too complicated for my usecase. I can try it back as I handle a bit more the idea of building a tap, if it helps getting things ready at a higher pace

Thanks for sharing your experience. And sorry to hear it was not more intuitive for you. We also have #C01QS0RV78D if it's ever helpful to come chat about what you are seeing and how to build out what you need to given your use cases.

I'll have a look! Thanks for your insights!

I think so - can you clarify a bit on the requirement? Do I understand correctly that you are calling a list-type endpoint followed by a describe endpoint on each entity? And that the list endpoint doesn't contain sufficient detail on it's own? If you don't mind sharing the API name or endpoint docs url, those would also be helpful. Thanks!

the api is this one: http://api.whise.eu/SystemIntegrator.html#section/Authentication And my current code is that: https://github.com/we-invest-real-estate/tap-whise#egg=tap-whise

I'm looking into the list option. The auth piece is also interesting though. We do have JWT support but most of our examples get a single tJWT token for all subsequent calls (with allowance for timeouts, etc.). Do I understand correctly the same extract process needs multiple tokens? Is there any option to simply request the needed scopes in the initial Auth flow?

The same extract process need on token for each office indeed! I'll definitely have a look at the parent-child stream. I think I used the None/ Custom in order to recreate the jwt token for each office

What's the approx. order of magnitude of offices are you dealing with? 10, 100, 1000+?

Nowadays more like 10. In the near future could be hundreds

This isn't a supported pattern as of now, and I'm not sure how you would implement it. Do you mind logging an issue on this within the SDK repo so we can discuss and explore options for future?: SDK Issue Tracker

Excellent! Thanks for your feedback and your insights. I'll do that and keep you posted with my answer 🙂

Much appreciated!

The multiple JWT are needed because of the role we have over the offices 🙂 Thank you ! 😄

Can you double-check the url? The above only has a partial org id but maybe that's just in what you pasted here.

Thanks for confirming. I don't have any other ideas unfortunately. I always find the setup.py process tricky to debug.

really appreciate all the help you've provided! Thanks and have a good afternoon 🙂

Happy to help! If testing using "regular pip/pipx", the idea was just to ignore meltano in order to test first if the install itself is working. Once you know the pip install is working, then you would use the

meltano add --custom extractor ...

which would run the pip instructions as you specify in the prompts.

Hello @aaronsteers, I was able to pip install my tap on my machine and it worked like a charm. But i'm still not able to install it with meltano 😕

Okay, cool. Glad to hear it!