https://meltano.com/ logo
Channels
Powered by
#announcements
Title
# announcements
c

curved-lunch-43500

01/22/2021, 10:11 PM
Hey Douwe! Quick question, singer requires pytz==2018.4, however my tap needs to use pytz==2020.5, because the older one is presenting issues with Day Light Saving for Brazil. If I add it to my tap 'setup.py' the requiremenet for pytz==2020.5, the extractor installation succeeds, but running etl fails.
✅ 1
r

ripe-musician-59933

01/22/2021, 10:13 PM
Hmm, it's odd that bumping pytz would cause it to no longer be able to find all of 
singer.catalog
c

curved-lunch-43500

01/22/2021, 10:14 PM
This is the error with log-level=debug:
r

ripe-musician-59933

01/22/2021, 10:15 PM
The first thing I'd check is the file path in that `cannot import name `singer.catalog`` error, that points at 
catalog.py
. Does that file actually have the 
write_catalog
function?
I wonder if you bumping 
pytz
led to an older singer-python being installed that didn't require 
==2018.4
, which doesn't have that function yet
Does your tap's 
setup.py
also pin a specific version of singer-python?
c

curved-lunch-43500

01/22/2021, 10:16 PM
Hmmm, awesome points, let me check
No, it allows any singer version
r

ripe-musician-59933

01/22/2021, 10:16 PM
All right, then I think you ended up with an old singer-python
I suggest pinning that to the desired version as well
pip install
(and 
meltano install
) may complain about the incompatibility between those requirements, though
But it's possible that it won't actually stop the tap from running despite that
c

curved-lunch-43500

01/22/2021, 10:18 PM
Great! Let me test 🙂
Thank you 😄
r

ripe-musician-59933

01/22/2021, 10:19 PM
Of course singer-python shouldn't actually pin to a specific version of pytz, it should at most list a minimum version, so I'd suggest contributing a PR to fix that. Looks like something similar has been requested before, though, to no response: https://github.com/singer-io/singer-python/issues/114
And there are more complaints about unnecessarily pinned dependencies: https://github.com/singer-io/singer-python/pull/135 https://github.com/singer-io/singer-python/issues/97
(@salmon-actor-23953 and I will try to make sure the new Singer SDK will have none of those issues 😄)
👍 1
c

curved-lunch-43500

01/22/2021, 10:24 PM
I searched in their repo, unfortunately they haven't respond to it.
What is the Singer SDK? Does it work on top of Singer, or is it a 'forked version'?
r

ripe-musician-59933

01/22/2021, 10:25 PM
It's a new framework we're building: https://gitlab.com/meltano/meltano/-/issues/2401
c

curved-lunch-43500

01/22/2021, 10:25 PM
Awesome! I'll take a look
r

ripe-musician-59933

01/22/2021, 10:26 PM
It's not quite ready yet 🙂 But it'll be the recommended way to build Singer taps instead of directly using singer-python. v0.0.1 is on the horizon
As for your current issue, did pinning both pytz and singer-python work or are you getting an error about the incompatibility?
If that fails, one workaround you could try is to directly specify 
pytz==2020.5
in the 
pip_url
in 
meltano.yml
, ahead of your reference to 
tap-bling-erp
, but that may lead to the same error
if we can't work around the incompatibility error, your best bet may be to fork singer-python, make the change, and directly reference your fork by Git URL from the 
pip_url
. You'd then also want to contribute that change back to the canonical repo, although it remains to be seen how responsive they'll be in terms of merging it
c

curved-lunch-43500

01/22/2021, 10:30 PM
Yes, it doesn't allow to install it:
r

ripe-musician-59933

01/22/2021, 10:31 PM
c

curved-lunch-43500

01/22/2021, 10:31 PM
The main problem with the old pytz is that Brazil is not in DSL because of a presidential order last year. The new version takes it into account the older one no 😞
r

ripe-musician-59933

01/22/2021, 10:32 PM
I guess singer-python wants us to forever be stuck in 2018
c

curved-lunch-43500

01/22/2021, 10:32 PM
Hahahaha
Maybe I'll try to find a workaround with timezones conversion. I didn't want to send a PR for Singer haha,
r

ripe-musician-59933

01/22/2021, 10:33 PM
Understandable
c

curved-lunch-43500

01/22/2021, 10:33 PM
But thank you!!
r

ripe-musician-59933

01/22/2021, 10:33 PM
Did you try adding pytz to 
pip_url
as well? I'm not confident it would work, but it's worth a try
Although I guess it still goes through 
pip install
, so there's no reason it would be less strict in this case
c

curved-lunch-43500

01/22/2021, 10:34 PM
Sorry, where does this pip_url go inside the meltano.yml?
r

ripe-musician-59933

01/22/2021, 10:34 PM
You probably already have one in the tap-bling-erp definition
What is it set to currently?
c

curved-lunch-43500

01/22/2021, 10:35 PM
Copy code
- name: tap-bling-erp
    namespace: tap_bling_erp
    pip_url: -e git+<https://github.com/Muriloo/tap-bling-erp.git#egg=tap-bling-erp>
    executable: tap-bling-erp
    capabilities:
    - catalog
    - discover
    - state
    settings:
    - name: api_url
    - name: api_token
    - name: start_date
r

ripe-musician-59933

01/22/2021, 10:35 PM
All right, I don't know if it'll work correctly with 
-e
, so try 
pip_url: pytz==2020.5 git+<https://github.com/Muriloo/tap-bling-erp.git#egg=tap-bling-erp>
And then 
meltano install extractor tap-bling-erp
c

curved-lunch-43500

01/22/2021, 10:36 PM
Ok!
It installs an older singer version
When I pin only the singer version, and try to add pytz to URL it doesn't allow due to compatibility issue
r

ripe-musician-59933

01/22/2021, 10:51 PM
😞
Then I'd go for forking it, or working around the issue somehow with the latest pytz
👍 1
c

curved-lunch-43500

01/22/2021, 11:30 PM
Thank you Douwe!
Hi @ripe-musician-59933! Sorry to come back to this again. It really bothers me that I can't do it properly because of Stitch requirement haha. There is no other way of installing pytz==2020.5 with for example an 'alias' as pytz20205, and force the tap code to import it via the package alias?
r

ripe-musician-59933

01/26/2021, 4:16 PM
@curved-lunch-43500 As far as I know, no, because a package always defines its own module structure and pytz uses pytz as the top level module. You can't override that at install time because the internal references in the package would also break...
You're trying to get pip to not do the very thing it was built for: dependency and version resolution 😄
😅 1
You may want to see if https://pypi.org/project/pipelinewise-singer-python/ / https://github.com/transferwise/pipelinewise-singer-python can act as a drop-in replacement for singer-python in your tap, so that you won't have the fork singer-python yourself.
c

curved-lunch-43500

01/26/2021, 4:57 PM
Thank you 😄
Powered by