https://linen.dev logo
Join Slack
Powered by
Hey Douwe! Quick question, singer requires pytz==2...
# announcements
m
Hey Douwe! Quick question, singer requires pytz==2018.4, however my tap needs to use pytz==2020.5, because the older one is presenting issues with Day Light Saving for Brazil. If I add it to my tap 'setup.py' the requiremenet for pytz==2020.5, the extractor installation succeeds, but running etl fails.
d
Hmm, it's odd that bumping pytz would cause it to no longer be able to find all of 
singer.catalog
m
This is the error with log-level=debug:
d
The first thing I'd check is the file path in that `cannot import name `singer.catalog`` error, that points at 
catalog.py
. Does that file actually have the 
write_catalog
function?
I wonder if you bumping 
pytz
led to an older singer-python being installed that didn't require 
==2018.4
, which doesn't have that function yet
m
Hmmm, awesome points, let me check
d
Does your tap's 
setup.py
also pin a specific version of singer-python?
m
No, it allows any singer version
d
All right, then I think you ended up with an old singer-python
I suggest pinning that to the desired version as well
pip install
(and 
meltano install
) may complain about the incompatibility between those requirements, though
But it's possible that it won't actually stop the tap from running despite that
m
Great! Let me test 🙂
Thank you 😄
d
Of course singer-python shouldn't actually pin to a specific version of pytz, it should at most list a minimum version, so I'd suggest contributing a PR to fix that. Looks like something similar has been requested before, though, to no response: https://github.com/singer-io/singer-python/issues/114
And there are more complaints about unnecessarily pinned dependencies: https://github.com/singer-io/singer-python/pull/135 https://github.com/singer-io/singer-python/issues/97
(@aaronsteers and I will try to make sure the new Singer SDK will have none of those issues 😄)
m
I searched in their repo, unfortunately they haven't respond to it.
What is the Singer SDK? Does it work on top of Singer, or is it a 'forked version'?
d
It's a new framework we're building: https://gitlab.com/meltano/meltano/-/issues/2401
m
Awesome! I'll take a look
d
It's not quite ready yet 🙂 But it'll be the recommended way to build Singer taps instead of directly using singer-python. v0.0.1 is on the horizon
As for your current issue, did pinning both pytz and singer-python work or are you getting an error about the incompatibility?
If that fails, one workaround you could try is to directly specify 
pytz==2020.5
in the 
pip_url
in 
meltano.yml
, ahead of your reference to 
tap-bling-erp
, but that may lead to the same error
if we can't work around the incompatibility error, your best bet may be to fork singer-python, make the change, and directly reference your fork by Git URL from the 
pip_url
. You'd then also want to contribute that change back to the canonical repo, although it remains to be seen how responsive they'll be in terms of merging it
m
Yes, it doesn't allow to install it:
d
m
The main problem with the old pytz is that Brazil is not in DSL because of a presidential order last year. The new version takes it into account the older one no 😞
d
I guess singer-python wants us to forever be stuck in 2018
m
Hahahaha
Maybe I'll try to find a workaround with timezones conversion. I didn't want to send a PR for Singer haha,
d
Understandable
m
But thank you!!
d
Did you try adding pytz to 
pip_url
as well? I'm not confident it would work, but it's worth a try
Although I guess it still goes through 
pip install
, so there's no reason it would be less strict in this case
m
Sorry, where does this pip_url go inside the meltano.yml?
d
You probably already have one in the tap-bling-erp definition
What is it set to currently?
m
Copy code
- name: tap-bling-erp
    namespace: tap_bling_erp
    pip_url: -e git+<https://github.com/Muriloo/tap-bling-erp.git#egg=tap-bling-erp>
    executable: tap-bling-erp
    capabilities:
    - catalog
    - discover
    - state
    settings:
    - name: api_url
    - name: api_token
    - name: start_date
d
All right, I don't know if it'll work correctly with 
-e
, so try 
pip_url: pytz==2020.5 git+<https://github.com/Muriloo/tap-bling-erp.git#egg=tap-bling-erp>
And then 
meltano install extractor tap-bling-erp
m
Ok!
It installs an older singer version
When I pin only the singer version, and try to add pytz to URL it doesn't allow due to compatibility issue
d
😞
Then I'd go for forking it, or working around the issue somehow with the latest pytz
m
Thank you Douwe!
Hi @douwe_maan! Sorry to come back to this again. It really bothers me that I can't do it properly because of Stitch requirement haha. There is no other way of installing pytz==2020.5 with for example an 'alias' as pytz20205, and force the tap code to import it via the package alias?
d
@murilo_tavolaro_de_nigris As far as I know, no, because a package always defines its own module structure and pytz uses pytz as the top level module. You can't override that at install time because the internal references in the package would also break...
You're trying to get pip to not do the very thing it was built for: dependency and version resolution 😄
You may want to see if https://pypi.org/project/pipelinewise-singer-python/ / https://github.com/transferwise/pipelinewise-singer-python can act as a drop-in replacement for singer-python in your tap, so that you won't have the fork singer-python yourself.
m
Thank you 😄
9 Views
Open in Slack
PreviousNext
Powered by