Hello everyone, for several days now I have been trying to expand the functionality of tap-slack. Now I'm stuck on the fact that I just can't do poetry install properly If someone understands what I need to do, please help i run with this manual

The library failing there (ciso8601) might be the issue. Do you know what references it, and if that dependency could be omitted?

The library failing there (ciso8601) might be the issue. Do you know what references it, and if that dependency could be omitted?

Honestly no, I'm new to python

I did some digging to refresh my memory. Our package

singer-sdk

used to depend on

pipelinewise-singer-python

, which relies on

ciso8601

. We removed that dependency in a newer version so as to resolve similar issues.

It's funny that when I played with it yesterday, everything was fine, then my python broke, I deleted everything related to python (somehow it turned out that there were 3 instances of python installed on my computer) Then I did a clean install via brew and ran into this today

That's telling you that in

pyproject.toml

, you need to bump the min python version to >=3.7.1.

But I have 3.9 installed and use it

Yeah, 3.9 is fine. Only 3.6 is the issue. Even though your local machine isn't trying to use 3.6, the

pyprojec.toml

is declaring that 3.6 is supported, which is the problem.

I realized that it seems that someone made some changes in this tap a long time ago and simply did not update the version to the current one. Right?

but I think it's a bad idea to change something here, because everything worked before me

Totally makes sense. The breakage here is three part: 1. We want to remove the dependency on

ciso

library to solve the install failure. 2. We bump the version of

singer-sdk

to remove the

ciso

library reference. 3. Bumping

singer-sdk

requires also bumping the min supported Python version.

I would be grateful as I don't want to break anything but just try to add a new stream inside

fyi https://github.com/MeltanoLabs/tap-slack/pull/13 cc @pat_nadolny @ken_payne

thankyou 👀

@aaronsteers ah ok so that was my original concern, I thought the pagination changes were required as part of the version bump. I can open a new PR to just bump the version

Yeah - I didn't think that was required - or rather than swap for a pre-built paginator we perhaps need to use the 'custom' one to avoid having to refactor and retest? @edgar_ramirez_mondragon - do you recall if some adaptation is needed?

So, I can only try to add what I need there after fixing the SDK version, right?

So, I can only try to add what I need there after fixing the SDK version, right?

@alex_dimov The version bump will get rid of the problematic dependencies

I didn’t think that was required - or rather than swap for a pre-built paginator we perhaps need to use the ‘custom’ one to avoid having to refactor and retest? @edgar_ramirez_mondragon - do you recall if some adaptation is needed

get_next_page_token

should continue to work without changes, so it’s not necessary to write a custom paginator at the moment

Hmmm, I bumped the version of

tap-slack

and found that the paginator did need reimplementing. I can’t recall why exactly, but I think it had something to do with how the tap implemented throttling on top of the paginator 🤔 I may have just responded to a depreciation warning, but as I was doing a few SDK version bumps at the time, I’d be surprised if I had gone to the extra effort if it wasn’t necessary 😅 Do we have a better mechanism for throttling now than adding sleeps to pagination?

I’ve used pyrate-limiter to configure client-side rate-limiting: https://github.com/MeltanoLabs/tap-stackexchange/blob/c0ecc6d94f2a506c03efb19ec4bb1a680342b6cf/tap_stackexchange/client.py#L73-L82

Cool, will check it out. In any case, for

tap-slack

I didn’t change the pagination/throttling behaviour, just migrated the logic to the new pattern. So it should be gtg. Will update the PR 👍

If I understand correctly, then you mean that now it would be enough just to add a new stream similar to users and everything should work? I tried, but tap made a request on my link (I saw in the logs) but the new collection did not appear, so I started implementing a custom paginator for my stream. I implemented it and if someone can look at it in my PR, I would be grateful, due to the fact that there is an old version of the SDK, I cannot run a local debug and see how everything works from the inside ( https://github.com/MeltanoLabs/tap-slack/pull/14

In

get_next_page_token

, you need

max_page = response.json()["paging"]["pages"]

Ahh

Looks like your

print

message got intercepted from stdout by

target-jsonl

or something? Not sure... Try logging with

<http://self.logger.info|self.logger.info>(f"next_page_token={next_page_token}")

sure, thanks. When I remove it, the application starts, but for some reason it does not make requests for the next page (returns only the first page)

I think your logic might have a bug - I'd suggest trying to step through the code with a debugger or print those token values so you can validate they're as expected

I can't run the debugger due to versioning issues inside tap-slack. I have to do meltano install extractor tap-slack --clean after each change and run to see the result

Probably in

get_next_page_token

again, you need a

<

instead of `==`:

if previous_token < max_page:
            token = previous_token + 1

I'm conscious that thread is getting quite long and off-topic from the original issue - maybe you should create a new one?

Yes, I used == for one of my checks, I already changed it to < sign. But I do not understand how to make it so that on the first request in the previous_value I get 1 instead of None

@alex_dimov I merged a couple PRs that bump the SDK version and other dependencies. If you pull those changes into your branch that might resolve your issues with being able to debug 👍

Thank you very much, I'll try it now