Can you please tell me what could be the reason that I am not getting any data other than users.* for SLACK? I don't see any errors in the logs

Hey @alex_dimov ! What do the logs look like when you run it? I dont know for sure whats happening but a few things you can try: • the start date you have looks to be incorrectly formatted, I would expect to see an error for that. You can change it to

2022-10-10

•

auto_join_channels

is false for you but in the hub docs it says "Whether the bot user should attempt to join channels that it has not yet joined. The bot user must be a member of the channel to retrieve messages." so maybe your bot doesnt have access to the channel data

yes, it's strange that I didn't find in the documentation what format the date should be in. Can I get data from /api/team.accessLogs, /api/team.integrationLogs, /api/files.list endpoints somehow?

Can I get data from /api/team.accessLogs, /api/team.integrationLogs, /api/files.list endpoints somehow?

It doesnt look like those streams are supported in the tap right now, I'm not super familiar with the slack API but you could create an issue/PR in that repo if you want to get them added.

I corrected the date and changed the flag(

auto_join_channels

) to true, but still I get only one table with users

Can you share the command you're using to run the sync? Also try running it with log level debug to get more info and share it if you can

my command

meltano run tap-slack target-bigquery

You can try running with

meltano invoke

to see if its potentially a state issue with run. Or run with

meltano run tap-slack target-bigquery --full-refresh

. What do the logs look like after the metadata? It looks like theyre cut off in your screenshot. And obvious question but I have to ask, the slack channels youre trying to pull from definitely have messages after your start date?

meltano invoke tap-slack

response

@alex_dimov hmm 🤔 can you try

curl -X GET -H 'Authorization: Bearer <Your Token>' <https://slack.com/api/conversations.list?exclude_archived=False>

with your token added? Thats the request that the tap makes to get channel data then all other streams (channel members, messages, threads) are children of that. If it doesnt return any channels potentially because of permissions then you wont get any data.

sure, one moment

Ok nice that helps! Did you get it working after adjusting the scopes?

I don't have scope extension rights. I asked the administrators. Now I'm trying to add another endpoint to slack-tap

Nice! Feel free to fork the repo and create a PR with any proposed changes

Okay, thanks, but it looks like I need to get this to work locally first. I'm trying to do this by analogy with the instructions for creating a Custom Extractor

A good first step could be to create an issue in the tap-slack repo with the features youre trying to add, then we can discuss there. To run locally you can execute the tap with poetry run tap-slack, I think the README has some instruction about that. If you want to run it in your meltano project you can edit the pip_url in your meltano.yml to point to your fork git url and run a clean install like

meltano install extractor tap-slack --clean

to pull in the new code.

based on documentation, i was found this variant

Can you say more about your issue? I'm not following. If you want to contribute a new feature create an issue in https://github.com/MeltanoLabs/tap-slack/issues describing it then if you have a PR I/we can review the changes!

I am trying to add another data source for slack. I will need to create a lot of custom taps and add the endpoints I need to the existing ones. So I decided to start with slack as I thought it shouldn't be hard, but it still doesn't work for me(

@alex_dimov it looks like theres 2 threads going on the same topic, can we continue over in https://meltano.slack.com/archives/CMN8HELB0/p1668527444888179 ?