https://linen.dev logo
Join Slack
Powered by
Is it possible to create a custom tap based on an ...
# getting-started
a
Is it possible to create a custom tap based on an existing one? I want to try adding integrationLogs inside slack-tap
s
Sure, just fork the github repository: https://github.com/MeltanoLabs/tap-slack.
a
I have already forked and uploaded it to the folder with the main meltano project. Should I do this item and everything after it? https://docs.meltano.com/tutorials/custom-extractor#5-install-the-newly-created-tap And should I somehow rename it so that I can use it in the main project?
s
You got it to work right? for reference: you just need to reference your new extractor as plugin. Here's an example: https://docs.meltano.com/tutorials/custom-extractor#2-edit-your-existing-meltanoyml-file.
a
It seems so, but I can't get any data
message has been deleted
message has been deleted
s
hm that says "api_key" is a required property that you did not supply if I read the debugging message correctly (yes I know, a lot of debugging messages)
a
but in .env i have api_key set
s
...
a
I didn't change anything inside the tap config. What I did was just add an IntegrationLogsStream and update STREAM_TYPES
or which config you mean?
s
...
a
yes, but i can't see the same field in your example here https://github.com/MeltanoLabs/tap-slack
it`s worked for me, but again only users (i check, and i has scope for team.integration_logs
s
I noticed, @pat_nadolny already authored a PR to fix that on the hub, in general you can refer to the code itself, but of course that's not optimal at all https://github.com/MeltanoLabs/tap-slack/blob/main/tap_slack/tap.py.
Maybe @pat_nadolny can help you? I see he did some work on the slack tap.
p
@alex_dimov thats strange your .env value should still work. Its probably something like 
TAP_SLACK_API_KEY
(is that whats in your .env?) so you shouldnt need to set it in the meltano.yml using templating 😄 . Try running with log level debug or 
meltano --environment=dev config tap-slack
to see whats configured.
a
Wow, it seems that this is not the latest version of my config, and in general it seems that this is what is left of the previous integration with slack tab from the uase repository
Can I somehow reset all configurations or update them to the latest version?
I thought that any changes inside the yaml file would be automatically caught
p
I'm not sure I follow your issue 🤔
You confirmed you have 
TAP_SLACK_API_KEY
in your .env file? Its not populating in the config in your screenshot so it looks to be missing
a
message has been deleted
So
Goal: Expand the list of tap-slack data sources Action (before that, tap-slack from github worked fine for me): 1. I forked tap-slack 2. I cloned the repository to the root of my meltano project 3. I added IntegrationLogsStream and added it to STREAM_TYPES 4. In emltano.yml in extractors -> tap-slack -> pip_url I replaced git+https://github.com/MeltanoLabs/tap-slack.git with -e ./tap-slack. 5. meltano install 6. meltano run tap-slack target-jsonl 7. Receive error
p
Can you run 
meltano install extractor tap-slack --clean
? That will clear out anything cached from the github git url install
a
So, i decide create new project, than put this and test
Copy code
version: 1
default_environment: dev
project_id: 7b3b22b3-0ea1-4a13-b384-71a7879c85d3
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-slack
    pip_url: -e ./tap-slack
    config:
      start_date: '2021-01-01'
    select:
      - users.*
      - channels.*
      - messages.*
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
again error 😞
message has been deleted
p
What is token in your config? I dont see that in the original tap.py it just has api_key. Did you change that in your fork or might it be configured incorrectly?
a
Now I'm trying to run this with a pure clone of the original tap-slack I set this api_key using 
meltano config tap-slack set --interactive
message has been deleted
Just tested and ran again using github link instead of -g ./ and in this case everything works. It turns out that it does not work only if I select a tap that is located locally
p
I suspect you've made a change in your fork, theres no 
token
setting referenced anywhere in this https://github.com/MeltanoLabs/tap-slack repo
a
i see the same, but i do not understand why...
@pat_nadolny @Sven Balnojan here is my PR and my changes initial new stream with integration logs
Another thing I did and I think the result is strange. 1. I pushed my changes to my repository 2. I changed meltano.yml (put pip_url in my repository) 3. I ran meltano install --clean 4. I ran meltano select tap-slack --list and expected to see new attributes there(ex. integration_logs.*), but as a result I see the same attributes as in the original repository. I just can't find that step that I've been missing somewhere all this time.
I just initialized a new project, added my tap-slack fork there, my integration_logs appeared in the select attributes list. Now I can say that the only problem that I have now is that for some reason when I call meltano config tap-slack set --interactive I am asked for the first element token (instead of api_key), I do not understand how my changes could affect this setting (I compared the tap.py file from the original repository with mine and there are no differences)
Maybe it’s possible to debug? And found reason why it require token instead api_key?
@pat_nadolny I would be very grateful for any help in this matter
I do not understand why i see 'token' in config instead of api_key
p
@alex_dimov related to token still being present, I wonder what your lock file looks like for the tap-slack plugin. I wonder if you have a cached lock file that holds metadata about a tap that is different. Can you share that file?
Related to the catalog/select issue you raised, theres an open issue somewhere (i'll try to find it) about catalog caches not being cleared so I bet thats what youre seeing. You can delete your 
.meltano/run/tap-slack
directory where that file might be cached
a
I just managed to run meltano run tap-slack target-jsonl , there were no errors, but in the results I saw only users.jsonl
But in the logs I see a request for my newly added endpoint
this file? tap-slack/poetry.lock
poetry.lock
p
no it should be in your meltano project directory 
plugins/extractors/tap-slack--meltanolabs.lock
good to hear your endpoint is being called. Did you confirm youre getting data from that API request elsewhere (curl/python/postman/etc.)?
a
yes
yes i could get the data via curl
my folder plugins/extractors/ is empty
message has been deleted
p
what meltano version are you on?
a
2.9.0
p
Also for debugging your tap changes you might want to consider running your tap in debug mode directly (not through meltano) - see https://sdk.meltano.com/en/latest/dev_guide.html#vscode-tips for doing that with VS code. Then you can step through and see what the response looks like
a
message has been deleted
is it possible to debug via pyCharm?
p
2.9.0
Hmm - youre in kind of a unique development state. If it were me I'd probably run 
meltano rm extractor tap-slack
then 
meltano add extractor tap-slack
which will install the default repo, then manually sub in your git url or local reference git url into the meltano.yml, then 
meltano install extractors tap-slack --clean
to pull the new code.
a
It seems I need to reset to create a clean project and try again
Regarding tap debugging without the participation of meltano. Is it possible to debug using pyCharm?
So I deleted everything I had. I created a new project, did as you wrote to me a little above, executed meltano config tap-slack set --interactive then meltano select tap-slack integration_logs meltano select tap-slack users meltano run tap-slack target-jsonl And yes, I have no errors, but so far I have not been able to get data on the new stream. I only got data about users. I will be grateful if you tell me how to debug me to find the reason why I am not getting data. It seems to me that when adding a new stream, I should have added some function inside that would determine how to work with pages. Since in this response we have a "paging" object: { "count": 100, "total": 13, "page": 1, "pages": 1 }
I am incredibly grateful to you. I wouldn't have made it without you (it's always hard to start working on something for the first time, especially if you don't know python)
@pat_nadolny perhaps you have seen my PR and can you tell me what is missing there? I don't quite understand how I can specify a link to the next page, or for example I can simply describe how the pagination object is stored and how to specify, for example, the size of the page
p
@alex_dimov I responded in the PR https://github.com/MeltanoLabs/tap-slack/pull/14. Can you start a new thread in #C01PKLU5D1R if you have more questions? This thread is getting long and isnt about getting-started anymore 😄
a
thanks a lot, i missed this chanell. Thank again
Regarding the fact that we can not make this stream available by default. I agree, I will be glad if you tell me what I should do to make this stream optional
p
Sounds good - would you mind putting any responses in that PR thread so all communications are in one place? Thanks!
a
yes, finally. I will formulate it
Open in Slack
PreviousNext
Powered by