brief question If I have two streams now in a tap and I want Meltano #singer-tap-development

brief question, If I have two streams now in a ta...

emcp

10/27/2021, 6:28 PM

brief question, If I have two streams now in a tap.. and I want to select the tap .. I thought I just added a

select

Copy code

- name: tap-ibkr-news
    select:
      - ib_news
    namespace: tap_ibkr
    executable: /home/ubuntu/stonks-tap-ibkr/tap-ibkr/tap-ibkr.sh
    config:
      host_tws_thrift: 1.2.3.4
      tws_thrift_port: 9090

but when I goto invoke

tap-ibkr-news

.. I get both streams.. I went through the academy example of creating a tap.. but it didn't really show.. once you have two streams.. how to invoke only 1 I am now wondering.. must I add the logic in my tap? in order to make these

select

statements work?? or is my

select

statement wrong? I've tried

Copy code

select:
      - ib_news
    select:
      - ib_news.*

emcp

10/27/2021, 6:30 PM

https://sdk.meltano.com/en/latest/stream_maps.html I think I am warmer.. but is the feature I am looking to get working the.. stream "map" ??

emcp

10/27/2021, 6:33 PM

hmm upon further reading this sounds wrong.. I want to avoid calling the 2nd stream entirely when I do this

select

but it says stream mapping happens after all that logic so.. I will keep looking..

visch

10/27/2021, 6:46 PM

Can you show your entire

meltano.yml

with the selects in place? Then run

meltano select --list tap-name

and show the output 😄

emcp

10/27/2021, 6:49 PM

I think after reading the docs a bit.. I need to learn catalogs better.. and yes perhaps with an already working end point

emcp

10/27/2021, 6:50 PM

Here's my meltano.yml

another.txt

emcp

10/27/2021, 6:51 PM

I think this is my answer

Copy code

ubuntu@juju-2dd159-248:~/meltano_proj_repo$ meltano select --list tap-ibkr-news
Cannot list the selected attributes: Could not find catalog. Verify that the tap supports discovery mode and advertises the `discover` capability as well as either `catalog` or `properties`

emcp

10/27/2021, 6:51 PM

but would help to maybe... get a link on how to add this ad a tap dev (but I know this is a niche user perhaps)

emcp

10/27/2021, 6:52 PM

in my git repo holding the tap.. I have this for meltano.yml

Copy code

plugins:
  extractors:
  - name: tap-ibkr
    namespace: tap_ibkr
    executable: ./tap-ibkr.sh
    capabilities:
    - state
    - catalog
    - discover
    config:
      host_tws_thrift: 10.1.2.3
      tws_thrift_port: 0000
      target_host: 10.1.2.3
      target_username: some_user
      target_password: 1234
    settings:
    - name: username

emcp

10/27/2021, 6:53 PM

so I've got both

discover

and

catalog

... I think i next need to experiment with implementing https://sdk.meltano.com/en/latest/classes/singer_sdk.Stream.html#singer_sdk.Stream.apply_catalog

emcp

10/27/2021, 6:54 PM

but this error sort of .. throws me off

visch

10/27/2021, 6:54 PM

I'd start with a tap that works with selects just to wrap your head around that side first but that's just me!

edgar_ramirez_mondragon

10/27/2021, 6:54 PM

I think i next need to experiment with implementing

You shouldn't need to override the implementation of

apply_catalog

emcp

10/27/2021, 6:57 PM

thanks Edgar, I will focus for now on selecting my csv tap.. it seems to advertise the feature

Copy code

ubuntu@juju-2dd159-248:~/meltano_proj_repo$ meltano select --list tap-csv
Legend:
	selected
	excluded
	automatic

Enabled patterns:
	*.*

Selected attributes:

Once I sort of get that.. then perhaps I will just review the code that enabled that csv tap

emcp

10/27/2021, 6:58 PM

and see where I am off.. one challenge I've noticed with doing that in past was, most if not all taps I had encountered were not looking like mine.. but now I get it that.. it's because most are REST based and hence do not need a

client.py

. I was afraid before that they were on a older singer spec

visch

10/27/2021, 9:55 PM

Not sure why that means anything here to be super honest with you!

emcp

10/28/2021, 6:49 AM

I see a path forward, basically I need to pick up as you said a tap that works with catalogs properly. get select to be something I understand.. then go back into my tap and see what code i am missing.. so far the best fully featured tap I can see using logic around the catalog is

tap-gitlab

.. which seems to have loads of logic in each stream checking if it's selected etc.

emcp

10/28/2021, 6:50 AM

I started printing out some debug statements in the

get_records

.. but it shows each time that both streams return

true

when calling

self.selected

emcp

10/28/2021, 6:51 AM

thanks for the advice Derek! I may pause on getting the stream selection working.. the good news is I can hard code simply returning in stream A for a while and try to make further progress in other parts.. before I come back to this

visch

10/28/2021, 12:41 PM

Pretty much every tap made with the sdk supports it, https://github.com/MeltanoLabs/tap-csv is an easier example you could get running real quickly :D

emcp

10/28/2021, 3:33 PM

I used this tap actually.. did some selects but.. it isn’t clear how to deselect… everytime I tried to exclude.. records were propagated

Open in Slack

Previous Next