https://linen.dev logo
Join Slack
Powered by
does anyone have an example of a tap with support ...
# singer-tap-development
e
does anyone have an example of a tap with support for multiple datasets or tables? I've had issues in past writing my tap to support selecting things at runtime.. how do I support that so I can (at runtime) run certain loading events in a certain order (it's required that I perform queries in a certain order because the results assist subsequent queries
h
I don't have an example ready, but this seems to fit into the parent-child stream paradigm https://sdk.meltano.com/en/latest/parent_streams.html
e
it's unfortunately a bit more complicated for me Haleemur. I download a large query of about 120K rows.. that needs to be pushed up into S3 BEFORE I can run child queries.. it'd be risky to code that into the tap.. so I think the cleaner approach for now has been to write separate taps unfortunately
with my current tap I tried listing the catalog.. and got an error.. so looks like my next step is to actually get that working and turn on discovery mode
Copy code
$ meltano select tap-ibkr-news --list --all
2023-05-23T19:56:15.067682Z [info     ] The default environment 'dev' will be ignored for `meltano select`. To configure a specific environment, please use the option `--environment=<environment name>`.
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Cannot list the selected attributes: Could not find catalog. Verify that the tap supports discovery mode and advertises the `discover` capability as well as either `catalog` or `properties`
as of now I am trying to debug via pycharm.. and this means setting some configs via I guess a json file.. but I cannot for the life of my find an example of this json file format.. according to chat gpt but this doesn't seem to work
Copy code
{
  "extractors": {
    "tap-ibkr": {
      "tap": "tap-ibkr",
      "properties": {
        "host_tws_thrift": "127.0.0.1",
        "tws_thrift_port": 9090,
        "tickers_db_host": "127.0.0.1",
        "tickers_db_username": "juju_meltano",
        "tickers_db_password": "1234",
        "tickers_db_type": "hive",
        "output_timestamp_format": "hive"
      }
    }
  }
}
UPDATE: I just tried using environment variables set in pycharm.. in order to push the config... still get the same error.. it almost feels like this does't work when debugging.. only when utilizing meltano to run a tap
Copy code
Config validation failed: 'host_tws_thrift' is a required property; 'tws_thrift_port' is a required property; 'tickers_db_host' is a required property; 'tickers_db_username' is a required property; 'tickers_db_password' is a required property; 'tickers_db_type' is a required property; 'output_timestamp_format' is a required property
JSONSchema was: {'type': 'object', 'properties': {'host_tws_thrift': {'type': ['string']}, 'tws_thrift_port': {'type': ['integer']}, 'tickers_db_host': {'type': ['string']}, 'tickers_db_username': {'type': ['string']}, 'tickers_db_password': {'type': ['string']}, 'tickers_db_type': {'type': ['string']}, 'output_timestamp_format': {'type': ['string']}, 'stream_maps': {'type': ['object', 'null'], 'properties': {}, 'description': 'Config object for stream maps capability. For more information check out [Stream Maps](<https://sdk.meltano.com/en/latest/stream_maps.html).>'}, 'stream_map_config': {'type': ['object', 'null'], 'properties': {}, 'description': 'User-defined config values to be used within map expressions.'}, 'flattening_enabled': {'type': ['boolean', 'null'], 'description': "'True' to enable schema flattening and automatically expand nested properties."}, 'flattening_max_depth': {'type': ['integer', 'null'], 'description': 'The max depth to flatten schemas.'}}, 'required': ['host_tws_thrift', 'tws_thrift_port', 'tickers_db_host', 'tickers_db_username', 'tickers_db_password', 'tickers_db_type', 'output_timestamp_format']}
ahhhh, now I just saw ``--config=ENV`` is required 😞 .. retrying now...
now I got it starting to work.. but then hit a weird behavior.. it's erroring on integers as if they are not integers...
message has been deleted
ill just drop this parameter... it never changes in my current design anyway
@haleemur_ali, I've started my first attempt at adding a parent child relationship.. but getting this error
Copy code
/home/emcp/.cache/pypoetry/virtualenvs/stonks-tap-ibkr-GGawKcX6-py3.10/bin/python /home/emcp/Dev/git/STONKS/stonks-tap-ibkr/tap-ibkr/tap_ibkr/tap.py --config=ENV --test 
Traceback (most recent call last):
  File "/home/emcp/Dev/git/STONKS/stonks-tap-ibkr/tap-ibkr/tap_ibkr/tap.py", line 30, in <module>
    Tapibkr.cli()
  File "/home/emcp/.cache/pypoetry/virtualenvs/stonks-tap-ibkr-GGawKcX6-py3.10/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/emcp/.cache/pypoetry/virtualenvs/stonks-tap-ibkr-GGawKcX6-py3.10/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/emcp/.cache/pypoetry/virtualenvs/stonks-tap-ibkr-GGawKcX6-py3.10/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/emcp/.cache/pypoetry/virtualenvs/stonks-tap-ibkr-GGawKcX6-py3.10/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/emcp/.cache/pypoetry/virtualenvs/stonks-tap-ibkr-GGawKcX6-py3.10/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 519, in cli
    tap = cls(  # type: ignore[operator]
  File "/home/emcp/.cache/pypoetry/virtualenvs/stonks-tap-ibkr-GGawKcX6-py3.10/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 103, in __init__
    self.mapper.register_raw_streams_from_catalog(self.catalog)
  File "/home/emcp/.cache/pypoetry/virtualenvs/stonks-tap-ibkr-GGawKcX6-py3.10/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 166, in catalog
    self._catalog = self.input_catalog or self._singer_catalog
  File "/home/emcp/.cache/pypoetry/virtualenvs/stonks-tap-ibkr-GGawKcX6-py3.10/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 287, in _singer_catalog
    for stream in self.streams.values()
  File "/home/emcp/.cache/pypoetry/virtualenvs/stonks-tap-ibkr-GGawKcX6-py3.10/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 128, in streams
    for stream in self.load_streams():
  File "/home/emcp/.cache/pypoetry/virtualenvs/stonks-tap-ibkr-GGawKcX6-py3.10/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 329, in load_streams
    parents = streams_by_type[stream_type.parent_stream_type]
KeyError: <class 'tap_ibkr.client.NewsStream'>
in pycharm I get no errors and the class exists... just seems i need to reread the directions once more and inject the class somewhere..
Copy code
from typing import List
from singer_sdk import Tap, Stream
from singer_sdk import typing as th  # JSON schema typing helpers

from tap_ibkr.streams import IBNewsStream, IBNewsBodyStream


STREAM_TYPES = [
    IBNewsStream,
    IBNewsBodyStream
]


class Tapibkr(Tap):
    """ibkr tap class."""
    name = "tap-ibkr"

    config_jsonschema = th.PropertiesList(
        th.Property("host_tws_thrift", th.StringType, required=True),
        th.Property("trino_host", th.StringType, required=True),
        th.Property("output_timestamp_format", th.StringType, required=True)
    ).to_dict()

    def discover_streams(self) -> List[Stream]:
        """Return a list of discovered streams."""
        return [stream_class(tap=self) for stream_class in STREAM_TYPES]


if __name__ == "__main__":
    Tapibkr.cli()
actually I think I cracked it .. but the example docs are extremely broken.. basically it's unclear where this change needs to go .. client.py , or streams.py .. because in fact as of now, I have it working by moving SOME of that example code into streams.py.. and some into client.py..
Open in Slack
PreviousNext
Powered by