https://linen.dev logo
Join Slack
Powered by
Hi, I would like to add <custom properties> stream...
# contributing
m
Hi, I would like to add custom properties stream into tap-github. Can someone advice the better way to do that?
I am currently creating in repository_streams.py
Copy code
class CustomPropertiesStream(RepositoryStream):
    """Defines 'custom_properties' stream."""

    name = "custom_properties"
    path = "/repos/{org}/{repo}/properties/values"
    primary_keys = ["property_name"]
    parent_stream_type = RepositoryStream
    ignore_parent_replication_key = False
    state_partitioning_keys = ["repo", "org"]
    selected_by_default = True

    schema = th.PropertiesList(
        # Parent Keys
        th.Property("repo", th.StringType),
        th.Property("org", th.StringType),
        th.Property("repo_id", th.IntegerType),
        # Custom Property Keys
        th.Property("property_name", th.StringType),
        th.Property("value", th.StringType),
    ).to_dict()
And adding 
CustomPropertiesStream
imports to stream.py Is my code correct? Anything else I should be aware of?
e
That looks good at first glance, and it makes sense to put it in that file. I'm happy to review a PR!
m
Thanks, I will do that, but I am still to figure out why my stream is not getting selected:
Copy code
root@b8857e052cea:/project# meltano select tap-github --list
2024-07-03T14:53:52.254688Z [info     ] The default environment 'dev' will be ignored for `meltano select`. To configure a specific environment, please use the option `--environment=<environment name>`.
Legend:
        selected
        excluded
        automatic

Enabled patterns:
        custom_properties.*

Selected attributes:
root@b8857e052cea:/project#
e
Can you do 
meltano install --clean
and try again?
m
I did that, but you know, for testing locally, I am directly modifying tap code in my 
.meltano/extractors/tap-github
folder
👀 1
😬 1
r
e
I did that, but you know, for testing locally, I am directly modifying tap code in my 
.meltano/extractors/tap-github
folder
Oh, sorry about that 😳
😂 1
m
Installing the plugin from 
pip_url: git+<https://github.com/datarts-tech/tap-github.git@add_custom_properties>
, solved the issue of not selecting it, but I see another problem when I try to run it: Maybe this is something very obvious for you?
Copy code
2024-07-04T07:11:51.002891Z [info     ]   File "/project/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 409, in get_replication_key_signpost cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-07-04T07:11:51.003641Z [info     ]     return utc_now() if self.is_timestamp_replication_key else None cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-07-04T07:11:51.004167Z [info     ]   File "/project/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 218, in is_timestamp_replication_key cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-07-04T07:11:51.004743Z [info     ]     return is_datetime_type(type_dict) cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-07-04T07:11:51.005519Z [info     ]   File "/project/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/helpers/_typing.py", line 140, in is_datetime_type cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-07-04T07:11:51.006143Z [info     ]     raise EmptySchemaTypeError cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-07-04T07:11:51.006741Z [info     ] singer_sdk.helpers._typing.EmptySchemaTypeError: Could not detect type from empty type_dict. Did you forget to define a property in the stream schema? cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-07-04T07:11:51.116180Z [info     ] Incremental state has been updated at 2024-07-04 07:11:51.116089.
2024-07-04T07:11:51.130659Z [error    ] Extractor failed
2024-07-04T07:11:51.131075Z [error    ] Block run completed.           block_type=ExtractLoadBlocks err=RunnerError('Extractor failed') exit_codes={<PluginType.EXTRACTORS: 'extractors'>: 1} set_number=0 success=False
👀 1
e
Nothing obvious at first glance. Are you maybe overriding a replication key?
m
Maybe it was the case, but I have made few commits and the issue is now solved.
e
Ok cool
20 Views
Open in Slack
PreviousNext
Powered by