Hi I have two questions 1 In the list of streams returned fr Meltano #troubleshooting

Hi I have two questions: 1. In the list of stream...

matt_elgazar

10/21/2023, 1:23 AM

Hi I have two questions: 1. In the list of streams returned from the

discover_streams

method, how can I manage the order in which streams are ran and pass data from the output of the first stream to the second stream? a. E.g I have two streams: StreamA and StreamB. When StreamA finishes, I need to pass data from StreamA

self.data_to_pass

into StreamB. 2. What’s the best way to pass the parameter passed to a CatalogEntry object? I want the

category

returned in some part of the CatalogEntry object. Below is my hack to do it but I know it’s the wrong way to do things. When doing it this way I get WARNING | tap-yfinance | Properties ('ticker',) were present in the 'crypto_tickers' stream but not found in catalog schema

Copy code

category_entry = CatalogEntry(
                tap_stream_id=table_name,
                stream=table_name,
                table=table_name,
                key_properties=required_fields,
                schema=Schema.from_dict(schema),
                replication_method=None,  # defined by user
                metadata=MetadataMapping.get_standard_metadata(
                    schema=schema,
                    schema_name=category,  # TODO: Fix this! This is just a placeholder.
                    replication_method=None,  # defined by user
                    key_properties=key_fields,
                    valid_replication_keys=None  # defined by user
                ),
                replication_key=None  # defined by user
            )

Open in Slack

Previous Next