We have a bunch of network devices all running the...
# getting-started
t
We have a bunch of network devices all running the same software. I need to query each of them with the same extractor. The queries will all be the same and the schemas will also all be the same. What's the correct way to do that? I'm planning on using the rest tap but can write my own if that's cleaner. Ideally I would be able to supply a list of hosts to reach out to based on the response from an initial api call to get a list of hosts
I should also mention that I want the data from all of this to end up in the same raw table
e
Ideally I would be able to supply a list of hosts to reach out to based on the response from an initial api call to get a list of hosts
That sounds like the perfect use case for Parent-Child streams.
t
I saw that and didn't see how it would fit into this. Are you suggesting I create a child class for each device?
e
So you would have two stream classes, e.g.
Hosts
and
Devices
. The
Hosts
stream class makes a single request for get a list of hosts and generates a context for each one. The
Devices
stream class then uses the context object that's automatically passed from the parent stream to configure the query for the corresponding device.
👍 1
t
Here's my stream that gets the hosts. I'm still confused about how to do this parent child thing. From the docs it looks like my OltStream should inherit RESTStream but specify the parent_stream_type as HostsStream. What's the correct way to create a context for every child OltStream?
Implement one of the below methods to pass context from the parent to the child:
1. If using
get_records
you can simply return a tuple instead of a
record
dictionary. A tuple return value will be interpreted by the SDK as
(record: dict, child_context: dict)
.
2. Override
get_child_context
to return a new child context object based on records and any existing context from the parent stream.
3. If you need to sync more than one child stream per parent record, you can override
generate_child_contexts
to yield as many contexts as you need.
I don't really understand what
get_records
is. The docs say that it needs to be overridden by the child class but at the same time it can optionally return a child context? So the child that's overriding it needs to create its own context? What is it returning a context for? To potentially another child?
e
The docs say that it needs to be overridden by the child class but at the same time it can optionally return a child context
Yeah, that's a bit of the older way of doing so I may need to update the docs. The way I'd recommend to do this is implementing generate_child_contexts in `HostStream`:
Copy code
class HostStream(RESTStream):
    def generate_child_contexts(self, record, context):
        yield {
            "hostname": ident.get("name"),
            "ip": ip_address.split("/")[0]
        }

class OltStream(RESTStream):
    parent_stream_type = HostStream
How would you pass the host information to the request for the Olt stream?