https://linen.dev logo
Join Slack
Powered by
If a json response from a stream has sub-objects i...
# getting-started
j
If a json response from a stream has sub-objects in it, what’s the best way to handle that? eg
Copy code
#invoice
{
  'id':12,
  'price': 23
  'shipment':{
     'uuid': ASDASD-23123,
     'status': 'pending'
  }
}
Looking to have two outputs invoice_id, price 12, 23 And a shipment info shipment_uuid, invoice_id (from higher in the body), status ASDADS-23123, 12, pending
a
Generally, you can declare the complex property as 
ObjectType
. The "author" example at the following link is a good example of this: https://sdk.meltano.com/en/latest/typing.html
These examples use the SDK's JSON Schema type helper classes, but you can do the same in native JSON if that's your preference. (The 
.to_dict()
at the bottom of that sample basically converts it to JSON Schema anyway.) The classes are more concise though, and you get the benefits of type checking, etc.
j
Is there a way to tell the object schema that the upper body’s id is also a field?
a
Just to confirm we're coming from the same context: I inferred from your question that you are building a tap on the SDK - is that right?
j
Correct, I’m working with the cookiecutter REST example. It’s working but I’m not extracting the second object yet
a
Cool. 👍 Then in your 
Stream
class declaration, you can add something like:
Copy code
class MyStream(RESTStream):
    schema = PropertiesList(
        Property("id", IntegerType, required=True),
        Property("price", IntegerType, required=True),
        Property(
            "shipment",
            ObjectType(
                Property("uuid", StringType),
                Property("status", StringType),
            )
    ).to_dict()
j
I’ll need to check that out, I’m concerned that a push to a SQL target would result in a second table like 
invoice_shipments
but only have 
uuid, status
without the id from invoice
a
Have you seen this before then, I presume? Generally this happens with arrays in the declaration, and not all targets have this behavior. Sometimes its configurable too, but that will depend on the implementation.
The above schema declaration uses sub-properties, but not array-typed subproperties, so I think you are safe. 🙂
j
It’s likely I’ve misunderstood in my prior tests. Thanks for your suggestion I’ll give a pass with the object types and get it into a sql target and follow up.
a
Sounds good! 👍
Open in Slack
PreviousNext
Powered by