Hi everyone! @edward_ryan and I are running into some trouble getting json objects from tap-shipstation to de-nest correctly. Any thoughts or suggestions would be appreciated! The Address and ordersItem objects are defined like this in the shipments.json schema:

{
  "type": ["null", "object"],
  "additionalProperties": false,
  "definitions": {
    "Address": {
      "type": ["null", "object"],
      "properties": {
        "name": { "type": ["null", "string"] },
        "company": { "type": ["null", "string"] },
        "street1": { "type": ["null", "string"] },
        "street2": { "type": ["null", "string"] },
        "street3": { "type": ["null", "string"] },
        "city": { "type": ["null", "string"] },
        "state": { "type": ["null", "string"] },
        "postalCode": { "type": ["null", "string"] },
        "country": { "type": ["null", "string"] },
        "phone": { "type": ["null", "string"] },
        "residential": { "type": ["null", "boolean"] },
        "addressVerified": { "type": ["null", "string"] }
      }
    },
     "OrderItem": {
      "type": ["null", "object"],
      "properties": {
        "orderItemId": { "type": ["null", "integer"] },
        "lineItemKey": { "type": ["null", "string"] },
        "sku": { "type": ["null", "string"] },
        "name": { "type": ["null", "string"] },
        "imageUrl": { "type": ["null", "string"] },
        "weight": { "$ref": "#/definitions/Weight" },
        "quantity": { "type": ["null", "integer"] },
        "unitPrice": { "type": ["null", "number"] },
        "taxAmount": { "type": ["null", "number"] },
        "shippingAmount": { "type": ["null", "number"] },
        "warehouseLocation": { "type": ["null", "string"] },
        "options": {
          "type": ["null", "array"],
          "items": { "$ref": "#/definitions/ItemOption" }
        },
        "productId": { "type": ["null", "integer"] },
        "fulfillmentSku": { "type": ["null", "string"] },
        "adjustment": { "type": ["null", "boolean"] },
        "upc": { "type": ["null", "string"] },
        "createDate": { "type": ["null", "string"] },
        "modifyDate": { "type": ["null", "string"] }
      }
    },

And they are referenced like this at the bottom of the file:

"properties": {
    "billTo": { "$ref": "#/definitions/Address" },
    "shipTo": { "$ref": "#/definitions/Address" },
    "items": {
      "type": ["null", "array"],
      "items": { "$ref": "#/definitions/OrderItem" }
    },

Both objects flatten correctly in target-redshift pipelinewise, but only the address object flattens in target postgres pipelinewise.

Hey @pat_nadolny, @aaronsteers,or @edgar_ramirez_mondragon any idea on this one?

This is tricky. As @edgar_ramirez_mondragon noted in our other thread, the

#ref

likely has something to do with this. Generally the target is responsible for the flattening function, and it does this based on stream-specific schema definitions for each stream. If the schema definitions contain 'ref' instances, this could hinder the parser's ability to flatten the schema. When you mentioned that you edited the references, do you mean that you ran discovery and replaced the catalog with a new version of the catalog where these refs were already expanded?

That makes a lot of sense. I edited the stream schema json, which created a disconnect that you mentioned between what the tap sent in the schema and what was actually coming from the API, so all of those new columns come in as null. I'm wondering if it’s possible to modify the Postgres target to be able to flatten the refs in the schema or if this is just part of using this kind of tap schema with a pg target @edward_ryan