Hi All , I have a weird response for which I cant define a Schema, Can someone help me if this use case can be handled with Meltano.

{
  "billable_info": {
    "U0632EWRW": {
      "billing_active": false
    },
    "U02UCPE1R": {
      "billing_active": true
    },
    "U02UEBSD2": {
      "billing_active": true
    }
  }
}

I have another endpoint that gives a gzip file as a response is there a way to unzip it and parse using meltano.

Is this a custom tap?

@Reuben (Matatika) yes this is something we are adding to slack tap

Can you define "weird response"? The snippet above just looks like nested object, which is definitely possible to describe with a schema. As for gzip, I don't know if there are better examples out there, but

tap-auth0

handles gzip content in `parse_response`: https://github.com/Matatika/tap-auth0/blob/76f6909613160bd4074f9bee8480436defeba068/tap_auth0/streams.py#L81:L86

@Reuben (Matatika) - this response’s are where value is used as key rather than name , for example in the example above “U02- - “ is a user id and the length of json changes for each fetch as org might have 100 users today and 101 tomorrow. So a new the key “U02- - “ is added as users are added . I am new to this stuff I feel having value as key is a blocker for schema generation.

The JSON schema spec and the singer sdk both support that type of variant objects. The SDK schema helpers have the

addiadditional_properties

parameter: https://sdk.meltano.com/en/v0.37.0/typing.html

Personally I'd go with mapping those fields to something else. If you know those are user id's then I'd make a mapping in the tap that does something like

"billable_info": [
        {
            "user_id": "U0632EWRW",
            "billing_active": false
        },
        {
            "user_id": "U02UCPE1R",
            "billing_active": true
        },
        {
            "user_id": "U02UEBSD2",
            "billing_active": true
        }
    ]

As someone in your transformation side is going to have to do this anyway. I try really hard to avoid doing this kind of thing but here's an example of one https://github.com/AutoIDM/tap-zohosprints/blob/main/tap_zohosprints/tests/tag_property_unfurl.json Test https://github.com/AutoIDM/tap-zohosprints/blob/main/tap_zohosprints/tests/test_core.py#L42 And code https://github.com/AutoIDM/tap-zohosprints/blob/52dc10d6912b74897ca5d62c55aa4ddd0b5fb333/tap_zohosprints/client.py#L201 Hope that helps!