Airbyte TAP s incremental issues I m having issue getting th Meltano #troubleshooting

Airbyte TAP’s incremental issues: I’m having issu...

jichao_sun

05/22/2024, 3:12 AM

Airbyte TAP’s incremental issues: I’m having issue getting the airbyte tap-amplitude to work with incremental extraction. It seems like the state messages meltano gets from airbyte tap is just a dict, but airbyte is expecting it to be a list of dicts. Where each dict is a state message for a stream, so tap-airbyte ends up keeping just a single state message for the latest incremental stream. Are there any plans to support this type of state messages in tap-airbyte?

👀 1

jichao_sun

05/22/2024, 3:59 AM

Narrowed this down to how airbyte is expecting the state.json vs how tap-airbyte-wrapper is passing the state. • For incremental to work, airbyte is expecting a state message in this form, a list of

AirbyteStateMessage

Copy code

[
    {
      "type": "STREAM",
      "stream": {
        "stream_descriptor": {
          "name": "events",
          "namespace": null
        },
        "stream_state": {
          "server_upload_time": "2024-05-13 19:35:13.833000"
        }
      }
    }
  ]

• The internal dict above is actually what gets sent by airbyte in a state message. But for some reason the tap-airbyte-wrapper unpack this and saves the unpacked version in the state table • This unpacked version is then fed back to airbyte on next invocation, which prevents certain airbyte extractors from not working. (For example intercom’s one works totally fine, while amplitude does not)

jichao_sun

05/22/2024, 5:50 AM

Tried out fix here for amplitude and it seems to work: https://github.com/MeltanoLabs/tap-airbyte-wrapper/pull/18

👀 1

2 Views

Open in Slack

Previous Next