Any examples of using stream maps flattening properties I m Meltano #troubleshooting

Any examples of using stream maps flattening prope...

marcin_wojciechowski

11/09/2023, 9:01 PM

Any examples of using stream maps flattening properties? I'm using tap-mongodb with strategy:envelope and I want to extract document.lastEdited date time field to be used as the replication-key.

marcin_wojciechowski

11/10/2023, 1:38 PM

So, I have a document coming out of the tap-mongodb like that:

Copy code

{
  "_id": "REP-1-2022",
  "document": {
    "_id": "REP-1-2022",
    "type": "safety-report",
    "created": "2022-04-11T14:34:01",
    "createdBy": "marcin",
    "createdById": "236520",
    "lastEdited": "2022-07-15T09:31:08"
  }
}

And I set the config as follows:

Copy code

config:
    plugins:
      extractors:
      - name: tap-mongodb
        config:
          stream_maps:
            t_ent_393_entity-report:
              properties:
                document:
                  properties:
                    lastEdited:
                      flatten: true
          strategy: envelope
          database_includes:
          - "t_ent_393"
          mongo:
            host: <mongodb://admin:password@localhost:27017/>
        select:
        - t_ent_393_entity-report.*
        metadata:
          '*':
            table-key-properties: _id
            replication-method: INCREMENTAL
            replication-key: document_lastEdited

however it fails on stream_map with the error as per the screenshot. Is this even the right approach?

edgar_ramirez_mondragon

11/10/2023, 7:24 PM

that stream maps config doesn't look right. Can you try:

Copy code

config:
    plugins:
      extractors:
      - name: tap-mongodb
        config:
          flattening_enabled: true
          flattening_max_depth: 2
          strategy: envelope
          database_includes:
          - "t_ent_393"
          mongo:
            host: <mongodb://admin:password@localhost:27017/>

marcin_wojciechowski

11/13/2023, 10:35 AM

Thanks for that. My problem is that this doesn't work with

strategy: envelope

an when I set it to

strategy:infer

my schema is not correctly detected as it's quite complicated. The problems I'm getting most is around string fields that are nullable. How can I deal with that?

marcin_wojciechowski

11/13/2023, 10:37 AM

message has been deleted

Open in Slack

Previous Next