I'm struggling with the meltanolabs hubspot tap. with this config it's grabbing duplicate company records like crazy, it doesn't seem to ever even end the import, i cut it off at 150k (3 times the amount of records there are even before the filter)

extractors:
    - name: tap-hubspot
      variant: meltanolabs
      pip_url: git+<https://github.com/MeltanoLabs/tap-hubspot.git>
      config:
        start_date: 2024-04-01T00:00:00+0000
        flattening_enabled: true
        flattening_max_depth: 2
        stream_maps:
          companies:
            __filter__: properties.lifecyclestage != 'lead'
      select:
        - users.*
        - deals.*
        - companies.*

@Jesse Johnson thanks for reporting this! Can you try and see if pinning an older version fixes the issue:

pip_url: git+<https://github.com/MeltanoLabs/tap-hubspot.git@9d62d3b90ce79d6d69eca544f1448d1049937131>

cc @Pat Nadolny (Arch) in case have you've ever seen this type of behavior

I solved this by going in to streams.py and selecting FULL instead of incremental for companies

Gotcha. Probably worth trying a replication-method override

extractors:
    - name: tap-hubspot
      metadata:
        companies:
          replication-method: FULL_TABLE

https://docs.meltano.com/concepts/plugins#metadata-extra

ah, thanks. i wasn't sure if that would work. i'll try that instead

> in case have you've ever seen this type of behavior I havent seen this myself but I'll report back if I do

I tried that, and it looks like it still uses the search endpoint

I'm not too familiar with the tap, but would the solution be for the it not to use the search endpoint?

depends on the use case, since i'm pulling tens of thousands of records then using a stream_map, i think the search endpoint just gets overwhelmed?

So it's too slow to even get the first response?

I'm not sure, it just runs until i stop it, I could try doing some more indepth troubleshooting at some point