Hello everyone, I’m having a rough time using Meltano at the moment. I’m trying to connect to hubspot API The hubspot API is a real mess : • It was updated for the last time in february 2019 • There are a lot of Pull Requests • Among those PR there are some that woud solve my problem My problem : The official hubspot tap doesn’t work well with BigQuery So I have 2 options : • Fork the official tap and create a variant oriented to BQ. But I have a lot of legacy code that would almost mean rewriting some chunck • Rewrite the whole thing using Meltano. But the SDK is a little bit immature and I don’t know how to test the scripts efficiently (for example the core tests I don’t know why don’t work) and even if they did, the integration to Meltano without the setup.py is going to be also a difficult step What is the right path when the 2 options are so steep ?

What's the reason the tap doesn't work well with target-bigquery? The number of columns?

The number of columns but also the type of the columns that are wrongly deduced from the tap as far as I understood

Hmm. Instead of creating your own fork, you can consider using one of these active ones as a jumping off point: https://github.com/goes-funky/tap-hubspot https://github.com/dreamdata-io/tap-hubspot https://github.com/pankajsaini-ldt/tap-hubspot (by @dry-dinner-58985) Some of your issues may already have been fixed

I am still working on improving tap-hubspot for limiting the number of columns, so the forked repo https://github.com/pankajsaini-ldt/tap-hubspot is not very updated. I am working on this for the next few days and will push my changes to the repo. I will also look the the problem with type of the columns and see if I can fix that.

Hi, @dry-dinner-58985 - I agree with @ripe-musician-59933 the path of least resistance here is probably to work with your own fork. Regarding the SDK issues, thanks for raising those. I incorporated your feedback (and the feedback of several other early adopters) into a set of fixes currently in review here: Add automated tests for cookiecutter generation (!23) This update (to be merged in the next two days) fixes a number of minor bugs in the SDK’s cookiecutter template, including the one you have pasted in your screenshot.

@dry-dinner-58985 Thanks for your help. I came across your fork but couldn’t make it work. I understand that is a work in progress so no worries. Thank you for your efforts 🙏

Hi @blue-continent-72423, I just pushed a change to my repo that skips property extras and versions if a config

skip_property_extras_and_versions=true

is passed. This is working well for me with pipelinewise-target-redshift , and the extras and versions are correctly removed. I however not had a chance to test it with tap-bigquery, but I assume that removing the extras and versions will dramatically reduce the number of columns created in bigquery.

Hello @dry-dinner-58985 I started to write my own tap but I’ll test yours. Could you put in the readme the way to use it with meltano please ? Specially the config part That way I’ll be able to test it quickly Thank you very much !

Hey Juan, I am planning to raise a PR to merge it with singer’s repo, so I don’t want to put meltano specific instructions, but If you want to give it a quick try with meltano, you can add the following under

plugins>extractors

in your meltano.yml

- name: tap-hubspot
      namespace: tap_hubspot
      pip_url: git+<https://github.com/pankajsaini-ldt/tap-hubspot.git>
      executable: tap-hubspot
      capabilities:
        - discover
        - state
        - properties
      settings:
        - name: hapikey
        - name: start_date
        - name: redirect_uri
        - name: client_id
        - name: client_secret
        - name: refresh_token
        - name: skip_property_extras_and_versions
      config:
        start_date: "2021-03-17T00:00:00Z"
        skip_property_extras_and_versions: true
      select:
        - companies.*
        - deals.*
        - owners.*
        - deal_pipelines.*

Ok, I’ll do that and let you know !

Please feel free to modify the entity and attribute selections under

select:

. Also you will need to pass the standard settings like redirect_uri, client_id, client_secret, refresh_token etc. You can set these in meltano.yml or better in env variables(please run

meltano config tap-hubspot list

to see the corresponding env variable names)

@dry-dinner-58985 I use your tap with BigQuery But I have this kind of stuff that is still appearing in BigQuery https://p479.p0.n0.cdn.getcloudapp.com/items/WnuYeEnq/bc1c466a-5097-4ced-8085-10ae3f09896d.jpg?v=f1222251e2c9ff61601f1dd2a5922fbc

As a quick fix can you try adding

'!contacts.associated-company'

instead of

contacts.*

in the select section in meltano.yml? I think a fix can be applied that only gives

assiciated-company id

instead of returning all the fields of the company.

The “select” feature of meltano was not working last week but I will try