Hello Someone understand this ? In my configuration file I select a specific field But in my destination I have a bunch of other stuff as if I wrote

contacts.*

does it show what you’d expect if you do

meltano select tap-hubspot --list --all

?

@taylor actually it doesn’t

Douwe helped debug a similar problem I think recently https://meltano.slack.com/archives/C013EKWA2Q1/p1614893097053400?thread_ts=1614892532.050300&cid=C013EKWA2Q1

Wait please I can’t save the log in a file with

meltano --log-level=debug invoke tap-hubspot > log.txt

oh wow - that’s super annoying. I’ll file an issue about that

Here you have

yep - looks like this bug then https://gitlab.com/meltano/meltano/-/issues/2627

@taylor That triggered the discovery https://p479.p0.n0.cdn.getcloudapp.com/items/6quQdJkv/a062d428-bddc-4ee8-8723-ee27dc3187f4.jpg?v=6840e072e42efc81162531531fce52b6

hrm that’s unfortunate. time to call in backup. @aaronsteers? Douwe is in several meetings at the moment unfortunately

3.8.5

dang - was hoping I could blame python 3.9 😆

I admit that the catalog file of hubspot is particularly long and complex

Hi, @juan_sebastian_suarez_valencia. Can I confirm the symptom we are diagnosing? It sounds like from the opening comment that you are filtering down the columns list for a specific table, but your target table is nevertheless being created with all columns. Do I have that correct?

Hello @aaronsteers I would like to filter the selection to a specific property in a specific field Hence, since the target is BigQuery, it should only create a column in a specific table It does create the right table : contacts But it creates almost all the columns of that field instead of just 1

Okay, I think I know what is going on here… One of the interesting nuances of the Singer spec is that the emitting of SCHEMA messages (which are used to create tables) is a separate message from the emitting of RECORD messages (which contain the actual data). In some tap implementations, the work of filtering down your RECORD data is performed perfectly but the SCHEMA message is not trimmed down at all. When this happens, the full table gets created with all columns, but then only the columns selected actually get populated. Random coincidence - I was actually just working on this related issue in the SDK so this could be automatic for taps that are built off of that framework. For other taps, however, such as with the Hubspot - it might take a code fix on their side (or a PR from us) to get that behavior implemented.

Kind of but not exactly I see that there’s no column called firstname although that is the specific property that I’d like However, I see a column called _property_firstname.value_ which is weird because the API doesn’t send the data like that None of the columns is populated though

It sounds like the nested column structures may be getting “flattened” on the way into the db… but good to hear they are at least not carrying over the data. The flattening affect does seem to match this screenshot: https://p479.p0.n0.cdn.getcloudapp.com/items/4gu1kNRe/5400d8eb-f89b-4999-bdd0-5204e89ac028.jpg?v=f20b5d0123f5967d0065ee032ad6515d

BigQuery

No worries at all. Whether rewriting or sending an MR, I’ll give you a link to the issue where we are doing this in the SDK anyway. And you can borrow that code when it’s ready, even if not building entirely on the SDK. Here’s where you can track progress: Draft: Resolve "Smartly manage `selected_properties` in `get_record_generator()`, `RECORD` messages, and `SCHEMA` messages" (!26) · Merge Requests · meltano / Singer SDK · GitLab

Ok thank you 🤝

Any time 🙂

I don’t know how to subscribe to an issue 😮

Easiest way is just hit the “thumbs up” (👍 ) but also, there’s a hidden sidebar to the right with a “Notifications” slider:

Ok got it 🙂

You’re in 🙂