https://linen.dev logo
Join Slack
Powered by
j
d
What does 
meltano select --list tap-hubspot-meister
say?
Does 
meltano elt tap-hubspot-meister target-bigquery --dump=catalog
have 
selected: true
on the 
companies
stream?
j
message has been deleted
message has been deleted
d
That looks right
Then it's likely on the side of the tap. It's built with the SDK? cc @aaronsteers
j
No, I built it myself @douwe_maan
But this part is not much more complicated than this
message has been deleted
d
I'd add some more debugging statements there to figure out what metadata it's seeing for the 
companies
stream
Just to confirm, you implemented the 
--catalog
flag, not the legacy 
--properties
? Because the 
capabilities
in 
meltano.yml
say 
catalog
j
I’ll add the debugging to understand the problem. Than you 🙏 I did implement 
--catalog
@douwe_maan I dumped the catalog in a file and it works
message has been deleted
So I think that it’s not the script the problem
d
Hmm
If you run with 
meltano --log-level=debug
and let Meltano generate the catalog itself, you should see the path to the catalog file printed. Does that have 
selected: true
in it?
j
I can’t find the catalog file printed
But I see this
message has been deleted
d
Do you see the 
Invoking:
line?
That should have the path to the generated catalog file
j
This ?
DEBUG Invoking: ['/home/juan_valencia/hubspot-meister/.meltano/extractors/tap-hubspot-meister/venv/bin/tap-hubspot-meister', '--config', '/home/juan_valencia/hubspot-meister/.meltano/run/
elt/alternative/fc2bb524-6d31-47ba-b326-91c2e2db22f0/tap.config.json', '--catalog', '/home/juan_valencia/hubspot-meister/.meltano/run/elt/alternative/fc2bb524-6d31-47ba-b326-91c2e2db22f0/tap.properties.json']
d
Yes! Does 
/home/juan_valencia/hubspot-meister/.meltano/run/elt/alternative/fc2bb524-6d31-47ba-b326-91c2e2db22f0/tap.properties.json
have the 
selected: true
we need?
j
No
message has been deleted
d
😕
And yet, 
--dump=catalog
does have 
selected: true
? That's very odd, that should just dump the exact same generated file
I can't explain right now why those would be different 😕
j
I’m sorry to bring this on your shoulders
d
Don't be!
Only issue is that I have to run right now for a dinner reservation
j
Yes ok no problem
I solved my problem
Bon appétit !
d
Thanks, I'll try to dig further tomorrow or @aaronsteers can help. If 
select
isn't working right (or if 
--dump=catalog
is giving something else than the actual catalog passed to the tap in sync mode) that's a bug we should dive into!
j
Hello @douwe_maan Any news on this topic ?
d
@juan_sebastian_suarez_valencia My bad, totally lost track of this. Can you confirm again that you get different catalog files when using 
--dump=catalog
and when looking at the catalog file pointed at by the 
Invoking:
log line?
j
Yes it was the case
Should I try again ?
d
Please do, I just want to make extra sure that's still the main symptom we're seeing
j
Ok
I just tried again and it works this time
d
😬
Nothing should have changed between then and now, at least not on Meltano's side
j
I will file an issue with the previous screenshots but you don’t have to work on it. And if it comes up again, you will be able to resume the investigation. What do you think ?
d
Since we're not sure if it's an actual bug or just some misconfiguration we didn't notice at the time, I'd hold off on creating an issue until you or someone else can reproduce this again. Otherwise the issue is just going to sit there indefinitely while it may not actually be "real"
Open in Slack
PreviousNext
Powered by