https://linen.dev logo
Join Slack
Powered by
I’d like to change the name of a table from the na...
# plugins-general
j
I’d like to change the name of a table from the name given in the tap to the name used for the table in the target. I’m using the Hubspot tap going to the Snowflake target. Full use case: I want to replicate the same table twice, once with the original name and second time with a modified name. Any known ways of accomplishing this?
Sounds a bit silly I know but the Hubspot api doesn’t provide a way for telling which contacts have been deleted. So the only way to identify deleted contacts in our warehouse is to compare the list of today’s contacts with the existing list of incrementally modified contacts. The list of today’s contacts will be dropped and replaced each day.
v
stream_maps could definitely do it, the mapper that meltano released should work
a
Yeah, was going to say - this is one of the use cases that stream maps will answer.
e
@josh_lloyd could you perhaps use metadata columns like 
__loaded___at
in a downstream step to determine if a contact was removed because it was not synced with the rest?
a
I'm guessing the tap and target are not SDK-based, right? @florian.hines's current WIP MR will add that functionality even for non-SDK taps and targets.
j
correct both tap and target are not SDK-based
a
v
Activate Version is the other thing that addresses this I believe. I cheat with my own target-mssql and say "I only support full table runs" which means either all the data get's loaded into a new table, or none which side steps all this as well (Same end result activate version)
*activate version is much better
j
@edgar_ramirez_mondragon I’m currently using a __sdc__batched_at and a __sdc_extracted__at metadata columns through the Snowflake target capabilities, but unfortunately neither captures the latest date that the row was included in the data replication
@visch I’m not sure I understand what “activate version” is or how to use it. Can you elaborate or provide a link?
v
j
oh, I see. Yeah that’s nice. Looks like I need to wait for some new features to come out to make this happen easily then. Thanks guys!
v
Plenty of work arounds right now, loaded_at column, transformation tap with stream maps 🤷
p
https://hub.meltano.com/singer/docs#activate-version - This is a description of the activate version concept
Open in Slack
PreviousNext
Powered by