hey guys - I'm curious if anyone else has come upo...
# singer-tapsp
pnadolny
05/03/2021, 2:44 PMhey guys - I'm curious if anyone else has come upon a use case for doing rolling look back windows using a tap. Our use case is that we have snowflake tables that we want to send to a target_sftp for sharing with some our partners and currently we make a snowflake base table with all our data (3+ years) and a view for each partner given their specific window of data they want (rolling 3 days, 7 days, 14 days) and then do a full table sync. The feature were thinking about adding is some metadata to tell the tap to do the look back for us. This would allow us to only manage the base DBT table and reuse it for many different rolling windows as needed. Has anyone heard of others doing something similar? Thoughts?
pnadolny
05/03/2021, 2:46 PMI was thinking about making it a feature of the full table sync option since it doesnt care about bookmarks or state really. Something like this
select * from x where column_ts >= dateadd(day, -7, sysdate())pnadolny
05/03/2021, 2:47 PMcolumn_ts, day, and 7 would all be configurable metadata
t
taylor
05/03/2021, 2:48 PMYeah, you could definitely do this - the tap would just have to be setup to take in a value to represent the lookback period.
taylor
05/03/2021, 2:49 PMEither way could work depending where you want to put the complexity. dbt could be nice b/c you could have a macro that represents the table and then just materialize whatever you need based on the timespan, or you could push this to the extract and load step - just really depends where you want to put some of the complexity
p
pnadolny
05/03/2021, 3:06 PMcool thanks - have you heard of any other taps with any similar functionality?
t
taylor
05/03/2021, 3:08 PMthere are several taps that have something like a
start_datedate_window_sizep
pnadolny
05/03/2021, 3:15 PMawesome thats helpful