https://linen.dev logo
Join Slack
Powered by
Using DBT models to derive which sources to query....
# plugins-dbt
v
Using DBT models to derive which sources to query. Not sure that this is the right place but wanted to share this Idea I've been using successfully. I think you'll want to run with this Note that I have my own runner here that bypasses meltano elt due to Native windows but the idea should be clear 
--tap_name=tap-oracle --target_name=target-mssql --select_filter=PS-PSM_STUDENTCONTACT,PS-PSM_STUDENTCONTACTTYPE,PS-STUDENTS,PS-SYNC_STUDENTMAP,PS-U_DEF_EXT_STUDENTS --dbt_modelfilter=+DB__activedirectory_2_oracle_student__target_contactemail
I'm populating select_filter automatically from 
dbt ls -s DB__activedirectory_2_oracle_student__target_contactemail --resource-type=source
What I do is walk through all of my target models I care about, get this information, and generate a few hundred runners automatically that pull exactly the data I need. This flips the integration with DBT on its head a bit, but I think it's much more powerful. Maybe this is more of a DBT feature?
Takes advantage of the DAG DBT generates, and hops on board. There's lot of room for improvement and native Meltano / DBT would make this pretty slick
Modified some of these names (they're a bit better on my side but stupidly long)
Another idea similar to this is I'd like to be able to run this and only have my source system get pulled if certain conditions are met (dbt's source freshness makes the most sense). If freshness is bad, Meltano needs to come help out.
Problem that comes after all of this is working is repeatability / testing. How do I know a change I"m making in one source impacts only the desired places. Or how do I make sure my targets are still working the way I'd expect. Maybe just schema/data tests will handle most of that
6 Views
Open in Slack
PreviousNext
Powered by