Question, I have a tap.. it 1. queries a api endpoint.. pushes out some relational data result.. lets call it pipeline A, this would run every weekend and right now can take anywhere from 45 minutes to 12 hours 2. Pipeline B .. would run daily and.. use the last known fresh results stored from pipeline A ... which in my case is being pushed to Postgresql... How do I write the 2nd pipeline B tap. logic.. I guess I can at first hardcode it to... go back into the postgresDB configured by pipeline A ... (the metadata of each is in the same tap in the yaml file) so I would use SQLAlchemy pypi library.. to pull that last known run out of DB X Table Y.. result from pipeline A .. in order to then query and fill up results for pipeline B Am I tapping things correct or?

2, is normally referring to as T in ELT. So a lot of people would say use DBT (including me 😄 )

ah hah... okay now I finally have a reason to use DBT (been reading it a lot but never had the clear use case) I refactored my python code which I was previously using the pull data out.. and was about to plug that into the tap.. but can def take a look at this instead

I'm not following so much on pipeline b. Generally it's something like, load your source data into something (raw schema in a datawarehouse is a common pattern) , now you are onto transformation step, everything after your raw data is loaded. There's a lot of conventions, the getting started tutorial at dbt really helped me grok modern data warehouses, and what dbt is, and some general patterns. Basically anything you want to do happens in the transform step. Normally in analytic work that's building stage models, analytical models (dimensions, wide tables, anything you want). I think about it more as transformation gets the data into the format I need to actually do things with.

I think the issue here is.. most people would use a child stream.. which I can grok would sync "everything" ... I'd have 1 pipeline run Query A and as it spits out it's values use some or all of that for Query B.. as B is a child a A.. but in my case.. I cannot run that Query A everytime.. it takes as I said 45 minutes to 12 hours right now.. depending on how deep my search can be set to go.. so I was thinking I simply use results of A.. PRIOR to the call.. by going back into the target and fetching the "last good result" of that long long query... It feels like this is a perhaps "anti-pattern" to what most good practices would suggest perhaps? but another option seemed to be use DBT after I've run A ... it will push most of it's results to a simple single table .. and DBT could then collect the last known run and push that to a single child table... thereby simplying what I have to do when I go to query B