Have a question regarding a sync I used tap covid 19 tap and

Question

Have a question regarding a sync I used tap covid 19 tap and target csv I set the start date of the tap to be 2021 01 01 I was thinking that maybe it would only pull records gt = that timestamp But it pulled the whole dataset I ran this last night meltano elt tap covid 19 target csv job id=covid19 to csv and the same command this morning The files that resulted were pretty much the same I would have expected the second run this morning to basically have no data My thought was that if the job id remained constant the line in the sand for the first full would have been the filter as the starting point for the second run After the second run this morning i ran meltano elt tap covid 19 target csv job id=covid19 to csv again and this time it told me no new data which is what I expected this morning to be

ripe-musician-59933 · Answer

< gray cricket 92960> It looks like `tap covid 19` doesn t use `start date` and the timestamp stored between runs to only select records created since that point but rather to only select files that were changed on GitHub since then Out of those matching files it looks like it still imports all records

ripe-musician-59933 · Answer

I think this matches what you re seeing your first and second runs both selected all records presumably because the file on GitHub changed in the mean time while your third run didn t select any records since the file didn t change

gray-cricket-92960 · Answer

Thanks for this Is the pattern those implementation details are always found in the sync file One more question about the elt pipeline If i haven t formally created a pipeline yet does meltano keep track of what i execute the job with e g tap target job id

ripe-musician-59933 · Answer

Not all taps use the same file structure but since handling `start date` and bookmarks is always the responsibility of the tap itself rather than Meltano you can usually find it by searching the repo for `start date` which is what I did here <https github com singer io tap covid 19 search q=start date>

ripe-musician-59933 · Answer

gt If i haven t formally created a pipeline yet does meltano keep track of what i execute the job with e g tap target job id Yep the runs end up in the `job` table in the <https meltano com docs project html system database|system database> with their `job id` whether Using `meltano schedule` and the `schedules` list in `meltano yml` is completely optional if you re happy constructing and running the appropriate `meltano elt` commands yourself

gray-cricket-92960 · Answer

Thanks for pointing out the system database I should have read the docs