I wrote a tap that gets info from our network devices using snmp. SNMP has no record of last time updated on these fields so I can't do replication. Currenly I throw away the whole schema before running the tap and get a fresh batch of data each time. We would like to change that and track historical data so that we know if two interfaces that had an OSPF adjacency changed their state. I'm looking for input on the correct way to do this. Do I need to put timestamps on every single record I store and just store everything for every day forever?

tap's do that for you 😄 https://sdk.meltano.com/en/latest/implementation/record_metadata.html Then if you wanted to keep the history you'd do "append-only" which means no primary key

So append only would never delete a row meaning this DB would fill up pretty quick right? I was hoping for something that would only insert a new row if there's a change from the previous record

"fill up pretty quick right"

how much data do you have? Not that odd for folks to store everything, compression is pretty killer 😉 and storage is fairly cheap Should be documented better, but I know it's here https://github.com/MeltanoLabs/target-postgres/blob/main/target_postgres/tests/data_files/test_no_pk_append.singer , remove the key properties. To remove key properties for your tap via meltano.yml go here https://docs.meltano.com/concepts/plugins/#how-to-use-2

Sure I can share it with you. I deviate quite a bit from the cookie cutter template and it's geared towards network devices

Sounds like you should try everything end to end then come back with your

meltano.yml

and specefic questions 🙂

tap-snmp.tar.gz

Would it be fair to say that tap is speceficto netbox? so maybe tap-netbox?

It uses netbox as the source of devices to crawl. It'd be easy to provide the list some other way