I wrote a tap that gets info from our network devi...
# singer-tap-development
t
I wrote a tap that gets info from our network devices using snmp. SNMP has no record of last time updated on these fields so I can't do replication. Currenly I throw away the whole schema before running the tap and get a fresh batch of data each time. We would like to change that and track historical data so that we know if two interfaces that had an OSPF adjacency changed their state. I'm looking for input on the correct way to do this. Do I need to put timestamps on every single record I store and just store everything for every day forever?
v
tap's do that for you 😄 https://sdk.meltano.com/en/latest/implementation/record_metadata.html Then if you wanted to keep the history you'd do "append-only" which means no primary key
t
So append only would never delete a row meaning this DB would fill up pretty quick right? I was hoping for something that would only insert a new row if there's a change from the previous record
I'm having a very hard time finding any documentation on how to actually set up my stream for "append-only"
v
> "fill up pretty quick right" how much data do you have? Not that odd for folks to store everything, compression is pretty killer 😉 and storage is fairly cheap > I'm having a very hard time finding any documentation on how to actually set up my stream for "append-only" Should be documented better, but I know it's here https://github.com/MeltanoLabs/target-postgres/blob/main/target_postgres/tests/data_files/test_no_pk_append.singer , remove the key properties. To remove key properties for your tap via meltano.yml go here https://docs.meltano.com/concepts/plugins/#how-to-use-2
Also any chance you want to publish the SNMP tap? I have a number of IT folks I work with who might like it, I've been debating on pushing IT folks a bit more to use this tool as I think it'd be fairly easy for them to setup now
t
Sure I can share it with you. I deviate quite a bit from the cookie cutter template and it's geared towards network devices
🦜 1
Do I need to remove primary keys from every stream?
Do I need to set "load_method" in my meltano.yml for the tap?
v
Sounds like you should try everything end to end then come back with your
meltano.yml
and specefic questions 🙂
t
tap-snmp.tar.gz
v
Would it be fair to say that tap is speceficto netbox? so maybe tap-netbox?
t
It uses netbox as the source of devices to crawl. It'd be easy to provide the list some other way
✅ 1