https://linen.dev logo
#singer-tap-development
Title
# singer-tap-development
j

joshuadevlin

05/09/2022, 6:35 AM
I have two streams with a parent/child relationship. The parent returns a list of records, and the child (unfortunately) has to hit the API for each individual record returned by the parent. The state for the child stream has partition bookmarks for all ~50k parent records, and the size of my meltano.db is increasing by ~2MB with each subsequent run. My questions: 1. Can I use
state_partitioning_keys
to reduce the state (ideally to zero) that's stored between invocations of the tap, and if so how? 2. Are there ways that I can purge some of the history of
meltano.db
to reduce its size to something more sensible?