https://meltano.com/ logo
#announcements
Title
# announcements
a

ancient-greece-23246

03/15/2021, 6:44 AM
I've got a feeling that this has been mentioned before, but I can't seem to find it by searching. How are people handling the fact that the
job
table in Meltano will grow indefinitely? I was thinking of modifying
orchestrate/dags/meltano.py
to include a task which will purge this table on a schedule, but just wondering if there's a better way of handling this?
s

salmon-salesclerk-77709

03/15/2021, 1:21 PM
What’s the concern with having the job table grow? Postgres can scale really well.
👍 1
a

ancient-greece-23246

03/15/2021, 10:22 PM
That's true in general but in this particular case I have a 10,000 row limit 😅 I'm using the Heroku postgres free tier as a proof of concept
This database is just used for the Meltano metadata, there's a separate bigger db for the actual warehouse 🙂
s

salmon-salesclerk-77709

03/16/2021, 2:12 PM
Ah ok 😅 Yeah, you can certainly add another DAG in the orchestrate/dags/ folder to do that. I don’t think it will break anything.