Hey all, I'm building a deployment template for Meltano on GitHub actions, and am just working on persisting the state between runs. In the interest of keeping things as slim as possible, I'm just going to use the SQLite database, but could do with some help working out which files to persist. I can see: •

meltano.db

•

meltano.db-shm

•

meltano.db-wal

What are the latter two files for? Will they all need persisting and retrieving for bookmark state to be restored properly?

wal - https://www.sqlite.org/wal.html

Thanks @visch. So, it sounds like all these files should be persisted between runs?

@niall_woodward I think it's safe to delete the

-wal

and

-shm

files (https://sqlite.org/forum/forumpost/6cd3d10b06) if

-wal

is empty. In general they shouldn't be there, but it seems like

meltano elt ...

leaves them there for some reason. I can confirm the suggestion in the linked post of running a select against the db should delete the files if there's no other process using it:

sqlite3 .meltano/meltano.db "select job_id from job limit 1"

I'm not sure about persisting the meltano db. Part of my feels like just passing state between runs would be better

Got it, thanks all

How would you handle concurrency?

Good question, and I'm deliberately not doing that @fred_reimer. We'll have a sqlite db per job.