#10379 perf: Add system database index to runs table on (`job_name`, `state`, `started_at`) to improve performance on startup queries
Pull request opened by
stevenlw-porpoise
Description
Adds an index to runs table on (job_name, state, started_at) to improve performance on startup queries. Setting this index in an internal deployment massively reduced the database load we observed due to expensive startup queries checking for already running jobs.
To keep this compatible with all supported databases, I changed the job_name and state columns with SQL Server to varchar(1024) and varchar(64) because sql server does not allow varchar(max) columns as index keys. I did not test this aspect because I do not have access to a sql server instance running a Meltano deployment.
Checklist
• I have read the
contribution guide
Was generative AI tooling used to co-author this PR?
• Yes (please specify the tool below)
Generated-by: Codex with GPT-6.1-Sol following
the agentic coding guidelines
Related Issues
• Addresses
#10373
Summary by Sourcery
Improve startup query performance by indexing job runs by name, state, and start time.
Enhancements:
• Add a composite index on runs by job name, state, and descending start time to improve startup queries that find active or recent jobs.
• Use database-specific bounded string lengths for SQL Server compatibility with the new index.
meltano/meltano