thomas_briggs
12/09/2022, 8:10 PM_sdc_batched_at > ...
clause; to get reasonable performance at scale that column needs to be indexed. I can add manually indexes easily enough but it feels wrong to be making changes to tables that the meltano pipeline is supposed to be maintaining. 🤔
I've found the schema_mapping
option within the pipelinewise Postgres target, which seems like a possible solution in our particular case but isn't a general answer. I also don't want to have to enumerate every table in our schema in the meltano.yml. 🙄 Arguably the target should create an index on that particular column automatically (which is does for the _sdc_deleted_at column) but it feels like there should be a more general solution out there too... this isn't the only column anyone will ever need to index, after all.
Anyway... any thoughts would be appreciated because as of now all I have is a short list of options I don't like. 😉