a small shell script, only 2 lines that has to be run before invoking meltano

export MELTANO_DEV_START_DATE=$(date -d '-2 day' '+%Y-%m-%dT00:00:00Z')
meltano "$@"
exit $?

and in the docker-compose the entrypoint is set to the script above. then, for each tap, its

dev

environment defines

config.start_date

to

$MELTANO_DEV_START_DATE

in our team, we have a docker based development workflow, so its safe to assume that all devs will test out their changes locally using

docker compose run meltano run <tap-name> target-jsonl

there are still a couple of traps folks might fall into. 1. the tap has been run locally but a few months ago, so Meltano picks up the start_date from the stored state. Its possible to ignore that by specifying

--full-refresh

, but it feels like a big risk to expect folks to remember to do this consistently. 2. the tap does not have a dev environment defined with

config.start_date

to

$MELTANO_DEV_START_DATE

yet. (e.g. when a new tap is being introduced and the changes are on the developer's local machine only). While this doesn't constitute a risk for the data engineering pipelines, being a tap that hasn't been deployed to production yet, it is still worthwhile to prevent quota exhaustion as these can be shared company-wide resources.