👋 Q - is there a way to have dynamic dates in meltano.yml? (e.g. -

start_date: some date variable or function for yesterday's date

)

Checkout https://stackoverflow.com/questions/70584527/how-to-provide-a-relative-start-date-for-meltano-ci-cd-pipelines-on-singer. I use that technique along with environments in https://github.com/meltano/squared/blob/main/data/environments/cicd.meltano.yml for CICD where I just want 1 day worth of data

You'll want to be mindful of differences between POSIX

date

and GNU

date

when using such utilities that exists in both GNU and POSIX

thanks to both of ya! makes sense, will try it soon

I haven't used dagster (yet), but it certainly would be within the realm of dagster to supply the input to your tap's

start_date

accordingly to the job's requirements. The

date

command from GNU coreutils will also do the job as shown in the SO topic. If you really want to go deep about falsehoods about time though, you can read a few examples here 😉 FalsehoodsAboutTime.com