A bit of a weird use case here, wondering if anyone has any ideas for me. I'm using tap-spreadsheets-anywhere in my project to pull Azure's public list of IP ranges. They provide it as a downloadable file at a url that looks like

<https://download.microsoft.com/download/7/1/D/71D86715-5596-4529-9B13-DA13A5DE5B63/ServiceTags_Public_202200808.json>

. Up until now I've hardcoded the path, including the date in my configs but it looks like it fails consistently every 14 days and the date needs to be updated. To avoid making a PR every 14 days I was hoping to fill the date portion dynamically but I'm not exactly sure how to pull that off. I need to consider my github actions CI pipeline and prod environment using the meltano docker image. It feels like the solution is to use env vars. I could use env vars that I manually update every 14 days to avoid the PRs but that also isnt the best, ideally I'd like to dynamically generate the date before each run but I'm not sure the best way to do it. I could do something like what I do for dynamic start_date https://github.com/meltano/squared/blob/main/.github/workflows/test.yml#L27 but I dont know how I would expose that to a prod job/schedule. I like the idea of solving it with a tiny python script as a utility that outputs the right date so its OS agnostic and I know python better than bash, but how would it pass the output to the tap or add it to the meltano environment variables? I call the tap using jobs so maybe I could do

my_date_util tap-spreadsheets-anywhere target-snowflake

, but would it be able to pass that date up to meltano env vars for use in the next plugin? I know this is hacky and I feel like thats not possible but wanted to see if others had ideas.

If the date changes in 14 day intervals, I would hard code that 14 change into the tap, no?

Did a quick google, It looks like PowerShell core may have something for you https://adamtheautomator.com/azure-ip-ranges/ https://docs.microsoft.com/en-us/powershell/module/az.network/get-aznetworkservicetag?view=azps-8.2.0&viewFallbackFrom=azps-5.1.0 Could use Powershell core (cross platform supposedly so it should work on Ubuntu) to pull this. Or maybe you could go look at the implementation of the function (it might actually be open source) and see what api endpoints they are hitting?

Another idea. date is on Unix by default too and pretty sweet. 1 sec

Another idea. date is on Unix by default too and pretty sweet. 1 sec

date +'%Y%m%d

Nice one! That's indeed one of the few true UNIX (POSIX)

date

command features!

If the date changes in 14 day intervals, I would hard code that 14 change into the tap, no?

@christoph I was hoping to keep using tap-spreadsheets-anywhere instead of managing a custom tap. I would normally agree though, if it was a normal single source tap

@pat_nadolny I think the bottom section I pointed to doesn't. It uses a different call

Another idea. date is on Unix by default too and pretty sweet. 1 sec

@alexander_butler thanks, I use jobs/schedules so I dont have the ability to add that

MY_MELTANO_VAR=..

to the beginning of my command. Maybe I could put it in my .env? I wonder if it would get evaluated properly 🤔

Kinda like @alexander_butler’s answer better now, I assumed the whole UUID was changing every time

@visch yeah I like that idea too, I'm going to see what I can do. I was hoping to find a way to do it with a python script because the bash is going to get messy since the 14 day rotating date is tricky. This was my excel equation

=DATE(2022,7,11)+14*ROUNDDOWN((NOW()-DATE(2022,7,11))/14)

- takes current date and finds the most recent date on the 14 day update schedule

haha got it! powershell smells nicer already

@visch thanks for the help, i'll report back what I end up doing

@pat_nadolny - I'm curious how this turned out for you. It seems like it would make life a lot easier if there was a file you could download called

ServiceTags_Public_latest.json

. And perhaps the file could have a createdDate property so you could detect how fresh it was - if you cared...

@chrish I still havent solved it, I just manually update it periodically still. I'll report back once I have a legit solution