It seems

tap-spreadsheets-anywhere

has less support for Azure than I thought, so I have created basic support for Azure blob storage. I’m new to the tap though, and there are some decisions that needs to be made before I can start thinking about creating a PR. I logged an issue on the repo to try to detail the remaining questions. If anyone feels like chiming in, it can be found here: https://github.com/ets/tap-spreadsheets-anywhere/issues/31

just kind of musing on the topic, but i wonder how hard it would be to wrap azcopy in meltano

What I did to work around the issue was to use

rclone

instead of

azcopy

@christoph Yeah… At one point Github Copilot even suggested to me that smart_open didn’t support azure, but I guess I was just lucky with the timing, that it’s there now. That if-else is one of the things I’m not very happy about…

Btw, I don't use Managed Identity either. I just use Firewall policies to allow Storage Account access for whole subscriptions, using Azure subscriptions as the trust boundary.

Oh… No we use managed identity all the way (sometimes we have to fall back to service principals), but some of the data integration happen across organisations and then SAS keys are the best option. I guess meltano use-cases are typically across some sort of boundary, which might mean SAS-keys are the primary access method for many. Back to the if-else statement, I have a feeling the best option is to add some kwarg stuff to it so that we can more easily add other parameters as well, based on the protocol. But I have to pick that one up again tomorrow.

commented on the if-else statement with a proposed tweak

You can always kick in the tires on the Airbyte wrapper https://github.com/MeltanoLabs/tap-airbyte-wrapper Example configs for Azure https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-file/integration_tests/client_storage_providers_test.py#L130 EDIT: updated example config link to master branch

@alexander_butler we are running meltano inside prefect on k8s, I spent some hours yesterday trying to find a good solution because I would have loved to use that, but… I didn’t find any solution that would be workable in practice.

@Henning Holgersen Spitballing. You could have a prefect agent running on a VM for blob ETL specifically. Then just a dedicated work queue for jobs that need it. It would still be orchestrated by prefect all the same.

I have had the same thought, and it is absolutely an option.

@Henning Holgersen @Matt Menzenski My gut feel on the kwargs is to go for that ... Haven't fully thought it through but it sounds more future proof than if-else

I threw in a comment too on what I'd do