A newbie in Meltano, facing problem building my first extractor for AzBlob I'm using tap-file . My .env file has all these attributes necessary to connect to az blob

TAP_FILE_AIRBYTE_CONFIG_PROVIDER_SAS_TOKEN='xxxx'
TAP_FILE_AIRBYTE_CONFIG_PROVIDER_STORAGE='AzBlob'
TAP_FILE_AIRBYTE_CONFIG_PROVIDER_STORAGE_ACCOUNT='xxxx'

My meltano.yaml looks like this

version: 1
default_environment: dev
project_id: eecac43d-8204-4503-b2e9-0519f72933c4
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-file
    variant: airbyte
    pip_url: git+<https://github.com/MeltanoLabs/tap-airbyte-wrapper.git>
    config:
      airbyte_config:
        url: 
          /di-data-extraction/xx/xx/20240726/
        dataset_name: xxxx
        format: csv

when i run meltano config tap-file test Getting the following error- raise AirbyteException( tap_airbyte.tap.AirbyteException: Discover failed with return code 2: usage: main.py [-h] {spec,check,discover,read} ... main.py: error: argument command: invalid choice: '--' (choose from 'spec', 'check', 'discover', 'read')

tap-airbyte-wrapper

is broken currently - see: https://github.com/MeltanoLabs/tap-airbyte-wrapper/issues/19 https://github.com/MeltanoLabs/tap-airbyte-wrapper/pull/20 You can pin to previous commit, which should fix the issue:

pip_url: git+<https://github.com/MeltanoLabs/tap-airbyte-wrapper.git@87da181>

oh thanks a lot

Oh, you will have to

meltano install --clean extractor tap-airbyte-wrapper

as well.

Did All that, now I'm getting this error- File "/usr/local/lib/python3.9/site-packages/azure/storage/blob/_container_client.py", line 158, in init raise ValueError("Please specify a container name.") ValueError: Please specify a container name. in this documentation - https://hub.meltano.com/extractors/tap-file/ I dont see any parameter to set container name. Also container name is already mentioned in the airbyte_config url as shown above. I do see container name in this documentation https://hub.meltano.com/loaders/target-azureblobstorage/ But this is for loader, not extractor

@Sreejit I took a peek at the source code just to see: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/storage/azure-storage-blob/azure/storage/blob/_blob_service_client.py https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/storage/azure-storage-blob/azure/storage/blob/_container_client.py#L142 Looks like the airbyte connector is not setting it? https://github.com/airbytehq/airbyte/blob/c9f45a0b85735f58102fcd78385f6f673e731aa6/airbyte-integrations/connectors/source-file/source_file/client.py#L236 Although I am not familiar with that source connector, it looks like the wrapper is working but the source might not be. Does it work in airbyte?

Well the tap-airbyte-connector has issues as pointed out by Reuben, so I'm using an older version. maybe that's why.I will wait to test all these when the tap-airbyte-connector issue is resolved. And test again when this pr - https://github.com/MeltanoLabs/tap-airbyte-wrapper/pull/20 is merged.