Hi, I’m trying to load some data from an sftp server, and have bumped into some issues. The

tap-sftp

keeps giving me an error saying

Extraction failed  code=1 message=KeyError: 'streams'

when I pass in a catalog json file that is similar to what has been posted as examples. Does anyone have a working example of a tap-sftp connection? My second attempt was to use tap-spreadsheets-anywhere, but it seems this needs a hardcoded sftp password in the

meltano.yml

file - once for each table. Has anyone found a way around this hardcoding?

For

tap-spreadsheets-anywhere

, you can write the

tables

as a YAML array in

meltano.yml

:

plugins:
  extractors:
  - name: tap-spreadsheets-anywhere
    variant: ets
    pip_url: git+<https://github.com/ets/tap-spreadsheets-anywhere.git>
    config:
      tables:
      - table_name: "nkoder"
        search_prefix: "/"
        search_pattern: "naer.*\\.csv$"
        key_properties: ["naerk"]
        delimiter: ";"
        encoding: "utf-8"

Thank you, that’s makes it a lot easier. Spreadsheets anywhere looks very promising. I now have the following readable and functioning config in meltano.yml:

- name: tap-spreadsheets-anywhere
  variant: ets
  pip_url: git+<https://github.com/ets/tap-spreadsheets-anywhere.git>
  config:
    tables:
      - path: "<sftp://my_user_name:my_pa55word1@ftp.server.com>"
        name: "nkoder"
        pattern: "naer.*"
        start_date: "2017-05-01T00:00:00Z"
        format: "csv"
        delimiter: ";"
        key_properties: ["naerk"]

Do you know if there is a way to use an environment-variable for the “path” entry so that I don’t have to hardcode the connection details?

@Henning Holgersen

Do you know if there is a way to use an environment-variable for the “path” entry so that I don’t have to hardcode the connection details?

Unfortunately there isn’t a way at the moment. See https://github.com/meltano/meltano/issues/3171.

is there any way of working around this? passing the env var via settings or something? Running into the same issue with the same tap 😬