Hi Team I am facing one issue with tap sftp ``` venv PS C Si Meltano #best-practices

Hi Team, I am facing one issue with tap-sftp. ``...

Siba Prasad Nayak

05/14/2025, 6:21 AM

Hi Team, I am facing one issue with tap-sftp.

Copy code

(venv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend>
 meltano --log-level=debug invoke tap-sftp
2025-05-14T06:09:15.617690Z [warning  ] Failed to create symlink to 'meltano.exe': administrator privilege required
2025-05-14T06:09:15.624021Z [debug    ] Meltano 3.6.0, Python 3.12.3, Windows (AMD64)
2025-05-14T06:09:15.632565Z [debug    ] Looking up time zone info from registry
2025-05-14T06:09:15.651188Z [info     ] Environment 'dev' is active   
2025-05-14T06:09:15.707195Z [debug    ] Creating DB engine for project at 'C:\\Siba_\\Work\\POC_ConnectorFactory\\Gerrit\\Connector_Factory_Development\\meltano-backend' with DB URI 'sqlite:/C:\\Siba_\\Work\\POC_ConnectorFactory\\Gerrit\\Connector_Factory_Development\\meltano-backend\\.meltano/meltano.db'
2025-05-14T06:09:16.045430Z [debug    ] Skipped installing extractor 'tap-sftp'
2025-05-14T06:09:16.046429Z [debug    ] Skipped installing 1/1 plugins
2025-05-14T06:09:16.231095Z [debug    ] Created configuration at C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\run\tap-sftp\tap.31d90bf6-66b2-4bc2-9d17-9305905bbcdf.config.json
2025-05-14T06:09:16.235112Z [debug    ] Could not find tap.properties.json in C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\tap.properties.json, skipping.
2025-05-14T06:09:16.238126Z [debug    ] Could not find tap.properties.cache_key in C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\tap.properties.cache_key, skipping.
2025-05-14T06:09:16.240124Z [debug    ] Could not find state.json in C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\state.json, skipping.
2025-05-14T06:09:16.248129Z [debug    ] Invoking: ['C:\\Siba_\\Work\\POC_ConnectorFactory\\Gerrit\\Connector_Factory_Development\\meltano-backend\\.meltano\\extractors\\tap-sftp\\venv\\Scripts\\tap-sftp.exe', '--config', 'C:\\Siba_\\Work\\POC_ConnectorFactory\\Gerrit\\Connector_Factory_Development\\meltano-backend\\.meltano\\run\\tap-sftp\\tap.31d90bf6-66b2-4bc2-9d17-9305905bbcdf.config.json']
C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\venv\Lib\site-packages\paramiko\pkey.py:59: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from cryptography.hazmat.primitives.ciphers.algorithms in 48.0.0.
  "cipher": algorithms.TripleDES,
C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\venv\Lib\site-packages\paramiko\transport.py:219: CryptographyDeprecationWarning: Blowfish has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.Blowfish and will be removed from cryptography.hazmat.primitives.ciphers.algorithms in 45.0.0.
  "class": algorithms.Blowfish,
C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\venv\Lib\site-packages\paramiko\transport.py:243: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from cryptography.hazmat.primitives.ciphers.algorithms in 48.0.0.
  "class": algorithms.TripleDES,
2025-05-14T06:09:17.479363Z [debug    ] Deleted configuration at C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\run\tap-sftp\tap.31d90bf6-66b2-4bc2-9d17-9305905bbcdf.config.json
(venv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend>

I have created a basic configuration in meltano.yml file.

Copy code

- name: tap-sftp
    namespace: tap_sftp
    pip_url: ./connectors/tap-sftp
    executable: tap-sftp
    config:
      host: 10.148.155.30
      port: 22
      username: ubuntu
      start_date: 2025-05-13
      private_key_file: bridgex.pem
      tables:
      - table_name: single_file_test
        search_prefix: /home/ubuntu
        search_pattern: 'wget-log'

Copy code

(venv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend>
 meltano config tap-sftp list
2025-05-14T06:16:25.653934Z [warning  ] Failed to create symlink to 'meltano.exe': administrator privilege required
2025-05-14T06:16:25.677427Z [info     ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.

Custom, possibly unsupported by the plugin:
host [env: TAP_SFTP_HOST] current value: '10.148.155.30' (from `meltano.yml`)
port [env: TAP_SFTP_PORT] current value: 22 (from `meltano.yml`)
username [env: TAP_SFTP_USERNAME] current value: 'ubuntu' (from `meltano.yml`)
start_date [env: TAP_SFTP_START_DATE] current value: '2025-05-13' (from `meltano.yml`)
private_key_file [env: TAP_SFTP_PRIVATE_KEY_FILE] current value: 'bridgex.pem' (from `meltano.yml`)
tables [env: TAP_SFTP_TABLES] current value: [{'table_name': 'single_file_test', 'search_prefix': '/home/ubuntu', 'search_pattern': 'wget-log'}] (from `meltano.yml`)
(venv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend>

Can anyone please help.

Reuben (Matatika)

05/14/2025, 11:25 AM

What's the issue you are seeing?

Siba Prasad Nayak

05/14/2025, 11:31 AM

My understanding is when I execute "meltano invoke tap-sftp" it should print the content on the remote file on the console. But in my case that's not happening due to some reason and giving some warnings. So I am not able to understand the issue because there is no error on the console

Reuben (Matatika)

05/14/2025, 11:32 AM

Custom, possibly unsupported by the plugin:

This is probably a giveaway. Looks like you are using a custom plugin, so you will need to define the settings in your

meltano.yml

Reuben (Matatika)

05/14/2025, 11:33 AM

Otherwise, the configuration will not be passed to the tap.

Siba Prasad Nayak

05/14/2025, 11:34 AM

You mean I need to provide the configuration values under "settings" instead of "config"?

Reuben (Matatika)

05/14/2025, 11:37 AM

You need to provide setting definitions for the config you want to use:

Copy code

settings:
    - name: host
    - name: port
      kind: integer
    - name: username
    - name: start_date
      kind: date_iso8601
    - name: private_key_file
    - name: tables
      kind: array
    config:
      host: 10.148.155.30
      port: 22
      username: ubuntu
      start_date: 2025-05-13
      private_key_file: bridgex.pem
      tables:
      - table_name: single_file_test
        search_prefix: /home/ubuntu
        search_pattern: 'wget-log'

https://docs.meltano.com/reference/plugin-definition-syntax#settings

Siba Prasad Nayak

05/14/2025, 11:50 AM

I am getting the same error with this configuration as well

Copy code

(sibaVenv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend> meltano --log-level=debug invoke tap-sftp
2025-05-14T11:47:17.284004Z [warning  ] Failed to create symlink to 'meltano.exe': administrator privilege required
2025-05-14T11:47:17.287255Z [debug    ] Meltano 3.6.0, Python 3.12.3, Windows (AMD64)
2025-05-14T11:47:17.289253Z [debug    ] Looking up time zone info from registry
2025-05-14T11:47:17.301296Z [info     ] Environment 'dev' is active   
2025-05-14T11:47:17.400846Z [debug    ] Creating DB engine for project at 'C:\\Siba_\\Work\\POC_ConnectorFactory\\Gerrit\\Connector_Factory_Development\\meltano-backend' with DB URI 'sqlite:/C:\\Siba_\\Work\\POC_ConnectorFactory\\Gerrit\\Connector_Factory_Development\\meltano-backend\\.meltano/meltano.db'
2025-05-14T11:47:17.698724Z [debug    ] Skipped installing extractor 'tap-sftp'
2025-05-14T11:47:17.699727Z [debug    ] Skipped installing 1/1 plugins
2025-05-14T11:47:17.836800Z [debug    ] Created configuration at C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\run\tap-sftp\tap.c9840cf2-cf7e-43d3-8e47-2458dbc4b06f.config.json
2025-05-14T11:47:17.839351Z [debug    ] Could not find tap.properties.json in C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\tap.properties.json, skipping.
2025-05-14T11:47:17.841331Z [debug    ] Could not find tap.properties.cache_key in C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\tap.properties.cache_key, skipping.
2025-05-14T11:47:17.843329Z [debug    ] Could not find state.json in C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\state.json, skipping.
2025-05-14T11:47:17.850493Z [debug    ] Invoking: ['C:\\Siba_\\Work\\POC_ConnectorFactory\\Gerrit\\Connector_Factory_Development\\meltano-backend\\.meltano\\extractors\\tap-sftp\\venv\\Scripts\\tap-sftp.exe', '--config', 'C:\\Siba_\\Work\\POC_ConnectorFactory\\Gerrit\\Connector_Factory_Development\\meltano-backend\\.meltano\\run\\tap-sftp\\tap.c9840cf2-cf7e-43d3-8e47-2458dbc4b06f.config.json']
C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\venv\Lib\site-packages\paramiko\pkey.py:59: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from cryptography.hazmat.primitives.ciphers.algorithms in 48.0.0.
  "cipher": algorithms.TripleDES,
C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\venv\Lib\site-packages\paramiko\transport.py:219: CryptographyDeprecationWarning: Blowfish has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.Blowfish and will be removed from cryptography.hazmat.primitives.ciphers.algorithms in 45.0.0.
  "class": algorithms.Blowfish,
C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\venv\Lib\site-packages\paramiko\transport.py:243: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from cryptography.hazmat.primitives.ciphers.algorithms in 48.0.0.
  "class": algorithms.TripleDES,
2025-05-14T11:47:19.628262Z [debug    ] Deleted configuration at C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\run\tap-sftp\tap.c9840cf2-cf7e-43d3-8e47-2458dbc4b06f.config.json
(sibaVenv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend>

The configuration is

Copy code

- name: tap-sftp
    namespace: tap_sftp
    pip_url: ./connectors/tap-sftp
    executable: tap-sftp
    settings:
    - name: host
    - name: port
      kind: integer
    - name: username
    - name: start_date
      kind: date_iso8601
    - name: private_key_file
    - name: tables
      kind: array
    config:
      host: 10.148.155.30
      port: 22
      username: ubuntu
      start_date: 2025-05-13
      private_key_file: bridgex.pem
      tables:
      - table_name: single_file_test
        search_prefix: /home/ubuntu
        search_pattern: 'wget-log'

Siba Prasad Nayak

05/14/2025, 11:56 AM

https://github.com/MeltanoLabs/tap-sftp this is the code I am taking reference from

Reuben (Matatika)

05/14/2025, 12:12 PM

That tap is not built with the SDK, so it looks you might need to pass it a catalog when `invoke`ing, or else it just no-ops.

Copy code

meltano invoke tap-sftp --discover > catalog.json
meltano invoke tap-sftp --catalog catalog.json

or possibly just

Copy code

meltano --log-level debug config tap-sftp test

Siba Prasad Nayak

05/14/2025, 12:27 PM

Copy code

(sibaVenv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend> meltano invoke tap-sftp --discover > catalog.json
2025-05-14T12:27:07.165415Z [warning  ] Failed to create symlink to 'meltano.exe': administrator privilege required
2025-05-14T12:27:07.203415Z [info     ] Environment 'dev' is active   
C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\venv\Lib\site-packages\paramiko\pkey.py:59: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from cryptography.hazmat.primitives.ciphers.algorithms in 48.0.0.
  "cipher": algorithms.TripleDES,
C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\venv\Lib\site-packages\paramiko\transport.py:219: CryptographyDeprecationWarning: Blowfish has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.Blowfish and will be removed from cryptography.hazmat.primitives.ciphers.algorithms in 45.0.0.
  "class": algorithms.Blowfish,
C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\venv\Lib\site-packages\paramiko\transport.py:243: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from cryptography.hazmat.primitives.ciphers.algorithms in 48.0.0.
  "class": algorithms.TripleDES,
INFO Starting discover
CRITICAL the JSON object must be str, bytes or bytearray, not list
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\venv\Scripts\tap-sftp.exe\__main__.py", line 7, in <module>
  File "C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\venv\Lib\site-packages\singer\utils.py", line 229, in wrapped
    return fnc(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\venv\Lib\site-packages\tap_sftp\__init__.py", line 82, in main
    do_discover(args.config)
  File "C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\venv\Lib\site-packages\tap_sftp\__init__.py", line 22, in do_discover
    streams = discover_streams(config, encoding_format)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\.meltano\extractors\tap-sftp\venv\Lib\site-packages\tap_sftp\discover.py", line 18, in discover_streams
    tables = json.loads(config['tables'])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\SiNayak\AppData\Local\Programs\Python\Python312\Lib\json\__init__.py", line 339, in loads
    raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not list
(sibaVenv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend>

Siba Prasad Nayak

05/14/2025, 12:30 PM

I guess I am passing the table values in a wrong way. Let me check

Reuben (Matatika)

05/14/2025, 12:35 PM

Oh, you will need to remove

kind

from the

tables

setting definition to treat the value as a string:

Copy code

- name: tables

Siba Prasad Nayak

05/14/2025, 12:36 PM

ohh ok

Reuben (Matatika)

05/14/2025, 12:37 PM

(or set `kind: string`; whatever you prefer)

Siba Prasad Nayak

05/14/2025, 12:38 PM

ok let me try

Siba Prasad Nayak

05/14/2025, 12:43 PM

- name: tables kind: string #value: '[{"table_name": "single_file_test", "search_prefix": "/home/ubuntu", "search_pattern": "wget-log", "delimiter": ","}]' config: host: 10.148.155.30 port: 22 username: ubuntu start_date: 2025-05-13 private_key_file: 'C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend\bridgex.pem' tables: '[{"table_name": "single_file_test", "search_prefix": "/home/ubuntu", "search_pattern": "wget-log", "delimiter": ","}]' with this I am able to move ahead but I am facing issue with Authentication.

Siba Prasad Nayak

05/14/2025, 12:45 PM

It works from the same shell when I do manual SFTP connection.

Reuben (Matatika)

05/14/2025, 1:03 PM

paramiko

is the SSH library that tap is using. Have a look through the issues: https://github.com/search?q=repo%3Aparamiko%2Fparamiko%20%22Authentication%20(publickey)%20failed%22&type=issues

Siba Prasad Nayak

05/14/2025, 1:04 PM

Steven Searcy

06/06/2025, 5:07 PM

Was this ever resolved? I'm running into the same issue.

Siba Prasad Nayak

06/07/2025, 4:18 AM

Yes the issue was resolved after upgrading the paramiko library and same change in setup.py I believe

ty 1

7 Views

Open in Slack

Previous Next