Hi all, I am new for Meltano, and trying SFTP extractor plug-in and facing some issue,

meltano config tap-sftp test
2023-11-08T08:11:39.323427Z [info     ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Plugin configuration is invalid
Catalog discovery failed: command ['/Users/kusansa/Documents/Oracle/workspace/pocMeltano/my-meltano-project/.meltano/extractors/tap-sftp/venv/bin/tap-sftp', '--config', '/Users/kusansa/Documents/Oracle/workspace/pocMeltano/my-meltano-project/.meltano/run/tap-sftp/tap.dbe373f8-9ed1-446b-8386-2c775acb33d8.config.json', '--discover'] returned 1 with stderr:
 /Users/kusansa/Documents/Oracle/workspace/pocMeltano/my-meltano-project/.meltano/extractors/tap-sftp/venv/lib/python3.9/site-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
CRITICAL Config is missing required keys: ['delimiter', 'table_name', 'search_prefix', 'search_pattern']
Traceback (most recent call last):
  File "/Users/kusansa/Documents/Oracle/workspace/pocMeltano/my-meltano-project/.meltano/extractors/tap-sftp/venv/bin/tap-sftp", line 8, in <module>
    sys.exit(main())
  File "/Users/kusansa/Documents/Oracle/workspace/pocMeltano/my-meltano-project/.meltano/extractors/tap-sftp/venv/lib/python3.9/site-packages/singer/utils.py", line 229, in wrapped
    return fnc(*args, **kwargs)
  File "/Users/kusansa/Documents/Oracle/workspace/pocMeltano/my-meltano-project/.meltano/extractors/tap-sftp/venv/lib/python3.9/site-packages/tap_sftp/tap.py", line 84, in main
    utils.check_config(table, REQUIRED_TABLE_SPEC_CONFIG_KEYS)
  File "/Users/kusansa/Documents/Oracle/workspace/pocMeltano/my-meltano-project/.meltano/extractors/tap-sftp/venv/lib/python3.9/site-packages/singer/utils.py", line 190, in check_config
    raise Exception("Config is missing required keys: {}".format(missing_keys))
Exception: Config is missing required keys: ['delimiter', 'table_name', 'search_prefix', 'search_pattern']

Hey @sanjeev, as said in the Exception you are missing these Config-Values

['delimiter', 'table_name', 'search_prefix', 'search_pattern']

Here is an Example of how the Config should look like in your

meltano.yml

- name: tap-sftp
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-sftp.git>
    config:
      host: YOUR_HOST
      port: YOUR_PORT
      username: YOUR_USERNAME
      start_date: ANY_DATE
      password: YOUR_PASSWORD
      tables:
      - table_name: TABLE_NAME
        search_pattern: ^.*\.csv$ # This is a regex pattern to match the file name
        search_prefix: /prod/outgoing/ # This is the path to the directory where the file is located
        key_properties: ['id'] # This is the primary key of the table
        encoding: utf-8
        sanitize_headers: false
        skip_rows: 0
        delimiter: ;

Thanks @nico_berchtold, I tried after fixing that and getting below error now for test

meltano config tap-sftp test
2023-11-08T12:17:27.014360Z [info     ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Plugin configuration is invalid
No RECORD message received

you need to specify a regex by

search_pattern

```2023-11-08T123243.197560Z [debug ] Invoking: ['/Users/kusansa/Documents/Oracle/workspace/pocMeltano/my-meltano-project/.meltano/extractors/tap-sftp/venv/bin/tap-sftp', '--config', '/Users/kusansa/Documents/Oracle/workspace/pocMeltano/my-meltano-project/.meltano/run/tap-sftp/tap.54fd7224-f346-4802-9094-226d4e7feb8d.config.json', '--catalog', '/Users/kusansa/Documents/Oracle/workspace/pocMeltano/my-meltano-project/.meltano/run/tap-sftp/tap.properties.json'] 2023-11-08T123243.299250Z [debug ] /Users/kusansa/Documents/Oracle/workspace/pocMeltano/my-meltano-project/.meltano/extractors/tap-sftp/venv/lib/python3.9/site-packages/paramiko/transport.py219 CryptographyDeprecationWarning: Blowfish has been deprecated 2023-11-08T123243.299418Z [debug ] "class": algorithms.Blowfish, 2023-11-08T123243.350539Z [debug ] INFO Starting sync. 2023-11-08T123243.414697Z [debug ] INFO Creating new connection to SFTP... 2023-11-08T123244.728423Z [debug ] INFO Connected (version 2.0, client OpenSSH_7.4) 2023-11-08T123246.555027Z [debug ] INFO Auth banner: b"Use of the Oracle network and applications is intended solely for Oracle's authorized users. The use of these resources by Oracle employees and contractors is subject to company policies, including the Code of Conduct, Acceptable Use Policy and Information Protection Policy; access may be monitored and logged, to the extent permitted by law, in accordance with Oracle policies. Unauthorized use may result in termination of your access, disciplinary action and/or civil and criminal penalties.\n" 2023-11-08T123246.556237Z [debug ] INFO Authentication (publickey) successful! 2023-11-08T123247.863232Z [debug ] INFO [chan 0] Opened sftp connection (server version 3) 2023-11-08T123247.863959Z [debug ] INFO Connection successful 2023-11-08T123247.864109Z [debug ] {"type": "STATE", "value": {}} 2023-11-08T123247.864225Z [debug ] {"type": "SCHEMA", "stream": "C_DATA_STORE", "schema": {"properties": {"DATA_STORE_KEY": {"type": ["null", "string"]}, "DATA_STORE_CODE": {"type": ["null", "string"]}, "DATA_STORE_LABEL": {"type": ["null", "string"]}, "DATA_STORE_DESCRIPTION": {"type": ["null", "string"]}, "IS_CLOUD_EXT_ENABLED": {"type": ["null", "integer", "string"]}, "QUERY_FILTER": {"type": ["null", "string"]}, "INACTIVE_FLAG": {"type": ["null", "integer", "string"]}, "LAST_EXTRACT_DATE": {"type": ["null", "string"]}, "IS_EFFECTIVE_DATE_DISABLED": {"type": ["null", "integer", "string"]}, "IS_FLEX_DATA_STORE": {"type": ["null", "integer", "string"]}, "SILENT_ERROR_ENABLED": {"type": ["null", "integer", "string"]}, "USE_ADF_QUERY": {"type": ["null", "integer", "string"]}, "IS_METADATA_GEN_VO": {"type": ["null", "integer", "string"]}, "CREATED_BY": {"type": ["null", "string"]}, "CREATION_DATE": {"type": ["null", "string"]}, "LAST_UPDATED_BY": {"type": ["null", "string"]}, "LAST_UPDATE_DATE": {"type": ["null", "string"]}, "LAST_UPDATE_LOGIN": {"type": ["null", "integer", "string"]}, "_sdc_source_file": {"type": "string"}, "_sdc_source_lineno": {"type": "integer"}, "_sdc_extra": {"items": {"type": "string"}, "type": "array"}}, "type": "object"}, "key_properties": []} 2023-11-08T123247.864646Z [debug ] INFO C_DATA_STORE: Starting sync 2023-11-08T123247.865248Z [debug ] INFO Syncing table "C_DATA_STORE". 2023-11-08T123247.865449Z [debug ] INFO Getting files modified since 2023-11-08 072002+00:00. 2023-11-08T123249.384343Z [debug ] INFO Found 1 files in "/home/opc/extractservice/SFTP/C_DATA_STORE" 2023-11-08T123249.384962Z [debug ] INFO Searching for files for matching pattern: C_DATA_STORE.csv 2023-11-08T123249.385236Z [debug ] INFO Found 1 files in "/home/opc/extractservice/SFTP/C_DATA_STORE" matching "C_DATA_STORE.csv" 2023-11-08T123249.385365Z [debug ] INFO Found file: /home/opc/extractservice/SFTP/C_DATA_STORE/C_DATA_STORE.csv 2023-11-08T123249.385685Z [debug ] INFO Found 0 files to be synced. 2023-11-08T12:32…

glad to hear 👍

Hi @nico_berchtold, how we can get the output file as .csv in stead of jsonl ?

you mean like a 1:1 copy from thr csv?

I mean in SFTP server we are having .csv file and we are running command : meltano run tap-sftp target-jsonl and output is .jsonl file , but i need output as .csv

well you kinda answered your question yourself 🙂 if you wanna use another destination than jsonl you have to you another target

ok thanks let me try, i tried it once didn't work for me let me do it again