Hi, I am using tap-sftp to extract CSV files from server, I need to skip few rows and specify that header and data starts from 6th row in CSV, Does anyone experienced the same issue and was able to resolve this?

@sumit_singh theres an open issue in the repo related to this feature https://github.com/MeltanoLabs/tap-sftp/issues/12. I dont know if @Henning Holgersen has had a chance to build it yet though

Yeah sorry @sumit_singh, I was supposed to add it to the main repo but attention drifted elsewhere once I got it working. We are using this fork in production today (and have been for a while), so it’s safe to use. I’ll get the PR to the main repo going and let you know.

@Henning Holgersen thank you, any expected timeline to merge it in main repo ?

@sumit_singh I’ll try to get a PR in by the end of the week. In the meantime, here is an example of using my fork:

extractors:
  - name: tap-sftp
    namespace: tap_sftp
    pip_url: git+<https://github.com/radbrt/tap-sftp.git>
    executable: tap-sftp
    capabilities:
    - state
    - catalog
    - discover
    settings:
    - name: username
    - name: password
      kind: password
    - name: start_date
      value: '2010-01-01T00:00:00Z'
    config:
      username: ftpuser
      host: <http://ftp.example.com|ftp.example.com>
      password: mySuperSecretPassword
      port: 22
      start_date: 2021-01-28
      tables:
      - table_name: test_table
        search_prefix: ''
        search_pattern: test_.*\.*
        key_properties: []
        delimiter: ','
        encoding: utf-8

Thank you @Henning Holgersen, so changing jsut the pip_url would work ?

That should work, yes.

@Henning Holgersen how can I use skip-rows branch ? What needs to be changed in pip_url ?

@sumit_singh you can use

pip_url: git+<https://github.com/meltanolabs/tap-sftp.git@skip-rows>

in the config.

@sumit_singh you'll need to run a clean reinstall once you update the pip url like

meltano install extractor tap-sftp --clean

PR is now merged (and a bug fixed), you should be able to use the main branch now @sumit_singh . If you tried to get the skip-rows branch to work without luck, it might be because Pat spotted a typo. Should work now anyways. Let me know when it is safe to delete the skip-rows branch.

@Henning Holgersen I got it working with skip-rows branch as well, now tested with main branch as well and its working as expected. Thank you so much for your efforts and guidancemelty bouncymelty bouncy