Meltano

Has anyone used <https://github.com/bonobos/tap-sftp|tap-sftp> before? Any clue how to automate the discovery process?

<@U06CQGUPTPB> yes I work at Bonobos (that fork owner) and we use it in production. What do you mean by automating discovery?

For our use case the files have a consistent prefix with a timestamp appended to each filename like `FilePrefix-20210915T000000.csv`  so we just set the tables config to `FilePrefix*`

<@U06D0TMQ2U8> Hi! Just meaning this config:

``` "tables": [
         {
             "table_name": "MyExportData",
             "search_prefix": "\/Export\/SubFolder",
             "search_pattern": "MyExportData.*\\.zip.gpg$",
             "key_properties": [],
             "delimiter": ",",
             "encoding": "utf-8"
         }
     ],```
So what does this section look like for you?

``` "tables": [
         {
             "table_name": "ExampleFile",
             "search_prefix": "/Export/SubFolder",
             "search_pattern": "FilePrefix-.*\\.csv$",
             "key_properties": [],
             "delimiter": ",",
             "encoding": "utf-8"
         }
     ],```

I havent tested that but thats what I expect to work

it will iterate all files matching that regex and read them as a single stream. I think it keeps track of the file modified timestamp in order to run incremental syncs

Ahh, that makes sense! Thanks Pat, will test on my side :slightly_smiling_face:

no problem - I also just updated the config above. I think the escaped slashes in the search prefix was a legacy thing that isnt needed anymore

let me know if you have issues and I can try to help