I am trying to use `tap sftp` but the files in the remote sf Meltano #getting-started

I am trying to use `tap-sftp`, but the files in th...

blake_blackshear

06/14/2023, 9:28 PM

I am trying to use

tap-sftp

, but the files in the remote sftp are gpg encrypted. I searched around a bit, but I don’t see a way to have the tap decrypt the file before ingesting. Is this possible?

taylor

06/14/2023, 9:41 PM

Hey Blake 🙂 Have you looked at a mapper as an option? https://docs.meltano.com/getting-started/part4#installing-the-transform-field-mapper the one shown there is a simple example, but you can do a lot with them. just may require a bit of coding

taylor

06/14/2023, 9:42 PM

https://github.com/meltano/sdk/pull/1617 is an example showing a more complicated mapper capability

taylor

06/14/2023, 9:43 PM

unfortunately tap-sftp isn’t based on the SDK so native stream maps for the tap aren’t available

blake_blackshear

06/14/2023, 9:48 PM

hmm. most of this seems to be about ways to map/transform/enrich fields within a file. tap-sftp cant even open the file at all. im not seeing a way it intercept the raw file stream and decrypt. tap-sftp is expecting a file encoding like

utf-8

and im getting

'utf-8' codec can't decode byte 0x85

blake_blackshear

06/14/2023, 9:50 PM

i think it would need to support something like this but for decryption: https://github.com/MeltanoLabs/tap-sftp/blob/main/tap_sftp/singer_encodings/compression.py

blake_blackshear

06/14/2023, 9:54 PM

actually, i think i see it in here: https://github.com/MeltanoLabs/tap-sftp/blob/main/tap_sftp/client.py#L156-L172

blake_blackshear

06/14/2023, 9:54 PM

https://hub.meltano.com/extractors/tap-sftp#decryption_configs-setting

blake_blackshear

06/14/2023, 9:56 PM

based on the docs there, i thought it had to do with a passphrase for the

private_key_file

blake_blackshear

06/14/2023, 10:01 PM

thank goodness we are in AWS and can use SSM. its the only supported way to store a private key to decrypt

blake_blackshear

06/14/2023, 10:02 PM

i will figure this out later

blake_blackshear

06/14/2023, 10:02 PM

thanks for rubber ducking 😉

user

06/14/2023, 10:02 PM

@blake_blackshear I worked on this tap a while ago when I was at my previous job and I believe this was our fork that we fixed up and added this decryption workflow to. Its a bit custom though because our key was stored in AWS SSM and had to get pulled down and installed each time it ran (required by our org). I'm sure it would be pretty easy to create a fork and modify it to not use SSM. Since its a meltanolabs variant PRs would be welcomed to make it a more generic solution thats not AWS specific

blake_blackshear

06/14/2023, 10:04 PM

it actually works ok for us i think. if i end up needing to modify, i will open a PR

Open in Slack

Previous Next