Hi guys, 👋 , just starting out w/ Meltano after tracking the project for awhile. Trying to get a POC up and running using the

tap-s3-csv

(singer) to

targe-postgres

(transferwise) working. I was able to troubleshoot in accordance with this thread https://meltano.slack.com/archives/C013EKWA2Q1/p1639488946123600. Running this cmd

meltano --log-level=debug elt tap-s3-csv target-postgres --job_id=test

seems to work (see that the test file is discovered and fields selected, but nothing materializes in postgres and there are no errors. I know there are other variants, but I would like to use this one as it supports gzip. Has anybody been able to get this tap working or does anyone have any recs on what to try next?

When I first tried sending data to PG it looked like nothing was getting loaded because my query tool wasn't showing the newly-created schema or anything in it. 😛 If it isn't anything embarrassingly simply like that I'd try running the tap manually and then inspect the JSON it produces; if that looks right run the target manually and see what that does. Meltano is great for defining and executing pipelines; for debugging I find working directly with the tap and target more effective.

Thanks for the debugging tips! Checked the information schema so I think I'm covered on the first one haha. I need to read the docs more to figure out how to do it

Should be something like...

meltano config tap-s3-csv > src_config.json

tap-s3-csv -c src_config.json > output.json

At least if the s3-csv tap is like the mysql tap anyway 😉

Hmm so the config as far as I can tell looks good. The output is blank. I'm guessing that should have records? Should I be using

meltano invoke

in front of the 2nd commpand?

Maybe? Personally when working with the taps and targets I clone their gitub repo and work directly with that. You should be able to work with the same code Meltano uses if you use the stuff in the .meltano/extractors/ directory though. That has a venv for the extractor you can use to run it manually.

@chris_evans I made a fork some time ago. Try it out. Contact me by DM if you have doubts. https://meltano.slack.com/archives/C013EKWA2Q1/p1639585900136700?thread_ts=1639488946.123600&cid=C013EKWA2Q1

Thanks @boggdan_barrientos, what was the issue you needed to solve for on the original repo to get it to work? The original repo has gzip/zip handling so would like to get it working if possible.