Hello, I've started working on a Target for DuckDB based on the Singer SDK and it's been a bit of a slog given the current state of the docs (clearly there is more love for adding new taps vs. new targets, and I understand why.) My feeling is that there is some sort of magic around the configuration/wiring up of the Target/Sink/SQLConnector relationship that I haven't quite grokked yet and was wondering if I could have some expert eyes on what I have right now: https://github.com/jwills/target-duckdb For the moment I only have the core unit tests in place and have been testing stuff by running

tap-carbon-intensity | target-duckdb --config /path/to/config.json

but I don't seem to be constructing the output files that I'm expecting right now

Thanks for the update, @josh_wills! Would love to help if I can. As you rightly call out, the target side is less well documented. I'm happy to get on a call with you if you have questions or feedback.

I managed to get a working version of the Target for DuckDB going this AM by throwing out my previous attempt and doing a copy/modify operation on the https://github.com/meltano/target-postgres target until I could get a very minimal setup working; how do y'all generally go about testing/hardening a brand new target against various kinds of data they're likely to see from different taps?

https://gitlab.com/autoidm/autoidm-target-mssql this is very hacky but it works @josh_wills , take all of the data* files (without an extension), run them against your Target ie

cat data_date | target-duckdb

I love it my man-- thank you so much!

Let me know if it helps you find anything, and we should add some to a list, and probably make this a part of the SDK or at least its own repo

it's really handy and your tests haven't defeated the target so far! 💪

@josh_wills we have https://gitlab.com/meltano/tap-smoke-test that @florian.hines made. Could be helpful 🙂

@josh_wills glad to hear of the progress. Very timely actually: https://meltano.slack.com/archives/C013X86PGUS/p1648668130566639

@taylor ftfy: friend of Meltano -> my archnemesis

“friend of Meltano” means different things depending on the subject being referenced.

Noting that my first rev of this target is based off an AGPL'd version, which I'm not an enormous fan of; thinking about trying the same trick with the datamill postgres target (which is MIT)