I have been looking into what it would take to get...
# singer-target-developmentm
michael_horvath
03/14/2022, 7:15 PMI have been looking into what it would take to get a basic Oracle target created using the Meltano SDK and I'm getting a bit stuck finding out where to start. I used the cookiecutter template to create a SQL serialization (option 3) based target, but the scaffolding it creates defaults to the single record serialization instead of https://sdk.meltano.com/en/latest/classes/singer_sdk.SQLSink.html, nor can I find any tutorials or code samples for building targets.
I'm an experienced python developer, but I'm new to the singer spec. Does anyone have any resources they could recommend?
michael_horvath
03/14/2022, 7:46 PMPartially answering my own question, I found a reference to a port for sqlite using the new sql-type classes, https://github.com/MeltanoLabs/target-sqlite/tree/port-to-sdk/singer_sqlite by @aaronsteers.
It might be that there is an upcoming branch already for the sdk repo https://gitlab.com/meltano/sdk/-/tree/main, but I wasn't able to track it down.
a
aaronsteers
03/15/2022, 11:09 PM@michael_horvath - Thanks for posting. Apologies if the docs or cookiecutters were pointing you in the wrong direction. The SQL-type target support is very new and benefits a lot from developers testing and providing feedback for improvement. I'll look into the issue you mention and will post back.
Are you blocked as of now or have you been able to see some records flowing into Oracle?
m
michael_horvath
03/16/2022, 12:29 PMI understand completely, I didn't realize it was so new at the time I started using it. I've been using your example to work toward an ambidextrous library, I'm not blocked. Thanks!
a
aaronsteers
03/16/2022, 12:32 PMAwesome. Glad to hear. Would love to hear from you any improvements you would propose. And we will gladly accept pull requests for docs, features, and bug fixes. 🙂
m
michael_horvath
03/18/2022, 12:25 PMI have run into a bit of an issue while testing a full table replication to my oracle target.
2022-03-17T210001.743673Z [info ] INFO Table IRB_i2b2_Direct_View proposes FULL_TABLE sync cmd_type=extractor job_id=eirb-to-oracle name=tap-eirb run_id=fc43d0e4-b3ec-4742-bc8a-1ef3a8d29208 stdio=stderr
2022-03-17T210001.744487Z [info ] INFO Table IRB_i2b2_Direct_View will use FULL_TABLE sync cmd_type=extractor job_id=eirb-to-oracle name=tap-eirb run_id=fc43d0e4-b3ec-4742-bc8a-1ef3a8d29208 stdio=stderrIt attempts to perform a "DELETE FROM IRB_i2b2_Direct_View WHERE _sdc_table_version <= 1647550801948". The problem here is that you can't start a column with an underscore in oracle unless you quote it. I've tried to set add_record_metadata to False by default (https://github.com/wake-forest-ctsi/target-oracle/blob/d12675f40e7f9b115cb5727aee003a101f7b1d0a/target_oracle/config.py#L11) to try and disable these columns. Is there something else that should be done to remove the integration metadata columns from the target?