Hello All, I have been using Meltano for a while t...
# singer-target-development
s
Hello All, I have been using Meltano for a while to perform our ELT processes. For a specific use-case, we now need to develop a reverse ETL pipeline. I was thinking about using Meltano for this as well (considering it is already integrated in our stack). The destination system for our reverse ETL flow is a REST API where we need to do POST/PUT requests. I can't seem to find some good examples on this subject. Does anyone have some good examples of Targets for REST API's in general? When doing the post, we also get back an import_id. How can we pull this import_id out of the target in order to monitor the status of the import? Or can we do this inside the meltano target as well?
v
"good examples" is subjective but there are some. Hotglue has some examples! Personally we have ~10 targets now (maybe more?) that do this but they are all internal (target-activedirectory, target-entraid, target-netsuite, etc) SaaS targets can be kept pretty simple luckily! Yes I think it's a great usecase
When doing the post, we also get back an import_id. How can we pull this import_id out of the target in order to monitor the status of the import? Or can we do this inside the meltano target as well?
Using state is the best way here, hotglue has some examples of how, we do this as well
👍 1
What we do most of the time is we just rely on the status code of the target to return as 0, then we design all integrations to be idempotent so it's not a big deal if/when something fails
There's a bunch of tricks you learn over time too, like if something fails in the target side you should try all records and just keep track of the failures, then fail at the end with a failed exit code (so one failure doesn't hold up your whole import process)
s
Thanks for the feedback. I found a nice tutorial in the hotglue docs. I will try this first.
1