https://linen.dev logo
Join Slack
Powered by
With target's in the SDK we have a `clean_up` mess...
# singer-tap-development
v
With target's in the SDK we have a 
clean_up
message re https://sdk.meltano.com/en/latest/classes/singer_sdk.Sink.html#singer_sdk.Sink.clean_up With taps we don't have the equivalent for Streams or taps, so what if we want to do something like a connection to a DB and then close the connection to the DB. Today even with the SQL Alchemy jobs we're making a seperate connection for each stream. Maybe I"m wrong and someone can prove it to me?
a
Technically you can instantiate the pool in the 
Tap
class and use 
atexit
stdlib module to register clean ups. 🤷
v
Yeah rely on signals is good idea, I like it
e
I thought I had created an issue to support signals in the SDK but I probably only dreamt it 😅. I agree that it’s a good idea.
v
Note: The functions registered via this module are not called when the program is killed by a signal not handled by Python, when a Python fatal internal error is detected, or when 
os._exit()
is called.
Interesting
Could register sigint, sigterm to the same function though with 
signal
still interesting
a
atexit
is pretty common to use in this situation where you want a clean up script in most situations. Going deeper into signal handling and trying to "overpower" the user or OS to run your cleanup even in the event of fatal failure is unnecessary or unexpected in most cases (if it even works)?
Also signal is funny cross-platform iirc so atexit abstracts some of that away including making sure its another thread that manages calling the registered functions
But you could swing either way for sure if you felt like using 
signal
, I just like the simplicity of 
atexit
the times I've needed to reach for it
v
iirc meltano sends a sigterm to the tap on a target failure. So you'd probably want both here
I think you can do it something like
Copy code
import signal
import atexit

def clean_up():
    print("cleaning up!")

signal.register(signal.SIGTERM, clean_up)
signal.register(signal.SIGINT, clean_up)
atexit.register(clean_up)
Theoretically would do both I think 🤷
Still a good idea, thank you @alexander_butler 😄
a
iirc meltano sends a sigterm to the tap on a target failure. So you'd probably want both here
Ah I see. I am imaging the experience for people using the tap/target in general and running 
kill
in their terminal in a frenzy to stop the accidental beast they've released on their prod system only to be met with a forced clean up LOL
Copy code
def catch_sig():
    exit(1)

def clean_up():
    print("cleaning up!")

signal.register(signal.SIGTERM, catch_sig)
signal.register(signal.SIGINT, catch_sig)
atexit.register(clean_up)
Open in Slack
PreviousNext
Powered by