https://linen.dev logo
Join Slack
Powered by
Hello there, I am trying to create a new tap, but...
# singer-tap-development
m
Hello there, I am trying to create a new tap, but I am having trouble saving the state in the database for each chunk of data, I use the 
self.finalize_state_progress_markers
method in my overwritten 
parse_reponse
method after the yield data, but it doesn't work, any idea?
e
Hi @michael_lan! You usually don't need to call 
finalize_state_progress_markers
yourself as it is called for you in a private class method. When you say
I am having trouble saving the state in the database for each chunk of data
how are you running the tap?
m
The problem is that if the extraction fails it does not save the status, it only saves when it completes successfully, I need it to save every certain chunk of data in case it fails to keep track of the last date it was extracted and start from the same point.
e
which target are you using? Every time the target writes a batch of data, it should output a state object that Meltano picks up
m
I run with command 
meltano elt tap-x target-jsonl --job_id=my_job_id
where tap-x is the tap i am creating
When fail the status in meltano.db is 
{"error": "ELT could not be completed: Extractor failed\nFor more detailed log messages, check the generated log file '/home/michaellan/Development/work/folder/data/collectors/meltano/meltano-projects/my-meltano-project/.meltano/logs/elt/job5/de6bcd03-4fc4-47b4-9133-c1c57bc44176/elt.log' or re-run the command using the '--log-level=debug' CLI flag."}
I need that when the process fails due to timeout or other problem the state is saved, for example 
{"singer_state": {"bookmarks": {"invoices": {"replication_key": "issuedAt", "replication_key_value": "2015-07-26 19:51:11"}}}}
.
Any idea?
e
Perhaps you can change the STATE_MSG_FREQUENCY of your stream?
m
Copy code
STATE_MSG_FREQUENCY = 20
It is already configured, when I run the tap without meltano the output file if does create the state every 20 records
It's working for me, the problem was that the error was caused by the lack of internet connection (this was done on purpose). When the error is caused by the requests the state is saved correctly.
e
by the lack of internet connection
interesting! I'm curious why that would cause problems for Meltano trying to save state. Is the database in the cloud?
m
No, I'm working locally
Open in Slack
PreviousNext
Powered by