Hello, has anyone has experimented this issue? I'm...
# troubleshooting
b
Hello, has anyone has experimented this issue? I'm using tap-mysql and I have updated to meltano 1.80.1 I check that it occurs when I'm running many elt in parallel. If I run the elt alone, is working well. ```meltano | This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: Instance '<Job at 0x7f1f90269780>' has been deleted, or its row is otherwise not present. (Background on this error at: https://sqlalche.me/e/14/7s2a) [2021-09-13 120836,509] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] Traceback (most recent call last): [2021-09-13 120836,526] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] File "/usr/local/lib/python3.6/site-packages/meltano/core/logging/output_logger.py", line 135, in redirect_logging [2021-09-13 120836,544] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] yield [2021-09-13 120837,209] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] File "/usr/local/lib/python3.6/site-packages/meltano/cli/elt.py", line 227, in _redirect_output [2021-09-13 120837,229] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] yield [2021-09-13 120837,250] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] File "/usr/local/lib/python3.6/site-packages/meltano/cli/elt.py", line 239, in _run_elt [2021-09-13 120837,270] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] await _run_extract_load(elt_context, output_logger) [2021-09-13 120837,288] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] File "/usr/local/lib/python3.6/site-packages/meltano/cli/elt.py", line 282, in _run_extract_load [2021-09-13 120837,308] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] loader_out=loader_out_writer, [2021-09-13 120837,329] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] File "/usr/local/lib/python3.6/site-packages/meltano/core/runner/singer.py", line 230, in run [2021-09-13 120837,346] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] loader_out=loader_out, [2021-09-13 120837,365] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] File "/usr/local/lib/python3.6/site-packages/meltano/core/runner/singer.py", line 152, in invoke [2021-09-13 120837,383] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] raise failed_future.exception() [2021-09-13 120837,400] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] File "/usr/local/lib/python3.6/site-packages/meltano/core/logging/utils.py", line 104, in capture_subprocess_output [2021-09-13 120837,768] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] if not await _write_line_writer(writer, line): [2021-09-13 120837,785] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] File "/usr/local/lib/python3.6/site-packages/meltano/core/logging/utils.py", line 81, in _write_line_writer [2021-09-13 120837,802] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] writer.writeline(line.decode()) [2021-09-13 120837,820] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin/singer/target.py", line 57, in writeline [2021-09-13 120837,842] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] job.payload["singer_state"] = new_state [2021-09-13 120837,859] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 481, in get [2021-09-13 120837,877] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] return self.impl.get(state, dict_) [2021-09-13 120837,894] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 926, in get [2021-09-13 120837,972] {{ecs.py:317}} INFO - [2021-09-13T120808.888000] value = self._fire_loader_callables(state, key, passive) [2021-09-13 120837,990] {{ecs.py:317}} INFO - [2021-09-13T120808.88…
d
@boggdan_barrientos Are you using an external (Postgres) system database, or the default SQLite one at
.meltano/meltano.db
?
b
I'm usign the default SQLite.
d
@boggdan_barrientos OK, is it an option to use an external Postgres database instead? SQLite can only handle one concurrent connection. See https://meltano.com/docs/settings.html#database-uri
b
Yes I can set an external postgres database. Thanks!
I'm setting the postgres database, exists any reference about the memory and disk for meltano use. Thanks.
d
No specific recommendations, but it can be very small, since it’s only used by Meltano to track the state of running jobs