Hey All, I'm trying to schedule pipelines and setup airflow for that. However, I keep getting below error while running the command "meltano invoke airflow scheduler". (meltanovenv) myhome@5401-IBANS:~/meltano-projects/myproj$ meltano invoke airflow scheduler ____________ _____________ __ |__( )_________ / /________ __ __ /| |_ /__ ___/_ /_ _ / _ \ | /| / / _ _ | / _ / _ __/ _ / / /_/ /_ |/ |/ / _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/ [2020-11-03 232453,033] {init.py:50} INFO - Using executor SequentialExecutor [2020-11-03 232453,047] {scheduler_job.py:1367} INFO - Starting the scheduler [2020-11-03 232453,048] {scheduler_job.py:1375} INFO - Running execute loop for -1 seconds [2020-11-03 232453,048] {scheduler_job.py:1376} INFO - Processing each file at most -1 times [2020-11-03 232453,049] {scheduler_job.py:1379} INFO - Searching for files in /home/myhome/meltano-projects/myproj/orchestrate/dags [2020-11-03 232453,050] {scheduler_job.py:1381} INFO - There are 1 files in /home/myhome/meltano-projects/myproj/orchestrate/dags [2020-11-03 232453,050] {scheduler_job.py:1438} INFO - Resetting orphaned tasks for active dag runs [2020-11-03 232453,067] {dag_processing.py:562} INFO - Launched DagFileProcessorManager with pid: 26007 [2020-11-03 232453,076] {settings.py:55} INFO - Configured default timezone <Timezone [UTC]> [2020-11-03 232453,089] {dag_processing.py:776} WARNING - Because we cannot use more than 1 thread (max_threads = 2) when using sqlite. So we set parallelism to 1. Traceback (most recent call last): File "/home/myhome/meltano-projects/meltanovenv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context cursor, statement, parameters, context File "/home/myhome/meltano-projects/meltanovenv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute cursor.execute(statement, parameters) sqlite3.OperationalError: locking protocol The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/myhome/meltano-projects/meltanovenv/lib/python3.6/site-packages/meltano/core/db.py", line 58, in init_hook function_map[engine.dialect.name](engine) File "/home/myhome/meltano-projects/meltanovenv/lib/python3.6/site-packages/meltano/core/db.py", line 67, in init_sqlite_hook engine.execute("PRAGMA journal_mode=WAL") File "/home/myhome/meltano-projects/meltanovenv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2237, in execute return connection.execute(statement, *multiparams, **params) File "/home/myhome/meltano-projects/meltanovenv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1003, in execute return self._execute_text(object_, multiparams, params) File "/home/myhome/meltano-projects/meltanovenv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1178, in _execute_text parameters, File "/home/myhome/meltano-projects/meltanovenv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context e, statement, parameters, cursor, context File "/home/myhome/meltano-projects/meltanovenv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception sqlalchemy_exception, with_traceback=exc_info[2], from_=e File "/home/myhome/meltano-projects/meltanovenv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_ raise exception File "/home/myhome/meltano-projects/meltanovenv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context cursor, statement, parameters, context File "/home/myhome/meltano-projects/meltanovenv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) locking protocol [SQL: PRAGMA journal_mode=WAL] (…

@ishan_b This suggests that multiple processes are trying to access the Meltano system database at

.meltano/meltano.db

at the same time, while SQLite only supports one concurrent access.

@douwe_maan No, I am only running one meltano command at a time. I am currently in POC phase but will definitely consider PosgtreSQL when we move to prod.

@ishan_b

meltano invoke airflow scheduler

ends up running

meltano schedule list --format=json

and

meltano elt <args>

, so it's possible that the nested

meltano

command executions run into a lock taken on the SQLite database by the top-level

meltano invoke airflow scheduler

below is the output I get .meltano/meltano.db

Can you verify that there is no

.meltano/meltano.db-shm

or

.meltano/meltano.db-wal

?

This is the content of .meltano folder: . .. analytics.json cache extractors files loaders logs meltano.db models orchestrators run However, I've observed that while the airflow scheduler is executing and running into repeated exceptions, below 2 files appear in .meltano directory: meltano.db-shm meltano.db-wal After I interrupt the airflow command, those 2 files get deleted.

@ishan_b OK, that's all as expected. What version of SQLite are you running?

@douwe_maan I haven't setup sqlite separately. The version of meltano is 1.54.0

@ishan_b What OS are you runinng?

@douwe_maan yes I forgot to mention earlier that I'm using wsl for POC since meltano doesn't run on windows. Guess that's the root cause. Thanks !

@ishan_b Glad we found the root cause! Would you like to create an issue for this so that we can consider showing a more informative (WSL-specific) error message?

@douwe_maan The issue seems to have gone away after upgrading to wsl2

@ishan_b Great, thanks for confirming!