https://linen.dev logo
Join Slack
Powered by
So we've been trying to get our docker image to de...
# announcements
m
So we've been trying to get our docker image to deploy and run in Google App Engine (GAE). We've set up a Google SQL database using postgres and have followed the guide for production best practices. Initially we were getting an issue where SQLite was failing with 
initdb
, so we also switched the 
MELTANO_DATABASE_URI
also to our postgres database, but we keep getting this error:
Copy code
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/meltano/core/db.py", line 58, in init_hook
    function_map[engine.dialect.name](engine)
  File "/usr/local/lib/python3.8/site-packages/meltano/core/db.py", line 67, in init_sqlite_hook
    engine.execute("PRAGMA journal_mode=WAL")
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2235, in execute
    return connection.execute(statement, *multiparams, **params)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1003, in execute
    return self._execute_text(object_, multiparams, params)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1172, in _execute_text
    ret = self._execute_context(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
    util.raise_(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) disk I/O error
Is there a reason why it's still using 
sqlite
? And the meltano docs mention not needing to change the 
MELTANO_DATABASE_URI
when deploying to production, so why is it breaking to begin with?
j
O error
O error
O error
O error
d
This suggests to me that the database_uri setting is not getting picked up correctly, because it shouldn't be doing anything with SQLite. Do you see the correct 
database_uri
in 
meltano config meltano
?
m
Still not working/running, but the URI wasn't getting passed in at build time. Since the 
database_uri
is needed for build time, is there a good way to set it when dealing with external databases?
And when the scheduler runs, it's not actually running any dags. It starts the scheduler, but then does nothing else:
Copy code
2020-12-04 22:38:18 default[20201204t151928]  [2020-12-04 22:38:18,677] {__init__.py:50} INFO - Using executor LocalExecutor
2020-12-04 22:38:18 default[20201204t151928]  [2020-12-04 22:38:18,881] {scheduler_job.py:1350} INFO - Starting the scheduler
2020-12-04 22:38:18 default[20201204t151928]  [2020-12-04 22:38:18,883] {scheduler_job.py:1358} INFO - Running execute loop for -1 seconds
2020-12-04 22:38:18 default[20201204t151928]  [2020-12-04 22:38:18,884] {scheduler_job.py:1359} INFO - Processing each file at most -1 times
2020-12-04 22:38:18 default[20201204t151928]  [2020-12-04 22:38:18,885] {scheduler_job.py:1362} INFO - Searching for files in /projects/orchestrate/dags
2020-12-04 22:38:18 default[20201204t151928]  [2020-12-04 22:38:18,916] {scheduler_job.py:1364} INFO - There are 2 files in /projects/orchestrate/dags
2020-12-04 22:38:19 default[20201204t151928]  [2020-12-04 22:38:19,578] {scheduler_job.py:1421} INFO - Resetting orphaned tasks for active dag runs
2020-12-04 22:38:19 default[20201204t151928]  [2020-12-04 22:38:19,821] {dag_processing.py:562} INFO - Launched DagFileProcessorManager with pid: 175
2020-12-04 22:38:19 default[20201204t151928]  [2020-12-04 22:38:19,837] {settings.py:55} INFO - Configured default timezone <Timezone [UTC]>
d
@michael_cooper If you happen to have time, wanna jump on a call for a second?
j
d
.meltano/run/airflow/logs/dag_processor_manager/dag_processor_manager.log
10 Views
Open in Slack
PreviousNext
Powered by