https://dagster.io/ logo
#announcements
Title
# announcements
b

Binh Pham

07/06/2020, 11:02 PM
Copy code
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/binhpham/.local/share/virtualenvs/limebi-flow-DASaHltL/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1278, in _execute_context
    cursor, statement, parameters, context
  File "/home/binhpham/.local/share/virtualenvs/limebi-flow-DASaHltL/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: disk I/O error

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/binhpham/.pyenv/versions/3.7.3/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/home/binhpham/.local/share/virtualenvs/limebi-flow-DASaHltL/lib/python3.7/site-packages/watchdog/observers/api.py", line 203, in run
    self.dispatch_events(self.event_queue, self.timeout)
  File "/home/binhpham/.local/share/virtualenvs/limebi-flow-DASaHltL/lib/python3.7/site-packages/watchdog/observers/api.py", line 376, in dispatch_events
    handler.dispatch(event)
  File "/home/binhpham/.local/share/virtualenvs/limebi-flow-DASaHltL/lib/python3.7/site-packages/watchdog/events.py", line 452, in dispatch
    super(PatternMatchingEventHandler, self).dispatch(event)
  File "/home/binhpham/.local/share/virtualenvs/limebi-flow-DASaHltL/lib/python3.7/site-packages/watchdog/events.py", line 336, in dispatch
    }[event.event_type](event)
  File "/home/binhpham/.local/share/virtualenvs/limebi-flow-DASaHltL/lib/python3.7/site-packages/dagster/core/storage/event_log/sqlite/sqlite_event_log.py", line 195, in on_modified
    self._process_log()
  File "/home/binhpham/.local/share/virtualenvs/limebi-flow-DASaHltL/lib/python3.7/site-packages/dagster/core/storage/event_log/sqlite/sqlite_event_log.py", line 185, in _process_log
    events = self._event_log_storage.get_logs_for_run(self._run_id, self._cursor)
  File "/home/binhpham/.local/share/virtualenvs/limebi-flow-DASaHltL/lib/python3.7/site-packages/dagster/core/storage/event_log/sql_event_log.py", line 130, in get_logs_for_run
    events_by_id = self.get_logs_for_run_by_log_id(run_id, cursor)
  File "/home/binhpham/.local/share/virtualenvs/limebi-flow-DASaHltL/lib/python3.7/site-packages/dagster/core/storage/event_log/sql_event_log.py", line 102, in get_logs_for_run_by_log_id
    results = conn.execute(query).fetchall()
  File "/home/binhpham/.local/share/virtualenvs/limebi-flow-DASaHltL/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1014, in execute
    return meth(self, multiparams, params)
  File "/home/binhpham/.local/share/virtualenvs/limebi-flow-DASaHltL/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/binhpham/.local/share/virtualenvs/limebi-flow-DASaHltL/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1133, in _execute_clauseelement
    distilled_params,
  File "/home/binhpham/.local/share/virtualenvs/limebi-flow-DASaHltL/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1318, in _execute_context
    e, statement, parameters, cursor, context
  File "/home/binhpham/.local/share/virtualenvs/limebi-flow-DASaHltL/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1512, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/home/binhpham/.local/share/virtualenvs/limebi-flow-DASaHltL/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/home/binhpham/.local/share/virtualenvs/limebi-flow-DASaHltL/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1278, in _execute_context
    cursor, statement, parameters, context
  File "/home/binhpham/.local/share/virtualenvs/limebi-flow-DASaHltL/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) disk I/O error
[SQL: SELECT event_logs.id, event_logs.event 
FROM event_logs 
WHERE event_logs.run_id = ? AND event_logs.id > ? ORDER BY event_logs.id ASC]
[parameters: ('c53b5d8e-f76a-4a46-94f1-554b94bdbd00', 252)]
(Background on this error at: <http://sqlalche.me/e/13/e3q8>)
m

max

07/07/2020, 12:42 PM
Hi Binh! If you're running heavy workloads, we recommend using Postgres instead of SQLite -- SQLite is really not built to handle concurrency
Is there anything special about the filesystem/underlying storage substrate you're running on
b

Binh Pham

07/07/2020, 5:18 PM
Hi Max, thanks I'll give that a try. No there is nothing special about my file system. But I'm suspecting that I'm running out of memory, could that be an issue?.