I recently upgrade to `v1 0 16` and getting `sqlite3 Program

Question

I recently upgrade to `v1 0 16` and getting `sqlite3 ProgrammingError SQLite objects created in a thread can only be used in that same thread ` errors however I m running it in k8s using a postgres backend for jobs schedules etc My jobs are using in memory output storage and I did notice this recent change <https github com dagster io dagster pull 10154> and the exceptions are happening in the pods generated for the kubernetes jobs The jobs seem to complete fine but I m getting at least one exception per run

Aaron Hoffer · Answer

Here s the full exception ``` Traceback most recent call last File usr local lib python3 8 site packages sqlalchemy pool base py line 753 in finalize fairy fairy reset pool File usr local lib python3 8 site packages sqlalchemy pool base py line 1004 in reset pool dialect do rollback self File usr local lib python3 8 site packages sqlalchemy engine default py line 683 in do rollback dbapi connection rollback sqlite3 ProgrammingError SQLite objects created in a thread can only be used in that same thread The object was created in thread id 139621353035584 and this is thread id 139620698298112 Traceback most recent call last File usr local lib python3 8 site packages sqlalchemy pool base py line 753 in finalize fairy fairy reset pool File usr local lib python3 8 site packages sqlalchemy pool base py line 1004 in reset pool dialect do rollback self File usr local lib python3 8 site packages sqlalchemy engine default py line 683 in do rollback dbapi connection rollback sqlite3 ProgrammingError SQLite objects created in a thread can only be used in that same thread The object was created in thread id 139621353035584 and this is thread id 139620698298112 ```

prha · Answer

I m surprised that the in memory storage is getting invoked at all especially since you ve configured storage to run on a postgres backend Can you confirm that your instance is configured with postgres Are you executing anything in process as part of your job Is there any more to the stack in the logs

Aaron Hoffer · Answer

```local artifact storage module dagster core storage root class LocalArtifactStorage config base dir opt dagster dagster home run storage module dagster postgres run storage class PostgresRunStorage event log storage module dagster postgres event log class PostgresEventLogStorage config compute logs module dagster core storage noop compute log manager class NoOpComputeLogManager config schedule storage module dagster postgres schedule storage class PostgresScheduleStorage config ```

Aaron Hoffer · Answer

gt Are you executing anything in process as Yes I m using the `in process executor`

Aaron Hoffer · Answer

I don t see anything in dagit just this logged to stderr ```Exception closing connection lt sqlite3 Connection object at 0x7f1ed40a2e40 gt Traceback most recent call last File usr local lib python3 8 site packages sqlalchemy pool base py line 753 in finalize fairy fairy reset pool File usr local lib python3 8 site packages sqlalchemy pool base py line 1004 in reset pool dialect do rollback self File usr local lib python3 8 site packages sqlalchemy engine default py line 683 in do rollback dbapi connection rollback sqlite3 ProgrammingError SQLite objects created in a thread can only be used in that same thread The object was created in thread id 139770470790976 and this is thread id 139769816073984 During handling of the above exception another exception occurred Traceback most recent call last File usr local lib python3 8 site packages sqlalchemy pool base py line 255 in close connection self dialect do terminate connection File usr local lib python3 8 site packages sqlalchemy engine default py line 689 in do terminate self do close dbapi connection File usr local lib python3 8 site packages sqlalchemy engine default py line 692 in do close dbapi connection close sqlite3 ProgrammingError SQLite objects created in a thread can only be used in that same thread The object was created in thread id 139770470790976 and this is thread id 139769816073984 ```

prha · Answer

Using the in process executor should be fine I m wondering if you re using the testing API on jobs or ops e g `my job execute in process `

prha · Answer

AFAICT the in memory storage implementations are only used in our unit tests I could see this impacting you if you were kicking off untracked runs on ephemeral storage by using our test APIs within a thread This is unrelated to the `mem io manager` which it sounds like you re using and does not use SQLite

Aaron Hoffer · Answer

Oh sorry not that I m aware of I do have `run monitoring` on only other random config I could think of

Issac Loo · Answer

< Nicolas May>

Alexander Osipov · Answer

Have same problem Suppose that the reason because using sensor and dagster build resources ``` sensor job= def some sensor with build resources some db pg database as resources ``` As I see calling `build resources` with no `instance` argument creates `DagsterInstance` using `DagsterInstance ephemeral ` method In this method `InMemoryRunStorage` and `InMemoryEventLogStorage` uses They use sqlite during initialization

Alexander Osipov · Answer

Yes it was the problem for my case Fixed with next changes ``` sensor job= def some sensor context with build resources some db pg database instance=context instance as resources ```

prha · Answer

The fix for this should be rolled out in this week s release <https github com dagster io dagster pull 10851>

Nicolas May · Answer

< Issac Loo>

prha · Answer

This is now out as part of `1 1 6` Thanks everyone