https://dagster.io/ logo
f

Félix Tremblay

02/16/2023, 3:16 PM
Hello 👋 It looks like the SQLite connection used by Dagster to log events is not thread-safe. The connection is created in one thread and is being used in another thread, causing the following error message :
Copy code
sqlalchemy.exc.ProgrammingError: (sqlite3.ProgrammingError) SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140024476800832 and this is thread id 140024381171456.
(Background on this error at: <https://sqlalche.me/e/14/f405>)
Here's how to reproduce the error:
Copy code
import concurrent.futures

from dagster import get_dagster_logger, graph, op


def my_function(x):
    logger = get_dagster_logger()
    <http://logger.info|logger.info>(f"Got {x=}")
    return x + 1


@op
def my_op():
    with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
        results = list(executor.map(my_function, range(3)))
    print(results)
    return results


@graph
def my_graph() -> None:
    my_op()


def main() -> None:
    my_graph.execute_in_process()


if __name__ == "__main__":
    main()
I was not able to fix the issue, unless I remove logging from
my_function
(but I need logging), or use a ProcessPoolExecutor (which in my case is significantly less performant than ThreadPoolExecutor)
I am on the latest version of dagster (1.1.18)
o

owen

02/17/2023, 7:04 PM
hi @Félix Tremblay! As of the version of dagster released yesterday (1.1.19), you shouldn't see this error anymore
❤️ 1
big dag eyes 1
f

Félix Tremblay

02/17/2023, 7:33 PM
Thank you @owen!
Hello @owen, after upgrading to 1.1.19, I still get the same error
a

alex

02/21/2023, 4:09 PM
Looks like the fix landed just after the branch cut so will actually be out in this weeks release on Thursday.
🎉 1
f

Félix Tremblay

02/24/2023, 8:25 PM
thank you! it works
2 Views