Félix Tremblay
02/16/2023, 3:16 PMsqlalchemy.exc.ProgrammingError: (sqlite3.ProgrammingError) SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140024476800832 and this is thread id 140024381171456.
(Background on this error at: <https://sqlalche.me/e/14/f405>)
Here's how to reproduce the error:
import concurrent.futures
from dagster import get_dagster_logger, graph, op
def my_function(x):
logger = get_dagster_logger()
<http://logger.info|logger.info>(f"Got {x=}")
return x + 1
@op
def my_op():
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
results = list(executor.map(my_function, range(3)))
print(results)
return results
@graph
def my_graph() -> None:
my_op()
def main() -> None:
my_graph.execute_in_process()
if __name__ == "__main__":
main()
my_function
(but I need logging), or use a ProcessPoolExecutor (which in my case is significantly less performant than ThreadPoolExecutor)owen
02/17/2023, 7:04 PMFélix Tremblay
02/17/2023, 7:33 PMalex
02/21/2023, 4:09 PMFélix Tremblay
02/24/2023, 8:25 PM