https://dagster.io/ logo
#ask-community
Title
# ask-community
j

Jeremy Chua

03/03/2023, 5:16 PM
Hello! Testing Q: Is there a way to call
job.execute_in_process()
in an async context? I'm trying to write a test for a sensor that terminates running jobs after some period of time (based off of this github answer). To test this, I was thinking of having a job that runs forever, while having the sensor periodically run in the background using threads. But when I try to do this, I get the following sqlite error:
(sqlite3.ProgrammingError) SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140180967843648 and this is thread id 140180440418048
The code roughly looks like:
Copy code
_run_forever_job = {...}

def test_sensor():
    instance = DagsterInstance.ephemeral()
    context = build_sensor_context(instance=instance)
    def periodically_run_sensor(context):
            while True:
                job_timeout_terminator_sensor(context)
                time.sleep(0.25)

    thread = threading.Thread(target=periodically_run_sensor, args=[context])
    thread.start()
    _run_forever_job.execute_in_process(instance=instance)
I also tried running
execute_in_process
in a thread, with the same result. Is there a better way to write a test for this?
d

daniel

03/03/2023, 5:31 PM
Hi Jeremy - what version of Dagster are you using?
j

Jeremy Chua

03/03/2023, 5:32 PM
I'm using
dagster = "1.1.14"
right now.
d

daniel

03/03/2023, 5:32 PM
Can you try upgrading? I think a fix for this may have gone out in the last couple of weeks
j

Jeremy Chua

03/03/2023, 5:34 PM
Will do and report back. Thanks for the tip.
Upgrading cleared the sqllite issue. However, it looks like the in memory launcher doesn't support termination yet, but I should be able to work around that for now. Thanks for the help!
d

daniel

03/03/2023, 5:59 PM
Ah, that's true, but as per the comment here that launcher is only intended for testing: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/launcher/sync_in_memory_run_launcher.py#L9
2 Views