Hello team! We have deployed our dagster instance on k8 and are facing this issue randomly while testing. Can someone help with this?

Mar 22 12:31:37.125 | dev-0 | INFO:dagster_telemetry_logger:{"action": "_logged_execute_pipeline_started", "client_time": "2022-03-22 07:01:37.124162", "elapsed_time": "None", "event_id": "2bd9050d-27b9-41b3-a9ac-67c8607d6de8", "instance_id": "45ac0356-b90e-4e3e-986a-f933b34d43c3", "pipeline_name_hash": "", "num_pipelines_in_repo": "", "repo_hash": "", "python_version": "3.6.15", "metadata": {}, "version": "0.2"}
Mar 22 12:31:37.127 | dev-0 | INFO:dagster_telemetry_logger:{"action": "update_repo_stats", "client_time": "2022-03-22 07:01:37.126882", "elapsed_time": "", "event_id": "7300aa49-b2a0-4eab-be3e-55d1c59da244", "instance_id": "45ac0356-b90e-4e3e-986a-f933b34d43c3", "pipeline_name_hash": "e74ca13b4c0a61dcf7746ff71c1238e2cd1b5026838c8b3c5e147595aa627025", "num_pipelines_in_repo": "1", "repo_hash": "eab79d95791949da51c2fbf1742d5dbf0e4621f546e46fe909f78efeb94c10f2", "python_version": "3.6.15", "metadata": {"source": "execute_pipeline"}, "version": "0.2"}
Mar 22 12:31:37.182 | dev-0 | 2022-03-22 07:01:37 - dagster - DEBUG - sleepy - 84d8d7ed-5573-4548-b280-5d9ac37463bb - 19 - PIPELINE_START - Started execution of pipeline "sleepy".
Mar 22 12:31:37.273 | dev-0 | 2022-03-22 07:01:37 - dagster - DEBUG - sleepy - 84d8d7ed-5573-4548-b280-5d9ac37463bb - 19 - ENGINE_EVENT - Executing steps using multiprocess executor: parent process (pid: 19)
Mar 22 12:31:37.284 | dev-0 | 2022-03-22 07:01:37 - dagster - ERROR - sleepy - 84d8d7ed-5573-4548-b280-5d9ac37463bb - 19 - PIPELINE_FAILURE - Execution of pipeline "sleepy" failed. An exception was thrown during execution.
Mar 22 12:31:37.285 | dev-0 | 
Mar 22 12:31:37.285 | dev-0 | ModuleNotFoundError: No module named 'runpy'
Mar 22 12:31:37.285 | dev-0 | 
Mar 22 12:31:37.285 | dev-0 | Stack Trace:
Mar 22 12:31:37.285 | dev-0 | File "/opt/bitnami/python/lib/python3.6/site-packages/dagster/core/execution/api.py", line 756, in pipeline_execution_iterator
Mar 22 12:31:37.285 | dev-0 | for event in pipeline_context.executor.execute(pipeline_context, execution_plan):
Mar 22 12:31:37.285 | dev-0 | File "/opt/bitnami/python/lib/python3.6/site-packages/dagster/core/executor/multiprocess.py", line 149, in execute
Mar 22 12:31:37.285 | dev-0 | term_events[step.key] = multiprocessing.Event()
Mar 22 12:31:37.285 | dev-0 | File "/opt/bitnami/python/lib/python3.6/multiprocessing/context.py", line 92, in Event
Mar 22 12:31:37.285 | dev-0 | return Event(ctx=self.get_context())
Mar 22 12:31:37.285 | dev-0 | File "/opt/bitnami/python/lib/python3.6/multiprocessing/synchronize.py", line 335, in __init__
Mar 22 12:31:37.285 | dev-0 | self._cond = ctx.Condition(ctx.Lock())
Mar 22 12:31:37.285 | dev-0 | File "/opt/bitnami/python/lib/python3.6/multiprocessing/context.py", line 67, in Lock
Mar 22 12:31:37.285 | dev-0 | return Lock(ctx=self.get_context())
Mar 22 12:31:37.285 | dev-0 | File "/opt/bitnami/python/lib/python3.6/multiprocessing/synchronize.py", line 162, in __init__
Mar 22 12:31:37.285 | dev-0 | SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
Mar 22 12:31:37.285 | dev-0 | File "/opt/bitnami/python/lib/python3.6/multiprocessing/synchronize.py", line 79, in __init__
Mar 22 12:31:37.285 | dev-0 | from .semaphore_tracker import register
Mar 22 12:31:37.285 | dev-0 | File "/opt/bitnami/python/lib/python3.6/multiprocessing/semaphore_tracker.py", line 21, in <module>
Mar 22 12:31:37.285 | dev-0 | from . import spawn
Mar 22 12:31:37.285 | dev-0 | File "/opt/bitnami/python/lib/python3.6/multiprocessing/spawn.py", line 13, in <module>
Mar 22 12:31:37.285 | dev-0 | import runpy

That’s strange. runpy should be installed with python. Have you tried using a a fresh virtualenv?

So this is happening when I am load testing my server. Once it goes down and restarts, we start getting such errors Here's the stack trace of another one of these random errors is:

Mar 22 16:51:09.633 | dev-0 | Traceback (most recent call last):
Mar 22 16:51:09.633 | dev-0 | File "<string>", line 1, in <module>
Mar 22 16:51:09.633 | dev-0 | File "/opt/bitnami/python/lib/python3.6/multiprocessing/spawn.py", line 105, in spawn_main
Mar 22 16:51:09.633 | dev-0 | exitcode = _main(fd)
Mar 22 16:51:09.633 | dev-0 | File "/opt/bitnami/python/lib/python3.6/multiprocessing/spawn.py", line 114, in _main
Mar 22 16:51:09.633 | dev-0 | prepare(preparation_data)
Mar 22 16:51:09.633 | dev-0 | File "/opt/bitnami/python/lib/python3.6/multiprocessing/spawn.py", line 225, in prepare
Mar 22 16:51:09.633 | dev-0 | _fixup_main_from_path(data['init_main_from_path'])
Mar 22 16:51:09.633 | dev-0 | File "/opt/bitnami/python/lib/python3.6/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
Mar 22 16:51:09.633 | dev-0 | run_name="__mp_main__")
Mar 22 16:51:09.633 | dev-0 | File "/opt/bitnami/python/lib/python3.6/runpy.py", line 263, in run_path
Mar 22 16:51:09.633 | dev-0 | pkg_name=pkg_name, script_name=fname)
Mar 22 16:51:09.633 | dev-0 | File "/opt/bitnami/python/lib/python3.6/runpy.py", line 96, in _run_module_code
Mar 22 16:51:09.633 | dev-0 | mod_name, mod_spec, pkg_name, script_name)
Mar 22 16:51:09.633 | dev-0 | File "/opt/bitnami/python/lib/python3.6/runpy.py", line 85, in _run_code
Mar 22 16:51:09.633 | dev-0 | exec(code, run_globals)
Mar 22 16:51:09.633 | dev-0 | File "/datax-dagster-poc/run.py", line 2, in <module>
Mar 22 16:51:09.633 | dev-0 | import uuid
Mar 22 16:51:09.633 | dev-0 | ModuleNotFoundError: No module named 'uuid'

I haven’t experienced this error myself. But it looks like the python environment your script is using does not have the required packages. Try running

pip freeze

and see if the packages are stilled as expected (cross reference this with you requirements.txt file) Also, are you using a virtual environment and is your instance using the packages in there? The line below makes me think that you’re running this off of your systems base python interpreter or something else that isn’t in an isolated virtual environment

/opt/bitnami/python/lib/python3.6/runpy.py