https://dagster.io/ logo
Powered by
#dagster-support
Title
# dagster-support
s

Stanley Yang

08/02/2022, 9:08 PM
Hi, my local dagit is hanging and I am not sure how to start debugging it. This is on dagster and dagit 
0.15.2
I see that the dagster code server starts and stops itself after a few minutes. I need to send a sigterm (
^\
) to kill it Terminal Output:
Copy code
▶ dagit --workspace local-workspace.yml
Using temporary directory /home/ystanley/src/cdf-pipeline/tmp76vm_hsr for storage. This will be removed when dagit exits.
To persist information across sessions, set the environment variable DAGSTER_HOME to a directory to use.

0it [00:00, ?it/s]
0it [00:00, ?it/s]
INFO: Pandarallel will run on 7 workers.
INFO: Pandarallel will use Memory file system to transfer data between the main process and workers.
INFO:dagster.code_server:Started Dagster code server for file /home/ystanley/src/cdf-pipeline/cdf_pipeline/repo.py in process 3932

INFO:dagster.code_server:Shutting down Dagster code server for file /home/ystanley/src/cdf-pipeline/cdf_pipeline/repo.py in process 3932
^C^C^C^C



^C^C^C
^\[1]    3916 quit       dagit --workspace local-workspace.yml
🤖 1
Bump, since this hurting our local workflow. We have determined it is something to do with how dagit is loading the workspace file. Probably some behavior change in 0.15. We are currently on 0.15.2 and did not see this issue on 0.14.3. The core issue is that it seems dagit's initial startup using a workspace file is choking when there are multiple venvs. In our current setup, we have a monorepo where we have multiple dagster repos and respective venvs. We have a workspace file that loads the "extra" venvs using 
executable_path
Example workspace.yml
Copy code
# This workspace file is for local development.
load_from:
  - python_file:
      relative_path: my_first_pipeline/repo.py
  - python_file:
      relative_path: my_second_pipeline/repo.py
      location_name: "second_repo"
      executable_path: my_second_pipeline/.venv/bin/python
If I remove 
second_repo
, dagit starts fine. The interesting thing is that 
second_repo
can be added back and loaded in after 
dagit
is running already. Our current workaround is: 1. Comment out the extra repos in the workspace.yml 2. Start dagit 3. Uncomment out the extra repos 4. Go to dagit's Workspace > Reload all The alternative workaround is just using --python-file instead of --workspace
🤖 1
d

daniel

08/03/2022, 2:32 AM
hey stanley - do you possibly have grpcio 1.48.0 installed?
they had to yank that release because it was causing mysterious deadlocks/hangs
s

Stanley Yang

08/03/2022, 1:59 PM
@daniel yes I do! I'll try deleting our lockfile and regenerating it
@daniel the regenerated piplock is using grpcio==1.47 and now smoothly starting up again 👍 Thanks!
8 Views
Powered by