Hello everybody I m new to dagster and I ve been running int dagster #announcements

Hello everybody! I'm new to dagster and I've been ...

David Smit

04/08/2021, 4:00 PM

Hello everybody! I'm new to dagster and I've been running into some issues trying to get pipelines to execute correctly in Dagit. The first solid will run and finish, but then nothing else happens until the process times out. I recreated the "complex_pipeline" from the tutorial (which I was able to get working in dagit on my personal machine) and the behavior was the same. If I run either pipeline from the command line, it executes without an issue.

David Smit

04/08/2021, 4:01 PM

Below is a screenshot of a section that is different between the dagit and command line executions. This is consistent between the (2) pipelines I tried

David Smit

04/08/2021, 4:01 PM

Any help is appreciated 🙂

David Smit

04/08/2021, 4:02 PM

This machine is a Windows 10 w/python 3.7.3

David Smit

04/08/2021, 4:16 PM

I was able to get it to work by creating a venv in the same directory and installing all of the packages I needed (used Pycharm to create the project, which creates the environment in a common folder). Is it best practice to keep the environment in the same directory, or is there a way to fix this through configuraiton?

prha

04/08/2021, 5:03 PM

Hi David! Trying to understand more about your setup to repro that error. Were you invoking both the

dagster

command and

dagit

in the same directory? What tool are you using to create your venv? Curious because we should be invoking the same python executable for this forked process (to manage compute log output), but for some reason it cannot find the installed dagster module…

David Smit

04/08/2021, 5:08 PM

I am in the same same directory for both commands. The env that didn't work was created when I created a new pycharm project:

David Smit

04/08/2021, 5:08 PM

David Smit

04/08/2021, 5:09 PM

The environment that did work I used venv to create a venv called venv in the project folder.

David Smit

04/08/2021, 5:09 PM

prha

04/08/2021, 5:27 PM

I think this is a bug in Python venv…. I found this open issue: https://bugs.python.org/issue38905

prha

04/08/2021, 5:30 PM

It looks like there might be a possible workaround using a

__PYVENV_LAUNCHER__

environment variable, but I need to do more research to see if this can be handled from within dagster or if it has to be set by each user.

prha

04/08/2021, 5:31 PM

Do you mind filing a ticket on Github with this issue? It’s not likely to be a quick fix and I’d like to make sure we can track it (and other Windows users running into this can find it).

David Smit

04/08/2021, 5:45 PM

Will do. Do you have a preferred format for submitting an issue?

prha

04/08/2021, 8:56 PM

I think there should be a pre-filled rubric when you create the issue.

🕺 1

David Smit

04/15/2021, 1:02 PM

https://github.com/dagster-io/dagster/issues/4051

2 Views

Open in Slack

Previous Next