https://dagster.io/ logo
Title
d

David Clements

09/29/2022, 11:02 PM
Hey there - I am doing local development via Docker containers. With mounted directories. I can see my code changes in the container when I change code locally. I am assuming that when I reload a repo in the dagit UI, that it would reload all my code into the UI and I would see the dag view change. But that does not appear to be happening. Is there a way to get my changes to reload in the UI?
s

Sterling Paramore

09/29/2022, 11:13 PM
Strange… I am doing the same and haven’t had that problem
d

daniel

09/30/2022, 12:06 AM
Hey David - unfortunately I believe the only way to do this right now if the gRPC server is in a separate process is to restart the container yourself. This is a known rough edge that we’re hoping to make better, - the changes should appear automatically in dagit once you restart the container though
d

David Clements

09/30/2022, 3:26 PM
Which container do I restart? Dagit? Is there a simpler way to get a more rapid dev cycle for local dev?
d

daniel

09/30/2022, 3:26 PM
You'd restart the container with your code in it
And dagit will realize that the code has changed and automatically reload
d

David Clements

09/30/2022, 3:27 PM
@Sterling Paramore - what is your local setup? Multiple containers?
d

daniel

09/30/2022, 3:28 PM
I could imagine setting up a watchdog hook that automatically reboots the container if any of the code being mounted as volumes changes - but we don't currently have anything like that built-in
d

David Clements

09/30/2022, 3:33 PM
@daniel what does reload in dagit do?
d

daniel

09/30/2022, 3:35 PM
It calls out to the gRPC server with your code on it and refreshes the information about your jobs that appears in dagit - which isn't super helpful in this particular case since the process needs to restart in order to pick up the changes to your code.
Where we'd like to take this is to give dagit enough information to restart or redeploy the server instead, which would make it do what you want
d

David Clements

09/30/2022, 3:36 PM
Ah cool. Which process has to restart in the container? That might be lighter weight faster cycle.
d

daniel

09/30/2022, 3:36 PM
The "dagster api grpc" one - should be the main entrypoint
👍 1
s

Sterling Paramore

09/30/2022, 4:06 PM
I’m using a single container. I’ve had plenty of times where I’ve made some kind of syntax error and then when I go into dagit, it will pop up and tell me I have an error. When I correct it and hit “reload”, it goes away. I did run into an issue yesterday like this, and then when I went to run the job, it was using the older code prior to my fix. But I could swear it’s worked in the past. I did recently start running the dagster daemon as part of my container startup script so I was wondering if that might be affecting something.
d

daniel

09/30/2022, 4:15 PM
Yeah, this is fairly confusing, I should have clarified - if dagit is responsible for spinning up the server itself (i.e. you're not running an external grpc server), it actually does do the process restart for you - it's easier to do that since dagit is managing the server directly. If you're running your grpc server in a separate docker container like David is, dagit doesn't know enough to go find it and tell docker to kill the container and restart it for you
d

David Clements

09/30/2022, 4:26 PM
@daniel does the grpc service have an api? That I could call to tell it to reload code?
d

daniel

09/30/2022, 4:27 PM
it does have an API, but since its loading the code in-process, right now it needs to restart. We actually do have a call that tells it to shut down though, and you can have docker set up so that it will restart the container whenever it goes down
So that could work - if you have a Dagster Python graphql client you could have a script that does
client.shutdown_repository_location("your_location_name)
whenever you want to reload, that should be reasonably quick
d

David Clements

09/30/2022, 5:36 PM
Cool thanks — as a workaround I am running the dagster api grpc command manually and restarting it when I am ready to restart.
d

daniel

09/30/2022, 5:40 PM
Roger, thanks for bearing with us - we've also discussed the possibility of having the server load the actual code in the subprocess, which we could then cycle from dagit without needing to restart the whole server
👍 1
m

miyamoto

12/07/2022, 2:34 PM
hey @daniel is this behaviour still present in the current release
1.1.5
? it could be that there is something wrong with my environment or I am doing something wrong, because in the case of this example, by following the process below, I am not getting the expected result 1. provide volume access of local file to docker container
user_code
to reflect local changes 2. change line 8 of file repo.py from
1
2
(changes are reflected in the docker container as well) 3. restart
user_code
,
dagit
,
daemon
docker instances 4. re-run the
my_step_isolated_job
5. expecting job to fail, but it succeeds The only way to make the job fail due to changes at step 2 is to re-build the whole docker deployment (which is not ideal). Am I doing something wrong?
d

daniel

12/07/2022, 2:46 PM
If you are using the Docker run launcher like in the example, That sounds like an issue with the volume not actually being mounted - each run launches in its own container, so if the changes aren’t showing up with rebuilding, that implies to me that the code isn’t coming from the volume. To answer your question there have been no recent changes to this behavior
m

miyamoto

12/07/2022, 7:02 PM
@daniel thank you for the quick reply regarding the volume mount and permissions, I think it should be OK because the initial load works, and I can also confirm the local changes take effect inside the docker instance Just to be safe, I've made the changes directly into the docker instance and made sure they are persistent after restart of
user_code
,
dagit
,
daemon
Still, there is no proper reload of the new code, unless I rebuild
user_code
d

daniel

12/07/2022, 7:03 PM
If you have a Github repository that we can clone that reproduces the problem, we'd be happy to take a look
m

miyamoto

12/07/2022, 7:03 PM
thank you @daniel I will share it here
@daniel here is the git repo I've added in the README the steps to reproduce my issue
d

daniel

12/07/2022, 7:53 PM
and where/how are you mounting the code as a volume?
I see it in the docker-compose.yaml but not on the run launcher: https://docs.dagster.io/deployment/guides/docker#mounting-volumes
"If you are mounting your code as a volume in your user code container and using
DockerRunLauncher
to launch each run in a new container, you must specify your volume mounts in the
DockerRunLauncher
config as well."
m

miyamoto

12/07/2022, 8:03 PM
aha I'm suffering from RTFM problems here 😊 let me give it a try then Looks like the mounting was missing from these examples also
d

daniel

12/07/2022, 8:04 PM
no worries - I wish it worked that way, it's annoying that you have to enter it twice
I don't think the example is intended to specifically demonstrate loading the code from a volume
m

miyamoto

12/07/2022, 8:09 PM
@daniel it worked thank you so much
:condagster: 2