https://dagster.io/ logo
#ask-community
Title
# ask-community
k

Kushagra Rajput

07/01/2022, 3:17 PM
Hey guys, I'm restarting the dagster user code container at the start of every pipeline
c

chris

07/01/2022, 8:20 PM
hey! checking in with the team on this.
k

Kushagra Rajput

07/03/2022, 9:41 AM
thanks
I'm kinda implementing this refresh logic in prod, so pls let me know if its a good idea or not xd
hey @chris any update on this ?
hey @johann can you pls advice me on it ?
j

johann

07/05/2022, 7:47 PM
What run launcher are you using
If the DefaultRunLauncher, restarting the grpc server would kill in progress runs
If a different run launcher (e.g. Docker, ECS, K8s), the runs should continue unaffected. It’s possible you’d hit timeouts on requests to the grpc server, e.g sensor ticks
k

Kushagra Rajput

07/06/2022, 3:48 PM
No, I'm using dockerunlauncher, I've multiple pipelines which needs to update some variables everytime they run
so Before running any pipeline, I'm restarting the grpc server by using python dagster client
Copy code
client.shutdown_repository_location( REPO_NAME
Since there are multiple pipelines whose schedules are independent, each of them will restart user code docker container on their run to fetch the updated var values before running
can this create any problem in long run due to this race condition ?
j

johann

07/06/2022, 4:07 PM
fetch the updated var values before running
I don’t really understand this- restarting the gRPC server will make it pick up changes to jobs etc., but your job is already launched right
k

Kushagra Rajput

07/06/2022, 4:08 PM
yeah, changes need not to be reflected at the very same run
but should be updated for the future run, would've liked a solution to update the variables before as well
but that doesnt seem to be possible, pls correct me If I'm wrong
j

johann

07/06/2022, 4:13 PM
env vars?
k

Kushagra Rajput

07/06/2022, 4:14 PM
some variables that I need to pass to pipeline , they are stored in a key value store
and they can updated by anyone at any time
So I was hoping to find a way to get the updated values, and pass them to pipeline before dagster run
j

johann

07/06/2022, 4:26 PM
Could you just reach out to the KVS from the run itself
k

Kushagra Rajput

07/06/2022, 4:28 PM
not an exact solution as even cron dagster schedule are stored in kvs as well
need to update them as well
j

johann

07/06/2022, 4:40 PM
hmm interesting. Well once the run is started, I think that would be the only way to get new var values for the run. Maybe you would combine that approach with periodically restarting the grpc server
It also seems like it may be better to just have a regular interval at which the grpc server restarts to pick up new values, on a separate schedule
killing it at the start of every run seems messy, e.g. you could have a bunch of runs start near the same time and restart it unnecessarily
k

Kushagra Rajput

07/06/2022, 4:43 PM
yeah, that seems like a safe solution. Just to be sure, Restarting it periodically wont cause any problems with the pipelines execution, right ? given I'm using dockerRunLauncher
j

johann

07/06/2022, 4:44 PM
It won’t interrupt running pipelines, no
it may stop a schedule or sensor tick, but they retry
👍 1