Why do I observe branch deploys with a TTL > th...
# dagster-plus
Why do I observe branch deploys with a TTL > than the one specified in the agent?
my runner is docker compose
Why is dagster starting 6 containers for this code location whilst sitting idle?
server_ttl: full_deployments: enabled: true ttl_seconds: 60 branch_deployments: ttl_seconds: 60
are set
but somehow it already has spun up 20 by now
The names and labels of the containers will probably give a bit more of an answer. Can you share the agent logs, which code locations you expect to exist, and which containers actually exist?
I have 1 single code location deployed and would expect 1 container to be up and running at any given time and perhaps some of if a job is launched. However, in docker I see > 1 container being up and running
<<codelocation1>>-eb3e4b, <<codelocation1>>-05f02b are online currently (I killed all the others and increased the timeoutes again
A few things jump out from the logs (where I only see reference to 3 containers): • were there multiple agents at one point? It looks like some of the code locations previously (or even currently) belonged to another agent and weren’t properly cleaned up • when a location is updated, the agent performs a blue/green deployment where it first stands up a new container and then shuts down the old one. Did you recently reload a location? It’s possible both the new and old containers are still up while it’s transitioning. • does the agent service any branch deployments?
let me kill all containers and observe it for a couple of hours again
I indeed had 1 agent and one for branch deployments
But 2 dagster deployments (both branch deploys enabled)
And dagster cloud seems to have a restriction that there can only be one agent offering branch deploys (for now)
it seems to be better now.
@jordan I can observe it is better. But not fixed. Runs no longer cause stale containers to be kept around. However, a new deployment (i.e. branch) or push to main/prod will keep the OLD version(s) still flying around and not shut them down. How can these be removed as well? Is this a bug in dagster-cloud?
Can you share the agent logs, which containers actually exist, and which containers you expect to exist?
I will do this after a next deployment as I deleted the containers manually. Or would the old logs from the last couple of days be fine?
I pm-ed you the logs