19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
dagster-de
dagster-cube
gigs-freelance
dagster-wandb
dagster-machine-learning
dagster-bigquery
tools
dagster-dask
dagster-snowflake
dagster-releases
data-platform-design
events
dagster-serverless
dagster-noteable
introductions
dagster-airflow
dagster-cloud
random
dagster-dbt
faq-read-me-before-posting
dagster-airbyte
dagster-spatial
dagster-support
announcements
github-discussions
豆瓣酱帮
dagster-feedback
dagstereo
dagster-ecs
dagster-kubernetes
jobs
dagster-showcase
Title
g
geoHeil
05/14/2023, 1:50 PMWhy do I observe branch deploys with a TTL > than the one specified in the agent?
my runner is docker compose
Why is dagster starting 6 containers for this code location whilst sitting idle?
server_ttl:
full_deployments:
enabled: true
ttl_seconds: 60
branch_deployments:
ttl_seconds: 60
are set
but somehow it already has spun up 20 by now
j
jordan
05/15/2023, 2:14 PMThe names and labels of the containers will probably give a bit more of an answer. Can you share the agent logs, which code locations you expect to exist, and which containers actually exist?
g
geoHeil
05/15/2023, 6:49 PMI have 1 single code location deployed and would expect 1 container to be up and running at any given time and perhaps some of if a job is launched. However, in docker I see > 1 container being up and running
<<codelocation1>>-eb3e4b, <<codelocation1>>-05f02b are online currently (I killed all the others and increased the timeoutes again
Here you go with some logs https://gist.github.com/geoHeil/347553af596fa0807e8e4b4641563cec
j
jordan
05/15/2023, 8:42 PMA few things jump out from the logs (where I only see reference to 3 containers):
• were there multiple agents at one point? It looks like some of the code locations previously (or even currently) belonged to another agent and weren’t properly cleaned up
• when a location is updated, the agent performs a blue/green deployment where it first stands up a new container and then shuts down the old one. Did you recently reload a location? It’s possible both the new and old containers are still up while it’s transitioning.
• does the agent service any branch deployments?
g
geoHeil
05/16/2023, 7:03 AMlet me kill all containers and observe it for a couple of hours again
I indeed had 1 agent and one for branch deployments
But 2 dagster deployments (both branch deploys enabled)
And dagster cloud seems to have a restriction that there can only be one agent offering branch deploys (for now)
it seems to be better now.
@jordan I can observe it is better. But not fixed. Runs no longer cause stale containers to be kept around. However, a new deployment (i.e. branch) or push to main/prod will keep the OLD version(s) still flying around and not shut them down. How can these be removed as well? Is this a bug in dagster-cloud?
j
jordan
05/24/2023, 9:06 PMCan you share the agent logs, which containers actually exist, and which containers you expect to exist?
g
geoHeil
05/25/2023, 3:23 AMI will do this after a next deployment as I deleted the containers manually. Or would the old logs from the last couple of days be fine?
I pm-ed you the logs