if I use a location that references the `latest` tag of an i dagster #deployment-ecs

if I use a location that references the `latest` t...

rectalogic

04/05/2023, 5:40 PM

if I use a location that references the

latest

tag of an image - what will trigger dagster to force a new version of the

TaskDefinition

so it pulls the new image?

daniel

04/05/2023, 5:42 PM

We generally recommend against using the latest tag for this reason - I think there is ECS-level caching at the tag level that dagster doesn't really have visibility into or the ability to override

daniel

04/05/2023, 5:43 PM

k8s has an ImagePullPolicy that you can set to Always, but i'm not aware of anything similar for ECS

daniel

04/05/2023, 5:43 PM

(at least, if you're using Fargate, which most Dagster ECS users are)

rectalogic

04/05/2023, 5:43 PM

yeah, with Fargate you need to force a new deployment for it to re-pull I believe

rectalogic

04/05/2023, 5:45 PM

so I was hoping to be able to configure a "static" code location in my

dagster.yaml

that referenced a

latest

image, and so on any git merge to master I would rebuild it and so dagster would always be running that latest code. Otherwise I guess I need to build specific tagged images and have the end users add/modify a new code location any time they merge to master?

Mike Atlas

04/05/2023, 5:45 PM

https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/pull-behavior.html

Mike Atlas

04/05/2023, 5:46 PM

@rectalogic check my replies in the prior thread just before yours. My team uses dagster with non "latest" tags on our user code image: during CI/CD we build a new image and then update our ECS task definition with the newly tagged version number of the image

daniel

04/05/2023, 5:46 PM

Ah thanks for the link Mike - "Fargate does not cache images" makes me think I said some incorrect things above

👍 1

daniel

04/05/2023, 5:48 PM

But yeah i think there are a lot of advantages to using specific tagged images - one of them is that if you make a breaking change there's no risk of it affecting queued runs that then pull the wrong image

🌈 1

☝️ 1

rectalogic

04/05/2023, 5:51 PM

hmm, but can I specify the TaskDefinition the dagster agent will use to run my image - isn't that the TaskDefinition I would need to modify the code image version in? The agent TaskDefinition references the

dagster/dagster-cloud-agent:1.2.4

image

Mike Atlas

04/05/2023, 5:54 PM

not sure about the agent base image (my team self-hosts) but here's what our user_code Dockerfile looks like https://gist.github.com/mikeatlas/e0a47400a75f288262617405287f0586

rectalogic

04/05/2023, 5:56 PM

I see, I'm using Dagster Cloud and it appears the agent dynamically creates a new service and

TaskDefinition

configured using the code location image

👍 1

rectalogic

04/05/2023, 7:09 PM

ok I think what I can do is on any merge to master when I build a new image, use the CLI

dagster-cloud workspace add-location

to add/modify a singleton

master

code location with the new uniquely tagged docker image

daniel

04/05/2023, 7:13 PM

that looks right to me - that is essentially what the Dagster Cloud github action does

9 Views

Open in Slack

Previous Next