Yeah it's a great question. The latter is definitely more what we have focused on so far (automatically building and pushing an image on each Github change). That has a number of nice advantages like plugging in cleanly into container-based systems like ECS and kubernetes. The demo video here demonstrates that with a Github action:
https://dagster.io/cloud it's from our cloud product but a similar flow is absolutely possible with the open-source dagster product as well.