Hi. We are trialing Dagster Serverless, and notice...
# dagster-serverless
Hi. We are trialing Dagster Serverless, and noticed that sometimes after deploying to prod, workspace and assets show not the latest commit just pushed to prod, but workspace and assets used to be defined in an old commit. If I rerun the Gitlab (we're using Gitlab) pipeline to redeploy, it shows correct workspace and assets, but I was wondering if this is a known behaviour.
Hi Ricky - when the workspace is showing old data, did the gitlab pipeline complete successfully? One thing we saw when writing our github action for serverless was that sometime if two commits landed close after each other, there could be an issue where they completed out of order and the earlier one ended up going in last. For that reason, we set up the built-in github action so that it only executes once at a time. Kind of a shot in the dark, but any chance that could be what's happening in your gitlab pipeline?
Specifically this is what we have in our github action to minimize the likelihood of CI/CD updates happening out of order: https://github.com/dagster-io/dagster-cloud-serverless-quickstart/blob/main/.github/workflows/deploy.yml#L7-L10
Copy code
  # Cancel in-progress deploys to main branch
  group: ${{ github.ref }}
  cancel-in-progress: true
Hi Daniel. Thank you for your answer. Regarding your question, yes gitlab pipeline completed successfully when this happened. And I will try to apply the similar setting to the gitlab pipeline, and see if it fixes the problem.
Hi Daniel. I have checked the concurrency issue, but I doubt it's the source of the issue. Currently I am the only one working on the repo. And the Gitlab setting for concurrency is set to run only one prod pipeline for the repo at any time. To give you a bit more context, I have added
argument when I deploy to Dagster Serverless, so that I can see the commit that is pushed is the correct one showing in the UI. Even when the commit has is correct, sometimes the workspace is showing assets, jobs from a commit a while ago (more than a week ago). The command I run to deploy is
dagster-cloud serverless deploy --location-name "moo" --package-name moo --deployment prod --commit-hash $CI_COMMIT_SHA
Could you please advise me further on this?
Got it - is your deployment currently in the bad state? If so, do you have a link to page that's showing old content currently?
Happy to jump on a call quickly if that would be helpful