Hi We are trialing Dagster Serverless and noticed that somet dagster #dagster-serverless

Hi. We are trialing Dagster Serverless, and notice...

Ricky Kim

09/14/2022, 1:56 PM

Hi. We are trialing Dagster Serverless, and noticed that sometimes after deploying to prod, workspace and assets show not the latest commit just pushed to prod, but workspace and assets used to be defined in an old commit. If I rerun the Gitlab (we're using Gitlab) pipeline to redeploy, it shows correct workspace and assets, but I was wondering if this is a known behaviour.

daniel

09/14/2022, 2:43 PM

Hi Ricky - when the workspace is showing old data, did the gitlab pipeline complete successfully? One thing we saw when writing our github action for serverless was that sometime if two commits landed close after each other, there could be an issue where they completed out of order and the earlier one ended up going in last. For that reason, we set up the built-in github action so that it only executes once at a time. Kind of a shot in the dark, but any chance that could be what's happening in your gitlab pipeline?

daniel

09/14/2022, 7:17 PM

Specifically this is what we have in our github action to minimize the likelihood of CI/CD updates happening out of order: https://github.com/dagster-io/dagster-cloud-serverless-quickstart/blob/main/.github/workflows/deploy.yml#L7-L10

daniel

09/14/2022, 7:17 PM

Copy code

concurrency:
  # Cancel in-progress deploys to main branch
  group: ${{ github.ref }}
  cancel-in-progress: true

Ricky Kim

09/15/2022, 7:24 AM

Hi Daniel. Thank you for your answer. Regarding your question, yes gitlab pipeline completed successfully when this happened. And I will try to apply the similar setting to the gitlab pipeline, and see if it fixes the problem.

Ricky Kim

09/16/2022, 2:49 PM

Hi Daniel. I have checked the concurrency issue, but I doubt it's the source of the issue. Currently I am the only one working on the repo. And the Gitlab setting for concurrency is set to run only one prod pipeline for the repo at any time. To give you a bit more context, I have added

--commit-hash

argument when I deploy to Dagster Serverless, so that I can see the commit that is pushed is the correct one showing in the UI. Even when the commit has is correct, sometimes the workspace is showing assets, jobs from a commit a while ago (more than a week ago). The command I run to deploy is

dagster-cloud serverless deploy --location-name "moo" --package-name moo --deployment prod --commit-hash $CI_COMMIT_SHA

Could you please advise me further on this?

daniel

09/16/2022, 2:55 PM

Got it - is your deployment currently in the bad state? If so, do you have a link to page that's showing old content currently?

daniel

09/16/2022, 2:56 PM

Happy to jump on a call quickly if that would be helpful

Open in Slack