Has anyone here built a successful strategy for us...
# ask-community
j
Has anyone here built a successful strategy for using lakefs with dagster, trying to find some inspiration how to think regarding it and the branching and branch control.
👍 1
r
We have a user that’s using LakeFS to introduce code and data versioning as part of their assets in Dagster. Then, on branch deployment, a new LakeFS reference is created, and is used in the branch deployment environment. By comparing the versioning of the assets from the branch deployment to that of production, the Dagster framework can then tell which assets should be materialized and tested in the LakeFS branch, before merging the new code to production.
j
Nice it is similar to what I am thinking as well except I haven't explored the dagster cloud capabilities for this. Do they include new branch on scheduled runs as well? I am thinking i would like to add a branch from run context, then on full completion of run merge it into main. Haven't found any clear place to jack in to the completion of jobs though.
I can init the run request but not sure how i get good post hook without defining a new class, maybe I am confused with dagster terminology still.