Hi I m trying to understand how to manage the `dagster yaml`

Question

Hi I m trying to understand how to manage the `dagster yaml` configuration From the <https docs dagster io docs apidocs internals instance|instance documentation> I understand that it is read from the `$DAGSTER HOME` environment If the `dagster yaml` has the logic configuration per project while the `$DAGSTER HOME` env is per user what is the correct manner to use the `dagster yaml` Where should it be saved and how could it be run per project

alex · Answer

this doc is better <https docs dagster io docs deploying instance>

alex · Answer

`$DAGSTER HOME` points at a directory that can contain a `dagster yaml` to configure your instance which is akin to deployment or install

sephi · Answer

Thx for your quick response As a newbee i m probably missing something out In <https docs dagster io docs deploying instance|the link> it is says gt Dagster checks whether the environment variable `DAGSTER HOME` is set Isn t this environment variable set per user then it will always point to one directory with one `dagster yaml` file However each project that I use will want to run a different `dagster yaml` file how do I configure that

alex · Answer

gt However each project that I use will want to run a different `dagster yaml` file Out of curiosity why is this Can you tell me more about your set up If you want to change your storage per project you will have to set up multiple folders with `dagster yaml` and change your `$DAGSTER HOME` based on which project you are currently working on Our tools are currently set up to handle different pipelines repositories using a shared storage so I would consider using one instance for working on your projects at least locally

sephi · Answer

Since each project is from a different domain the logs would be saved in different databases We will consider your suggestion

alex · Answer

ya this is a workflow style we haven t optimized for yet I would encourage you to file an issue describing what you think an ideal workflow would be for multi instance work

sephi · Answer

Just submitted an issue <https github com dagster io dagster issues 2444>

sephi · Answer

I set my `DAGSTER HOME` environment to a folder on my dagster machine I set the dagster yaml configuration to be exactly <https docs dagster io docs apidocs internals dagster core storage event log SqliteEventLogStorage|as in the documentation> just updating the base dir I set the permissions of the folder to allow everyone to write to it When running the dagster the job completes without any errors but I don t see any sqlite file Do I need to generate a sqlite file or will be generated automatically I m I missing something

alex · Answer

hmm I wonder if this is just an issue with `base dir` do you see the sqlite file in your `DAGSTER HOME` directory instead