Dear Dagster Community 🙏 I have a question regarding the asset runtime configuration concept (described here: https://docs.dagster.io/concepts/configuration/config-schema#specifying-runtime-configuration) I don't understand the intended best practice for how I can pass in configuration depending on different environment (e.g. local while using dagster dev, and "in production" within containers in a pod on K8s). I can successfully configure an asset using Launchpad in the UI, but I don't understand how to supply such configuration "from code" (or from a config.yaml) in a running deployment of Dagster. Let's say I have a very basic asset

class AssetConfig(Config):
    mlflow_endpoint: str

@asset
def my_model_asset(config: AssetConfig) -> None:
    # Use config.mlflow_endpoint in some way, e.g. to track an ML experiment

And I'd have a very simple configuration

# config.prod.yaml

ops:
  model:
    config:
      mlflow_endpoint: "http://..."

How would I supply this configuration to a production deployment of Dagster, while at the same time have a different configuration (e.g.

config.dev.yaml

) for a local defelopment environment (dagster dev), and maybe yet another configuration for a CI/CD pipeline? To me, none of the 3 options in https://docs.dagster.io/concepts/configuration/config-schema#specifying-runtime-configuration seem to apply to this situation, while I assume that the use case I've described is a common scenario? I might be completely thinking in the wrong direction, thank you for any best practices you could share for how to properly do this. All the best

One option could be to use an environment variable that gets injected into the container you're running in at build time to switch between configs:

deployment = os.getenv("DEPLOYMENT")
if deployment == "dev":
  with open("config.dev.yaml", "r") as conf_in
    config = conf_in.read()
elif deployment == "prod":
  with open("config.prod.yaml", "r") as conf_in
    config = conf_in.read()

job = define_asset_job(AssetSelection.assets(my_model_asset), config=config)

If you're using Dagster Cloud, they inject env vars that indicate the deployment the code is executing on which you could also use for deciding how to load deployment-specific configurations

Hi @Zach, thank you very much for taking the time to answer and help! dagster yay Ah, that makes sense! As I didn't find an example of this scenario in the official Dagster documentation, I didn't think of this / wasn't sure that config reading/validation logic is expected to be implemented by the user - I may have expected Dagster to provide "API support" for this scenario out of the box 🙂 You introduced

job

into the picture, is there a specific necessity for this in this scenario, or would my original example make sense combined with your suggestion as well? I.e. something like this:

class AssetConfig(Config):
    mlflow_endpoint: str
 
deployment = os.getenv("DEPLOYMENT")
if deployment == "dev":
  with open("config.dev.yaml", "r") as conf_in:
    config = conf_in.read()
elif deployment == "prod":
  with open("config.prod.yaml", "r") as conf_in:
    config = conf_in.read()
 
@asset
def my_model_asset(config: AssetConfig = config) -> None:
    # Use config.mlflow_endpoint

All the best

I just pretty much always use asset jobs because there's a bunch of stuff that the base

@asset

definition doesn't seem to provide, like binding static config to the asset. If you don't want to make it into a job then I'm not really sure how to bind static configuration to the asset. You might be able to do it like you suggest, but Dagster might complain that it's missing config as I don't know if it's able to resolve defaults for config parameters. If you don't plan on changing the config in the launchpad you could just do

class AssetConfig(Config):
    mlflow_endpoint: str
 
deployment = os.getenv("DEPLOYMENT")
if deployment == "dev":
  with open("config.dev.yaml", "r") as conf_in:
    config = conf_in.read()
elif deployment == "prod":
  with open("config.prod.yaml", "r") as conf_in:
    config = conf_in.read()

asset_config = AssetConfig(**config)

@asset
def my_model_asset() -> None:
    mlflow_endpoint = asset_config.mlflow_endpoint

For this kind of use-cases, I tend to use resources instead. Resources are declared in Definitions object, and you can specify there that you want to make it use an environment variable by default (it is still editable via the launchpad when required). Example :

class MyWebServiceAccess(ConfigurableResource):
    url: str = Field(description="access url to ...")


@asset
def my_asset(my_web_service: MyWebServiceAccess):
    request(my_web_service.url)


definitions = Definitions(
    assets = [my_asset],
    resources = { 
        "my_web_service": MyWebServiceAccess(url=EnvVar(MY_WEB_SERVICE_URL))
    }
)

Hi @Alexis Manin and @Zach Thank you for your help! I haven't yet read into the concept of Resources, thanks for pointing them out, sounds like a good fit for the scenario I'm trying to solve. I'll experiment also with this approach 🙂 Many thanks and all the best

Yeah resources can be a nice way to handle this. You could do the same thing with configs too

About that. I have not watched it yet, but a "Deep dive: Configuration and resources" video has been posted yesterday :