https://dagster.io/ logo
Title
c

Chaitya

04/03/2023, 8:01 PM
hello, is there any reason why pythonic run configuration classes cannot be dataclasses? https://docs.dagster.io/1.1.21/guides/dagster/pythonic-config#pythonic-run-configuration Specifically the error I get when trying to create a dataclass to represent the fields is:
TypeError: "Config" is immutable and does not support item assignment
Specifically i've got a dataclass containing runtime configuration parameters for a particular asset. I'd like to wire these into the asset using the same schema of that dataclass. It would be nice to just reuse the dataclass for the config type hint, but this error is preventing that
The alternative right now is:
@dataclass
class Config():
   some: int
   params: str

class DagsterConfig(Config):
   some: int
   params: str

# some logic that populates this object
config: Config = load_config()

@asset
def some_asset(config: DagsterConfig):
    return something

materialize([asset], run_config={"ops": {"asset": config.asdict()}})
basically i would like to avoid having two separate definitions for Config and DagsterConfig
If there are any workarounds that might support something like this, please let me know!
(unfortunately im constrained to dagster 1.1.21 for now)
s

sandy

04/04/2023, 12:04 AM
@ben - mind chiming in on this one?
b

ben

04/04/2023, 12:08 AM
Hi Chaitya, could you elaborate a bit on the motivation for using
dataclass
here?
dagster.Config
wraps
pydantic.BaseModel
which acts similarly to a base Python dataclass. You should be able to do e.g.
class DagsterConfig(Config):
    some: int
    params: str

config: DagsterConfig = load_config()

@asset
def some_asset(config: DagsterConfig):
    return "foo"

materialize([asset], run_config=RunConfig(ops={"some_asset": config}))
Here,
RunConfig
will marshall
DagsterConfig
into the right dict format to pass into the asset
c

Chaitya

04/04/2023, 12:16 AM
Unfortunately the logic in
load_config
requires that the config datatype be defined as a dataclass directly
I wonder if theres a way for me to enable mutability on a dataclass such that it can be treated as a pydantic basemodel
I may be able to do something like the example here, where i nest the dataclass within the dagster config type https://docs.pydantic.dev/usage/dataclasses/#use-of-stdlib-dataclasses-with-basemodel
Let me give it a shot, will report back.
b

ben

04/04/2023, 12:24 AM
Yeah, that seems promising. I was able to get this snippet working for me:
from dataclasses import dataclass

from dagster import Config
from pydantic import create_model
from pydantic.dataclasses import dataclass as pydantic_dataclass


@dataclass
class MyConfig:
    foo: str
    bar: int


MyDagsterConfig = create_model(
    "MyDagsterConfig", __base__=(pydantic_dataclass(MyConfig).__pydantic_model__, Config)
)

config = MyDagsterConfig(foo="foo", bar=2)
print(config.foo)
which converts base dataclass -> pydantic and then uses multiple inheritance w/ Config to create a new model class
Extrapolating, this works (but probably ways to tidy it up)
from dataclasses import dataclass

from dagster import Config, RunConfig, asset, materialize
from pydantic import create_model
from pydantic.dataclasses import dataclass as pydantic_dataclass


@dataclass
class MyConfig:
    foo: str
    bar: int


MyDagsterConfig = create_model(
    "MyDagsterConfig", __base__=(pydantic_dataclass(MyConfig).__pydantic_model__, Config)
)


@asset
def some_asset(config: MyDagsterConfig):
    return config.foo


config = MyConfig(foo="foo", bar=2)

result = materialize(
    [some_asset], run_config=RunConfig(ops={"some_asset": config.__dict__})
)

print(result.asset_value("some_asset"))
c

Chaitya

04/04/2023, 3:07 PM
I think this approach will work for my use-case. Thanks a ton Ben!