https://dagster.io/ logo
#ask-community
Title
# ask-community
c

Chaitya

04/03/2023, 8:01 PM
hello, is there any reason why pythonic run configuration classes cannot be dataclasses? https://docs.dagster.io/1.1.21/guides/dagster/pythonic-config#pythonic-run-configuration Specifically the error I get when trying to create a dataclass to represent the fields is:
Copy code
TypeError: "Config" is immutable and does not support item assignment
Specifically i've got a dataclass containing runtime configuration parameters for a particular asset. I'd like to wire these into the asset using the same schema of that dataclass. It would be nice to just reuse the dataclass for the config type hint, but this error is preventing that
The alternative right now is:
Copy code
@dataclass
class Config():
   some: int
   params: str

class DagsterConfig(Config):
   some: int
   params: str

# some logic that populates this object
config: Config = load_config()

@asset
def some_asset(config: DagsterConfig):
    return something

materialize([asset], run_config={"ops": {"asset": config.asdict()}})
basically i would like to avoid having two separate definitions for Config and DagsterConfig
If there are any workarounds that might support something like this, please let me know!
(unfortunately im constrained to dagster 1.1.21 for now)
s

sandy

04/04/2023, 12:04 AM
@ben - mind chiming in on this one?
b

ben

04/04/2023, 12:08 AM
Hi Chaitya, could you elaborate a bit on the motivation for using
dataclass
here?
dagster.Config
wraps
pydantic.BaseModel
which acts similarly to a base Python dataclass. You should be able to do e.g.
Copy code
class DagsterConfig(Config):
    some: int
    params: str

config: DagsterConfig = load_config()

@asset
def some_asset(config: DagsterConfig):
    return "foo"

materialize([asset], run_config=RunConfig(ops={"some_asset": config}))
Here,
RunConfig
will marshall
DagsterConfig
into the right dict format to pass into the asset
c

Chaitya

04/04/2023, 12:16 AM
Unfortunately the logic in
load_config
requires that the config datatype be defined as a dataclass directly
I wonder if theres a way for me to enable mutability on a dataclass such that it can be treated as a pydantic basemodel
I may be able to do something like the example here, where i nest the dataclass within the dagster config type https://docs.pydantic.dev/usage/dataclasses/#use-of-stdlib-dataclasses-with-basemodel
Let me give it a shot, will report back.
b

ben

04/04/2023, 12:24 AM
Yeah, that seems promising. I was able to get this snippet working for me:
Copy code
from dataclasses import dataclass

from dagster import Config
from pydantic import create_model
from pydantic.dataclasses import dataclass as pydantic_dataclass


@dataclass
class MyConfig:
    foo: str
    bar: int


MyDagsterConfig = create_model(
    "MyDagsterConfig", __base__=(pydantic_dataclass(MyConfig).__pydantic_model__, Config)
)

config = MyDagsterConfig(foo="foo", bar=2)
print(config.foo)
which converts base dataclass -> pydantic and then uses multiple inheritance w/ Config to create a new model class
Extrapolating, this works (but probably ways to tidy it up)
Copy code
from dataclasses import dataclass

from dagster import Config, RunConfig, asset, materialize
from pydantic import create_model
from pydantic.dataclasses import dataclass as pydantic_dataclass


@dataclass
class MyConfig:
    foo: str
    bar: int


MyDagsterConfig = create_model(
    "MyDagsterConfig", __base__=(pydantic_dataclass(MyConfig).__pydantic_model__, Config)
)


@asset
def some_asset(config: MyDagsterConfig):
    return config.foo


config = MyConfig(foo="foo", bar=2)

result = materialize(
    [some_asset], run_config=RunConfig(ops={"some_asset": config.__dict__})
)

print(result.asset_value("some_asset"))
c

Chaitya

04/04/2023, 3:07 PM
I think this approach will work for my use-case. Thanks a ton Ben!
Revisiting this - Unfortunately, when I receive the dagster config as an input to an asset, mypy doesn't like it. I assume its because the type is dynamically generated. Any thoughts on what to do to make mypy happy in the above example @ben?
Copy code
MyDagsterConfig is not valid as a type  [valid-type]
The above workaround (using TYPE_CHECKING) worked for me... but its super hacky and I'd like to avoid it. So any alternative ideas would be super helpful!