https://dagster.io/ logo
Join Slack
Powered by
hello, is there any reason why pythonic run config...
# ask-community
c
hello, is there any reason why pythonic run configuration classes cannot be dataclasses? https://docs.dagster.io/1.1.21/guides/dagster/pythonic-config#pythonic-run-configuration Specifically the error I get when trying to create a dataclass to represent the fields is:
Copy code
TypeError: "Config" is immutable and does not support item assignment
Specifically i've got a dataclass containing runtime configuration parameters for a particular asset. I'd like to wire these into the asset using the same schema of that dataclass. It would be nice to just reuse the dataclass for the config type hint, but this error is preventing that
The alternative right now is:
Copy code
@dataclass
class Config():
   some: int
   params: str

class DagsterConfig(Config):
   some: int
   params: str

# some logic that populates this object
config: Config = load_config()

@asset
def some_asset(config: DagsterConfig):
    return something

materialize([asset], run_config={"ops": {"asset": config.asdict()}})
basically i would like to avoid having two separate definitions for Config and DagsterConfig
If there are any workarounds that might support something like this, please let me know!
(unfortunately im constrained to dagster 1.1.21 for now)
s
@ben - mind chiming in on this one?
b
Hi Chaitya, could you elaborate a bit on the motivation for using 
dataclass
here? 
dagster.Config
wraps 
pydantic.BaseModel
which acts similarly to a base Python dataclass. You should be able to do e.g.
Copy code
class DagsterConfig(Config):
    some: int
    params: str

config: DagsterConfig = load_config()

@asset
def some_asset(config: DagsterConfig):
    return "foo"

materialize([asset], run_config=RunConfig(ops={"some_asset": config}))
Here, 
RunConfig
will marshall 
DagsterConfig
into the right dict format to pass into the asset
c
Unfortunately the logic in 
load_config
requires that the config datatype be defined as a dataclass directly
I wonder if theres a way for me to enable mutability on a dataclass such that it can be treated as a pydantic basemodel
I may be able to do something like the example here, where i nest the dataclass within the dagster config type https://docs.pydantic.dev/usage/dataclasses/#use-of-stdlib-dataclasses-with-basemodel
Let me give it a shot, will report back.
b
Yeah, that seems promising. I was able to get this snippet working for me:
Copy code
from dataclasses import dataclass

from dagster import Config
from pydantic import create_model
from pydantic.dataclasses import dataclass as pydantic_dataclass


@dataclass
class MyConfig:
    foo: str
    bar: int


MyDagsterConfig = create_model(
    "MyDagsterConfig", __base__=(pydantic_dataclass(MyConfig).__pydantic_model__, Config)
)

config = MyDagsterConfig(foo="foo", bar=2)
print(config.foo)
which converts base dataclass -> pydantic and then uses multiple inheritance w/ Config to create a new model class
Extrapolating, this works (but probably ways to tidy it up)
Copy code
from dataclasses import dataclass

from dagster import Config, RunConfig, asset, materialize
from pydantic import create_model
from pydantic.dataclasses import dataclass as pydantic_dataclass


@dataclass
class MyConfig:
    foo: str
    bar: int


MyDagsterConfig = create_model(
    "MyDagsterConfig", __base__=(pydantic_dataclass(MyConfig).__pydantic_model__, Config)
)


@asset
def some_asset(config: MyDagsterConfig):
    return config.foo


config = MyConfig(foo="foo", bar=2)

result = materialize(
    [some_asset], run_config=RunConfig(ops={"some_asset": config.__dict__})
)

print(result.asset_value("some_asset"))
c
I think this approach will work for my use-case. Thanks a ton Ben!
Revisiting this - Unfortunately, when I receive the dagster config as an input to an asset, mypy doesn't like it. I assume its because the type is dynamically generated. Any thoughts on what to do to make mypy happy in the above example @ben?
Copy code
MyDagsterConfig is not valid as a type  [valid-type]
The above workaround (using TYPE_CHECKING) worked for me... but its super hacky and I'd like to avoid it. So any alternative ideas would be super helpful!
Open in Slack
PreviousNext
Powered by