Hello all, New to dagster so need a little bit of your guidance for solving these kind of errors. Error message:

dagster._core.errors.DagsterInvalidConfigError: Error in config for job
    Error 1: Missing required config entry "io_manager" at path root:resources. Sample config for missing entry: {'io_manager': {'config': {'s3_bucket': '...'}}}

Job decorator looks like this:

@job(
    resource_defs={"io_manager": s3_pickle_io_manager, "s3": s3_resource, "params": make_values_resource()},
    executor_def=k8s_job_executor,
)

run config:

run_config={
    "resources": {
        "params": {
            "config": {
                "script_path": "spark_dagster_test.py",
                "cluster_name": "spark_pod_name",
                "logs_output": "spark_dagster_test.txt",
            }
        }
    }
}

The problem: I could not define s3 required values in dagit UI before launching run since dagit immediately throws an error from the above during workspace loading phrase. Combination of

make_values_resource()

and

io_manager/s3

in resource_defs seems does not work together since individually code works as expected. Does anybody knows where the problem is hidding and could give some advices to solve it?

Hi Domantas. If you provide a

config

argument to your job, Dagster assumes that it is the full config (so contains all necessary configuration for all resources). If you want to provide partial config for just a single resource, you can use the

configured

API instead. configured API: https://docs.dagster.io/concepts/configuration/configured

@job(
    resource_defs={
        "io_manager": s3_pickle_io_manager,
        "s3": s3_resource,
        "params": make_values_resource().configured(
            {
                "script_path": "spark_dagster_test.py",
                "cluster_name": "spark_pod_name",
                "logs_output": "spark_dagster_test.txt",
            }
        ),
    },
    # executor_def=k8s_job_executor,
)

It works, thank you very much! 🙂