https://dagster.io/ logo
#ask-community
Title
# ask-community
d

Domantas M

08/11/2022, 1:08 PM
Hello all, New to dagster so need a little bit of your guidance for solving these kind of errors. Error message:
Copy code
dagster._core.errors.DagsterInvalidConfigError: Error in config for job
    Error 1: Missing required config entry "io_manager" at path root:resources. Sample config for missing entry: {'io_manager': {'config': {'s3_bucket': '...'}}}
Job decorator looks like this:
Copy code
@job(
    resource_defs={"io_manager": s3_pickle_io_manager, "s3": s3_resource, "params": make_values_resource()},
    executor_def=k8s_job_executor,
)
run config:
Copy code
run_config={
    "resources": {
        "params": {
            "config": {
                "script_path": "spark_dagster_test.py",
                "cluster_name": "spark_pod_name",
                "logs_output": "spark_dagster_test.txt",
            }
        }
    }
}
The problem: I could not define s3 required values in dagit UI before launching run since dagit immediately throws an error from the above during workspace loading phrase. Combination of
make_values_resource()
and
io_manager/s3
in resource_defs seems does not work together since individually code works as expected. Does anybody knows where the problem is hidding and could give some advices to solve it?
c

claire

08/11/2022, 5:59 PM
Hi Domantas. If you provide a
config
argument to your job, Dagster assumes that it is the full config (so contains all necessary configuration for all resources). If you want to provide partial config for just a single resource, you can use the
configured
API instead. configured API: https://docs.dagster.io/concepts/configuration/configured
Copy code
@job(
    resource_defs={
        "io_manager": s3_pickle_io_manager,
        "s3": s3_resource,
        "params": make_values_resource().configured(
            {
                "script_path": "spark_dagster_test.py",
                "cluster_name": "spark_pod_name",
                "logs_output": "spark_dagster_test.txt",
            }
        ),
    },
    # executor_def=k8s_job_executor,
)
d

Domantas M

08/17/2022, 10:44 AM
It works, thank you very much! 🙂
2 Views