https://dagster.io/ logo
Title
j

Josh Lloyd

07/16/2021, 8:42 PM
getting the following when trying to execute a pipeline from dagit:
dagster.core.errors.DagsterInvalidConfigError: Error in config for pipeline collective_pipeline
Error 1: Received unexpected config entries "['s3_bucket', 's3_prefix']" at path root:resources:io_manager:config. Expected: "['base_dir']."
my pipeline’s ModeDefinition has a lin in the
resource_defs
that read
"io_manager": s3_pickle_io_manager,
. and I have the run config in the playground set as follows:
resources:
  io_manager:
    config:
      s3_bucket: <bucket>
      s3_prefix: <prefix>
I don’t understand where i’m messing up here ….
a

alex

07/16/2021, 9:20 PM
hmmmm * double check you have the right mode selected in the playground * make sure processes are up to date if youve been changing the python code what exactly is happening? when do you see this error?
j

Josh Lloyd

07/16/2021, 9:21 PM
first double check done second double check: I rebuilt my images and reployed in local docker-compose
this is like the first message after is says the process started
so message #2 or 3 (not looking at it currently)
a

alex

07/16/2021, 9:22 PM
but dagit thought the config was fine when you clicked launch?
j

Josh Lloyd

07/16/2021, 9:23 PM
yeah, that’s the funny thing is that the playground expected the
s3_bucket
and the
s3_prefix
lines, gave me an error if I put in
base_dir
a

alex

07/16/2021, 9:25 PM
So something unexpected is happening when the run gets handed off for proper execution - with likely a stale or wrong container / image being used. Are you using the DockerRunLauncher?
j

Josh Lloyd

07/16/2021, 9:25 PM
yes
a

alex

07/16/2021, 9:28 PM
Do you configure the default image for that in your instance yaml?
the image to execute in is sourced from that by default if set, otherwise you can set
DAGSTER_CURRENT_IMAGE
in your “user code” deployment to have that be sourced https://github.com/dagster-io/dagster/blob/0.12.2/examples/deploy_docker/docker-compose.yml#L16-L34 https://docs.dagster.io/deployment/guides/docker#launching-runs-in-containers
j

Josh Lloyd

07/16/2021, 10:36 PM
you nailed it on the head. I recently switched to specifying tags rather than defaulting to the latest image. Forgot to specify the tag on the
DAGSTER_CURRENT_IMAGE
env var