My config validation is passing but then as soon as the job begins it fails on the config. V. confused at the min.

Hmm. Are you filling out the

execution

block at all?

No I'm not. And don't think I need to? Has a default doesn't it?

Yea that's correct. If I had to bet, we're injecting legacy config somewhere and causing this bug, just not sure where that would be, so trying to narrow it down

Do you think if I renamed the pipeline I could escape the problem?

I'd be surprised if the pipeline name had anything to do with it. Are you using the pipeline decorator or the job decorator here?

So I've resolved it for the timebeing by reverting it from a job to a pipeline. I was trying to migrate it to being a job, it's the oldest script I have, everything else I've written is now a job and I could have sworn I'm following the same patterns as those which work.

Interesting. Any way you could post some of the code here?

engineering_mongo_warehouse:
    config:
      mongo_host:
        env: ENG_MONGO_HOST
      mongo_password:
        env: ENG_MONGO_PASSWORD
      mongo_user:
        env: ENG_MONGO_USER
  ml_mongo_warehouse:
    config:
      mongo_host:
        env: ML_MONGO_HOST
      mongo_password:
        env: ML_MONGO_PASSWORD
      mongo_user:
        env: ML_MONGO_USER
ops:
  authenticate_api:
    config:
      cronjob_password:
        env: CRONJOB_PASSWORD
      cronjob_user:
        env: CRONJOB_USER

With job def as:

@job(
     resource_defs={
         "engineering_mongo_warehouse": mongo_handler_resource,
         "ml_mongo_warehouse": mongo_handler_resource,
         "io_manager": fs_io_manager, 
         "s3": s3_resource,
    },
     config=config_from_files([
                 # Config from file sorted but still not very secure because it shows the values via the UI.
                 # Should just load the config within the MongoStore initialization ??
                 file_relative_path(
                     __file__, "../datastores/config/connection_details.yaml"
             )
         ]
     )
)

Hmm I can't find anywhere in the codebase we could be injecting

multiprocess

... and just to confirm,

connection_details.yaml

doesn't set

execution: multiprocess

?

Launchpad, no subselection

Gotcha. For the moment, I'm going to file a bug report, not really sure what could be causing this. https://github.com/dagster-io/dagster/issues/7651

Sorry I'll provide more info on Tuesday. happy to hop on a call