So I tweaked the Advanced example and got it working for my POC: Kubernetes + Celery + S3!! For local development, I want to remove S3 from the equation. Is it possible to use _fs_io_manager_ with Kubernetes + Celery? I am getting the following after the first solid runs.

FileNotFoundError: [Errno 2] No such file or directory: 'my_dagster_run/19287725-103a-4344-8541-4b073cb20e91/refresh_views/refresh_complete'

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/core/errors.py", line 182, in user_code_error_boundary
    yield
,  File "/usr/local/lib/python3.7/site-packages/dagster/core/execution/plan/inputs.py", line 448, in _load_input_with_input_manager
    value = input_manager.load_input(context)
,  File "/usr/local/lib/python3.7/site-packages/dagster/core/storage/fs_io_manager.py", line 109, in load_input
    with open(filepath, self.read_mode) as read_obj:

Do you have multiple replicas of the worker? I wouldn't expect that to go well.

Yes. I have 2

How would that work then?

I am not stuck to using fs_io_manager ... I am trying to not use S3 or other service for local development. If there are other options that would be great.

I don't know of any io managers that ship with Dagster that would work well in a K8s setup 🙂

Is it possible to use _fs_io_manager_ with Kubernetes + Celery?

Most users do local development with a purely local deployment of Dagit, e.g. in process executor. This will work well without s3 or any external storage. Is there a reason for you to need k8s deployment during development?

(the only two as filesystem pickles and in-memory)

@johann Let me get with devops tomorrow ... really we are trying to come up with our local dev strategy. We know dont want to use S3 buckets for every dev.

Plus one to a purely local deployment of dagit - makes attaching your debugger to the solid as it’s executed by Dagit easy