https://dagster.io/ logo
#ask-community
Title
# ask-community
m

Mike Logaciuk

08/18/2022, 8:06 PM
Is there an option to use own S3 instance (minIO/lakeFS) as IO manager while using self hosted docker multi container instance? I found this in docs:
Copy code
resources:
    io_manager:
        config:
            s3_bucket: my-cool-bucket
            s3_prefix: good/prefix-for-files-
However it doesn't describe how to use it with local S3.
D 1
s

sandy

08/18/2022, 8:45 PM
hi @Mike Logaciuk - dagster doesn't have an out-of-the-box minIO/lakeFS resource, but you could write your own I'm not sure exactly how minIO works. there are two approaches you could take depending on how it works: • if it exposes a client object that's compatible with the S3 boto client, you could make your own version of the
s3_resource
that returns that minIO client object: https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/s3/resources.py#L38 • otherwise, then you'd need create your own version of the
s3_io_manager
to get it to work with minIO: https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/s3/io_manager.py
m

Mike Logaciuk

08/18/2022, 8:51 PM
@sandy Thanks for reply, however the K8s runner has a description in docs, how to use it with MinIO:
Using Local S3 - Minio#
Skip this step if you're using AWS S3
First, set up minio locally:
Copy code
brew install minio/stable/minio # server
brew install minio/stable/mc    # client
mkdir $HOME/miniodata           # Prepare a directory for data
minio server $HOME/miniodata    # start a server with default user/pass and no TLS
mc --insecure alias set minio <http://localhost:9000> minioadmin minioadmin
# See it work
mc ls minio
date > date1.txt # create a sample file
mc cp date1.txt <minio://testbucket/date1.txt>

export AWS_ACCESS_KEY_ID="minioadmin"
export AWS_SECRET_ACCESS_KEY="minioadmin"
# See the aws cli work
aws --endpoint-url <http://localhost:9000> s3 mb <s3://test-bucket>
aws --endpoint-url <http://localhost:9000> s3 cp date1.txt <s3://test-bucket/>
Now, create your k8s AWS secrets:
Copy code
$ kubectl create secret generic dagster-aws-access-key-id --from-literal=AWS_ACCESS_KEY_ID=minioadmin
$ kubectl create secret generic dagster-aws-secret-access-key --from-literal=AWS_SECRET_ACCESS_KEY=minioadmin
Unfortunately there is nothing in docs for docker runner about how to use local Minio instead of AWS S3.
z

Zach

08/18/2022, 10:17 PM
I think you might be able to just configure the S3 resource that the
s3_io_manager
depends on to point to your local minio endpoint using the config schema it provides as it has endpoint_url and profile config options. you might be able to just write your own S3 resource, returning an S3 resource object as described here, then use that resource as the one that the
s3_io_manager
depends on
51 Views