I did not see a way to configure a PV or PVC in the

dagster-user-deployments

. Should we manage it outside that Helm chart ?

You need to use IOManagers too keep that data outside of the Op (S3 / GCS / etc…) https://docs.dagster.io/concepts/io-management/io-managers Otherwise as soon as the k8s pod is gone also all the data on it is gone.

Thanks Andrea. I know that I have to use an IOManager, but what’s not clear for me is what is the default implementation if none is specified (memory or filesystem)?

Which value of the helm chart are you referring to exactly?

volumes:

is mentioned multiple times in the values.yaml file Anyway, I think that might give you a volume on the user-code-deployments, but not in the jobs you run. I believe IOmanagers are the best way to solve this issue, both in terms of scalability and complexity.

Hi Andrea, it looks we don’t understand each others. So let me retry to explain my thoughts.

Hi Nicolas 👋 I believe I understand your problem, but I think the solution you are suggesting is not optimal. Having a volume that moves around for every pipeline run is not recommended for a couple of reasons: • Volumes need to be attached to nodes where the pod is running. Every time a pod is scheduled on a different node, the volume must be attached to a new node. This takes time since this detach/attach can happen for every step of your pod. • What about volume sizing? is the amount of data you are generating even predictable in advance? • What about parallel steps? You can’t attach a volume to multiple pods (except for a couple of special volumes that tend to be like NFS file shares, so super-slow) For all the above reasons, I still think the best way of doing this is using IO managers. There is an IOManager for every major cloud provider. Instead of using a local volumes dagster will transparently upload/download your data from the object-storage of your choice. You won’t need any volume-magic and you won’t face any of the problems mentioned above.

Fair enough ! But I don’t understand what might be the usage in the

dagster-user-deployments

chart for volume mounts in that case.

I think it’s on an asset-level -> https://docs.dagster.io/concepts/io-management/io-managers#using-io-managers-with-software-defined-assets

True 😉

I don’t know of any public benchmark… but in my experience it’s pretty fast since it’s using the official libraries of the cloud provider. also, you don’t have the “will I have enough disk space for this file” problem… which is a huge plus for me 🙂 Which cloud provider are you using?

We are using GCP !

I use GCP as well 🙂 The single-bucket/multiple-bucket question is very subjective. If you have a lot of pipelines I would have multiple buckets, one per code-location. If you are running fewer pipelines maybe a single-bucket with different subfolders might be enough. But that’s going to be an issue if you start using dagster extensively

I just saw we can define the IOManager at repository level here:

@asset
def asset1():
    # create df ...
    return df

@asset
def asset2(asset1):
    return df[:5]

@repository
def repo():
    return with_resources(
        [asset1, asset2],
        resource_defs={
            "io_manager": gcs_pickle_io_manager.configured(
                {"gcs_bucket": "my-cool-bucket", "gcs_prefix": "my-cool-prefix"}
            ),
            "gcs": gcs_resource,
        },
    )
)

it’s this one right? https://docs.dagster.io/concepts/io-management/io-managers#applying-io-managers-to-assets

Perfect 😛 Thanks a lot I’m going to try that !

Happy to help !

Just a last question for you because your are using GCP. Did you configure the pods to use Workload Identity to access the GCP services ?

Yeah, I didn’t have any particular issue with it. Are you having troubles?

No no, just wanted to be sure I can set the service account with the annotation to the pods triggered by the K8s jobs

Currently i use a single serviceaccount (with workload identity) for all the pods. Then you just add it here https://github.com/dagster-io/dagster/blob/master/helm/dagster/values.yaml#L15 and it works 🙂 Multiple service accounts / workload identities should be manageable but maybe a bit tricky

Should it be only at code location level (here) ?

that should work if you want to have a separate service-account per code-location, but i haven’t tried it myself

Oki. Thanks again !

Sure thing, I will send you a quick DM!

Sure !