The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

heya folks, I’ve got a quick ask around reading/writing to GCS bucket.
Namely what’s the prefered way moving forward, I See that
there is a file manager <https://docs.dagster.io/_modules/dagster_gcp/gcs/file_manager#GCSFileHandle>
for gcs, can that be used as an “io manager”

the file manager is used as a resource within the actual op / asset to manage file I/O - it’s kind of an old pattern

we have a `gcs_pickle_io_manager` that should work as long as your outputs are pickle-able

otherwise yea you can make a custom filesystem IO manager

<https://dagster.slack.com/archives/C01U954MEER/p1683570683201389?thread_ts=1683539184.530739&amp;cid=C01U954MEER> just noticing this thread as well thanks chris