The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Guys, I could not find it in the docs, but may be you can point me out to the possibility. I am looking for a way to store execution status on S3 without AWS RDS. Is there such possiblity?

when you say execution status, what do you mean exactly?

Probably <https://docs.dagster.io/docs/apidocs/execution#dagster.PipelineExecutionResult>

I would like to avoid using RDS for any state.

hm, we haven't built an S3-backed RunStorage or EventLogStorage, but in principle it should be possible

it might not be responsive enough for dagit to really feel good

you could use the local (sqlite) storages instead on a networked file system

you could also perhaps execute pipelines without shared storage and then serialize the result objects to s3

This is probably the best way at this moment.

Where does dagster store execution process? Before results are ready there is a status of job, right?

yes, there is an event stream created by execution that is stored in the EventLogStorage