06/06/2020, 6:56 AM
Guys, I could not find it in the docs, but may be you can point me out to the possibility. I am looking for a way to store execution status on S3 without AWS RDS. Is there such possiblity?
06/06/2020, 8:06 AM
when you say execution status, what do you mean exactly?
06/06/2020, 8:07 AM
I would like to avoid using RDS for any state.
06/06/2020, 8:56 AM
hm, we haven't built an S3-backed RunStorage or EventLogStorage, but in principle it should be possible
it might not be responsive enough for dagit to really feel good
you could use the local (sqlite) storages instead on a networked file system
you could also perhaps execute pipelines without shared storage and then serialize the result objects to s3
06/06/2020, 8:59 AM
This is probably the best way at this moment.
Where does dagster store execution process? Before results are ready there is a status of job, right?
Is it only in memory or also persisted?
06/06/2020, 5:48 PM
yes, there is an event stream created by execution that is stored in the EventLogStorage