Do `run_keys` get persisted when running dagster '...
# ask-community
get persisted when running dagster 'in production'? At the moment I'm experimenting with a sensor locally, executing the script with
dagster dev -f
. When I run the sensor first time against some files, it will run a job, subsequent invocations of that sensor skip the files due to the key already existing. If I kill and restart the dagster process, the run keys have disappeared, then the same files get parsed again.
dagster bot answered by content 1
yes the run_keys get persisted when not running via the default
dagster dev
this is because
dagster dev
defaults to temp dir as the storage. To designate a more permanent home for your runs or logs, you can set the
environment variable to a folder on your filesystem. Dagster will then use the specified folder for storage on all subsequent runs of
dagster dev
. more details here:
Great, thanks for the info