Kobroli05/31/2022, 11:52 AM
(such that run concurrencies can be defined, but no storages are persisted to disk)? Thanks a lot in advance!
Daniel Mosesson05/31/2022, 12:32 PM
which you can set to put things in a custom path
Kobroli05/31/2022, 3:23 PM
directory disappear as well?
Daniel Mosesson05/31/2022, 3:27 PM
Kobroli05/31/2022, 3:28 PM
. Desirable would be something like, i.e., a mix of environment variable and path. Is sth like that possible?
run_storage: module: dagster.core.storage.runs class: SqliteRunStorage config: base_dir: env: LOCALAPPDATA/dagster/history
Daniel Mosesson05/31/2022, 3:30 PM
Kobroli05/31/2022, 3:30 PM
What other folders are you talking about?
-- all of them are created in the same directory where my
sits and that
owen05/31/2022, 6:13 PM
Kobroli06/01/2022, 7:29 AM
is set to one directory which every user gets to see, hence the
environment variable in
which is user-specific)
unset, but that would block me from using the
which is required in my case. It'd be cool if dagster offered a bit more flexibility in this matter (e.g. one central
that all storages use, or the option to choose an ephemeral instance, or an option to configure the daemon and the storages separately), but I totally understand that that might not be high prio 🙂
This sounds interesting -- do you mean mimicking it via
you can probably circumvent this by mimicking the ephemeral dagster instance
or programmatically? For the latter, it'd be helpful if you could point me to a starting point 🙂
owen06/01/2022, 4:45 PM
) wouldn't quite work, as some of the daemon operations depend on having access to the same run storage as dagit has (and if this run storage was in memory, then there would be no way to accomplish this).
thing, I was actually thinking that each user might have their own individual
(roughly the same path as whatever
was going to be). This would require N copies of that
file (or at least some sort of symlink setup), but depending on how much control you have over those things, that might work out ok.