Andy Carter03/06/2023, 12:27 PM
(DH) as I am a little confused. Do I have this right?
- Without DH set, dagster outputs logs to a local tmp directory, which is cleaned up after dagit shuts down. No info regarding runs is persisted when dagit is shut down.
- Not setting DH is useful when you are still developing/breaking things and don't care about logging long term.
- DH should probably be a directory outside of your python project, because...
should not be under change control'; it's something you can customize on the fly for each deployment location, more like a
file in that sense.
- It's common to have DH set but NOT create a dagster.yaml as some kind of halfway between testing and deployment.
- DH is both
1) a place to keep persistent data between dagit sessions, AND
2) a place to find
, which can redirect dagster data & logs that would otherwise go to DH to other IO sinks (postgres/S3 etc).
chris03/06/2023, 9:51 PM
Vinnie03/07/2023, 12:13 PM
shouldn’t be in change control?
Here’s my current setup in full OSS, roughly (might be forgetting a thing or two):
image with dagit, the daemon,
is set within the container
• project/user code locations don’t have
set or a
file, as they just run the gRPC and all logs and storage are managed by the central dagster instance
for local development that mimics my entire prod/staging environment: overrides my
package, especially relevant for the
. Storage is a postgres spun up with docker-compose and persisted in my local machine
• for quicker iteration/whenever I don’t need to see how the whole system will run, I might just run
from a project folder, in this case I have my
set to a location in my disk as well, but only because I needed it for