Is there a way to configure where the schedule log...
# announcements
r
Is there a way to configure where the schedule logs are written? By default they seem to be at
$DAGSTER_HOME/schedules/logs/<schedule_id>/
I was hoping to be able to do something like this in the dagster.yaml file:
Copy code
scheduler:
   module: dagster_cron.cron_scheduler
   class: SystemCronScheduler
   config:
      base_dir: /var/lib/persistent_storage
I’ve set up the event_log_storage, which contains most of the same info, but it would be good to have the schedule logs as well.
i
Copy code
event_log_storage in dagster yaml
and you can redifine every backend...with your own logger
put into the solid, but if there is a better way I am curious too
r
Thanks Istvan - I’m currently using the event_log_storage to log events. The information in these event logs is duplicated in
$DAGSTER_HOME/schedules/logs/<schedule_id>/
to schedule.log (with some other info in
<timestamp>_<schedule_id>.result
. I was hoping that there was additional info logged there, specifically about how the cron schedule is behaving, rather than just the output logged from events. But, it seems that everything in
$DAGSTER_HOME/schedules/logs/
is just used for Inter Process Communication, but it would be good to have someone from the Dagster team confirm this.
👀 1
a
ya that directory is just for local IPC and also the only place we can write things down if we can’t connect to the database from the
schedule_storage
section of config. That DB holds our “tick” structure which is what we use to track when cron has fired and capture any errors related to kicking off the scheduled run.
This all changes a bit in our upcoming
0.10.0
release with our new scheduler which @daniel can answer any questions about
d
Yes! The good news is the new scheduler logs will have a lot more information about how the cron schedule is behaving - it's not currently planned to be broken out by schedule in the same way, but it should be very easy to grep for information about the specific schedule that you're looking for.
r
Great, thanks for the info - I’ll hold off until
0.10.0
until trying to incorporate this.