Hello Team, I hope you are doing well. I am currently working on a monitoring tool and I am attempting to get all run IDs for a particular partition_key. I was thinking of either using the GraphQL api or somehow utilizing the dagster instance to get the runstorage. Would this be the best way to achieve this or is there a better method? Thanks in advance

hi @Jay! I wrote the answer up into a github discussion for better visibility here: https://github.com/dagster-io/dagster/discussions/14763. I focused on the python api, as that's somewhat simpler, but equivalent queries could be made via graphql with a bit of effort (you'd want to access the assetMaterializations on the AssetNode of the asset you're interested in)

Hey @owen, Thanks for the information, unfortunately I have tried a similar approach and our team does not have a dagster.yml set up, meaning that the

DagsterInstance.get()

will fail. Is there anyway to access the event records without the instance?

What error message are you getting? I wouldn't expect the call to fail even without a dagster.yaml file (you might get a warning), but even then I'd recommend just creating an empty

dagster.yaml

file in

~/.dagster/

The error that I am getting is the following

dagster._core.errors.DagsterInvariantViolationError: $DAGSTER_HOME "/opt/dagster/dagster_home" is not a directory or does not exist. Dagster requires this environment variable to be set to an existing directory in your filesystem

. Just for context, our team uses Postgres to store run history information, so would that mean that by adding the following directory and then adding an empty dagster.yaml file I would be able to access the event_storage?

re: using Postgres to store the run history info, I assume this means that wherever you've deployed dagster does have a

dagster.yaml

file, it's just that you don't have a copy of that locally? If so, you're correct that the empty dagster.yaml file wouldn't get you access to that remote storage.

for testing purposes it would be nice to have the script run locally, but the end goal would be to deploy it as a job that would run hourly on dagster

I see -- if you're running it in a job, then you can get access to the instance that the job is running against from the relevant

context

object, i.e.

@op
def watcher_op(context: OpExecutionContext):
    context.instance.get_event_records(...)

would that get the event_records for all assets?

it sounds like that'd require getting all of the materialization events for all assets across all time, is that right? it's doable, but would end up straining your database if you didn't have some cursor or something to keep track of which run_ids you'd already looked at

getting the metadata (start time, end time, etc)

might it make sense to do this in reverse order then? that'd be first getting all of the recent runs, then seeing which assets / partitions those runs materialized

that would also work. but how would I get that information given the issues I am facing with the dagster_home, is it simply just adding that directory where we deployed dagster?

the instance object should be available to you from the op context as shown above without needing to call DagsterInstance.get()

and that would allow to access the event_records for all assets then? and not just one?

depends on what you pass in as the EventRecordsFilter (this takes an optional asset_key argument to filter for events specific to a single asset key, but that does not need to be set)

I see, great. Will try it out. Thanks for all of your help. I appreciate your time.

event log records are the structured events that happen within a run, and run records are the representations of the runs themselves. so if you're curious in high-level stats about a run (did it succeed, when did it start, etc.), then you'd want RunRecord, and if you want to know specific details about events that happened (when was this asset / partition materialized), you'd want the EventLogRecord

I see thanks