https://dagster.io/ logo
Powered by
#announcements
Title
# announcements
c

Charm Malubag

11/16/2020, 12:49 PM
Hi! Is there a way to limit the number of runs fetched from the dagster DB when opening up the runs page in the dagit UI? Or if none, is there a way to safely reset the DB so the historical runs get removed from the DB? We have tried 
dagster run wipe
and 
dagster schedule wipe
, and though these were able to reset the runs and schedules from the schedules page, the DB still seems to contain historical data, and the runs page still takes long to load.
s

sashank

11/16/2020, 1:25 PM
We already do paginate fetching runs across all of Dagit, so there must be something else going on here. Would it be possible for you to run a few commands to profile what’s going on?
c

Charm Malubag

11/16/2020, 1:36 PM
Sure, can do. We are using an older version of dagster though (0.7.13) so there could be differences in behavior
s

sashank

11/16/2020, 1:41 PM
Ah yeah I am not sure what the behavior or the runs page was back then
Let me try putting together a script that works with 0.7.13
Once the page eventually loads, you don’t see any runs, correct? Or do you not even get past the loading spinner?
Here’s a small bash script that directly queries the dagster GraphQL API for the data needed for that runs page, it should help us narrow down what’s going on. It would be helpful to make sure you wipe the DB again before running it. It will create a 
dump-*
folder with the results, feel free to DM it to me and I can take a look.
Copy code
#!/bin/bash

# Dagster 0.7.13 runs query profile

NOW="dump-$(date +%s)"
mkdir $NOW

echo "Dumping RunsRootQueryRun with limit=1..."
time curl <http://localhost:3333/graphql> -X POST -H "Content-type: application/json" -d '{"query":"query RunsRootQuery($limit: Int, $cursor: String, $filter: PipelineRunsFilter!) {\n  pipelineRunsOrError(limit: $limit, cursor: $cursor, filter: $filter) {\n    ... on PipelineRuns {\n      results {\n        ...RunTableRunFragment\n        __typename\n      }\n      __typename\n    }\n    ... on InvalidPipelineRunsFilterError {\n      message\n      __typename\n    }\n    ... on PythonError {\n      message\n      __typename\n    }\n    __typename\n  }\n}\n\nfragment RunTableRunFragment on PipelineRun {\n  runId\n  status\n  stepKeysToExecute\n  canCancel\n  mode\n  rootRunId\n  parentRunId\n  pipelineSnapshotId\n  pipeline {\n    __typename\n    ... on PipelineReference {\n      name\n      __typename\n    }\n    ... on Pipeline {\n      pipelineSnapshotId\n      solids {\n        name\n        __typename\n      }\n      __typename\n    }\n  }\n  stats {\n    __typename\n    ... on PipelineRunStatsSnapshot {\n      stepsSucceeded\n      stepsFailed\n      startTime\n      endTime\n      expectations\n      materializations\n      __typename\n    }\n    ...PythonErrorFragment\n  }\n  tags {\n    key\n    value\n    __typename\n  }\n  __typename\n}\n\nfragment PythonErrorFragment on PythonError {\n  __typename\n  message\n  stack\n  cause {\n    message\n    stack\n    __typename\n  }\n}\n","variables":{"filter":{},"limit":1}}' > "$NOW/RunsRootQuery1.json"

echo "Dumping RunsRootQueryRun with limit=25..."
time curl <http://localhost:3333/graphql> -X POST -H "Content-type: application/json" -d '{"query":"query RunsRootQuery($limit: Int, $cursor: String, $filter: PipelineRunsFilter!) {\n  pipelineRunsOrError(limit: $limit, cursor: $cursor, filter: $filter) {\n    ... on PipelineRuns {\n      results {\n        ...RunTableRunFragment\n        __typename\n      }\n      __typename\n    }\n    ... on InvalidPipelineRunsFilterError {\n      message\n      __typename\n    }\n    ... on PythonError {\n      message\n      __typename\n    }\n    __typename\n  }\n}\n\nfragment RunTableRunFragment on PipelineRun {\n  runId\n  status\n  stepKeysToExecute\n  canCancel\n  mode\n  rootRunId\n  parentRunId\n  pipelineSnapshotId\n  pipeline {\n    __typename\n    ... on PipelineReference {\n      name\n      __typename\n    }\n    ... on Pipeline {\n      pipelineSnapshotId\n      solids {\n        name\n        __typename\n      }\n      __typename\n    }\n  }\n  stats {\n    __typename\n    ... on PipelineRunStatsSnapshot {\n      stepsSucceeded\n      stepsFailed\n      startTime\n      endTime\n      expectations\n      materializations\n      __typename\n    }\n    ...PythonErrorFragment\n  }\n  tags {\n    key\n    value\n    __typename\n  }\n  __typename\n}\n\nfragment PythonErrorFragment on PythonError {\n  __typename\n  message\n  stack\n  cause {\n    message\n    stack\n    __typename\n  }\n}\n","variables":{"filter":{},"limit":25}}' > "$NOW/RunsRootQuery25.json"

echo "Done."
(Should add that the data will contain info that is usually visible from that runs page such as pipeline names, but shouldn’t contain any other PII or run configuration)
c

Charm Malubag

11/16/2020, 4:15 PM
I don't get passed the loading spinner when I open the runs page
I'm running the script now and this will probably take a while as well. Will let you know once I have the output. Thanks!
Hi @sashank, the results are just blank after wiping:
Copy code
{
  "data": {
    "pipelineRunsOrError": {
      "results": [],
      "__typename": "PipelineRuns"
    }
  }
}
however, when one of the pipelines runs and then I run the script, it takes long to query (limit=1 takes about 11mins). Could this be a performance issue in our DB?
Just to clarify about the expected behavior -- should running wipe delete the records for historical runs in the DB? or does it just mark it as deleted?
2 Views
Powered by