My pods are erroring with 429 Client Error Too Many Requests

Question

My pods are erroring with 429 Client Error Too Many Requests for url <https formenergy agent dagster cloud graphql> I m running about 70 step pods concurrently across 2 nodes and each one only takes a couple seconds I assume this means I m getting rate limited on the GraphQL API side Is there a way to get a higher quota

Shalabh Chaturvedi · Answer

Hi Mark we are seeing a large spike in the event logs written and these are getting rate limited Usually `429` responses should auto retry after a delay and we do not expect failures This helps spread out the heavy load Are you seeing failures in your logs or do the runs eventually succeed Is it also possible to reduce your usage of `context log` writes That is the only request that is hitting the rate limit

Mark Fickett · Answer

The pods are failing with error In the pod logs I do see retries but then they say too many retries In theory we could reduce log usage but not easily It would be a slow process to comb through our logs and try to find ones which we might not need And generally we want all logs to be captured since it s useful to see logs in Dagit One of the main things we want out of Dagster Cloud is the ability to scale horizontally significantly I m worried that this logging bottleneck means we can t scale to more parallel tasks in our job For example in the future we might want 2 10x the number of tasks running concurrently those are guesses but basically we want lots of headroom to grow and that s our main use case for Dagster

Mark Fickett · Answer

Related we ve hit some slowness in the Dagit UI due to large log volume before <https github com dagster io dagster issues 7821> We usually don t have a lot of logs per step something like 10 but we do have a large fan out of dynamic outputs 1 5k depending on stage of the pipeline

Shalabh Chaturvedi · Answer

gt In theory we could reduce log usage but not easily It would be a slow process to comb through our logs and try to find ones which we might not need And generally we want all logs to be captured since it s useful to see logs in Dagit Sometimes we see accidental heavy log usage eg dumping large data in a tight loop Good to know this is not the case and these logs are actually useful Note that scaling the log writes and scaling the database in general is an active project we are working on and a key priority for us for the next couple of months For this specific job I noticed that the spike of log writes stopped before 9am pacific time today Was this a one off job or scheduled job If we increase the rate limit are you able to retry this job

Mark Fickett · Answer

Thanks yes I can retry This is a development job I m not running on a schedule but it is full scale representative of runs that we ll be doing on a daily schedule I m working on making sure our full daily runs succeed in the cloud environment before we finally flip the switch from on prem to Dagster Cloud so these are just the kinds of errors I m hoping to flush out

daniel · Answer

Compute logs can also help here if you have a high volume of logs that you want to be displayed in dagit those have much higher limits at play

Mark Fickett · Answer

What s the difference Or how would I issue a compute log instead of an event log

Mark Fickett · Answer

Ah OK Yes I think it would be fine to move our application logs over to just be visible in raw compute logs I m not sure what the change to the python logs config would look like What I want is a all logs go to the root logger b root logger has our two custom handlers attached I guess I would replace the `managed python loggers` with a console handler ```python logs python log level INFO managed python loggers Capture logs from all Python loggers into Dagster s UI I assume this is what we d change to go to compute logs but we still want context log calls to funnel into the root logger root Specify additional handlers We want these to get all the log messages from context log and application logs the Python root logger dagster handler config handlers combined json class processing common log JsonFileHandler level INFO otel span class form observability OtelSpanEventHandler level INFO```

daniel · Answer

I don t think you need to do anything special to make things show up in compute logs as long as they re appear ing in stdout stderr In fact I explicitly anti recommend capturing the root logger that will send things into the structured part which is what s getting rate limited

Mark Fickett · Answer

So Dagster will still attach my JsonFileHandler and OtelSpanEventHandler to the root logger even if it s not managed

daniel · Answer

Er I don t think it will dagster handler config only applies to context log calls logs that are managed by dagster I can check if there s a way to change the logging config for all logs even logs that aren t going into the structured dagster logs

daniel · Answer

If dagster no longer needs to be in the mix for those handlers is that something where you could use the regular python logging configuration APIs

Mark Fickett · Answer

Kinda For our application logs it sounds like I could switch from using `context log` to some other `logging getLogger ` A big find replace but not hard And then I would just attach these handlers in our own setup maybe in a resource There are some Dagster framework logs that are nice to see alongside the log message we write So I guess we might want to also attach the handlers via `dagster handler config` for those It would be too bad not to see the logs interleaved normally in the compute log output though Overall sounds like switching to compute logs is a potential workaround but if we re not really abusing the log system and if raising the event log quota is possible that would be a easier and result in a bit more streamlined product experience

daniel · Answer

I certainly don t think you re abusing it in the sense of any kind of intentional abuse but I think we have to make some improvements on our side before we can support high volumes of runs with tens or hundreds thousands of structured log lines at scale

daniel · Answer

Mark do you have an example run where 429s failed a run I wouldn t expect that to happen for a context log failure

Mark Fickett · Answer

Yep <https formenergy dagster cloud td dev runs 63290100 aacb 48ab a332 049e26aad1d8>

daniel · Answer

Are you sure that the 429s are what s failing the kubernetes jobs I see a lot of Discovered failed Kubernetes job but I actually wouldn t expect the 429s to be what made them fail very possible i m mistaken

Mark Fickett · Answer

When I described the failed pods it looked like they had failed because of an exit status 1 and the pod logs ended with a too many retries for status 429 which is interpreted the 429s as the cause of error

daniel · Answer

That does sound fairly conclusive

daniel · Answer

I ll see if we can reproduce that

Mark Fickett · Answer

For example step data pipe graph sasquatch anode raw data graph normalize task 0ue9r plus 25 ends with a discovered failed pod but right before that I see a too many retries ```An exception was thrown during step execution that is likely a framework error rather than an error in user code dagster cloud cli core errors GraphQLStorageError Max retries 6 exceeded too many 429 error responses Stack Trace File usr local lib python3 10 site packages dagster cli api py line 441 in execute step command body yield from execute plan iterator File usr local lib python3 10 site packages dagster core execution api py line 1190 in iter yield from self iterator File usr local lib python3 10 site packages dagster core execution plan execute plan py line 114 in inner plan execution iterator for step event in check generator File usr local lib python3 10 site packages dagster core execution plan execute plan py line 333 in dagster event sequence for step yield step failure event from exc info File usr local lib python3 10 site packages dagster core execution plan objects py line 122 in step failure event from exc info return DagsterEvent step failure event File usr local lib python3 10 site packages dagster core events init py line 802 in step failure event return DagsterEvent from step File usr local lib python3 10 site packages dagster core events init py line 418 in from step log step event step context event File usr local lib python3 10 site packages dagster core events init py line 297 in log step event step context log log dagster event File usr local lib python3 10 site packages dagster core log manager py line 387 in log dagster event self log level=level msg=msg extra= DAGSTER META KEY dagster event File usr local lib python3 10 site packages dagster core log manager py line 402 in log self log level msg args kwargs File usr local lib python3 10 logging init py line 1624 in log self handle record File usr local lib python3 10 logging init py line 1634 in handle self callHandlers record File usr local lib python3 10 logging init py line 1696 in callHandlers hdlr handle record File data pipeline orchestration log py line 96 in handle super handle record File usr local lib python3 10 logging init py line 968 in handle self emit record File usr local lib python3 10 site packages dagster core log manager py line 267 in emit handler handle dagster record File usr local lib python3 10 logging init py line 968 in handle self emit record File usr local lib python3 10 site packages dagster core instance init py line 181 in emit self instance handle new event event File usr local lib python3 10 site packages dagster core instance init py line 1587 in handle new event self event storage store event event File usr local lib python3 10 site packages dagster cloud storage event logs storage py line 371 in store event self execute query File usr local lib python3 10 site packages dagster cloud storage event logs storage py line 248 in execute query res = self graphql client execute File usr local lib python3 10 site packages dagster cloud cli core graphql client py line 111 in execute raise GraphQLStorageError The above exception was caused by the following exception requests exceptions HTTPError 429 Client Error Too Many Requests for url <https formenergy agent dagster cloud graphql> Stack Trace File usr local lib python3 10 site packages dagster cloud cli core graphql client py line 72 in execute return self execute retry query variable values headers File usr local lib python3 10 site packages dagster cloud cli core graphql client py line 157 in execute retry response raise for status File usr local lib python3 10 site packages requests models py line 1021 in raise for status raise HTTPError http error msg response=self 11 36 00 058 data pipe graph asquatch anode raw data graph normalize t 0ue9r plus 25 Step data pipe graph sasquatch anode raw data graph normalize task 0ue9r plus 25 failed health check Discovered failed Kubernetes job dagster step e5abc3ae8aad5ae2b7ebf2295ce0fde4 for step data pipe graph sasquatch anode raw data graph normalize task 0ue9r plus 25 ```

Mark Fickett · Answer

I confirmed that removing `root` from the `managed python loggers` allowed the run to complete successfully But I haven t done the other cleanup verification to make sure the logs get to where we need them otherwise

Mark Fickett · Answer

Hope y all had great holidays Looking for next steps here what s your recommendation Sounds like the options are increase the quota for context log calls to accommodate the big fan out easiest for me Dagster side infrastructure improvements to handle larger log volume what s the timeline on this switch to compute logs I d need to figure out how to change our log configuration to attach handlers in the right place and switch context log calls to plain Python logger calls some reduction in utility of the Dagit UI for searching logs not my preference but may be fastest

daniel · Answer

Hi Mark I think the last bullet is the way to go for now log volume improvements are still a ways out and when I looked at the volume it was high enough that I don t think the fix was a simple as just increasing the quota