I m bumping into a `botocore exceptions ClientError An error

Question

I m bumping into a `botocore exceptions ClientError An error occurred SlowDown when calling the PutObject operation reached max retries 4 ` error when using a possibly overloaded internal S3 compatible blobstore to store job compute logs see thread for details Since this doesn t actually cause any of the `op ` in the pipeline to fail I m wondering if there is a way to mark this error as ignorable

mrdavidlaing · Answer

```dagster core errors DagsterSubprocessError During multiprocess execution errors occurred in child processes In process 29 botocore exceptions ClientError An error occurred SlowDown when calling the PutObject operation reached max retries 4 Please reduce your request rate Stack Trace File usr local lib python3 7 site packages dagster core executor child process executor py line 65 in execute command in child process for step event in command execute snip File usr local lib python3 7 site packages s3transfer upload py line 694 in main client put object Bucket=bucket Key=key Body=body extra args File usr local lib python3 7 site packages botocore client py line 391 in api call return self make api call operation name kwargs File usr local lib python3 7 site packages botocore client py line 719 in make api call raise error class parsed response operation name In process 977 botocore exceptions ClientError An error occurred SlowDown when calling the PutObject operation reached max retries 4 Please reduce your request rate Stack Trace File usr local lib python3 7 site packages dagster core executor child process executor py line 65 in execute command in child process for step event in command execute File usr local lib python3 7 site packages dagster core executor multiprocess py line 83 in execute instance=instance ```

mrdavidlaing · Answer

Alternatively is there perhaps a way to configure the underlying boto library being used with a different set of retry options <https boto3 amazonaws com v1 documentation api latest guide retries html>

prha · Answer

Hi David Thanks for reporting I think we should probably guard against compute log manager errors and yield a special failure event instead of letting the error mark the run as failed Re configuring the underlying boto library any chance you can set an AWS config file on your run workers

prha · Answer

< Dagster Bot> issue Guard compute log manager errors to yield a custom failure event instead of failing the run

Dagster Bot · Answer

Created issue at <https github com dagster io dagster issues 5920>

mrdavidlaing · Answer

gt Re configuring the underlying boto library any chance you can set an AWS config file on your run workers If that means setting an env variable then yes I just can t figure out which env var to set slightly smiling face

prha · Answer

hmm not a boto expert does `AWS MAX ATTEMPTS` do what you want from <https botocore amazonaws com v1 documentation api latest reference config html>

mrdavidlaing · Answer

I ll give that a try Feels like a bit of a magic setting but if it works I ll submit a PR to document it better

mrdavidlaing · Answer

I can confirm that setting the `AWS MAX ATTEMPTS` env var does appear to have made the problem go away for me