While running the pipeline in dagster we are randomly getting the error below –

docker.errors.APIError: 500 Server Error for <http+docker://localhost/v1.41/images/create?tag=5.2.144&fromImage=><jfrog_url>%2F<project>%2Fdev%2F<container_image>: Internal Server Error ("Head https://<jfrog_url>/v2/<project>/dev/<container_image>/manifests/5.2.144: unknown: Authentication is required")
 File "/usr/local/lib/python3.8/dist-packages/dagster/core/instance/__init__.py", line 1732, in launch_run
   self._run_launcher.launch_run(LaunchRunContext(pipeline_run=run, workspace=workspace))
 File "/usr/local/lib/python3.8/dist-packages/dagster_docker/docker_run_launcher.py", line 152, in launch_run
   self._launch_container_with_command(run, docker_image, command)
 File "/usr/local/lib/python3.8/dist-packages/dagster_docker/docker_run_launcher.py", line 110, in _launch_container_with_command
   client.images.pull(docker_image)
 File "/usr/local/lib/python3.8/dist-packages/docker/models/images.py", line 465, in pull
   pull_log = self.client.api.pull(
 File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 429, in pull
   self._raise_for_status(response)
 File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 270, in _raise_for_status
   raise create_api_error_from_http_exception(e) from e
 File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
   raise cls(e, response=response, explanation=explanation) from e
The above exception was caused by the following exception:
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: <http+docker://localhost/v1.41/images/create?tag=5.2.144&fromImage=><jfrog_url>%2Fdatamax%2Fdev%2F<container_image>
 File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 268, in _raise_for_status
   response.raise_for_status()
 File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 1021, in raise_for_status
   raise HTTPError(http_error_msg, response=self)

We are using two celery workers and rabbitmq. We are using the dagster.yaml file where we tried specifying the jfrog credentials using the following syntax -

registry:
      url: "<<https://registry.gitlab.com/v2>>"
      username: "myusername"
      password:
        env: DAGSTER_CONT_REGISTRY_DEPLOY_TOKEN

This issue is happening intermittently • We have tried restarting the docker services but in vain. • We tried to perform the docker login from inside the container and it worked but the pipeline was still failing intermittently. • We tried restarting all the containers, but the same issue was repeated. • We have a similar setup in another environment with the same container images and those are executing just fine. The docker version in the environment where it’s running and this environment where it's failing intermittently is the same.

Hi Shaonauk - i'm not seeing anything obvious in dagster that would be causing this, it looks like it's failing when pulling the image and that goes through the standard Docker python client here: https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-docker/dagster_docker/docker_run_launcher.py#L98-L111 The client logs in, tries to create an image - if it fails, it then pulls and tries again. The only thing I can possibly think of is that the attempt to create an image might be taking long enough that the login is no longer valid? But that's a total guess. If this is working just fine with Dagster and the same images in another environment then there must be something that's different, no? Is there any way to triple check that there's nothing different about the two environments?