https://dagster.io/ logo
Join Slack
Powered by
Hi, I’m running a docker container via the `docker...
# dagster-plus
p
Hi, I’m running a docker container via the 
docker_container_op
. I’ve had success doing this locally with 
dagster dev
, but it’s been failing on Dagster Cloud. I’ve tried this with both the very simple example of using the busybox container to echo something, and using a private container registry. Both work locally, but fail on Dagster Cloud. The error I get seems to be unrelated to my own code.
Copy code
dagster._check.CheckError: Failure condition: Couldn't import module dagster_cloud_serverless_agent.serverless.user_code_launcher when attempting to load the configurable class dagster_cloud_serverless_agent.serverless.user_code_launcher.ServerlessUserCodeLauncher
I’ll post the full error in the thread. 🧵 Is Dagster Cloud supposed to be able to support this feature?
Here’s the full error, using the simple busybox example:
Copy code
dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "first_op":

  File "/venvs/543a943e79c6/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_plan.py", line 262, in dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "/venvs/543a943e79c6/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_step.py", line 369, in core_dagster_event_sequence_for_step
    for user_event in check.generator(
  File "/venvs/543a943e79c6/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_step.py", line 90, in _step_output_error_checked_user_event_sequence
    for user_event in user_event_sequence:
  File "/venvs/543a943e79c6/lib/python3.8/site-packages/dagster/_core/execution/plan/compute.py", line 192, in execute_core_compute
    for step_output in _yield_compute_results(step_context, inputs, compute_fn):
  File "/venvs/543a943e79c6/lib/python3.8/site-packages/dagster/_core/execution/plan/compute.py", line 161, in _yield_compute_results
    for event in iterate_with_context(
  File "/venvs/543a943e79c6/lib/python3.8/site-packages/dagster/_utils/__init__.py", line 447, in iterate_with_context
    return
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/venvs/543a943e79c6/lib/python3.8/site-packages/dagster/_core/execution/plan/utils.py", line 84, in op_execution_error_boundary
    raise error_cls(

The above exception was caused by the following exception:
dagster._check.CheckError: Failure condition: Couldn't import module dagster_cloud_serverless_agent.serverless.user_code_launcher when attempting to load the configurable class dagster_cloud_serverless_agent.serverless.user_code_launcher.ServerlessUserCodeLauncher

  File "/venvs/543a943e79c6/lib/python3.8/site-packages/dagster/_core/execution/plan/utils.py", line 54, in op_execution_error_boundary
    yield
  File "/venvs/543a943e79c6/lib/python3.8/site-packages/dagster/_utils/__init__.py", line 445, in iterate_with_context
    next_output = next(iterator)
  File "/venvs/543a943e79c6/lib/python3.8/site-packages/dagster/_core/execution/plan/compute_generator.py", line 124, in _coerce_op_compute_fn_to_iterator
    result = invoke_compute_fn(
  File "/venvs/543a943e79c6/lib/python3.8/site-packages/dagster/_core/execution/plan/compute_generator.py", line 118, in invoke_compute_fn
    return fn(context, **args_to_pass) if context_arg_provided else fn(**args_to_pass)
  File "/venvs/543a943e79c6/lib/python3.8/site-packages/dagster/_annotations.py", line 106, in inner
    return target(*args, **kwargs)
  File "/venvs/543a943e79c6/lib/python3.8/site-packages/dagster_docker/ops/docker_container_op.py", line 172, in docker_container_op
    execute_docker_container(context, **context.op_config)
  File "/venvs/543a943e79c6/lib/python3.8/site-packages/dagster/_annotations.py", line 106, in inner
    return target(*args, **kwargs)
  File "/venvs/543a943e79c6/lib/python3.8/site-packages/dagster_docker/ops/docker_container_op.py", line 109, in execute_docker_container
    if isinstance(context.instance.run_launcher, DockerRunLauncher)
  File "/venvs/543a943e79c6/lib/python3.8/site-packages/dagster_cloud/instance/__init__.py", line 353, in run_launcher
    return self.user_code_launcher.run_launcher()
  File "/venvs/543a943e79c6/lib/python3.8/site-packages/dagster_cloud/instance/__init__.py", line 342, in user_code_launcher
    self._user_code_launcher_data.rehydrate(as_type=DagsterCloudUserCodeLauncher)  # type: ignore  # (possible none)
  File "/venvs/543a943e79c6/lib/python3.8/site-packages/dagster/_serdes/config_class.py", line 82, in rehydrate
    check.failed(
  File "/venvs/543a943e79c6/lib/python3.8/site-packages/dagster/_check/__init__.py", line 1669, in failed
    raise CheckError(f"Failure condition: {desc}")

The above exception occurred during handling of the following exception:
ModuleNotFoundError: No module named 'dagster_cloud_serverless_agent'

  File "/venvs/543a943e79c6/lib/python3.8/site-packages/dagster/_serdes/config_class.py", line 80, in rehydrate
    module = importlib.import_module(self.module_name)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
j
The import error is a bit odd - I think it’s trying to import a private module that we use for maintaining serverless but isn’t actually installed in your container. But in general, serverless agents run everything on AWS Fargate and AWS Fargate doesn’t make the Docker socket available to end users. I would expect pretty much any docker-in-docker operation - not just dagster specific ones - to fail there. https://github.com/aws/containers-roadmap/issues/1356
Can you say a bit more about your 
docker_container_op
use case?
p
The real use case is to be able to run code written in other languages, such as R. But this is failing even for this:
Copy code
first_op = docker_container_op.configured(
    {
        "image": "busybox",
        "command": ["echo", "HELLO"],
    },
    name="first_op",
)
second_op = docker_container_op.configured(
    {
        "image": "busybox",
        "command": ["echo", "GOODBYE"],
    },
    name="second_op",
)


@job
def busybox_echo():
    second_op(first_op())
👍 1
If I wanted to use Dagster Cloud to orchestrate code written in languages other than python, what would you suggest?
@jordan: Any suggestions? Being able to orchestrate code written in languages other than python is a requirement for my team. We’re on GCP, so I’m guessing having DAGster interact with Cloud Run via an op is the way this could work. But I’m wondering if you have any suggestions.
j
Any of the hybrid agents should be able to support docker-in-docker patterns (given that you’re comfortable exposing the docker sock to your dagster containers). It would change how much infra you’d need to manage on your end though. Or what you suggested about having dagster orchestrate other services (like asynchronously calling cloud run or something) would be a reasonable idea too. There’s probably a third option that involves colocating your dagster code and the other language code its orchestrating in the same docker image and invoking via 
dagster-shell
. That might be a bit hard to scale depending on how many different things you’re orchestrating.
p
Thank you
f
Hey @Phil Dreizen - Good to see you on the demo/AMA earlier!
I hope Jordan got you unstuck on the Docker issue.
p
Yes, it was helpful. Thank you!
👍 1
g
Why don`t you put all code location code (including non python code) inside the docker image of each code location
no need for docker-in docker and simply invoke the non python code using the execute bash operator
b
Hey @Phil Dreizen, what was the verdict? Did you get the non-Python code to work?
16 Views
Open in Slack
PreviousNext
Powered by