When running spark in local mode inside the docker container dagster #ask-community

When running spark (in local mode) inside the dock...

geoHeil

07/08/2022, 5:20 PM

When running spark (in local mode) inside the docker container of the dagster job/workspace how can I expose the port (4040) for its UI when using the DockerRunner of dagster?

geoHeil

07/08/2022, 5:21 PM

(and also map it so it is available at the host)

johann

07/08/2022, 5:26 PM

I believe this should be possible using

container_kwargs

geoHeil

07/08/2022, 5:26 PM

how? Cause I think if 2 runs are started in parallel it would fail (to map to the same port)

geoHeil

07/08/2022, 5:31 PM

https://docker-py.readthedocs.io/en/stable/containers.html

None

, to assign a random host port. For example,

{'2222/tcp': None}

. might work

geoHeil

07/08/2022, 5:32 PM

ports: - 4040: None

geoHeil

07/08/2022, 5:32 PM

I hope this will work - let`s see

johann

07/08/2022, 5:32 PM

Ah got it

johann

07/08/2022, 7:31 PM

Any luck?

geoHeil

07/08/2022, 7:31 PM

I am still working on some other steps / issues. Will update you once I know more.

👍 1

geoHeil

07/09/2022, 9:27 AM

not fully there yet. But I already notice that when the job consists of multiple steps >> 1 spark session is instanciated. How can I prevent this and use a single shared sparksession (for multiple steps/ops/assets in a job)?

sandy

07/11/2022, 3:32 PM

If you use the in-process executor and the pyspark resource, that sharing should happen automatically. If you're using any of the other executors, this isn't possible, because they execute each step in its own process and a constraint of spark itself is that spark sessions can't be shared across multiple processes

geoHeil

07/11/2022, 3:34 PM

I am using the

docker_executor

is it possible to use the in_process one inside the docker container?

sandy

07/11/2022, 3:51 PM

I can't think of any reason why those wouldn't work together, but @johann would know better than me

johann

07/11/2022, 5:29 PM

DockerRunLauncher

and

in_process

are compatible ✅

geoHeil

07/11/2022, 8:42 PM

how can I pass in_process_executor/executor_def to define_asset_job?

sandy

07/11/2022, 8:44 PM

Right now, the recommendation is to supply it via the repository:

Copy code

@repository(default_executor_def=in_process_executor):
def repo():
    ...

But if you anticipate wanting different executors for different jobs or assets in the same repository, let us know, and it shouldn't be difficult for us to make that possible. cc @chris

geoHeil

07/11/2022, 8:45 PM

Normally I would want the higher parallel multi process one (in the docker run launcher) and only for specific ones the in process one

geoHeil

07/11/2022, 8:49 PM

But I still get number of parallel tasks spark sessions instanciated even when using the in_process executor - that is strange.

geoHeil

07/22/2022, 9:15 PM

@johann Using dagster 0.15.7 and the docker run launcher I observe a weird error message: AttributeError: 'list' object has no attribute 'items' File "/opt/conda/lib/python3.9/site-packages/dagster/core/instance/__init__.py", line 1729, in launch_run self._run_launcher.launch_run(LaunchRunContext(pipeline_run=run, workspace=workspace)) File "/opt/conda/lib/python3.9/site-packages/dagster_docker/docker_run_launcher.py", line 152, in launch_run self._launch_container_with_command(run, docker_image, command) File "/opt/conda/lib/python3.9/site-packages/dagster_docker/docker_run_launcher.py", line 100, in _launch_container_with_command container = client.containers.create( File "/opt/conda/lib/python3.9/site-packages/docker/models/containers.py", line 877, in create create_kwargs = _create_container_args(kwargs) File "/opt/conda/lib/python3.9/site-packages/docker/models/containers.py", line 1095, in _create_container_args create_kwargs['host_config'] = HostConfig(**host_config_kwargs) File "/opt/conda/lib/python3.9/site-packages/docker/types/containers.py", line 392, in init self['PortBindings'] = convert_port_bindings(port_bindings) File "/opt/conda/lib/python3.9/site-packages/docker/utils/utils.py", line 100, in convert_port_bindings for k, v in iter(port_bindings.items()): When:

Copy code