Hey all – how can I get Spark driver/executor logs to show up in Dagit?

Hi Kevin - how are you running Spark? I think getting the executor logs to show up in Dagit would be tough (other than maybe a link), but in many cases it shouldn't be hard to get the driver logs to show up.

Hey Sandy – I'm starting a new container per asset build with the

DockerRunLauncher

, and then using

dagster_pyspark

to get a spark session

Have you checked the compute logs to see if it's there? Along with the "managed logs", Dagster also records stdout and stderr from the processes it manages. There's a toggle to access these on the run page:

yep. the only thing I see there are the Python loggers I enabled in that block above, but it's not the typical nicely-formatted Spark logs, it's a bunch of control statements from py4j like this:

DEBUG:py4j.clientserver:__ASSET_JOB_0 - 8283175e-b8ca-4e80-abae-4884af8242db - [redacted] - Answer received: !yv
DEBUG:py4j.clientserver:__ASSET_JOB_0 - 8283175e-b8ca-4e80-abae-4884af8242db - [redacted] - Command to send: c
o62
collectToPython
e

Just to make sure, you're seeing that when you click the icon with the "Raw compute logs" tooltip? Asking because I would expect the

managed_python_loggers

to only affect the logs that show up when the other icon ("Structured event logs") is selected.

yep, they appear in both for me actually

got it - this is not my area of expertise, but forwarding this question to the experts

thanks, appreciate it! for the other connectors that you guys have that talk w/ Spark (databricks, snowflake, etc) – do they give back a stream of logs from the driver + executor that you guys are explicitly piping into Dagit? wonder if we could do the same

are you just using the default compute log manager? https://docs.dagster.io/deployment/dagster-instance#compute-log-storage I believe you either need to set up volume mounts to ensure the files written out for log capture are available outside the container or use one of the blob storage backed log managers

standard compute log manager right now, yes. would it be right to say that if they're not showing up in Dagit (because host process is not Dagster managed, etc.), that the compute log manager won't make a difference? that's just where the files are stored? or is the compute log manager actually responsible for capturing logs as well? I'm using the

DockerRunLauncher

and a docker image that includes Spark and all our Python dependencies (I believe this is in-line w/ the rec in the docs for k8s/docker deployment). I believe part of the complexity in using PySpark is that the Spark backend is a Java process regardless of which language you write your transforms in – so both the execution and logging are handled by Java – PySpark is just a Python wrapper around them. They make calls to the Java API through py4j and logging through log4j (latter is a Java library) – the little logs I was able to get to show up so far are by declaring py4j as a Dagster-managed logger, but it's not very useful output. With that said, though – if they are making it to stdout on my Docker container, I feel like we should be able to capture them somehow, and PySpark is pretty widely used in terms of supported Dagster integrations so hopefully many will benefit if we can figure it out.

compute log manager is responsible for how logs are captured as well. The issue with the default one and docker run launcher is that the default one stores the captured logs in the filesystem which unless you set up volume mounts only exist ephemerally within the container launched for the run. you’ll need to either add config to the run launcher to set up a mapping from the containers file system to the host filesystem using volume mounts, or just switch to a compute log manager that uploads to blob storage ie s3.

Hi Alex, do you have an example on how to configure the docker run launcher to make the compute logs available in the UI? How to setup the container volume? Thanks!

I do not have an example on hand. Heres some more specific docs links how you set the directory for compute logs: https://docs.dagster.io/deployment/dagster-instance#localcomputelogmanager an example of volume mounting via container kwargs on the run launcher https://docs.dagster.io/deployment/guides/docker#mounting-volumes

Thanks for the help Alex, I've figured it out!

@Edo Do you mind sharing how you managed? I’m still stuck here. So far: I’ve mounted the volume in Docker and added this volume to the respective containers. I my dagster.yml, I then added the following:

compute_logs:
  module: dagster.core.storage.local_compute_log_manager
  class: LocalComputeLogManager
  config:
    base_dir: /opt/dagster/logs

Being

/opt/dagster/logs

my mounted volume. I can see my volume in Docker, and all of the subfolders being created by each run, I don’t see any files. ``` ```