https://dagster.io/ logo
#ask-ai
Title
# ask-ai
j

Jamie

02/20/2024, 11:24 PM
How can I pipe stdout from a docker container executed with execute_docker_container() into something dagster can read?
s

Scout

02/20/2024, 11:24 PM
To capture stdout from a Docker container executed with
execute_docker_container()
in Dagster, you can utilize the logging facilities provided by Dagster Pipes. According to the documentation, the
PipesContext
object within the
open_dagster_pipes
context manager provides a
log
property, which is a standard Python
logging.Logger
. You can add a custom handler to this logger to capture stdout and stderr. Here's an example from the documentation on how to add a stream handler to the
PipesContext
logger to output log messages to stdout:
Copy code
python import sys import logging from dagster_pipes import open_dagster_pipes  with open_dagster_pipes(...) as pipes:     handler = logging.StreamHandler(sys.stdout)     handler.setLevel(logging.DEBUG)     formatter = logging.Formatter('%(levelname)s - %(message)s')     handler.setFormatter(formatter)     pipes.log.addHandler(handler)          <http://pipes.log.info|pipes.log.info>(f"Run ID: {pipes.run_id}")
In this example, a
StreamHandler
is added to the
PipesContext
logger, which directs the log output to
sys.stdout
. This setup allows you to capture and display log messages from the external process running in the Docker container. Please note that Dagster Pipes is an experimental feature, and the APIs may change in future versions. If you need further assistance or have more specific requirements, please refer to the Dagster documentation or consider reaching out to the Dagster community for more detailed guidance.
6 Views