@Scout I want to move from print lines to using a logger. Explain to me the different options for loggers and biasing for simplicity

We understand the importance of moving from print statements to a structured logging system. In Dagster, you have several options for logging, which allow you to capture and structure your logs in a way that's conducive to both development and production environments. Here are the different options for loggers in Dagster, with a bias for simplicity: 1. Built-in Loggers: Dagster provides built-in loggers that you can easily configure and use. For instance, there's a

json_console_logger

that prints JSON-formatted single-line messages to the console. This is useful if you want your logs to be easily parseable by other tools or services. 2. Configuring Loggers: You can configure the log level of the built-in loggers through your run config. For example, to filter all messages below

ERROR

out of the colored console logger, you can add the following to your config YAML:

yaml    loggers:      console:        config:          log_level: ERROR

When a job with the above config is executed, you'll only see the

ERROR

level logs. 3. Custom Loggers: If the built-in loggers don't meet your needs, you can define custom loggers using the

@logger

decorator. This allows you to create a logger that fits your specific requirements. The decorated function should take a single argument, the

init_context

, and return a

logging.Logger

. 4. Python Logging: You can also use Python's standard logging module in conjunction with Dagster. To create a logger that's captured by Dagster without modifying your

dagster.yaml

file, you can use the

get_dagster_logger

utility function. This is useful for logging from inside nested functions or other cases where passing the context parameter is inconvenient. Here's a simple example of using `get_dagster_logger`:

python from dagster import get_dagster_logger, op  @op def my_op():     logger = get_dagster_logger()     <http://logger.info|logger.info>("This is an info message")

5. Environment-Specific Logging: Logging can be environment-specific. For example, you might want to use different logging configurations for development, testing, and production. Dagster allows you to attach loggers to jobs so that you can seamlessly switch environments without changing any code. For simplicity, starting with the built-in loggers and configuring them as needed is often the easiest path. If you find that you need more control over your logging, you can then move on to defining custom loggers or integrating with Python's logging module. Remember that all these configurations and customizations are meant to help you transition from simple print statements to a more robust and scalable logging system that can grow with your application's needs.

@Scout is the logger from get_dagster_logger() thread safe or should I create a new instance per thread?

The

get_dagster_logger()

function provides you with a Dagster-managed logger that you can use anywhere in your code. However, the documentation does not explicitly state whether the logger obtained from

get_dagster_logger()

is thread-safe. In general, Python's built-in logging module, which Dagster's logging is based on, is thread-safe. The logging module ensures that individual log records are locked during emission, which means that multiple threads can safely use the same logger instance without corrupting each other's log messages. However, there was an issue reported where the

DagsterLogHandler

was not thread-safe, which could potentially lead to lost log messages when logging from multiple threads. This issue was specifically related to the internal handling of Dagster events and has since been closed, indicating that it was addressed. Given this context, while the base logging functionality should be thread-safe, if you are working with multiple threads and have concerns about thread safety, especially in relation to Dagster's event system, it might be prudent to review the most recent Dagster release notes or check with the latest documentation to ensure that any past issues have been resolved. If you encounter any unexpected behavior or if the documentation is not clear on the current state of thread safety with

get_dagster_logger()

, it would be advisable to reach out to the Dagster community or maintainers for clarification.