What’s the best way to log from inside resources?

When logging from inside resources in Dagster, you can use the

get_dagster_logger

utility function to create a logger that's captured by Dagster without modifying your

dagster.yaml

file. This is useful when logging from inside nested functions or other cases where it would be inconvenient to thread through the context parameter to enable calls to

context.log

. Here's an example from the documentation on how to use `get_dagster_logger`:

python from dagster import get_dagster_logger, graph, op  @op def ambitious_op():     my_logger = get_dagster_logger()     try:         x = 1 / 0         return x     except ZeroDivisionError:         my_logger.error("Couldn't divide by zero!")      return None

In the context of a resource, you would similarly instantiate a logger using

get_dagster_logger

and then use it to log messages. The logger obtained in this way will be configured according to the settings in your

dagster.yaml

or the default Dagster logging configuration if the

dagster.yaml

is not customized for logging. Remember that the

context.log

object is available when you are within the execution context of an op, and it exposes the usual

debug

,

info

,

warning

,

error

, and

critical

methods. However, when you are in a resource and do not have access to the

context

object,

get_dagster_logger

is the recommended way to log messages that will be captured by Dagster's logging system.

@Scout Can you provide an example of logging inside a io manager class than inherits from dagster.UPathIOManager

Certainly! When you're creating a custom IO manager that inherits from

dagster.UPathIOManager

, you can use the

get_dagster_logger

function to obtain a logger for logging within the IO manager methods. Below is an example of how you might implement logging inside a custom IO manager:

python from dagster import IOManager, get_dagster_logger, io_manager from dagster.core.storage.file_manager import UPathIOManager  class MyCustomIOManager(UPathIOManager):     def handle_output(self, context, obj):         logger = get_dagster_logger()         <http://logger.info|logger.info>(f"Handling output for: {context.asset_key}")         # Your custom logic to handle the output         super().handle_output(context, obj)      def load_input(self, context):         logger = get_dagster_logger()         <http://logger.info|logger.info>(f"Loading input from: {context.upstream_output.asset_key}")         # Your custom logic to load the input         return super().load_input(context)  # Then, you can define your IO manager as a resource @io_manager def my_custom_io_manager():     return MyCustomIOManager()  # And include it in your job definition @job(resource_defs={"io_manager": my_custom_io_manager}) def my_job():     # Your job definition here

In this example,

MyCustomIOManager

inherits from

UPathIOManager

, which is a utility for file-based IO managers. Inside the

handle_output

and

load_input

methods, we first obtain a logger using

get_dagster_logger

and then log some informational messages. After logging, we call the superclass's method to handle the actual file operations. By using

get_dagster_logger

, the logs emitted from the IO manager will be captured by Dagster's logging system and will be available in the Dagster event log, which can be viewed in Dagit, Dagster's web interface.

@Scout Can I use logger = get_dagster_logger() outside of the class?

Yes, you can use

get_dagster_logger()

outside of a class to obtain a logger that is managed by Dagster. This function can be called from anywhere in your code, not just within the context of a class or a Dagster op or asset. The logger obtained in this way will be configured according to the settings in your

dagster.yaml

or the default Dagster logging configuration if the

dagster.yaml

is not customized for logging. Here's an example of using

get_dagster_logger()

at the module level:

python from dagster import get_dagster_logger  # Obtain a Dagster-managed logger at the module level logger = get_dagster_logger()  def some_function():     # You can use the logger inside other functions     <http://logger.info|logger.info>("This is an info message from some_function.")  class SomeClass:     def some_method(self):         # You can also use the logger inside class methods         <http://logger.info|logger.info>("This is an info message from SomeClass.some_method.")

In this example,

logger

is defined at the module level and can be used throughout the module in functions and class methods. Messages logged using this logger will be captured by Dagster's logging system and will be associated with the relevant Dagster computations when used within the context of a Dagster pipeline or job execution.