https://dagster.io/ logo
#ask-ai
Title
# ask-ai
f

Fernando Cagua

02/28/2024, 8:24 PM
What’s the best way to log from inside resources?
s

Scout

02/28/2024, 8:24 PM
When logging from inside resources in Dagster, you can use the
get_dagster_logger
utility function to create a logger that's captured by Dagster without modifying your
dagster.yaml
file. This is useful when logging from inside nested functions or other cases where it would be inconvenient to thread through the context parameter to enable calls to
context.log
. Here's an example from the documentation on how to use `get_dagster_logger`:
Copy code
python from dagster import get_dagster_logger, graph, op  @op def ambitious_op():     my_logger = get_dagster_logger()     try:         x = 1 / 0         return x     except ZeroDivisionError:         my_logger.error("Couldn't divide by zero!")      return None
In the context of a resource, you would similarly instantiate a logger using
get_dagster_logger
and then use it to log messages. The logger obtained in this way will be configured according to the settings in your
dagster.yaml
or the default Dagster logging configuration if the
dagster.yaml
is not customized for logging. Remember that the
context.log
object is available when you are within the execution context of an op, and it exposes the usual
debug
,
info
,
warning
,
error
, and
critical
methods. However, when you are in a resource and do not have access to the
context
object,
get_dagster_logger
is the recommended way to log messages that will be captured by Dagster's logging system.
f

Fernando Cagua

02/29/2024, 8:57 AM
@Scout Can you provide an example of logging inside a io manager class than inherits from dagster.UPathIOManager
s

Scout

02/29/2024, 8:57 AM
Certainly! When you're creating a custom IO manager that inherits from
dagster.UPathIOManager
, you can use the
get_dagster_logger
function to obtain a logger for logging within the IO manager methods. Below is an example of how you might implement logging inside a custom IO manager:
Copy code
python from dagster import IOManager, get_dagster_logger, io_manager from dagster.core.storage.file_manager import UPathIOManager  class MyCustomIOManager(UPathIOManager):     def handle_output(self, context, obj):         logger = get_dagster_logger()         <http://logger.info|logger.info>(f"Handling output for: {context.asset_key}")         # Your custom logic to handle the output         super().handle_output(context, obj)      def load_input(self, context):         logger = get_dagster_logger()         <http://logger.info|logger.info>(f"Loading input from: {context.upstream_output.asset_key}")         # Your custom logic to load the input         return super().load_input(context)  # Then, you can define your IO manager as a resource @io_manager def my_custom_io_manager():     return MyCustomIOManager()  # And include it in your job definition @job(resource_defs={"io_manager": my_custom_io_manager}) def my_job():     # Your job definition here
In this example,
MyCustomIOManager
inherits from
UPathIOManager
, which is a utility for file-based IO managers. Inside the
handle_output
and
load_input
methods, we first obtain a logger using
get_dagster_logger
and then log some informational messages. After logging, we call the superclass's method to handle the actual file operations. By using
get_dagster_logger
, the logs emitted from the IO manager will be captured by Dagster's logging system and will be available in the Dagster event log, which can be viewed in Dagit, Dagster's web interface.
f

Fernando Cagua

02/29/2024, 9:00 AM
@Scout Can I use logger = get_dagster_logger() outside of the class?
s

Scout

02/29/2024, 9:00 AM
Yes, you can use
get_dagster_logger()
outside of a class to obtain a logger that is managed by Dagster. This function can be called from anywhere in your code, not just within the context of a class or a Dagster op or asset. The logger obtained in this way will be configured according to the settings in your
dagster.yaml
or the default Dagster logging configuration if the
dagster.yaml
is not customized for logging. Here's an example of using
get_dagster_logger()
at the module level:
Copy code
python from dagster import get_dagster_logger  # Obtain a Dagster-managed logger at the module level logger = get_dagster_logger()  def some_function():     # You can use the logger inside other functions     <http://logger.info|logger.info>("This is an info message from some_function.")  class SomeClass:     def some_method(self):         # You can also use the logger inside class methods         <http://logger.info|logger.info>("This is an info message from SomeClass.some_method.")
In this example,
logger
is defined at the module level and can be used throughout the module in functions and class methods. Messages logged using this logger will be captured by Dagster's logging system and will be associated with the relevant Dagster computations when used within the context of a Dagster pipeline or job execution.
6 Views