https://dagster.io/ logo
#ask-ai
Title
# ask-ai
j

Joshua Greenhalgh

02/25/2024, 4:54 PM
I have the following code;
Copy code
from dagster import Definitions
from dagster_dev.assets import test_asset
from dagster_dev.loggers import custom_logger

defs = Definitions(assets=[test_asset],loggers={'custom_logger':custom_logger})
Copy code
from dagster import logger

@logger
def custom_logger(init_context):
    assert False
Copy code
from dagster import asset, AssetExecutionContext

@asset
def test_asset(context: AssetExecutionContext) -> int:
    <http://context.log.info|context.log.info>("Some Log")
    return 42
The logger init is never called!? I expect to see an assertion failure why do I not?
s

Scout

02/25/2024, 4:54 PM
Based on the documentation, when you define a custom logger using the
@logger
decorator, the decorated function should accept an
InitLoggerContext
and return an instance of
logging.Logger
. This function will become the
logger_fn
of an underlying
LoggerDefinition
. The custom logger is then supposed to be automatically invoked whenever the methods on
context.log
are called from within job compute logic. In your code, you have defined a custom logger
custom_logger
that asserts
False
when initialized. However, you're expecting to see an assertion failure when the logger is initialized, but you're not seeing it. There are a few potential reasons why the logger initialization is not being called: 1. The logger is not correctly configured in the job or pipeline that is executing the asset. The
Definitions
object you're creating with
defs = Definitions(assets=[test_asset], loggers={'custom_logger': custom_logger})
is not by itself sufficient to ensure that the logger is used during execution. You need to make sure that the job or pipeline that includes
test_asset
is also configured to use
custom_logger
. 2. The logger might not be getting instantiated due to a misconfiguration or an issue with the execution environment. For example, if the job or pipeline is not executed, or if there is an error in the configuration that prevents the logger from being instantiated, you would not see the assertion failure. 3. The asset execution might not be reaching the point where it logs information. If there is an error in the asset code before the logging statement, or if the asset is not being executed at all, the logger would not be invoked. To debug this issue, you can check the following: - Ensure that the job or pipeline that includes
test_asset
is configured to use
custom_logger
. This typically involves setting the
logger_defs
argument in the job definition to include your custom logger. - Verify that the job or pipeline is being executed correctly and that there are no errors preventing the execution from reaching the point where
<http://context.log.info|context.log.info>
is called. - Check the execution environment and configuration to ensure that there are no issues preventing the logger from being instantiated. If you've confirmed that the job or pipeline is set up correctly and the asset is being executed without errors, but you're still not seeing the assertion failure, it may be necessary to look into the specific details of your execution environment or consider other potential issues that could be affecting the logger initialization.
j

Joshua Greenhalgh

02/25/2024, 4:56 PM
@Scout In response to; "to use
custom_logger
. This typically involves setting the
logger_defs
argument in the job definition to include your custom logger." I do not make use of an explicit Job - I am on;y using the @asset decorator - so how do I set the
logger_defs
s

Scout

02/25/2024, 4:56 PM
In Dagster, when you're using assets without an explicit job definition, you can still configure loggers by using a repository definition that includes your assets and loggers. Here's how you can do it: 1. Define your custom logger using the
@logger
decorator, as you have done. 2. Create a repository that includes your assets and logger definitions. 3. Use the repository in Dagit or when executing your assets programmatically. Here's an example of how you might define a repository with your custom logger and asset:
Copy code
python from dagster import repository, asset, logger, AssetExecutionContext  # Define your custom logger @logger def custom_logger(init_context):     assert False  # Define your asset @asset def test_asset(context: AssetExecutionContext) -> int:     <http://context.log.info|context.log.info>("Some Log")     return 42  # Define a repository that includes your asset and custom logger @repository def my_repository():     return {         "assets": [test_asset],         "loggers": {"custom_logger": custom_logger},     }
When you load this repository in Dagit, it should automatically pick up the custom logger definition. When you execute your asset, the custom logger should be initialized, and you should see the assertion failure as expected. If you're executing your assets programmatically, you can use the repository definition to configure the execution context with the custom logger. Here's an example of how you might do that:
Copy code
python from dagster import execute_pipeline, build_pipeline_definition  # Assuming you have a repository as defined above pipeline_def = build_pipeline_definition(my_repository)  # Execute the pipeline with the custom logger result = execute_pipeline(pipeline_def)
Make sure that your repository is correctly configured and recognized by Dagster, and that you're using the correct execution commands that take the repository's configuration into account.
j

Joshua Greenhalgh

02/25/2024, 4:58 PM
@Scout but I am not using repository derictly either - I have specified the loggers in the Definitions like;
Copy code
defs = Definitions(assets=[test_asset],loggers={'custom_logger':custom_logger})
s

Scout

02/25/2024, 4:58 PM
If you are using the
Definitions
class to configure your assets and loggers, and you are not seeing the logger initialization being called, it could be due to the way you are executing your assets. The
Definitions
class is used to implicitly create a repository under the hood, which is then used by Dagster's tooling. To ensure that your custom logger is being used when executing your assets, you should execute your assets in a context that respects the
Definitions
configuration. This typically involves using Dagster's command-line interface (CLI) or Dagit UI, which will automatically pick up the
defs
variable from your module and use it to construct the execution environment. Here's what you need to do: 1. Make sure your module (let's call it
my_module.py
) with the
Definitions
is structured like this:
Copy code
python
from dagster import asset, logger, Definitions, AssetExecutionContext

# Define your custom logger
@logger
def custom_logger(init_context):
    assert False

# Define your asset
@asset
def test_asset(context: AssetExecutionContext) -> int:
    <http://context.log.info|context.log.info>("Some Log")
    return 42

# Create Definitions with your asset and custom logger
defs = Definitions(assets=[test_asset], loggers={'custom_logger': custom_logger})
2. Use the Dagster CLI to execute your assets. If you're using the CLI, you can run the following command:
Copy code
bash
dagster asset execute -f my_module.py
This command tells Dagster to execute the assets defined in
my_module.py
, and Dagster will use the
defs
variable to construct the execution environment, including the custom logger. 3. If you're using Dagit, simply start Dagit while pointing to the directory containing
my_module.py
, and Dagit will automatically load the
defs
variable and use it to construct the execution environment.
Copy code
bash
dagit -f my_module.py
When you execute your assets through these methods, Dagster should use the custom logger defined in your
Definitions
, and you should see the assertion failure as expected. If you are still not seeing the logger initialization being called, you may want to verify that there are no issues with the way you are executing your assets or with the environment in which Dagster is running. Additionally, ensure that there are no errors in your asset or logger definitions that could be preventing the logger from being initialized. Loggers | Dagster button Dagster log handler re-logs messages to ... button Dagster Docs button "DagsterInvariantViolationError: Provide... button Thank you for your feedback!