https://dagster.io/ logo
#ask-ai
Title
# ask-ai
a

Alex Berndt

03/06/2024, 9:15 AM
I have an asset which needs to have access to external API credentials. During testing, I do not have access to these credentials, so I would like to mock this. Please show me a code snippet of how I can achieve this using resources. The example should show the asset using the API credentials in production as well as a test which materializes the asset using the mocked resource.
s

Scout

03/06/2024, 9:15 AM
Certainly! To achieve this, you would define a resource that provides access to the external API credentials and then use different implementations of this resource for production and testing environments. Here's a step-by-step guide based on the documentation: 1. Define a resource that encapsulates the API client, which requires credentials. In production, this resource will use the real credentials to access the API. 2. Define a mock resource for testing that simulates the behavior of the API without requiring real credentials. 3. Create an asset that depends on this resource. The asset will use the resource to interact with the API. 4. In your tests, materialize the asset using the mocked resource to avoid the need for real credentials. Here's a code snippet that demonstrates this pattern:
Copy code
python from dagster import resource, asset, Field, StringSource, build_asset_context  # Define a resource for the production API client @resource({     "api_key": Field(StringSource, description="API key for the external API") }) def api_client_resource(context):     api_key = context.resource_config["api_key"]     # Replace 'ExternalAPIClient' with your actual client class     return ExternalAPIClient(api_key)  # Define a mock resource for testing @resource def mock_api_client_resource(_):     # Replace 'MockExternalAPIClient' with your mock client class     return MockExternalAPIClient()  # Define an asset that uses the API client resource @asset(required_resource_keys={"api_client"}) def my_asset(context):     api_client = context.resources.api_client     # Use the API client to interact with the external API     data = api_client.get_data()     return data  # Example test that materializes the asset using the mocked resource def test_my_asset():     with build_asset_context(resources={"api_client": mock_api_client_resource}) as context:         result = my_asset(context=context)         # Perform assertions on 'result'
In this example,
ExternalAPIClient
would be a class that interacts with the real external API, and
MockExternalAPIClient
would be a class that simulates the API for testing purposes. The
api_client_resource
function creates an instance of the real API client using the provided API key, while
mock_api_client_resource
creates an instance of the mock client. When running tests, you use
build_asset_context
to provide the
mock_api_client_resource
to the asset, which allows you to test the asset's logic without needing real API credentials. Please note that the actual implementation of
ExternalAPIClient
and
MockExternalAPIClient
will depend on the specifics of the external API you're interacting with and how you want to simulate it for testing.
a

Alex Berndt

03/06/2024, 9:17 AM
@Scout please update the provided example by using the pydantic resource definition instead of the legacy resource approach.
s

Scout

03/06/2024, 9:17 AM
Certainly! With the introduction of Pythonic resources in Dagster, you can define resources using Pydantic models. Here's how you can update the example to use Pydantic resources: First, you'll need to define a Pydantic model for your resource configuration:
Copy code
python from pydantic import BaseModel from dagster import asset, build_asset_context from dagster_pydantic import PydanticConfigurableResource  # Define a Pydantic model for the API client configuration class APIClientConfig(BaseModel):     api_key: str  # Define a resource for the production API client using the Pydantic model class APIClientResource(PydanticConfigurableResource[APIClientConfig]):     def __init__(self, config: APIClientConfig):         # Replace 'ExternalAPIClient' with your actual client class         self.client = ExternalAPIClient(config.api_key)  # Define a mock resource for testing class MockAPIClientResource(PydanticConfigurableResource):     def __init__(self):         # Replace 'MockExternalAPIClient' with your mock client class         self.client = MockExternalAPIClient()  # Define an asset that uses the API client resource @asset(required_resource_keys={"api_client"}) def my_asset(context):     api_client = context.resources.api_client.client     # Use the API client to interact with the external API     data = api_client.get_data()     return data  # Example test that materializes the asset using the mocked resource def test_my_asset():     with build_asset_context(resources={"api_client": MockAPIClientResource()}) as context:         result = my_asset(context=context)         # Perform assertions on 'result'
In this updated example,
APIClientConfig
is a Pydantic model that defines the configuration schema for the API client resource.
APIClientResource
and
MockAPIClientResource
are resource classes that inherit from
PydanticConfigurableResource
, with
APIClientResource
taking an instance of
APIClientConfig
as its configuration. When you materialize the asset in your tests, you instantiate
MockAPIClientResource
directly without any configuration, since it's a mock that doesn't require real credentials. Please ensure that you have the
dagster_pydantic
package installed to use
PydanticConfigurableResource
. If it's not already installed, you can add it to your project with
pip install dagster_pydantic
. Remember to replace
ExternalAPIClient
and
MockExternalAPIClient
with the actual implementations that correspond to your production and mock API clients, respectively.
2 Views