hello all! We are trying to use Azure ADLS2 as a resource for an OP and Sensor. But we already struggle with getting ADLS2 running (

adls2_resource

) with the example in the documentation: https://docs.dagster.io/_apidocs/libraries/dagster-azure

from dagster import job, op
from dagster_azure.adls2 import adls2_resource

@op(required_resource_keys={'adls2'})
def example_adls2_op(context):
    return list(context.resources.adls2.adls2_client.list_file_systems())

@job(resource_defs={"adls2": adls2_resource})
def my_job():
    example_adls2_op()

We want to pass the config as YAML, as it is described in the docu:

resources:
  adls2:
    config:
      storage_account: my_storage_account
      # str: The storage account name.
      credential:
        sas: my_sas_token
        # str: the SAS token for the account.
        key:
          env: AZURE_DATA_LAKE_STORAGE_KEY
        # str: The shared access key for the account.
        DefaultAzureCredential: {}
        # dict: The keyword arguments used for DefaultAzureCredential
        # or leave the object empty for no arguments
        DefaultAzureCredential:
            exclude_environment_credential: true

But how do we pass this YAML to dagster? Because in dagit, we always get this error:

KeyError: 'adls2'

When we pass the config directly in the OP decorator, it works. But it doesn't solve our problem because ultimately we want to use the ADLS2 Resource in a Sensor - which doesn't take the the config via the decorator...

hi @David Weber! to use resources in sensors in general, you'll want to use the build_resources API (https://docs.dagster.io/concepts/partitions-schedules-sensors/sensors#using-resources-in-sensors). Unfortunately, these examples don't actually show how to add config blobl grimace to do that, you'll want to pass

{"adls2": adls2_resource.configured(<config>)}

into build_resources, where

<config>

is the python dictionary representation of everything under

resources:
  adls2:
    config:

i.e.:

{
    "storage_account": "my_storage_account",
    "credential": {
        "sas": "my_sas_token",
        "key": {
            "env": "AZURE_DATA_LAKE_STORAGE_KEY"
        },
        "DefaultAzureCredential": {
            "exclude_environment_credential": True
        }
    }
}

Hi @owen, thanks for your reply! We discovered this fortunately yesterday. However, we found the documentation very much confusing and lacking this exact information.