Hey! How would I add configuration to a sensor, li...
# announcements
r
Hey! How would I add configuration to a sensor, like with a solid please?
p
Sensors are evaluated outside of a pipeline context, so they aren’t configurable in the same way solids are. Would love to hear more about what your use case is though… always trying to find ways to make the API better.
r
Thanks, essentially I would like a SSH resource configured for different environments that checks whether new files are in a remote folder then triggers a download
p
Ah, okay. When would you specify which environment you’d select for the sensor to evaluate? At definition time, or when you turn the sensor on/off, or …?
r
At definition time
p
Makes sense. I think we might be able to write some helper functions in the
dagster-aws
library to help with some of this boilerplate, but it wouldn’t be a resource, per-se
Would need to make sure it wouldn’t get confused with the current S3 resource which is used and configured for solids
r
Oh, to confirm - it's SSH
p
Ah right, you did say that… sorry about that!
r
No worries! Thanks for your help
p
Should also point out that the
@sensor
decorator is a wrapper around
SensorDefinition
. You could write your own factory method
my_sensor_factory
that takes in a mode and produces the
SensorDefinition
that you want. It may unblock you in the time being.
🙇 1
@Richard Whitefoot Also, might be worth mentioning, when writing sensors, I’ve found that having factories and configuration wrappers tend to be less reusable than having good helper methods… Would something like this work for you?
Copy code
def get_files_for_remote_directory(remote_directory):
    # connect and list remote directory

def sensor_for_mode(mode):
    if mode === 'prod':
        sensor_name = "my_prod_sensor"
        remote_directory_name = 'my_prod_directory'
    else:
        sensor_name = "my_dev_sensor"
        remote_directory_name = 'my_dev_directory'

    @sensor(name=sensor_name, pipeline_name="my_pipeline", mode=mode)
    def my_sensor_definition(_):
        files = get_files_for_remote_directory(remote_directory_name)
        for filename in files:
            yield RunRequest(run_key=filename, run_config={})

    return my_sensor_definition
r
Not quite clear on the pattern. Is get_files_for_remote_directory() just a helper method, so won't have access to the resources config?