https://dagster.io/ logo
#announcements
Title
# announcements
r

Richard Whitefoot

01/19/2021, 4:17 PM
Hey! How would I add configuration to a sensor, like with a solid please?
p

prha

01/19/2021, 4:20 PM
Sensors are evaluated outside of a pipeline context, so they aren’t configurable in the same way solids are. Would love to hear more about what your use case is though… always trying to find ways to make the API better.
r

Richard Whitefoot

01/19/2021, 4:21 PM
Thanks, essentially I would like a SSH resource configured for different environments that checks whether new files are in a remote folder then triggers a download
p

prha

01/19/2021, 4:26 PM
Ah, okay. When would you specify which environment you’d select for the sensor to evaluate? At definition time, or when you turn the sensor on/off, or …?
r

Richard Whitefoot

01/19/2021, 4:26 PM
At definition time
p

prha

01/19/2021, 4:28 PM
Makes sense. I think we might be able to write some helper functions in the
dagster-aws
library to help with some of this boilerplate, but it wouldn’t be a resource, per-se
Would need to make sure it wouldn’t get confused with the current S3 resource which is used and configured for solids
r

Richard Whitefoot

01/19/2021, 4:31 PM
Oh, to confirm - it's SSH
p

prha

01/19/2021, 4:32 PM
Ah right, you did say that… sorry about that!
r

Richard Whitefoot

01/19/2021, 4:33 PM
No worries! Thanks for your help
p

prha

01/19/2021, 4:36 PM
Should also point out that the
@sensor
decorator is a wrapper around
SensorDefinition
. You could write your own factory method
my_sensor_factory
that takes in a mode and produces the
SensorDefinition
that you want. It may unblock you in the time being.
🙇 1
@Richard Whitefoot Also, might be worth mentioning, when writing sensors, I’ve found that having factories and configuration wrappers tend to be less reusable than having good helper methods… Would something like this work for you?
Copy code
def get_files_for_remote_directory(remote_directory):
    # connect and list remote directory

def sensor_for_mode(mode):
    if mode === 'prod':
        sensor_name = "my_prod_sensor"
        remote_directory_name = 'my_prod_directory'
    else:
        sensor_name = "my_dev_sensor"
        remote_directory_name = 'my_dev_directory'

    @sensor(name=sensor_name, pipeline_name="my_pipeline", mode=mode)
    def my_sensor_definition(_):
        files = get_files_for_remote_directory(remote_directory_name)
        for filename in files:
            yield RunRequest(run_key=filename, run_config={})

    return my_sensor_definition
r

Richard Whitefoot

01/19/2021, 7:19 PM
Not quite clear on the pattern. Is get_files_for_remote_directory() just a helper method, so won't have access to the resources config?
3 Views