Richard Whitefoot
01/19/2021, 4:17 PMprha
01/19/2021, 4:20 PMRichard Whitefoot
01/19/2021, 4:21 PMprha
01/19/2021, 4:26 PMRichard Whitefoot
01/19/2021, 4:26 PMprha
01/19/2021, 4:28 PMdagster-aws
library to help with some of this boilerplate, but it wouldn’t be a resource, per-seprha
01/19/2021, 4:31 PMRichard Whitefoot
01/19/2021, 4:31 PMprha
01/19/2021, 4:32 PMRichard Whitefoot
01/19/2021, 4:33 PMprha
01/19/2021, 4:36 PM@sensor
decorator is a wrapper around SensorDefinition
. You could write your own factory method my_sensor_factory
that takes in a mode and produces the SensorDefinition
that you want. It may unblock you in the time being.prha
01/19/2021, 7:15 PMdef get_files_for_remote_directory(remote_directory):
# connect and list remote directory
def sensor_for_mode(mode):
if mode === 'prod':
sensor_name = "my_prod_sensor"
remote_directory_name = 'my_prod_directory'
else:
sensor_name = "my_dev_sensor"
remote_directory_name = 'my_dev_directory'
@sensor(name=sensor_name, pipeline_name="my_pipeline", mode=mode)
def my_sensor_definition(_):
files = get_files_for_remote_directory(remote_directory_name)
for filename in files:
yield RunRequest(run_key=filename, run_config={})
return my_sensor_definition
Richard Whitefoot
01/19/2021, 7:19 PM