https://dagster.io/ logo
#announcements
Title
# announcements
d

Deveshi

04/01/2021, 3:10 PM
Hi, I have probably a very silly question related to sensors. I can run my pipeline with a preset config using this command dagster pipeline execute -f pipeline_script.py --preset default I have a sensor (a copy of my_s3_sensor from dagster examples) to monitor files in the bucket. How do I set parameters for my run_config in the sensor to call this pipeline ( say example_pipeline). In the dagster examples, the run_config parameter includes solids. I am a bit confused here as I only need to run my pipeline , since all the solid dependencies are defined there. Also, it says Expected: "['execution', 'intermediate_storage', 'loggers', 'resources', 'solids', 'storage']." for config entries. Thanks in advance!
a

alex

04/01/2021, 3:14 PM
you can reference your presets
run_config
from the sensor when you return the
RunRequest
this example should be sort of  relevant https://docs.dagster.io/concepts/partitions-schedules-sensors/schedules#using-a-preset-in-a-schedule-definition
the ergonomics here are a bit awkward and something we are working towards improving
d

Deveshi

04/01/2021, 3:23 PM
Thank you @alex, I will give it a try
Hi @alex another question, I am not using a run_config as such from my preset, I am using a yaml file via this option config_files=["resources.yaml"]. Is there a way to utilize this in RunRequest instead?
a

alex

04/01/2021, 4:16 PM
ya so you are using that to create a
PresetDefinition
right? The idea here is to be able to reference that
PresetDefinition
instance from your run request. If you are defining the object directly on your pipeline you’ll have to move it so you can reference it from both places
d

Deveshi

04/01/2021, 4:16 PM
This is what it looks like
default = ModeDefinition(
name="default",
resource_defs={"s3": s3_resource_for_profile, "redshift": redshift_resource})
#main pipeline
@pipeline(mode_defs=[default],
preset_defs =[
PresetDefinition.from_files(
name="default",
mode="default",
config_files=["resources.yaml"]
)
]
)
def example_pipeline():
a

alex

04/01/2021, 4:17 PM
yep so the same you way you moved
ModeDefinition
you’ll have to move
PresetDefinition
so you can reference it and call
run_config
on the instance
d

Deveshi

04/01/2021, 4:21 PM
ah got it! thanks!