https://dagster.io/ logo
#ask-community
Title
# ask-community
c

Chris Nogradi

05/10/2023, 7:58 PM
With the new Pythonic Config system using Pydantic, what is the best way to configure a set of resource/iomanagers from a yaml? Use the old method or use Pydantic from yaml library or does dagster provide its own method?
Maybe an example somewhere to look at would be good. Thanks!
dagster bot not a thread 1
s

sean

05/12/2023, 6:46 PM
Hi Chris, you can still pass
run_config
in the old format (directly loaded from YAML). This is unlikely to go away anytime soon because this is also how config needs to come in from non-Python APIs (CLI, dagit)
❤️ 1
m

Megan Beckett

06/05/2023, 10:03 AM
Hi, Im also interested in this. Looking at migrating my legacy resources to the new pythonic resources as I specifically want to use resources in sensors to query a database. However, the way I currently do it, is provide the connection details for the config for a resource at run time in a schedule using a yaml file. Do you need to add resources to the
Definitions
using this new way? or, just leave them out and still be able to pass in a yaml file to
run_config
in the
RunRequest
in a schedule or sensor?
@Chris Nogradi Did you find any examples of this that were helpful?
c

Chris Nogradi

06/21/2023, 3:54 PM
@Megan Beckett I am sorry I missed your question. I have not yet coded this fully but yes it would go along with the sensor. Here is something to look at:
def make__sensor(sensor_job, tables_partitions_def, prefix=None, run_config=None) -> SensorDefinition:
name = tables_partitions_def.name.replace('-','_')
config = {"ops": {f'{name}': {"config": { "file_url": ""} } } }
if run_config:
config = deep_merge_dicts(config, run_config)
url_config = config['ops'][f'{name}']['config']
@sensor(job=sensor_job, name=f'{name}_sensor')
def sensor(context: SensorEvaluationContext,
sensor_resource: SensorResource):
...
run_requests = [RunRequest(... run_config=update_file_url(config, url_config,f'construct_url_w_{key}')) for key in keys]
...
return SensorResult(run_requests=run_requests, dynamic_partitions_requests=[tables_partitions_def.build_add_request(new_keys)])
return sensor
3 Views