https://dagster.io/ logo
Powered by
#ask-community
Title
# ask-community
q

Quy

03/12/2022, 12:07 PM
Hi, I was wondering if there is a way to have config for a job is a mix between 
config_from_files
, and 
partitioned_config
to illustrate
Copy code
config=[
    partitioned_config,
    config_from_files(
        [file_relative_path(__file__, os.path.join("..", "..", "run_config", "pipeline.yaml"))]
    )
]
in pipeline.yaml, I want to config resource 
io_manager
e.g: 
s3_bucket
…
❤️ 1
j

johann

03/14/2022, 11:20 PM
Hi @Quy, are you looking to merge the contents of 
pipeline.yaml
with each output of the partitioned config?
If you look at the definition for 
config_from_files
, it just reads the files and returns the parsed yaml. In your partition definition function, you could merge the result of 
file_relative_path(__file__, os.path.join("..", "..", "run_config", "pipeline.yaml"))
with whatever you’re currently returning
q

Quy

03/15/2022, 12:26 AM
@johann I want to merge my resource configs and partition configs. In my case, my resource configs is on 
pipeline.yaml
file
i tried this,
Copy code
config=deep_merge_dicts(
        partitioned_config(),
        config_from_files(
            [file_relative_path(__file__, os.path.join("..", "..", "run_config", "pipeline.yaml"))]
        )),
but got error
Copy code
Failure condition: config param must be a ConfigMapping, a PartitionedConfig, or a dictionary, but is an object of type
j

johann

03/15/2022, 12:34 AM
Do you have a partition config function? Something like
Copy code
@daily_partitioned_config(start_date=datetime(2020, 1, 1))
def my_partitioned_config(start: datetime, _end: datetime):
    ...
My suggestion would be do the merge inside that function
q

Quy

03/15/2022, 12:38 AM
yes, I did used a a partition config function. is there anyway to separate of business concerns by merge configs in a higher level?
j

johann

03/15/2022, 12:43 AM
Hmm, I don’t think so currently. cc @prha to confirm
p

prha

03/15/2022, 12:46 AM
No, there’s not. We’ve talked about having some utility functions to integrate multi-dimensional partition schemes into dagit / backfills / cli, but right now we force everything to resolve to a full run config, so that we don’t need to figure out config resolution / merging.
q

Quy

03/15/2022, 1:28 AM
Thanks guys. I really wish dagster would support it
Powered by