https://dagster.io/ logo
Powered by
y

Yan

04/07/2021, 10:35 AM
Hi Is there any way to combine preset and run config? I want to build pex zip of my pipeline with preset config yamls inside. And then do spark submit of this zip, but would like to override some values defined in that yamls. If i use pipeline execute cli i am getting error that preset and config cannot be specified at the same time. Thank you.
j

Jeff Hulbert

04/07/2021, 1:15 PM
You can copy the preset to a dict with
Copy code
config = prod_preset.run_config.copy()
then modify it or add keys as needed and use "config" as your new run config.
y

Yan

04/07/2021, 1:37 PM
It seems that execute pipeline api requires preset to be passed and it should have the same run config
j

Jeff Hulbert

04/07/2021, 1:38 PM
did you try to pass the mode and then the modified run config?
if you used the code above you've already copied the preset into the run config so you don't need the preset when you run execute_pipeline
y

Yan

04/07/2021, 2:06 PM
This api requires to pass preset It should be one of presets defined for pipeline If i pass mothing it throws exception
j

Jeff Hulbert

04/07/2021, 2:07 PM
This is how I call it in a bunch of my tests
Copy code
result = execute_pipeline(
            my_solid,
            mode="local",
            run_config=config_dict,
        )
y

Yan

04/07/2021, 2:10 PM
Do you have presets specified in my_solid?
j

Jeff Hulbert

04/07/2021, 2:11 PM
yes
y

Yan

04/07/2021, 2:19 PM
Yes it works Thanks a lot!
👍 1
4 Views
Powered by