https://dagster.io/ logo
Powered by
e

esztermarton

03/19/2021, 7:08 PM
Hi all 👋 I'm attempting to deepcopy a ModeDefinition object. It's giving me an error. Am attaching the code snippet - would be great to understand - is it wrong to be doing this? I want to do it so that I can have a default mode defined, and allow pipelines to customise this (without mutating the original object).
a

alex

03/19/2021, 8:38 PM
hm im not sure what exactly is going wrong there with the deepcopy - i am guessing something deep in the config schema system
what are you trying to add to the mode copies? new resources?
e

esztermarton

03/19/2021, 8:39 PM
exactly
and maybe in the future i'll want to remove resources too 🤔
a

alex

03/19/2021, 8:40 PM
ModeDefinition
is just a 
namedtuple
with some top level 
list
and 
dicts
as long as you copy those over shouldn’t have to worry about going any deeper
e

esztermarton

03/19/2021, 8:40 PM
i imagine the rest won't change very frequently
Yeah, I did that as a workaround - I wrote util function - this look ok to you?
Copy code
def _copy_mode_definition(mode: ModeDefinition):
    """
    Creates a copy of the mode definition.
    """
    return ModeDefinition(
        name=mode.name,
        resource_defs=copy(mode.resource_defs),
        executor_defs=copy(mode.executor_defs),
        logger_defs=copy(mode.loggers),
        intermediate_storage_defs=copy(mode.intermediate_storage_defs),
        description=mode.description,
    )
a

alex

03/19/2021, 8:45 PM
yep - nice i was just writing something similar
Copy code
def adjust_resources(
    mode: ModeDefinition,
    new_resources: Dict[str, ResourceDefinition], 
):
    resource_defs = dict(mode.resource_defs)
    for key, resource in new_resources.items():
        resource_defs[key] = resource
    # _replace is a tuple method that copies over all other properties
    return mode._replace(resource_defs=resource_defs)
e

esztermarton

03/19/2021, 8:47 PM
Beautiful 👌 Thank you Alex
a

alex

03/19/2021, 8:47 PM
the other thing to consider is refactoring to late bind the mode
so instead of working from a base mode that you make copies of - you work form a resource_defs dictionary that you bind into a mode only once you attach it to pipeline def
🤔 1
👍 1
or something like that
e

esztermarton

03/19/2021, 8:51 PM
Thanks, I'll ponder on that too to see if that's nice in our setup. Just trying to keep it as simple as possible 🙂
2 Views
Powered by