https://dagster.io/ logo
Title
a

Alexis

04/16/2021, 10:50 AM
Hi everyone, I have a repository with 2 pipelines using the same solids and I’m getting the “Duplicate solids found in repository with name ‘name’. Solid definition names must be unique within a repository” message. The obvious fix would be to give each solid an alias with a prefix for corresponding pipeline or something like that but I’m wondering if there isn’t a better way to do this ? Maybe at the pipeline level, so all solids within the pipeline get a prefixed name or end up unique ? Thanks for your help !
I don’t get why if you din’t specify an explicit solid name, it will take the method name and there will be no duplicate message but when you give it an explicit name, it will clash between pipelines ?
a

alex

04/16/2021, 2:24 PM
so there is a difference between solid definitions and solid invocations invocations of the same solid will get aliased automatically we currently don’t allow duplicate definitions to allow for unique identifications of the solids that are invoked in potentially many places within a repository
👍 1
using the same solids
if you are attempting to use the same solid definition - you just need to make sure you are using the same definition object instance
a

Alexis

04/16/2021, 2:48 PM
Yes I’m currently calling the same solid in both pipelines so what you’re saying is if I were to remove the “name” argument it would work because each invocation would get aliased automatically otherwise I would have to use a
dagster.SolidDefinition
instead of a
@dagster.solid
? How would one do that ?
a

alex

04/16/2021, 4:21 PM
@solid
makes a
SolidDefinition
f I were to remove the “name” argument
what are you passing name to in this context?
@solid
? Might be useful to send a small code snippet
a

Alexis

04/16/2021, 4:56 PM
class SimpleProcessor(BaseProcessor, ABC):
    @classmethod
    def definition(cls, name: Optional[str] = None) -> SolidDefinition:
        return solid(
            name=name if name is not None else cls.info.name,
            description=cls.info.description,
            input_defs=cls._input_defs,
            output_defs=cls._output_defs,
            config_schema=cls._config_schema,
        )(cls._algorithm)
And then this is called this way in two different pipelines:
result = SimpleProcessor.definition(
        "signal_filter"
    )(signal=signal)
Yes I’m passing
name
to solid and then calling the resulting
SolidDefinition
with an argument once in the first pipeline and once in the second pipeline, do I have to give them different names for them to be in the same repository ? Thanks I really appreciate your help
a

alex

04/16/2021, 5:46 PM
ya giving them different names will work and will be the easiest fix if everything is actually the same between these two SolidDefinitions - you just need to make this function return the same instance of the object ie memoizing manually or using something like
@lru_cache
.
example of manual memoization for clarity
class SimpleProcessor(BaseProcessor, ABC):
    @classmethod
    def definition(cls, name: Optional[str] = None) -> SolidDefinition:
        if name not in cls._cached_solids:
            cls._cached_solids[name] = solid(
                name=name if name is not None else cls.info.name,
                description=cls.info.description,
                input_defs=cls._input_defs,
                output_defs=cls._output_defs,
                config_schema=cls._config_schema,
            )(cls._algorithm)
        return cls._cached_solids[name]
a

Alexis

04/16/2021, 7:24 PM
Ah great ! thanks for the memoization tip 👍