Hey all, just a quick question/suggestion on the l...
# announcements
a
Hey all, just a quick question/suggestion on the latest addition to the static type check analysis relating to the source to destination input mappings. I’ve updated my code to use the latest version and am now getting an DagsterInvalidDefinitionError. This error is thrown when I try to pass the input of a composite solid into a nested solid within the composite where the composite solid’s input definition is of a specific DataFrame type but the nested solid is a generic DataFrame type. Since the sub-solid accepts a generic DataFrame type, I believe it should be legal to pass inputs from composite solid to it’s sub solids where a composite solid’s type maybe different from the sub solid and that the check happening just in time. This issue may be happening since I’m using a DataFrame which may need some inference logic to enforce the correct type validation. I could also be thinking about this wrong or not using the correct pattern, any suggestions on how to handle this scenario?
a
Hmm interesting I didnt consider this case when I added the check. The motivation for enforcing this is how
input_hydration_config
and the config schema interacted with the types not aligning through composite mapping.
Do these
DataFrame
s all have the same
input_hydration_config
?
a
No, they don’t and some of them are just regular dataframes
a
Also just to make sure I have the right context, could I get the composite function definition and the nested solid definition in question?
a
The composite definition takes in a ProgramDataFrame and passes it to the solids that either accepts a generic pd.DataFrame or ProgramPartialDataFrame.
a
What is the difference in loading from config that makes it so they cant share input hydration config? My current assessment is that changing the restriction to that those must be the same as opposed to the whole type is reasonable and might be a good path forward here.
a
I’m not sure if I’m understanding so please feel free to let me know. Just a correction, the input_hydration_configs are the same for the types listed above outside of the pd.DataFrame. All the input hydration config does in my case is take in a some parameters to get a csv file and convert it to a dataframe. Checking to see if the input_hydration_config are the same may work to solve this issue but it may not cover types that don’t have an input_hydration_config set but is in fact the same type using a different name
a
hmm true true. Thanks for reporting - this has sparked a lot of good discussion in the office this morning about what to do here. Complex issues.
A workaround in the short term if you want to be able to upgrade would be to introduce a “pass through” solid in the composite since we dont have any restrictions on solid to solid dependencies
👍🏿 1
Copy code
@composite_solid
def composite(df: SpecialDataFrame):
    general(df)
    partial(df)
would become
Copy code
@composite_solid
def composite(df: SpecialDataFrame):
    hax = pass_through(df) # solid just returns input
    general(hax)
    partial(hax)
👍🏿 1