https://dagster.io/ logo
Title
p

Peter B

06/17/2021, 5:51 AM
Hi all, how do we use part of our Solid Input definition into our Output definition? The output definition has no context to draw from what I understand?
o

owen

06/17/2021, 4:11 PM
Hi @Peter B! metadata for OutputDefinitions is intended to be constant (i.e. not dependent on runtime things), so you're not able to use it in this way unfortunately. Just based off of how this is structured, I'm guessing you're structuring it this way so that your output can be managed by an IOManager? If so, I think there are a few different options for you depending on the specifics of your pipeline
If you don't expect the solid to change which table it's writing to once it's used in a pipeline (and so you're just writing it this way so that it can be reused in different pipelines for different tables), you might want to try out the solid factory pattern (https://docs.dagster.io/concepts/solids-pipelines/solids#solid-factory), which would allow you to "bake in" the metadata for that output, instead of getting it from the inputs. However, that may not work for you depending on what logic is actually going on in the solid
The other option would be to bundle more information into the output itself by creating some sort of wrapper object (maybe a namedtuple that contains the data you're currently putting in the output + fields target_schema/target_table) and returning that instead. This would give the IOManager enough information to figure out where to put that data
p

Peter B

06/18/2021, 4:19 AM
Hi Owen, that's perfect! Thanks so much for your help. Yes, it was just to use a custom IO_Manager for prod vs. fs_io for local. But, in future we will also want to set it as input parameters that translate to static outputs that vary, so that we can re-use the same code but point the different departments to different schemas/tables yet in the same format. Thanks!!
o

owen

06/18/2021, 4:30 AM
happy to help 😄