https://dagster.io/ logo
Title
c

Charles Lariviere

11/29/2021, 4:45 PM
Hey 👋 I’m looking for the most dagster-y way to have an op that takes either a config or an input. I’m writing a reusable
create_database_schema()
op that I would like to either accept a
name
config or input from another upstream op. I can make them both optional and raise an error if both are
None
, but curious if there’s a better construct in the Dagster framework for something like this?
m

max

11/29/2021, 5:04 PM
Hi Charles! As you can imagine we have gone back and forth on this quite a bit. What API would feel most intuitive here?
c

Charles Lariviere

11/29/2021, 5:48 PM
Hey Max! Good question — I get why inputs and config were created as 2 different concepts, but for this particular scenario, it feels like having a single construct for inputs (that can be configured) would help. I don’t know that it’s the best generalized solution though 🤔 For instance, I would like to define this op as:
@op()
def create_database_schema(context, name: str):
  context.database.create_schema(name)
And then be able to use it either as:
@job
def my_job():
  schema = create_database_schema() # this is defined in the config
  ... = another_op(schema)
Or as:
@job
def my_job():
  get_list_of_schema().map(lambda schema: create_database_schema(schema))
m

max

11/29/2021, 5:59 PM
ah great -- all inputs can be configured
c

Charles Lariviere

11/29/2021, 6:42 PM
Ahh I see! Oops, I didn’t know this was possible — thanks! 🙏
👍 1