Hello. I started to use dagster and I have a question regarding the

context

argument. Using

context

seems to "break" Python functions, as you have to provide this argument when you call the function directly. On the other hand, this argument is not explicit when you call an op from a graph/job. It's an important buy-in to migrate an existing code base to Dagster. Is there a way around it? Would it be better to use context as an optional keyword argument instead of a positional argument ?

Hi mederic - you’re not required to define a context argument when defining an op. Eg you can do @op def my_op(): …

Yes, but if I do that, I can't use the configuration/resource system properly. It's either "adapt all your code to get more features" or "keep it as is, but you don't get our features".

Hi @Médéric Hurier I'm evaluating dagster as a framework for our business process automations. We have been using out homegrown framework for this for a couple of years, but we have new requirements that dagster can probably help with. I'm about to start migrating existing (python) business processes to dagster. Interestingly, our own framework was broken down into

tasks

and

functions

, where the

functions

are like

ops

and take a single

context

parameter. Because of this similarity, I'm hoping that the migration process will be relatively easy... In your case, where your existing code is quite different, maybe it would make sense to do a gradual approach where you write dasgter

ops

which interact with the context, getting the logger and relevant resources and then call into your existing functionality - i.e. the

ops

would be thin wrappers which conform your existing code, like shims.

Thanks for your feedback @Stefan Adelbert. Indeed, I think it makes more sense to clearly separate business functions from dagster ops. Even if dagster ops behaves 90% likes plain Python functions, they have subtle differences that could generate conflicts.

@Médéric Hurier - is there a way you envision ops could work that would allow them to work more like plain Python functions?

I see 3 possibilities: 1) pass the context object as an optional keyword argument to limit refactoring (e.g., def f(a, b, context=None)) 2) get the context object inside the function body (like in flask) and the new dagster logging 0.13.0 or 3) prevent the use of context objects inside ops altogether, and rely either on 3.1) parameters to pass config arguments (like prefect) or 3.2) let jobs manage configurations as this is the only object launched with a config in the end.

Thanks for the detailed response - that's helpful. The way we've been thinking about it, there are two situations that any particular op can be in: • The body of the op uses configuration and/or resources. In this case, the op needs a context argument, because that's the way for it to access them (without doing flask-style magic). • The body of the op does not use configuration or resources. In this case, the ops doesn't need a context argument. When you talk about context as an optional keyword argument, are you envisioning a middle-ground where some ops will use configuration and/or resources if they're provided, but can get by without them if they're not provided?

@sandy Indeed, the two situations are well explained in the documentation. However, there is no easy transition from one to the other. On one hand, you have to refactor the code outside jobs (testing, debugging, ...). On the other hand, I find it "magical" to automatically pass the context argument in jobs, whever the ops use a context or not.