I have a n00b question regarding ops and functions. The docs suggest splitting your code up into

@ops

over regular python functions and I understand the reasoning. However is it still possible to interweave regular python functions with

@ops

when it comes to small helper functions?

Yes absolutely. I've got a bunch of little helpers in a lot of my jobs

@Ben Gatewood Thanks. How do you determine what should be an

@op

and what is a regular python function? Is there a general rule?

For me an op represents some significant part of the pipeline or an artefact thereof. I split things like formatters/convertors out into helpers

That helps. So it doesn’t just have to be the assets that get materialized and it doesn’t have to be all

@ops

but just a sensible balance in between where it makes sense?

Yeah pretty much in my view. For instance, retrieving a bunch of raw inputs from an API makes sense to be an op to me whereas the little function that generates the appropriate query params for the API based on a list of dates or similar doesn't need to be

And can you pass in-memory assets between

@ops

the same as functions? Like a list obtained from an API which isn’t yet materialized?

You can pass basically anything as the output of one op into a another one

Awesome, thanks. Do you have any tips for converting existing python scripts to Dagster? I presume it’s better to create a project structure using

dagster project scaffold

and then build up your assets, ops, and functions? It seems like software-defined assets is the favoured approach?

I started before the CLI was available so I built up my first ones from scratch and new ones are cloned from a template based on the earlier ones. There's a docs page on situations that are well suited to SDAs and others that are not so much.....

Yeah it takes a bit of a mental shift defining assets composed of jobs and ops but it makes a lot of sense. “*If assets are known*” seems to be the golden rule coming through there. 🙂

We run in k8s with separate user code repos based (mostly) on official helm charts

Ian one thing I want to make sure is clear is that ops can absolutely call out to other python functions, but everything inside your dagster job does need to happen inside an op. If you put code inside a Dagster job that isn't inside an op, that code will execute when the job is defined, not when you launch a run.

@daniel That’s good to know. So as long as I’m pulling in those functions to my ops then all is good?

That's right - I just wanted to make sure you weren't interleaving ops and non-ops within the DAG itself

I’m completely new to Dagster and still building my understanding so it may well have been something I tried at some point. 😄 Thanks for the heads-up.