Hey group. A friend of mine referred me to dagster and I found it interesting. Currently, we use airflow to schedule work on kubernetes cluster via kubernetes job api. The feature that dagster appeals to me the most is its airflow integration and an web ui editor that allows you to write dags interactively. Being able to develop dags interactively has been a shortcoming for airflow imo. Here is my question: what's it like to develop dags with dagster and deploy the dag to a remote airflow instance? Skimming through the documentation, I thought a user could write a dag on dagster-git, deploy dag to a repo, then have airflow dagster plugin pull from that repo. Am I close?

Hi @yx! I think that makes sense. The way that our Airflow integration is currently conceived is that 1) you first write pipelines in Dagster 2) then you write a little bit of code in a DAG definition (e.g. https://github.com/dagster-io/dagster/tree/master/python_modules/dagster-airflow#running-uncontainerized-dagsterpythonoperator) that imports the Dagster pipeline and wraps it in an Airflow DAG.

It does make sense, although I was scratching my head when reading

dag, steps

returned from

make_airflow_dag

, I assume airflow would have just scanned the dag with all the tasks (or

steps

in that example) properly included in that dag.

the

dag

is the Airflow DAG that contains all of the tasks -- we could maybe adjust

make_airflow_dag

so that it doesn't return the

steps

as well

Good to hear that you are also considering working on a kubernetes oriented integration with Airflow!

😄 We would love your input on what kind of Kubernetes integration would be the most helpful!

@max I have been cooking one idea related to kubernetes which may interest you. Without me knowing how much of the project scope you are planning nor having read the dagster's source code, I would love to see the user be able to specify compute resource for a step, such as:

@kube(cpu=8, mem=250g, gpu=1)
@lambda_solid(...)
def compute()

this is a neat feature from netflix's metaflow () You could leverage taints in kubernetes for advanced scheduling (https://kubernetes.io/blog/2017/03/advanced-scheduling-in-kubernetes/)

Very interesting! My reflex is that we might want to keep hints to the compute substrates out of the code, and instead specify them in config -- so that you could run the same solid code in local test, or containerized in Airflow, or even perhaps directly on Kubernetes, and just change config blocks to tell the engines what constraints to impose on solid execution. Does that feel burdensome or unnatural?

Yes, config sounds good. It looks like dagster's configuration is the right place for that (because it's really just pipeline metadata).