you can do it one by one but it's painful. you have to manually clear state from the failed operator instance

and then re-run just that instance

yeah that doesn’t sound reliable enough to make it the default process

nate can you shed any color on the impetus was behind "in dev, all operators are a no-op" -- just to test dag construction / scheduling?

yes, so job testing and airflow testing were two separate processes

and the latter was just to ensure that the DAGs were valid, you didn't introduce cycles, etc.

the Slack data eng team built something better for this, they have an automated testing tool to inspect your airflow code and fail loudly during CI/CD if something's wrong

but I don't think most people are that sophisticated

I know Ananth well, we should grab time with him at some point

a ton of our airflow jobs used versions of this file: https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py

and this is what hooks look like https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/gcp_dataproc_hook.py

ananth's talk is truly amazing. they wrote their own layer on top of airflow to surface cross-dag dependencies

yah, he’s awesome

it seems total scattershot which operators make it into airflow-core vs. contrib: https://github.com/apache/airflow/tree/master/airflow/operators https://github.com/apache/airflow/tree/master/airflow/contrib/operators

so very interesting conversation with james. he’s interested in working on this. Making a dagster-friendly k8s executor (e.g. making it easy for our containers to delegate compute to a pod in k8s) is of interest to him, and he thinks it could very useful to community, as the built in k8s operator has a bunch of problems. He is speaking at a conf in April as welll (April 30), so we agreed to reconnect then. Given this conversation I’m definitely leaning towards contracting him, especially that he was able to identity a self-contained problem that he could execute on independently.

great, i think a k8s operator is a really natural next step

i could try to push for an earlier engagement if you think we are ready

but my thought is that we are still working on fairly fundamental things in our integration strategy

also he was a very intrigued by the airflow-ception idea

awesome. the two of you probably have a better sense than I do of how self-contained something like that would be, but more firepower seems like a good thing 🙂

actually that's a good solution to our expanding-logs-inline issue ! cc @Ben Gotow

Yeah full screen log view

https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753

In lieu of a growing list of functionality-specific Operators, we believe that there should be a single, bug-free Operator that would be able to execute any arbitrary task

they went hard in the other direction

@nate this article is what pushed us to go Dockerized-first and prioritize k8s second vs. the approach we talked about at lunch

Great article

so, i figured out why the dockerized airflow airline-demo tests stubbornly refuse to pass, and it's a basic misconception baked into the airline demo, which was written for local execution. in this context, and in a project-oriented workflow, it felt natural to me to operate on the file system and to pass paths around from solid to solid. (we even wrote a fancy FileExistsAtPath type that does a runtime os.isfile check). this obviously breaks when we are operating dockerized and blowing the file system away between each solid. i'm flagging this because it's not clear to me that we are going to be able to get our users not to use the filesystem. i also think we are probably going to want to expose an object store in the pipeline execution context.

if we want to operate seamlessly between S3-backed and locally-backed execution environments, we are going to need to rethink the idea of the Path type too i think

it might be reasonable to operate on some combo of a Filelike and a rich Path