03/29/2019, 9:43 PM
so, i figured out why the dockerized airflow airline-demo tests stubbornly refuse to pass, and it's a basic misconception baked into the airline demo, which was written for local execution. in this context, and in a project-oriented workflow, it felt natural to me to operate on the file system and to pass paths around from solid to solid. (we even wrote a fancy FileExistsAtPath type that does a runtime os.isfile check). this obviously breaks when we are operating dockerized and blowing the file system away between each solid. i'm flagging this because it's not clear to me that we are going to be able to get our users not to use the filesystem. i also think we are probably going to want to expose an object store in the pipeline execution context.