Yeah we need a context aware file object type

i wonder if it really needs to be context-aware. we could pass a Filelike (binary stream) around, and let the intermediates manager do all the writing to/from disk, and then also expose like, context.storage, which could tell you what it was (context.storage.storage_mode) and also give you Paths (context.storage.put_object() -> Path)

context.storage.url_for_object(paths=['foo', 'bar'], protocol='s3a://')

following this line of thought, stuff like

skip_if_present

is really a property of certain kinds of object stores

it's to do with memoization/reexecution

sharing here, I just went through this https://medium.com/@srivathsankr7/apache-airflow-a-practical-guide-5164ff19d18b to set up airflow locally w/ mysql/celery/rabbitmq, seems to be working fine for me (and permits individual task execution)

eh, trickier than i thought at first to switch the DagsterOperator seamlessly from operating containerized to uncontainerized

since we're using the container to enforce implicitly that there be an obvious repository.yml etc

i think we should stay focused on the uncontainerized locally, containerized in the cluster use case

regardless, the way we're scaffolding the DagsterOperator needs to change to support an uncontainerized case

since we're not specifying a repository or pipeline in code right now

ah

i think we are talking about different things

by executing locally do you mean executing airflow locally, or executing the pipeline within dagster/dagit locally

i mean executing the pipeline outside of a container

nothing we've written makes assumptions about where airflow is executing

call starting

i gtg

@nate @max just checked out 1097

it’s sweet but has interesting tradeoffs

dynamically generating the dag is more elegant but has some risk

we could easily codegen and output to a scaffolded file somewhere also

or indeed provide a switch for dynamic generation/scaffolding in both the PythonOperator and the DockerOperator-based approaches

yeah I’m just wondering if we can consolidate those codepaths considerably

e.g. what is the minimal viable artifact we need to generate in order to load the containerized case in a reliable fashion

e.g. can we just codegen a serialized format of the execution plan

and then dynamically construct the airflow DAG, which is safe

it would reduce the amount of codegen considerably

that solves the performance issue, but we lose the human factors of heavy codegen

?