i suspect the vast,vast majority of users are on pre-1.10.3 revs

i believe this is because of a transitive Jinja dep

so it can be solved that way as well, probably

(i got a notif from another repo)

Hi everyone, I'm trying to run an uncontainerized pipeline in airflow using the docs: https://dagster.readthedocs.io/en/0.5.7/sections/deploying/airflow.html. In the example DAG file there is the note:

you must ensure that dagster_examples.toys.sleepy is installed or available on sys.path

. How do I install

dagster_examples.toys.sleepy

?

I have '/home/ubuntu/dagster/examples' in my sys path

Nevermind, I restarted the airflow webserver and now it seems to be working 🙂

👍 let us know if you hit any other issues with the Airflow integration! are you planning on running some Dagster pipelines on Airflow?

Hey 👋 I’d love to run Dagster in a Kubernetized context on GCP using Cloud Composer. I understand there’s quite a bit on the roadmap to get that to happen, I’m generally curious of the priority surrounding things like supporting GCS for dagster-airflow storage as well as being able to create GKEPodOperator Airflow tasks from Dagster?

Or maybe even how do you see it being implemented eventually? Will you prefer to stay more general and have it use DagsterKubernetesPodOperator or somehow allow for the creation and use of a DagsterGKEPodOperator if using GCP?

hey Zachary! that’s awesome. we haven’t done any work w/ Cloud Composer specifically yet, but I don’t see any reason why our

dagster-airflow

integration wouldn’t work on CC. For other GCP integrations, we’ve done some early work w/ BQ and Dataproc in a library here: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-gcp - generally we’re trying to build an ecosystem of modular libraries that work on top of Dagster for things like this. So, I think either

dagster-airflow

or

dagster-gcp

would be the right place to experiment with a

DagsterGKEPodOperator

, but this isn’t something we’ve explicitly worked on yet!

Hey all! I'm getting this weird errors while running my pipelines on airflow. The first error I receive is the weakref error to a temp file that is not on my system. I was wondering if this is caused by the storage environment configuration setting but am not sure how to debug it.

I’m experiencing some flakiness in my pipelines while running on airflow

hmm, can you provide some more details on what exactly you are observing?

A lot of my tasks are randomly/passing and failing that would normally pass when executed. It takes several tries to get a pipeline to run successfully. I am seeing error outputs relating duplicate run ids

interesting - can you send over some gists of the errors you are seeing (scrubbed or sent privately)?

Sure but outside of the error id, I’m not seeing much logging from my dagster tasks. I’ve tried running on airflow version 1.10.5, 1.10.6, and 1.10.7…https://gist.github.com/amandajcrawford/1a70923fe1903b0d0a825a0dbe357d47

This is one common output but most of my failed tasks don’t produce any output info

alright interesting - thanks for sending this over @max and I will take a look

do you have

$DAGSTER_HOME

set on the airflow machine?

or have you configured the dagster instance at all?

No, I don’t have that set on my docker instance, should I? What should I set that too?

I don’t seem to have the value set on my local machine either

for context

so whats happening is, we are sharing a temp directory that all the dagster python operators are going to share for storing things

this is desired for passing data from task to task

but it seems that the shared sqlite runs database is suffering from race conditions - you can see

get_or_create

in the stack trace - it seems that multiple operators are hitting that at roughly the same time and trying to write the run to the DB after first querying and not finding an entry for that ID

if you set up a

DagsterInstance

which uses for example postgres for runs/events - than you could see the dagster artifacts from the airflow runs via dagit