https://dagster.io/ logo
Join Slack
Powered by
Hey all! I'm getting this weird errors while runni...
# integration-airflow
a
Hey all! I'm getting this weird errors while running my pipelines on airflow. The first error I receive is the weakref error to a temp file that is not on my system. I was wondering if this is caused by the storage environment configuration setting but am not sure how to debug it.
a
Can you share a copy of the error you are getting?
a
a
ah interesting - I think that 
KeyError
might be the more interesting of the errors there. Seems like we have a bug where we always expect 
message
but there are cases when it is not present
a
Ah ok!
a
what version of dagster are you using?
a
0.6.4
Should I upgrade to 0.6.5?
a
i don’t think that will fix anything - just gathering information
👍🏿 1
n
yeah AFAIK we didn’t update this stuff in 0.6.5 - agree w/ Alex that 
KeyError
seems like a bug on our end
a
Thanks for taking a look. Would you all be able to provide a fix?
n
yes, we’ll dig into this today
a
Awesome, thanks guys!
a
https://dagster.phacility.com/D1525 - fixes the error you are seeing - though getting a 
PythonError
here indicates you are likely hitting another problem
a
Hmmm once that bug is fixed would it give the ability to view the python error?
a
ya trying to think of best way to avoid so much latency
maybe well publish something akin to a nightly build you can install this afternoon
👍🏿 1
a
Also I ran into another error. When I try to use s3 as my storage enviroment, I get the following error:
a
you need to set the 
system_storage_defs
to include 
s3
as an option - the example linked above demonstrates how
a
Ah ok! I will give it a try
Thanks!
a
No problem!
If you are still stuck on this you can try to grab the nightly build of dagster-graphql to get the python error message https://pypi.org/project/dagster-graphql-nightly/
a
great! Thanks! I'll do that
How do I install the nightly version since dagster looks for dagster_graphql instead of dagster_graphql_nightly?
a
dagster-graphql-nightly will provide the dagster_graphql module
Copy code
(dagenv36) ~/dagster:master$ python
Python 3.6.8 (default, Dec  2 2019, 12:06:17)
[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from dagster_graphql.version import __nightly__
>>> __nightly__
'2019.12.03'
python environment stuff is not my forte either - so i might have tested incorrectly
but I am able to execute dagit in a set up like this: here is a slice from 
pip list
dagit 0.6.5 dagster 0.6.5 dagster-graphql-nightly 2019.12.3
a
Hmm...when I'm getting errors on error since dagster-airflow is looking for dagster-graphql. I testing out 
pip install dagster-graphql[nightly]
a
try 
pip install dagster-graphql-nightly
you may need to explicitly 
pip uninstall dagster-graphql
also
you’re the first person to try to use these nightly so thanks for helping us work the kinks out
plus1 2
🦜 1
I have an interview coming up so @max is here to help as well
m
👋 hi @Amanda Crawford
a
Great! I did the uninstall and install using 
pip install dagster-graphql-nightly
and tried to run and that when I ran to the error message
m
are you hitting an error message in dagit?
a
No, when I run my pipelines on airflow it looks like dagster-airflow is looking for dagster-graphql but can't find it when I install using the nightly build
m
can you paste the error message?
a
Yep, I will post it in a few minutes
Copy code
>>> from  dagster_graphql.version import __nightly__
>>> __nightly__
'nightly-2019.11.28'
Ah I figured it out!
🎊 2
I'm seeing another error '
m
i think storage might not be configured in the config for your pipeline run?
does it have 
storage: s3:
or equivalent?
a
yep
Copy code
storage:
  filesystem:
    config:
      base_dir: /tmp/dagster-airflow/entity_type_pipeline
m
interesting
@prha ^^
we're going to look into the strange SIGTERM that is being generated
but regardless of that, the issue at the top of that stacktrace is that the intermediates aren't being found by downstream steps
which points to a storage issue
but you're using the 
DagsterPythonOperator
hmm
a
can you send over a gist with the full error https://gist.github.com/
a
Sorry! I had to head out.
Good morning! Would someone be open to meeting today on a call to work through this issue?
I went the route of downloading the dagster repo, switching to the nightly branch, and then installing the dagster-graphql and dagster-airflow module
I'm still getting the same error with the latest code. I have successful run on a few other tasks but am seeing errors in the logs from dagster checks. It may be easier to explain over a video chat.
a
awesome, thanks for sending over the gist
I (and likely max) should be available to jump on VC and figure this out
a
Great! I have a cutoff time of 4:30 EST today. What would be the earliest time you all are available for a VC session?
a
I can hop on now but I have ~0 airflow knowledge
I’ll go hop on http://zoom.us/j/2758662566 til about 9:30a PST if you are available
a
Jumping on now
m
@prha want to make sure you saw this gist
a
update for others - think we figured out the issue
a
Ah!!! I'm still back to the original error
I do see more of the output now in the intermediates folder
a
send a gist over when you can
a
a
whats in the logs for 
create_intermediate_entity_table
?
a
a
hmm well the step is failing - not clear why
oop ok its buried in there
a
Interesting, I guess I assumed it was passing again since it's marked as success in Airflow 😲
a
StepFailureData(error=SerializableErrorInfo(message=‘sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused\n\tIs the server running on host “localhost” (127.0.0.1) and accepting\n\tTCP/IP connections on port 5433?\ncould not connect to server: Address not available\n\tIs the server running on host “localhost” (::1) and accepting\n\tTCP/IP connections on port 5433?\n\n(Background on this error at: http://sqlalche.me/e/e3q8)\n’, stack=None, cls_name=None), user_failure_data=None), message=None)
I guess I assumed it was passing again since it’s marked as success in Airflow
ya this is the terrible bug that @nate is fixing
a
Interesting....
Thanks for finding it! I'll figure out why my pipeline is failing to connect
a
👍 no problem good luck dealing docker shenanigans
😄 1
n
yeah @Amanda Crawford I should have a fix for the airflow failed task status out this afternoon
😃 1
a
Ok got my first full green pipeline 🙂 Thanks all!
🦜 2
It's really great to have your support!
n
amazing! partywizard
hey @Amanda Crawford just wanted to let you know that I just landed a fix for the Airflow tasks not being marked as failed here https://dagster.phacility.com/D1543 - this is available on master now, but will also let you know when it gets picked up by a release!
a
Ok great! thanks for the update. Please let me know when it's been added to a release.
a
0.6.6 just went out with the fix
Open in Slack
PreviousNext
Powered by