https://dagster.io/ logo
Powered by
a

Amanda Crawford

12/02/2019, 3:00 PM
Hey all! I'm getting this weird errors while running my pipelines on airflow. The first error I receive is the weakref error to a temp file that is not on my system. I was wondering if this is caused by the storage environment configuration setting but am not sure how to debug it.
a

alex

12/02/2019, 3:12 PM
Can you share a copy of the error you are getting?
a

Amanda Crawford

12/02/2019, 3:53 PM
a

alex

12/02/2019, 4:10 PM
ah interesting - I think that 
KeyError
might be the more interesting of the errors there. Seems like we have a bug where we always expect 
message
but there are cases when it is not present
a

Amanda Crawford

12/02/2019, 4:11 PM
Ah ok!
a

alex

12/02/2019, 4:23 PM
what version of dagster are you using?
a

Amanda Crawford

12/02/2019, 4:25 PM
0.6.4
Should I upgrade to 0.6.5?
a

alex

12/02/2019, 4:27 PM
i don’t think that will fix anything - just gathering information
👍🏿 1
n

nate

12/02/2019, 4:39 PM
yeah AFAIK we didn’t update this stuff in 0.6.5 - agree w/ Alex that 
KeyError
seems like a bug on our end
a

Amanda Crawford

12/02/2019, 4:45 PM
Thanks for taking a look. Would you all be able to provide a fix?
n

nate

12/02/2019, 4:45 PM
yes, we’ll dig into this today
a

Amanda Crawford

12/02/2019, 4:46 PM
Awesome, thanks guys!
a

alex

12/02/2019, 4:49 PM
https://dagster.phacility.com/D1525 - fixes the error you are seeing - though getting a 
PythonError
here indicates you are likely hitting another problem
a

Amanda Crawford

12/02/2019, 4:52 PM
Hmmm once that bug is fixed would it give the ability to view the python error?
a

alex

12/02/2019, 4:57 PM
ya trying to think of best way to avoid so much latency
maybe well publish something akin to a nightly build you can install this afternoon
👍🏿 1
a

Amanda Crawford

12/02/2019, 4:57 PM
Also I ran into another error. When I try to use s3 as my storage enviroment, I get the following error:
a

alex

12/02/2019, 5:02 PM
you need to set the 
system_storage_defs
to include 
s3
as an option - the example linked above demonstrates how
a

Amanda Crawford

12/02/2019, 5:12 PM
Ah ok! I will give it a try
Thanks!
a

alex

12/02/2019, 5:20 PM
No problem!
If you are still stuck on this you can try to grab the nightly build of dagster-graphql to get the python error message https://pypi.org/project/dagster-graphql-nightly/
a

Amanda Crawford

12/03/2019, 6:31 PM
great! Thanks! I'll do that
How do I install the nightly version since dagster looks for dagster_graphql instead of dagster_graphql_nightly?
a

alex

12/03/2019, 9:35 PM
dagster-graphql-nightly will provide the dagster_graphql module
Copy code
(dagenv36) ~/dagster:master$ python
Python 3.6.8 (default, Dec  2 2019, 12:06:17)
[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from dagster_graphql.version import __nightly__
>>> __nightly__
'2019.12.03'
python environment stuff is not my forte either - so i might have tested incorrectly
but I am able to execute dagit in a set up like this: here is a slice from 
pip list
dagit 0.6.5 dagster 0.6.5 dagster-graphql-nightly 2019.12.3
a

Amanda Crawford

12/03/2019, 9:45 PM
Hmm...when I'm getting errors on error since dagster-airflow is looking for dagster-graphql. I testing out 
pip install dagster-graphql[nightly]
a

alex

12/03/2019, 9:47 PM
try 
pip install dagster-graphql-nightly
you may need to explicitly 
pip uninstall dagster-graphql
also
you’re the first person to try to use these nightly so thanks for helping us work the kinks out
plus1 2
🦜 1
I have an interview coming up so @max is here to help as well
m

max

12/03/2019, 9:57 PM
👋 hi @Amanda Crawford
a

Amanda Crawford

12/03/2019, 10:00 PM
Great! I did the uninstall and install using 
pip install dagster-graphql-nightly
and tried to run and that when I ran to the error message
m

max

12/03/2019, 10:04 PM
are you hitting an error message in dagit?
a

Amanda Crawford

12/03/2019, 10:05 PM
No, when I run my pipelines on airflow it looks like dagster-airflow is looking for dagster-graphql but can't find it when I install using the nightly build
m

max

12/03/2019, 10:09 PM
can you paste the error message?
a

Amanda Crawford

12/03/2019, 10:11 PM
Yep, I will post it in a few minutes
Copy code
>>> from  dagster_graphql.version import __nightly__
>>> __nightly__
'nightly-2019.11.28'
Ah I figured it out!
🎊 2
I'm seeing another error '
m

max

12/03/2019, 10:23 PM
i think storage might not be configured in the config for your pipeline run?
does it have 
storage: s3:
or equivalent?
a

Amanda Crawford

12/03/2019, 10:25 PM
yep
Copy code
storage:
  filesystem:
    config:
      base_dir: /tmp/dagster-airflow/entity_type_pipeline
m

max

12/03/2019, 10:42 PM
interesting
@prha ^^
we're going to look into the strange SIGTERM that is being generated
but regardless of that, the issue at the top of that stacktrace is that the intermediates aren't being found by downstream steps
which points to a storage issue
but you're using the 
DagsterPythonOperator
hmm
a

alex

12/03/2019, 11:14 PM
can you send over a gist with the full error https://gist.github.com/
a

Amanda Crawford

12/04/2019, 2:21 AM
Sorry! I had to head out.
Good morning! Would someone be open to meeting today on a call to work through this issue?
I went the route of downloading the dagster repo, switching to the nightly branch, and then installing the dagster-graphql and dagster-airflow module
I'm still getting the same error with the latest code. I have successful run on a few other tasks but am seeing errors in the logs from dagster checks. It may be easier to explain over a video chat.
a

alex

12/04/2019, 3:57 PM
awesome, thanks for sending over the gist
I (and likely max) should be available to jump on VC and figure this out
a

Amanda Crawford

12/04/2019, 4:01 PM
Great! I have a cutoff time of 4:30 EST today. What would be the earliest time you all are available for a VC session?
a

alex

12/04/2019, 4:35 PM
I can hop on now but I have ~0 airflow knowledge
I’ll go hop on http://zoom.us/j/2758662566 til about 9:30a PST if you are available
a

Amanda Crawford

12/04/2019, 4:46 PM
Jumping on now
m

max

12/04/2019, 6:02 PM
@prha want to make sure you saw this gist
a

alex

12/04/2019, 6:42 PM
update for others - think we figured out the issue
a

Amanda Crawford

12/04/2019, 7:13 PM
Ah!!! I'm still back to the original error
I do see more of the output now in the intermediates folder
a

alex

12/04/2019, 7:26 PM
send a gist over when you can
a

Amanda Crawford

12/04/2019, 7:29 PM
a

alex

12/04/2019, 7:31 PM
whats in the logs for 
create_intermediate_entity_table
?
a

Amanda Crawford

12/04/2019, 7:34 PM
a

alex

12/04/2019, 7:35 PM
hmm well the step is failing - not clear why
oop ok its buried in there
a

Amanda Crawford

12/04/2019, 7:36 PM
Interesting, I guess I assumed it was passing again since it's marked as success in Airflow 😲
a

alex

12/04/2019, 7:36 PM
StepFailureData(error=SerializableErrorInfo(message=‘sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused\n\tIs the server running on host “localhost” (127.0.0.1) and accepting\n\tTCP/IP connections on port 5433?\ncould not connect to server: Address not available\n\tIs the server running on host “localhost” (::1) and accepting\n\tTCP/IP connections on port 5433?\n\n(Background on this error at: http://sqlalche.me/e/e3q8)\n’, stack=None, cls_name=None), user_failure_data=None), message=None)
I guess I assumed it was passing again since it’s marked as success in Airflow
ya this is the terrible bug that @nate is fixing
a

Amanda Crawford

12/04/2019, 7:37 PM
Interesting....
Thanks for finding it! I'll figure out why my pipeline is failing to connect
a

alex

12/04/2019, 7:40 PM
👍 no problem good luck dealing docker shenanigans
😄 1
n

nate

12/04/2019, 7:55 PM
yeah @Amanda Crawford I should have a fix for the airflow failed task status out this afternoon
😃 1
a

Amanda Crawford

12/04/2019, 8:21 PM
Ok got my first full green pipeline 🙂 Thanks all!
🦜 2
It's really great to have your support!
n

nate

12/04/2019, 8:44 PM
amazing! partywizard
hey @Amanda Crawford just wanted to let you know that I just landed a fix for the Airflow tasks not being marked as failed here https://dagster.phacility.com/D1543 - this is available on master now, but will also let you know when it gets picked up by a release!
a

Amanda Crawford

12/05/2019, 8:36 PM
Ok great! thanks for the update. Please let me know when it's been added to a release.
a

alex

12/06/2019, 7:02 PM
0.6.6 just went out with the fix
Powered by