https://dagster.io/ logo
Powered by
q

QianXiao

07/30/2020, 1:31 PM
Hi, everyone. I am new to Dagster and I meet a strange problem, when I run the python file using dagit, I find that the order of importing python library matters. And the detail is showed in https://github.com/dagster-io/dagster/issues/2767. Any help will be appreciated.
m

max

07/30/2020, 2:00 PM
thanks @QianXiao! this is a confusing one
cc @alex-- i wonder if 
sync_get_external_repositories
is not playing nicely with relative imports / pythonpath -- i also wonder if this is windows- or anaconda-only
@QianXiao it might be helpful to see the output of 
conda list
d

daniel

07/30/2020, 2:02 PM
I can repro this on my mac with the same two files
m

max

07/30/2020, 2:07 PM
that's great
q

QianXiao

07/30/2020, 2:14 PM
@max the output of 
conda list
is a very long
d

daniel

07/30/2020, 2:22 PM
Similar issue happening on that file in master right now, just at a slightly different place (and maybe a bit more info on py38 about why the import is failing?): https://dagster.phacility.com/P69
m

max

07/30/2020, 2:30 PM
Copy code
from . import numeric\n  File "/Users/dgibson/.pyenv/versions/3.8.1/envs/dagster-3.8.1/lib/python3.8/site-packages/numpy/core/numeric.py", line 23, in <module>\n    from . import shape_base\n  File "/Users/dgibson/.pyenv/versions/3.8.1/envs/dagster-3.8.1/lib/python3.8/site-packages/numpy/core/shape_base.py", line 9, in <module>\n    from . import numeric as _nx\nImportError: cannot import name \'numeric\' from partially initialized module \'numpy.core\' (most likely due to a circular import)
wild, that looks like a circular import in numpy but it can't be
q

QianXiao

07/30/2020, 2:40 PM
and the error occured, if we 
import numpy as np
in the message.py
a

alex

07/30/2020, 2:41 PM
what happens if you run 
dagit -f simple_task.py -d .
?
q

QianXiao

07/30/2020, 2:43 PM
@alex get the same error
a

alex

07/30/2020, 2:44 PM
So we recently introduced this 
working_directory
/ 
-d
aspect to file based loads to ensure we capture if the current working directory needs to be set, in your case 
from message import Message
only works if the working directory is the one containing the two files. Though its only supposed to warn if its not set, and you are still getting the error with 
-d
so…. very puzzling
q

QianXiao

07/30/2020, 2:53 PM
yep, the working directory is the one containing the two files,
and run the command in the directory
@alex I don't know whether is helpful, but I 
import pandas as pd
also encounters the error
m

max

07/30/2020, 3:07 PM
i think we are close to a fix
q

QianXiao

07/30/2020, 3:07 PM
a

alex

07/30/2020, 3:07 PM
are you saying that if you move 
pandas
above the 
message
import it works as well?
q

QianXiao

07/30/2020, 3:10 PM
yep, that works as well
and some libs that python built-in, do not have the problem
a

alex

07/30/2020, 3:19 PM
message.py is just those 5 lines?
q

QianXiao

07/30/2020, 3:20 PM
right, just 5 lines
a

alex

07/30/2020, 3:35 PM
great thanks for the report, pretty sure we found the issue and a fix should go out in 0.8.10 later today
s

sashank

07/31/2020, 12:07 AM
hey @QianXiao we just pushed a release with the fix for this - let us know if you’re still running into issues
q

QianXiao

07/31/2020, 3:58 AM
@sashank You guys are awesome, I test the latest version and it works. Thanks everyone for your help @max @alex @daniel
🙌 1
d

daniel

07/31/2020, 12:55 PM
No problem, this was actually the root cause of another mysterious bug and your report helped us get to the bottom of it, so thank you!
Powered by