https://dagster.io/ logo
#announcements
Title
# announcements
b

Ben Fogelson

12/17/2019, 7:49 PM
Hi Dagster folks! I’m running into an issue where a pipeline fails in dagit but works when run with
execute_pipeline
. I posted about it on StackOverflow, was wondering if anyone can help. https://stackoverflow.com/questions/59379523/dagster-pipeline-executes-successfully-when-run-with-execute-pipeline-but-not
m

max

12/17/2019, 8:01 PM
this is interesting! i will try to reproduce this afternoon
seems like
Failed in nopython mode pipeline
is the key
a

abhi

12/17/2019, 8:03 PM
I am having trouble reproducing this. What version of python are you using? What OS? Also where is your code using numba? I am seeing this LoweringError pop up a ton in the numba repository on github.
This leads me to believe it might be an issue with numba, try getting rid of that dependency and see if it works. That should at least point us in the right direction
b

Ben Fogelson

12/17/2019, 8:56 PM
Hi @abhi and @max, thanks for checking it out! Python 3.6.9, OS X (10.15.1).
numba
is being called from the
umap-learn
package in
UMAP.fit()
, I don’t think it would be possible to remove the dependency on
numba
while still using
umap-learn
, which is a pretty central part of the pipeline. It is some subtle issue in how
numba
is doing its compiling. What’s strange is that the compiling works when the pipeline is called using
execute_pipeline
. Any idea what would be different in
dagit
?
a

abhi

12/17/2019, 9:15 PM
Ah I see that's a good point. I just repro'd it. Yeah that is weird. There shouldn't be anything majorly different. Dagit is also eventually calling execute_pipeline as well however, it does so from a forked process. I wonder if it's an issue with numba not playing well with multiprocessing when it tries to use the JIT compiler. Let me poke around a bit and get back to you.
m

max

12/17/2019, 10:20 PM
it also looks like the JIT may not play nicely with pickle
@Ben Fogelson what config are you using when executing with dagit?
b

Ben Fogelson

12/17/2019, 10:25 PM
Completely blank config
2 Views