Hi Dagster folks! I’m running into an issue where a pipeline fails in dagit but works when run with

execute_pipeline

. I posted about it on StackOverflow, was wondering if anyone can help. https://stackoverflow.com/questions/59379523/dagster-pipeline-executes-successfully-when-run-with-execute-pipeline-but-not

this is interesting! i will try to reproduce this afternoon

I am having trouble reproducing this. What version of python are you using? What OS? Also where is your code using numba? I am seeing this LoweringError pop up a ton in the numba repository on github.

Hi @abhi and @max, thanks for checking it out! Python 3.6.9, OS X (10.15.1).

numba

is being called from the

umap-learn

package in

UMAP.fit()

, I don’t think it would be possible to remove the dependency on

numba

while still using

umap-learn

, which is a pretty central part of the pipeline. It is some subtle issue in how

numba

is doing its compiling. What’s strange is that the compiling works when the pipeline is called using

execute_pipeline

. Any idea what would be different in

dagit

?

Ah I see that's a good point. I just repro'd it. Yeah that is weird. There shouldn't be anything majorly different. Dagit is also eventually calling execute_pipeline as well however, it does so from a forked process. I wonder if it's an issue with numba not playing well with multiprocessing when it tries to use the JIT compiler. Let me poke around a bit and get back to you.

it also looks like the JIT may not play nicely with pickle

Completely blank config