19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
#announcements
Title
# announcements
n
Noah K
12/31/2020, 3:04 PMOh but it doesn't log any more than is sent back
d
daniel
12/31/2020, 3:34 PMI’m not totally clear what the call site is where you’re running into this - could you reply to the thread here with the full stack trace? Is this running against master?
n
Noah K
12/31/2020, 3:38 PMUnfortunately I don't have it
It's from here
And grpcio doesn't actually report the stack trace. I just set up a fork to add that but building the wheel is proving difficult
On the dagit side
d
daniel
12/31/2020, 4:29 PMAh I see. And you’re running the gRPC server yourself right? You’re sure that that command is using the same version of dagster code as dagit is? I could imagine these symptoms happening if dagit was on master but the gRPC server was not, for example
n
Noah K
12/31/2020, 4:31 PMYes, and very sure of that.
I initially thought that was the issue so I went through and confirmed everything is identical, all using a shared base container image
It doesn't seem like a gRPC protocol error anyway, it's something in the response data from an API-side thing
I've finally got a version of gRPC build that should give me an api-server track trace 🙂
d
daniel
12/31/2020, 4:35 PMGot it. This used to be working right? Do you know the change that caused it to transition from working to not working - was it rebasing to master?
n
Noah K
12/31/2020, 4:36 PMYeah, switching from 0.9.20 to my branch off master
If my reading of the code is right
That must be getting 0 bytes of serialized data
d
daniel
12/31/2020, 4:40 PMGot it. The next question id have is whether or not the issue is happening with a simple toy repo, or it’s something about your repo specifically. If it’s possible to send your branch over we can try to reproduce as well
Id expect there to be some useful error in the grpc server stderr if it wasn’t returning anything, so if that’s not happening something strange is going on
n
Noah K
12/31/2020, 4:42 PMThe repo is basically a hello world test
Plus https://github.com/geomagical/dagster-geomagical/blob/main/dagster_geomagical/workspace.py for pipeline extraction
Copy code
- command:
- dagster
- api
- grpc
- '--port=9000'
- '--host=0.0.0.0'
- '--module-name=dagster_geomagical.workspace'd
daniel
12/31/2020, 4:44 PMCould we use that GitHub repo to reproduce ourselves?
n
Noah K
12/31/2020, 4:45 PMNo, that repo is just the support library
The pipeline repo isn't public
But the only python code in is it that one file I snippet'd
hello_world.py 🙂
d
daniel
12/31/2020, 4:46 PMDoes that repo load in dagit if you run it against a clean dagster in master without any changes?
n
Noah K
12/31/2020, 4:48 PMI have no way to do that
I guess I can try it locally
d
daniel
12/31/2020, 4:48 PMThat’s what I was thinking yeah - just trying to isolate the problem if we can
Anything is possible but I’m fairly confident that master isn’t broken, so it should either be a problem with the repo or something with the changes on top of master
n
Noah K
12/31/2020, 4:52 PMAHA
thank you
That pointed at a different thing 🙂
Different generator than the one I had been looking at
Buried a lot deeper in some of the plumbing code
d
daniel
12/31/2020, 4:53 PMAh good!
n
Noah K
12/31/2020, 4:55 PMNew builds working their way through CI but I have a good feeling about this 🙂
d
daniel
12/31/2020, 4:56 PMIs there anything dagit/the gRPC api should be doing differently to better capture/surface the error here? Or is it somewhat specific to your setup
n
Noah K
12/31/2020, 4:58 PMJust having it log errors would help a lot
I think some of that is limited by the grpc library though?
It seems to have very poor error management 😞
d
daniel
12/31/2020, 4:59 PMHm in theory we should be capturing and returning errors, there may be something about this one causing it to slip through the cracks though
n
Noah K
12/31/2020, 4:59 PMIt does capture them, but all you get is
str(exception)No stack trace
Which makes debugging difficult at best
d
daniel
12/31/2020, 5:00 PMI might need to understand the error better / where the plumbing code comes into play
n
Noah K
12/31/2020, 5:00 PMAnd beyond that, the gRPC daemon just produces very little logging. I don't think many folks are running it as an independent daemon like I'm doing 😄
d
daniel
12/31/2020, 5:00 PMIf you had, say, a syntax error in your repo
code that you posted earlier, that should be surfaced with a clear stack trace
Oh, yeah, if you have a custom gRPC server of some kind that might be it. I don’t have a great understanding of your setup
n
Noah K
12/31/2020, 5:02 PMIt's not custom, it's just the
dagster api grpcI run them independently from dagit so they can be versioned on their own
d
daniel
12/31/2020, 5:03 PMGot it. I guess I’m confused what the plumbing code you mentioned earlier is then that had the issue?
n
Noah K
12/31/2020, 5:04 PMSorry, in the dagster plumbing code, I changed some stuff in the code loading code to try and improve handling in threads
And broke it in the process apparently 🙂
d
daniel
12/31/2020, 5:05 PMAhhhh, ok, yes all bets are off then I think :) no worries.
n
Noah K
12/31/2020, 5:09 PMYeah, I think the fixes for this need to be in grpc, but after that we should add a log level config option to
dagster api grpcd
daniel
12/31/2020, 5:12 PMThat seems reasonable
4 Views