Oh but it doesn't log any more than is sent back
# announcements
n
Oh but it doesn't log any more than is sent back
d
I’m not totally clear what the call site is where you’re running into this - could you reply to the thread here with the full stack trace? Is this running against master?
n
Unfortunately I don't have it
It's from here
And grpcio doesn't actually report the stack trace. I just set up a fork to add that but building the wheel is proving difficult
On the dagit side
d
Ah I see. And you’re running the gRPC server yourself right? You’re sure that that command is using the same version of dagster code as dagit is? I could imagine these symptoms happening if dagit was on master but the gRPC server was not, for example
n
Yes, and very sure of that.
I initially thought that was the issue so I went through and confirmed everything is identical, all using a shared base container image
It doesn't seem like a gRPC protocol error anyway, it's something in the response data from an API-side thing
I've finally got a version of gRPC build that should give me an api-server track trace 🙂
d
Got it. This used to be working right? Do you know the change that caused it to transition from working to not working - was it rebasing to master?
n
Yeah, switching from 0.9.20 to my branch off master
If my reading of the code is right
That must be getting 0 bytes of serialized data
d
Got it. The next question id have is whether or not the issue is happening with a simple toy repo, or it’s something about your repo specifically. If it’s possible to send your branch over we can try to reproduce as well
Id expect there to be some useful error in the grpc server stderr if it wasn’t returning anything, so if that’s not happening something strange is going on
n
The repo is basically a hello world test
Copy code
- command:
            - dagster
            - api
            - grpc
            - '--port=9000'
            - '--host=0.0.0.0'
            - '--module-name=dagster_geomagical.workspace'
d
Could we use that GitHub repo to reproduce ourselves?
n
No, that repo is just the support library
The pipeline repo isn't public
But the only python code in is it that one file I snippet'd
hello_world.py 🙂
d
Does that repo load in dagit if you run it against a clean dagster in master without any changes?
n
I have no way to do that
I guess I can try it locally
d
That’s what I was thinking yeah - just trying to isolate the problem if we can
Anything is possible but I’m fairly confident that master isn’t broken, so it should either be a problem with the repo or something with the changes on top of master
n
AHA
thank you
That pointed at a different thing 🙂
Different generator than the one I had been looking at
Buried a lot deeper in some of the plumbing code
d
Ah good!
n
New builds working their way through CI but I have a good feeling about this 🙂
d
Is there anything dagit/the gRPC api should be doing differently to better capture/surface the error here? Or is it somewhat specific to your setup
n
Just having it log errors would help a lot
I think some of that is limited by the grpc library though?
It seems to have very poor error management 😞
d
Hm in theory we should be capturing and returning errors, there may be something about this one causing it to slip through the cracks though
n
It does capture them, but all you get is
str(exception)
No stack trace
Which makes debugging difficult at best
d
I might need to understand the error better / where the plumbing code comes into play
n
And beyond that, the gRPC daemon just produces very little logging. I don't think many folks are running it as an independent daemon like I'm doing 😄
d
If you had, say, a syntax error in your repo code that you posted earlier, that should be surfaced with a clear stack trace
Oh, yeah, if you have a custom gRPC server of some kind that might be it. I don’t have a great understanding of your setup
n
It's not custom, it's just the
dagster api grpc
daemon running in a pod
I run them independently from dagit so they can be versioned on their own
d
Got it. I guess I’m confused what the plumbing code you mentioned earlier is then that had the issue?
n
Sorry, in the dagster plumbing code, I changed some stuff in the code loading code to try and improve handling in threads
And broke it in the process apparently 🙂
d
Ahhhh, ok, yes all bets are off then I think :) no worries.
n
Yeah, I think the fixes for this need to be in grpc, but after that we should add a log level config option to
dagster api grpc
to enable the logging that does exist in there 🙂
d
That seems reasonable