19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hi team, I am trying to backfill a pipeline but th...
# ask-communitym
Marco
04/27/2022, 5:08 PMHi team, I am trying to backfill a pipeline but the process keeps on failing with the following message. Any idea how to solve?
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:status = StatusCode.DEADLINE_EXCEEDEDdetails = "Deadline Exceeded"debug_error_string = "{"created":"@1651039595.624784023","description":"Error received from peer ipv6:[::1]:4266","file":"src/core/lib/surface/<http://call.cc|call.cc>","file_line":1074,"grpc_message":"Deadline Exceeded","grpc_status":4}"Marco
04/27/2022, 5:45 PMI guess I need to pass/set somewhere an increased grpc timeout value (defaulting to
DEFAULT_GRPC_TIMEOUTd
daniel
04/27/2022, 5:57 PMHi Marco - do you have a full stack trace for this? This indicates that a gRPC call is taking more than 60 seconds to run, which would typically indicate user code that takes a really long time to run, or a bug on our side making it take much longer than it should be. Is it possible to share the code for the parittion set that's taking a really long time to generate config for the backfill?
m
Marco
04/27/2022, 6:08 PMHi Daniel, thanks. the pipeline runs a solid that takes a long time to execute - might this be the issue? Or it is really the config generation? If you can narrow it down I might be able to share specific bits.
d
daniel
04/27/2022, 6:08 PMThe full stack trace of the deadline exceeded would be really useful - a slow op shouldn't make a difference here
daniel
04/27/2022, 6:09 PMsince the actual runs happen in a different process - this is likely just generating config for hte backfill
m
Marco
04/27/2022, 6:15 PMDoes this help?
d
daniel
04/27/2022, 6:21 PMIt does - what version of dagster is this, and is it possible to share the code of the partition set being backfilled?
m
Marco
04/27/2022, 6:25 PM0.11.11
Marco
04/27/2022, 6:26 PMI'll try to share the code but it's not going to be trivial as there are partials, configs etc
Marco
04/27/2022, 6:27 PMDo you have a feeling
For what is mostly relevant?
d
daniel
04/27/2022, 6:27 PMif upgrading would ever be on the table, there have been substantial performance improvements and bugfixes since 0.11.11
daniel
04/27/2022, 6:28 PMthe most likely cause for this is calling the function on your partition set that generates the partitions taking a very long time
m
Marco
04/27/2022, 9:13 PMThe code that generates the partitions config is the same that I use elsewhere (normally without problems). That said, I did have similar issues before. I have reduced the number of concurrent runs, hopefully that will help. I’ll also look into unpgrading; I am a bit concerned with backward compatibility - do you expect there might be issues in this respect?
d
daniel
04/27/2022, 9:14 PMI don't expect significant backwards-compatible issues with your code - you'd need to migrate your storage though. The list of breaking changes between minor releases can be found here: https://github.com/dagster-io/dagster/blob/master/MIGRATION.md
3 Views