Hey team we are experiencing an issue with our dagster user

Question

Hey team we are experiencing an issue with our dagster user deployments where it is constantly running an SQL query that is definesd in a partition set aka it is constantly running the partition set fn this is causing our dagit deployment to die and our DB to become overloaded any ideas As you can see in the picture its constantly calling a function that reads a partition set from BQ hence the download statements and this is being triggered constantly

daniel · Answer

Hi Ruben in the short term is this something that you could cache memoize within your python code So that repeated calls only make the SQL query once

Rubén Lopez Lozoya · Answer

would need to get back to the team but is this issue I am experiencing something you guys are aware of just to make sure that this behavior is something inherent to dagster and not something we may be doing wrong

daniel · Answer

This is the function that returns the list of partitions

Rubén Lopez Lozoya · Answer

yep

daniel · Answer

and what version of dagster is this

Rubén Lopez Lozoya · Answer

we have other partition sets defined but this one was recently added and only 3 days after deploying it it has started this never ending loop of constantly running and now no matter how back we rollback it wont stop unless we stop inserting the set in the repository

Rubén Lopez Lozoya · Answer

0 14 8

daniel · Answer

and the intention is that on each schedule tick it will run on every partition hence the partition selector

Rubén Lopez Lozoya · Answer

so what I am trying to achieve is to have the whole list of partitions be run once per day at 14 00

Rubén Lopez Lozoya · Answer

so Id expect this schedule to trigger the partition fn once per day at the desired time and then run all items returned by that fn

daniel · Answer

is there any way to get a stack trace when its running the partition function repeatedly If you add something like ```import traceback traceback print stack ``` inside get company partitions that would tell us exactly where its being called

Rubén Lopez Lozoya · Answer

this is something I can try yes

Rubén Lopez Lozoya · Answer

does the use case I depicted make sense with my current implementation

daniel · Answer

it seems reasonable to me

daniel · Answer

I think the stack trace will give us some useful clues I was under the impression that it should only be called if you re running a backfill or once per day like you said

Rubén Lopez Lozoya · Answer

```Error 2022 04 22T14 36 38 925203914Z File opt pysetup venv lib python3 8 site packages dagster grpc server py line 521 in ExternalScheduleExecution Error 2022 04 22T14 36 38 925294492Z get external schedule execution Error 2022 04 22T14 36 38 925378256Z File opt pysetup venv lib python3 8 site packages dagster grpc impl py line 252 in get external schedule execution Error 2022 04 22T14 36 38 925474883Z return schedule def evaluate tick schedule context Error 2022 04 22T14 36 38 925559545Z File opt pysetup venv lib python3 8 site packages dagster core definitions schedule definition py line 442 in evaluate tick Error 2022 04 22T14 36 38 925648461Z result = list ensure gen execution fn context Error 2022 04 22T14 36 38 925754251Z File opt pysetup venv lib python3 8 site packages dagster core definitions partition py line 594 in execution fn Error 2022 04 22T14 36 38 925851548Z missing partition names = Error 2022 04 22T14 36 38 925938358Z File opt pysetup venv lib python3 8 site packages dagster core definitions partition py line 597 in lt listcomp gt Error 2022 04 22T14 36 38 926027145Z if partition name not in self get partition names context scheduled execution time Error 2022 04 22T14 36 38 926113111Z File opt pysetup venv lib python3 8 site packages dagster core definitions partition py line 520 in get partition names Error 2022 04 22T14 36 38 926201454Z return part name for part in self get partitions current time Error 2022 04 22T14 36 38 926298093Z File opt pysetup venv lib python3 8 site packages dagster core definitions partition py line 510 in get partitions Error 2022 04 22T14 36 38 926391442Z return self partitions def get partitions current time Error 2022 04 22T14 36 38 926482648Z File opt pysetup venv lib python3 8 site packages dagster core definitions partition py line 356 in get partitions Error 2022 04 22T14 36 38 926576250Z partitions = self partition fn current time Error 2022 04 22T14 36 38 926666711Z File opt pysetup venv lib python3 8 site packages dagster core definitions partition py line 440 in wrap partition fn Error 2022 04 22T14 36 38 926789120Z obj list = partition fn type ignore Error 2022 04 22T14 36 38 926887857Z File opt dagster app maquinillo dagster resources underwriting pipeline defs post company metrics to client api pipeline py line 409 in get company partitions Error 2022 04 22T14 36 38 926987185Z traceback print stack ```

Rubén Lopez Lozoya · Answer

this is the stack trace sorry if its a bit messy

daniel · Answer

That s no problem i m a bit confused though because that evaluate tick call should only be called when the cron schedule triggers once per day in this case Are there any logs from your daemon that might help explain why it s repeatedly making that ExternalScheduleExecution call

Rubén Lopez Lozoya · Answer

```Información 2022 04 22T14 40 26 941372516Z external schedule execution args Información 2022 04 22T14 40 26 941375803Z File usr local lib python3 7 site packages dagster grpc client py line 119 in streaming query Información 2022 04 22T14 40 26 941379619Z raise DagsterUserCodeUnreachableError Could not reach user code server from e Información 2022 04 22T14 40 26 941384093Z Información 2022 04 22T14 40 26 941387633ZThe above exception was caused by the following exception Información 2022 04 22T14 40 26 941391342Zgrpc channel MultiThreadedRendezvous lt MultiThreadedRendezvous of RPC that terminated with Información 2022 04 22T14 40 26 941397975Z status = StatusCode DEADLINE EXCEEDED Información 2022 04 22T14 40 26 941402632Z details = Deadline Exceeded Información 2022 04 22T14 40 26 941407134Z debug error string = created 1650638426 930071599 description Error received from peer ipv4 10 88 0 193 3030 file src core lib surface call cc file line 903 grpc message Deadline Exceeded grpc status 4 Información 2022 04 22T14 40 26 941412324Z gt Información 2022 04 22T14 40 26 941416031Z Información 2022 04 22T14 40 26 941419939ZStack Trace Información 2022 04 22T14 40 26 941423891Z File usr local lib python3 7 site packages dagster grpc client py line 117 in streaming query Información 2022 04 22T14 40 26 941433701Z yield from response stream Información 2022 04 22T14 40 26 941437169Z File usr local lib python3 7 site packages grpc channel py line 426 in next Información 2022 04 22T14 40 26 941441036Z return self next Información 2022 04 22T14 40 26 941444936Z File usr local lib python3 7 site packages grpc channel py line 826 in next Información 2022 04 22T14 40 26 941449299Z raise self```

Rubén Lopez Lozoya · Answer

it says it cannot reach user code but in the user code I am seeing that it constantly runs the partition fn

daniel · Answer

Ah is that firing in a loop

daniel · Answer

that timeout

Rubén Lopez Lozoya · Answer

so maybe there is a weird loop

Rubén Lopez Lozoya · Answer

yes

daniel · Answer

Is it possible that this get company partitions call is taking more than 60 seconds to execute

Rubén Lopez Lozoya · Answer

no it shouldnt at least when I execute it it doesnt

Rubén Lopez Lozoya · Answer

I mean if I go to the pipeline page and I click on the partitions section

Rubén Lopez Lozoya · Answer

and then I load the partitions it takes maybe 3 seconds tops

Rubén Lopez Lozoya · Answer

but after some point my whole deployment crashes bc of this ongoing loop of errors

Rubén Lopez Lozoya · Answer

my db is overloaded

Rubén Lopez Lozoya · Answer

but until that the partition works fine

daniel · Answer

Ah so maybe the timeout is what happens after the deployment crashes

Rubén Lopez Lozoya · Answer

but the daemon and user deployments are not crashed right now its dagit

daniel · Answer

Can you give more output from the scheduler daemon maybe as a text file I m trying to figure out if its actually calling ExternalScheduleExecution frequently or if something else is going on

daniel · Answer

This should be much more performant in the 0 14 11 release that just went out