Hi I had a scheduler to run a test job run every 15 seconds

Question

Hi I had a scheduler to run a test job run every 15 seconds After confirming it worked the scheduler was then deactivated switched off using dagit ui but the task still runs every 15 seconds I closed the dagster daemon and dagit to stop that but it starts again as soon as I restart dagster daemon and it repeats the grpc error message below How can I hard reset this so that this weird behavior stops The

jasono · Answer

2022 03 07 20 21 50 0800 dagster daemon SchedulerDaemon WARNING Could not load location auto tests py to check for schedules due to the following error Exception Timed out waiting for gRPC server to start with arguments E Data dagster venv Scripts python exe m dagster api grpc lazy load user code port 62844 heartbeat heartbeat timeout 120 fixed server id 75ba823e e008 4b26 a0b4 de63f6482425 log level WARNING use python environment entry point f E Data dagster repo auto tests py Most recent connection error dagster core errors DagsterUserCodeUnreachableError Could not reach user code server Stack Trace File E Data dagster venv lib site packages dagster grpc server py line 937 in wait for grpc server client ping File E Data dagster venv lib site packages dagster grpc client py line 123 in ping res = self query Ping api pb2 PingRequest echo=echo File E Data dagster venv lib site packages dagster grpc client py line 110 in query raise DagsterUserCodeUnreachableError Could not reach user code server from e The above exception was caused by the following exception grpc channel InactiveRpcError lt InactiveRpcError of RPC that terminated with status = StatusCode UNAVAILABLE details = failed to connect to all addresses debug error string = created 1646713277 274000000 description Failed to pick subchannel file src core ext filters client channel client channel cc file line 3135 referenced errors created 1646713277 274000000 description failed to connect to all addresses file src core lib transport error utils cc file line 163 grpc status 14

daniel · Answer

Hi would you mind posting the contents of your workspace yaml file or the arguments that you re starting your daemon with even if you don t have any schedules running the daemon is still going to try to load the code in your workspace to see if there are any schedules there it looks like that s the part that s failing

daniel · Answer

if its not sensitive maybe you could post the contents of `repo auto tests py`

jasono · Answer

Sure please see below Workspace yaml `load from ` ` python file repo hello cereal py` ` python file repo auto tests py` ` python file repo dynamic tests py`

jasono · Answer

Auto tests py `from dagster import RunRequest ScheduleDefinition job op repository sensor DefaultScheduleStatus` `import great expectations as ge` ` op` `def hello world ` `pass` ` job` `def ge job ` `context = ge data context DataContext context root dir= y great expectations ` `checkpoint result = context run checkpoint ` `checkpoint name = my checkpoint ` ` ` `job1 schedule = ScheduleDefinition ` `job=ge job ` `cron schedule= 15 ` `execution timezone= US Pacific ` `default status=DefaultScheduleStatus RUNNING` ` ` ` repository` `def ge auto test repository ` `return ge job job1 schedule `

jasono · Answer

And finally the daemon server my command is the bare bones ```dagster daemon run```

daniel · Answer

Ok so you turned off the schedule in the dagit UI but the schedule never stopped If your dagit and your daemon are using the same DAGSTER HOME and are using the same workspace yaml and running in the same python environment stopping the schedule in dagit should work If you don t mind you could post or DM the contents of your schedules db file in your DAGSTER HOME folder and we could take a look

daniel · Answer

The most likely reason for the grpc error that I can think of is that if a run is getting launched every 15 seconds eventually it s more runs than your machine can handle so things have trouble starting up

jasono · Answer

1 602a2e394a98ddbe30f3f19bbde31399b6f0488b SUCCESS SCHEDULE 2022 03 08 02 00 00 000000 class TickData cursor null error null failure count 0 job name ge job schedule job origin id 602a2e394a98ddbe30f3f19bbde31399b6f0488b job type enum InstigatorType SCHEDULE origin run ids run ids 57d91897 097f 4f34 857b a39cff9ffe61 run keys skip reason null status enum TickStatus SUCCESS timestamp 1646704800 0 2022 03 08 02 00 39 2022 03 08 02 00 39 2 602a2e394a98ddbe30f3f19bbde31399b6f0488b SUCCESS SCHEDULE 2022 03 08 02 15 00 000000 class TickData cursor null error null failure count 0 job name ge job schedule job origin id 602a2e394a98ddbe30f3f19bbde31399b6f0488b job type enum InstigatorType SCHEDULE origin run ids run ids a5aeb129 03bb 4db4 afe5 8577a65cdfd7 run keys skip reason null status enum TickStatus SUCCESS timestamp 1646705700 0 2022 03 08 02 15 04 2022 03 08 02 15 04 3 602a2e394a98ddbe30f3f19bbde31399b6f0488b SUCCESS SCHEDULE 2022 03 08 02 30 00 000000 class TickData cursor null error null failure count 0 job name ge job schedule job origin id 602a2e394a98ddbe30f3f19bbde31399b6f0488b job type enum InstigatorType SCHEDULE origin run ids run ids 2e602097 48f5 4b13 934e cc5e1d2526f2 run keys skip reason null status enum TickStatus SUCCESS timestamp 1646706600 0 2022 03 08 02 30 03 2022 03 08 02 30 03 4 602a2e394a98ddbe30f3f19bbde31399b6f0488b SUCCESS SCHEDULE 2022 03 08 03 00 00 000000 class TickData cursor null error null failure count 0 job name ge job schedule job origin id 602a2e394a98ddbe30f3f19bbde31399b6f0488b job type enum InstigatorType SCHEDULE origin run ids run ids 9062d0eb f100 417e a235 26a69624c4af run keys skip reason null status enum TickStatus SUCCESS timestamp 1646708400 0 2022 03 08 03 00 26 2022 03 08 03 00 26 5 602a2e394a98ddbe30f3f19bbde31399b6f0488b SUCCESS SCHEDULE 2022 03 08 03 15 00 000000 class TickData cursor null error null failure count 0 job name ge job schedule job origin id 602a2e394a98ddbe30f3f19bbde31399b6f0488b job type enum InstigatorType SCHEDULE origin run ids run ids f9f3f5a6 bcc6 4bd2 9783 dbc1fc55fe5a run keys skip reason null status enum TickStatus SUCCESS timestamp 1646709300 0 2022 03 08 03 15 04 2022 03 08 03 15 04

jasono · Answer

The above is the content of the job table

jasono · Answer

actually job ticks table

jasono · Answer

below is the job table

jasono · Answer

602a2e394a98ddbe30f3f19bbde31399b6f0488b bb5b6fe5f352839b38265dfa66badc6a78fb22a0 STOPPED SCHEDULE class InstigatorState job specific data class ScheduleInstigatorData cron schedule 15 start timestamp null job type enum InstigatorType SCHEDULE origin class ExternalJobOrigin external repository origin class ExternalRepositoryOrigin repository location origin class ManagedGrpcPythonEnvRepositoryLocationOrigin loadable target origin class LoadableTargetOrigin attribute null executable path E Data dagster venv Scripts python exe module name null package name null python file E Data dagster repo auto tests py working directory null location name auto tests py repository name ge auto test repository job name ge job schedule status enum InstigatorStatus STOPPED 2022 03 08 03 38 59 2022 03 08 03 38 59

daniel · Answer

are you sure that your daemon and your dagit are using the same value of DAGSTER HOME From that table you posted it looks like the schedule is stopped If it s not could you post the output of the daemon process where it s saying that the schedule is still running

jasono · Answer

Yes they are using the same Dagster Home I checked the environment variables and also the Daemon server appears on Dagit s Daemon list in the Status menu

jasono · Answer

Here is the daemon output I terminated daemon last night and just restarted it but it s still having the issue ```$ dagster daemon run 2022 03 08 09 02 46 0800 dagster daemon INFO instance is configured with the following daemons BackfillDaemon SchedulerDaemon SensorDaemon Calculating Metrics 100 | | 15 15 00 00 lt 00 00 68 57it s E Data dagster venv lib site packages jinja2 environment py 1088 DeprecationWarning soft unicode has been renamed to soft str The old name will be removed in MarkupSafe 2 1 return concat self root render func self new context vars Calculating Metrics 100 | | 15 15 00 00 lt 00 00 96 00it s 2022 03 08 09 03 34 0800 dagster daemon SensorDaemon WARNING Could not load location auto tests py to check for sensors due to the following error Exception Timed out waiting for gRPC server to start with arguments E Data dagster venv Scripts python exe m dagster api grpc lazy load user code port 50654 heartbeat heartbeat timeout 120 fixed server id 81c77561 3f33 4be9 afd4 71f36336169b log level WARNING use python environment entry point f E Data dagster repo auto tests py Most recent connection error dagster core errors DagsterUserCodeUnreachableError Could not reach user code server Stack Trace File E Data dagster venv lib site packages dagster grpc server py line 937 in wait for grpc server client ping File E Data dagster venv lib site packages dagster grpc client py line 123 in ping res = self query Ping api pb2 PingRequest echo=echo File E Data dagster venv lib site packages dagster grpc client py line 110 in query raise DagsterUserCodeUnreachableError Could not reach user code server from e The above exception was caused by the following exception grpc channel InactiveRpcError lt InactiveRpcError of RPC that terminated with status = StatusCode UNAVAILABLE details = failed to connect to all addresses debug error string = created 1646759002 877000000 description Failed to pick subchannel file src core ext filters client channel client channel cc file line 3135 referenced errors created 1646759002 877000000 description failed to connect to all addresses file src core lib transport error utils cc file line 163 grpc status 14 gt Stack Trace File E Data dagster venv lib site packages dagster grpc client py line 107 in query response = getattr stub method request type kwargs timeout=timeout File E Data dagster venv lib site packages grpc channel py line 946 in call return end unary response blocking state call False None File E Data dagster venv lib site packages grpc channel py line 849 in end unary response blocking raise InactiveRpcError state Stack Trace File E Data dagster venv lib site packages dagster core host representation grpc server registry py line 207 in get grpc endpoint server process = GrpcServerProcess File E Data dagster venv lib site packages dagster grpc server py line 1083 in init self server process self port = open server process on dynamic port File E Data dagster venv lib site packages dagster grpc server py line 1031 in open server process on dynamic port server process = open server process File E Data dagster venv lib site packages dagster grpc server py line 1008 in open server process wait for grpc server server process client subprocess args timeout=startup timeout File E Data dagster venv lib site packages dagster grpc server py line 943 in wait for grpc server raise Exception 2022 03 08 09 03 34 0800 dagster daemon SensorDaemon INFO Not checking for any runs since no sensors have been started 2022 03 08 09 03 34 0800 dagster daemon SchedulerDaemon WARNING Could not load location auto tests py to check for schedules due to the following error Exception Timed out waiting for gRPC server to start with arguments E Data dagster venv Scripts python exe m dagster api grpc lazy load user code port 50654 heartbeat heartbeat timeout 120 fixed server id 81c77561 3f33 4be9 afd4 71f36336169b log level WARNING use python environment entry point f E Data dagster repo auto tests py Most recent connection error dagster core errors DagsterUserCodeUnreachableError Could not reach user code server Stack Trace File E Data dagster venv lib site packages dagster grpc server py line 937 in wait for grpc server client ping File E Data dagster venv lib site packages dagster grpc client py line 123 in ping res = self query Ping api pb2 PingRequest echo=echo File E Data dagster venv lib site packages dagster grpc client py line 110 in query raise DagsterUserCodeUnreachableError Could not reach user code server from e The above exception was caused by the following exception grpc channel InactiveRpcError lt InactiveRpcError of RPC that terminated with status = StatusCode UNAVAILABLE details = failed to connect to all addresses debug error string = created 1646759002 877000000 description Failed to pick subchannel file src core ext filters client channel client channel cc file line 3135 referenced errors created 1646759002 877000000 description failed to connect to all addresses file src core lib transport error utils cc file line 163 grpc status 14 gt Stack Trace File E Data dagster venv lib site packages dagster grpc client py line 107 in query response = getattr stub method request type kwargs timeout=timeout File E Data dagster venv lib site packages grpc channel py line 946 in call return end unary response blocking state call False None File E Data dagster venv lib site packages grpc channel py line 849 in end unary response blocking raise InactiveRpcError state Stack Trace File E Data dagster venv lib site packages dagster core host representation grpc server registry py line 207 in get grpc endpoint server process = GrpcServerProcess File E Data dagster venv lib site packages dagster grpc server py line 1083 in init self server process self port = open server process on dynamic port File E Data dagster venv lib site packages dagster grpc server py line 1031 in open server process on dynamic port server process = open server process File E Data dagster venv lib site packages dagster grpc server py line 1008 in open server process wait for grpc server server process client subprocess args timeout=startup timeout File E Data dagster venv lib site packages dagster grpc server py line 943 in wait for grpc server raise Exception 2022 03 08 09 03 49 0800 dagster daemon SensorDaemon WARNING Could not load location auto tests py to check for sensors due to the following error Exception Timed out waiting for gRPC server to start with arguments E Data dagster venv Scripts python exe m dagster api grpc lazy load user code port 50654 heartbeat heartbeat timeout 120 fixed server id 81c77561 3f33 4be9 afd4 71f36336169b log level WARNING use python environment entry point f E Data dagster repo auto tests py Most recent connection error dagster core errors DagsterUserCodeUnreachableError Could not reach user code server Stack Trace File E Data dagster venv lib site packages dagster grpc server py line 937 in wait for grpc server client ping File E Data dagster venv lib site packages dagster grpc client py line 123 in ping res = self query Ping api pb2 PingRequest echo=echo File E Data dagster venv lib site packages dagster grpc client py line 110 in query raise DagsterUserCodeUnreachableError Could not reach user code server from e The above exception was caused by the following exception grpc channel InactiveRpcError lt InactiveRpcError of RPC that terminated with status = StatusCode UNAVAILABLE details = failed to connect to all addresses debug error string = created 1646759002 877000000 description Failed to pick subchannel file src core ext filters client channel client channel cc file line 3135 referenced errors created 1646759002 877000000 description failed to connect to all addresses file src core lib transport error utils cc file line 163 grpc status 14 gt Stack Trace File E Data dagster venv lib site packages dagster grpc client py line 107 in query response = getattr stub method request type kwargs timeout=timeout File E Data dagster venv lib site packages grpc channel py line 946 in call return end unary response blocking state call False None File E Data dagster venv lib site packages grpc channel py line 849 in end unary response blocking raise InactiveRpcError state Stack Trace File E Data dagster venv lib site packages dagster core host representation grpc server registry py line 207 in get grpc endpoint server process = GrpcServerProcess File E Data dagster venv lib site packages dagster grpc server py line 1083 in init self server process self port = open server process on dynamic port File E Data dagster venv lib site packages dagster grpc server py line 1031 in open server process on dynamic port server process = open server process File E Data dagster venv lib site packages dagster grpc server py line 1008 in open server process wait for grpc server server process client subprocess args timeout=startup timeout File E Data dagster venv lib site packages dagster grpc server py line 943 in wait for grpc server raise Exception 2022 03 08 09 03 49 0800 dagster daemon SensorDaemon INFO Not checking for any runs since no sensors have been started 2022 03 08 09 03 49 0800 dagster daemon SchedulerDaemon WARNING Could not load location auto tests py to check for schedules due to the following error Exception Timed out waiting for gRPC server to start with arguments E Data dagster venv Scripts python exe m dagster api grpc lazy load user code port 50654 heartbeat heartbeat timeout 120 fixed server id 81c77561 3f33 4be9 afd4 71f36336169b log level WARNING use python environment entry point f E Data dagster repo auto tests py Most recent connection error dagster core errors DagsterUserCodeUnreachableError Could not reach user code server Stack Trace File E Data dagster venv lib site packages dagster grpc server py line 937 in wait for grpc server client ping File E Data dagster venv lib site packages dagster grpc client py line 123 in ping res = self query Ping api pb2 PingRequest echo=echo File E Data dagster venv lib site packages dagster grpc client py line 110 in query raise DagsterUserCodeUnreachableError Could not reach user code server from e The above exception was caused by the following exception grpc channel InactiveRpcError lt InactiveRpcError of RPC that terminated with status = StatusCode UNAVAILABLE details = failed to connect to all addresses ```

daniel · Answer

so what that error is saying is that it s trying to spin up a process that loads your code but that process is taking more than 60 seconds to start up In the past i ve seen this when the box that is running dagster is just overworked running out of resources is there a way to check whether that s the case You could try running that same command that it s saying is timing out and see if it s able to run and start up a code server ```E Data dagster venv Scripts python exe m dagster api grpc lazy load user code port 50654 heartbeat heartbeat timeout 120 fixed server id 81c77561 3f33 4be9 afd4 71f36336169b log level WARNING use python environment entry point f E Data dagster repo auto tests py```