Folks been trying to run some sensors on ECS with dagster Ev

Question

Folks been trying to run some sensors on ECS with dagster Everything works locally but on our staging prod env we are getting an exception `dagster core errors SensorExecutionError Error occurred during the execution of evaluation fn for sensor dagster task success` Did some research here and couldn t find much

Bianca Rosa · Answer

Stacktrace ```dagster core errors SensorExecutionError Error occurred during the execution of evaluation fn for sensor dagster task success File usr local lib python3 8 site packages dagster grpc impl py line 289 in get external sensor execution return sensor def evaluate tick sensor context File usr local lib python3 8 contextlib py line 131 in exit self gen throw type value traceback File usr local lib python3 8 site packages dagster core errors py line 191 in user code error boundary raise error cls The above exception was caused by the following exception sqlalchemy exc StatementError sqlalchemy exc ResourceClosedError This Connection is closed SQL SHOW FULL TABLES FROM `workflows` File usr local lib python3 8 site packages dagster core errors py line 184 in user code error boundary yield File usr local lib python3 8 site packages dagster grpc impl py line 289 in get external sensor execution return sensor def evaluate tick sensor context File usr local lib python3 8 site packages dagster core definitions sensor definition py line 354 in evaluate tick result = list self evaluation fn context File usr local lib python3 8 site packages dagster core definitions sensor definition py line 509 in wrapped fn for item in result File usr local lib python3 8 site packages dagster core definitions run status sensor definition py line 437 in wrapped fn context instance get event records File usr local lib python3 8 site packages dagster core definitions sensor definition py line 106 in instance DagsterInstance from ref self instance ref File usr local lib python3 8 site packages dagster core instance init py line 451 in from ref unified storage = instance ref storage File usr local lib python3 8 site packages dagster core instance ref py line 413 in storage return self storage data rehydrate if self storage data else None File usr local lib python3 8 site packages dagster serdes config class py line 85 in rehydrate return klass from config value self result value File usr local lib python3 8 site packages dagster core storage legacy storage py line 98 in from config value run storage = ConfigurableClassData File usr local lib python3 8 site packages dagster serdes config class py line 85 in rehydrate return klass from config value self result value File usr local lib python3 8 site packages dagster mysql run storage run storage py line 115 in from config value return MySQLRunStorage inst data=inst data mysql url=mysql url from config config value File usr local lib python3 8 site packages dagster mysql run storage run storage py line 66 in init table names = retry mysql connection fn db inspect self engine get table names File usr local lib python3 8 site packages dagster mysql utils py line 96 in retry mysql connection fn return fn File lt string gt line 2 in get table names File usr local lib python3 8 site packages sqlalchemy util deprecations py line 128 in warned return fn args kwargs File usr local lib python3 8 site packages sqlalchemy engine reflection py line 198 in get table names tnames = self dialect get table names File lt string gt line 2 in get table names File usr local lib python3 8 site packages sqlalchemy engine reflection py line 52 in cache ret = fn self con args kw File usr local lib python3 8 site packages sqlalchemy dialects mysql base py line 2530 in get table names rp = connection execute File usr local lib python3 8 site packages sqlalchemy engine base py line 2191 in execute return connection execute statement multiparams params File usr local lib python3 8 site packages sqlalchemy engine base py line 976 in execute return self execute text object multiparams params File usr local lib python3 8 site packages sqlalchemy engine base py line 1145 in execute text ret = self execute context File usr local lib python3 8 site packages sqlalchemy engine base py line 1177 in execute context self handle dbapi exception File usr local lib python3 8 site packages sqlalchemy engine base py line 1481 in handle dbapi exception util raise File usr local lib python3 8 site packages sqlalchemy util compat py line 178 in raise raise exception File usr local lib python3 8 site packages sqlalchemy engine base py line 1173 in execute context conn = self revalidate connection File usr local lib python3 8 site packages sqlalchemy engine base py line 463 in revalidate connection raise exc ResourceClosedError This Connection is closed ```

Bianca Rosa · Answer

The sensor itself is pretty simple ``` run status sensor pipeline run status=DagsterRunStatus SUCCESS minimum interval seconds=1 def dagster task success context RunStatusSensorContext gt None try our code except Exception logger exception Error saving dagster job status exc info=True ```

Bianca Rosa · Answer

And we are not even catching the exception which makes me think something went wrong inside of the decorator

Bianca Rosa · Answer

Other things like schedules work fine so i don t see a reason why dagsterdaemon is unable to hold that db conn

daniel · Answer

Hi Bianca what tasks do you have running in ECs Is there a separate one running a grpc server

Bianca Rosa · Answer

Yes there is Been following the examples on the `deploy ecs` folder

daniel · Answer

Are any env vars needed by MySQL available in that gRPC server task That s where the sensor executes

daniel · Answer

It seems like that task is having trouble accessing the mysql db anything else that would be different about it compared to the other tasks

Bianca Rosa · Answer

All env vars are the same on all task tasks thinking face

Bianca Rosa · Answer

gt Are any env vars needed by MySQL available in that gRPC server task That s where the sensor executes That s helpful I m gonna isolate the user used to connect to the db to make sure its grabbing a connection when it starts

Bianca Rosa · Answer

So the user s access is not getting denied afaik If I change the password to a bad password I get a proper error and if the user doesn t have permissions to log into the db I also get a proper error

daniel · Answer

What version of dagster is this

Bianca Rosa · Answer

Running 0 15 5

Bianca Rosa · Answer

MySQL is in an older version tho

Bianca Rosa · Answer

Running `5 7 mysql aurora 2 07 2`

daniel · Answer

this happens 100 of the time on every sensor tick for that sensor

Bianca Rosa · Answer

yep

Bianca Rosa · Answer

it never worked on that env

daniel · Answer

The reason it s firing just on that sensor is that run status sensors have to hit the DB to learn what runs succeeded or failed but I don t yet understand why it can t connect to the DB especially if it s fine connecting to it in e g dagit

Bianca Rosa · Answer

We ve had a similar error when we tried to create our own queue coordinator maybe for the same reason

Bianca Rosa · Answer

It looks like <https github com dagster io dagster blob master python modules libraries dagster mysql dagster mysql run storage run storage py L66|here> when `retry mysql connection fn` is running its always returning a connection that s already closed by some reason thinking face

daniel · Answer

Yeah i don t immediately see why that would fire in the user code task but not the dagit task they both go through that same codepath

Bianca Rosa · Answer

Hey < daniel> Morning I got some updates enabled mysql general logs and there is a quite weird behavior on the logs I am no database specialist tho ```2022 07 25T15 30 07 427308Z11132431 Query SELECT jobs job body FROM jobs WHERE jobs job type = SENSOR 2022 07 25T15 30 07 428497Z11132431 Query rollback 2022 07 25T15 30 07 430312Z11132431 Quit ```

Bianca Rosa · Answer

There s a bunch of these logs connects succesfully runs query tries to commit and then just rollback

daniel · Answer

could your queries be hitting some kind of a timeout Is there a way to configure the server to increase timeout limits

Bianca Rosa · Answer

thinking face maybe

Bianca Rosa · Answer

it does look like the rollback is expected tho so if there is a commit there shouldn t be anything wrong <https stackoverflow com questions 52287683 general log of mysql server show a record query rollback after inserting to t|very trustful font>

Nick Thorne · Answer

If the time outs involve database queries have you looked at adding some indexes