https://dagster.io/ logo
#ask-community
Title
# ask-community
r

Robert Wade

03/14/2023, 4:08 PM
RE: Time change issue On March 10th I created a schedule in Dagster which executes a job. The schedule is set for everyday at 2am. I did not consider the upcoming time change when choosing this time. When I go look at the runs in Dagster I see the following (I am listing these from the bottom of the screen to the top): - March 11 2am - March 12 3am - March 13 1am - March 14 1am - March 14 2am As a reminder, the time change happened on March 12. The March 11 execution is obviously as expected. On March 12 2am instantly become 3am, so I guess that makes sense too. The March 13 1am doesn’t make sense to me, and the 2 March 14 executions definitely don’t make sense to me. Any ideas about why this 2am schedule was handled like this? Thanks in advance
d

daniel

03/14/2023, 4:11 PM
Hi Robert - what timezone did you use and what version of Dagster are you on?
Is it possible to post your schedule code where you specified the timezone and the daily schedule?
r

Robert Wade

03/14/2023, 4:15 PM
We are using eastern time -- I am checking to see if someone can help me get the exact timezone settings. Dagster version 1.1.14
d

daniel

03/14/2023, 4:15 PM
No problem - if its possible to post a code snippet I can see if we can reproduce the problem
r

Robert Wade

03/14/2023, 4:16 PM
the schedule is annotated as follows: @schedule(job=populate_job, cron_schedule="0 2 * * *", execution_timezone="US/Eastern")
d

daniel

03/14/2023, 5:07 PM
How odd... we even have a test for this that verifies that it still runs exactly one run for each day, at either 2AM or 3AM on the correct day. I tried changing it to 2023 and US/Eastern and it still passed, so something strange is definitely happening. Two questions: would it be possible to check what version of the
pendulum
package you have installed? And any chance you could pass along the logs from the daemon during those unexpected ticks on 3/13 and 3/14?
r

Robert Wade

03/14/2023, 5:09 PM
Pendulum is 2.1.2
where would i find the daemon logs?
d

daniel

03/14/2023, 5:10 PM
That will depend on how you're deploying dagster - if you're not sure, it may be difficult to grab the logs, but often in production they would be in some container logs somewhere
If you're running dagster locally, they would be in the console output of
dagster dev
Can I also ask what version of
croniter
you have installed? Still seeing if I can try to reproduce some version of this problem myself
r

Robert Wade

03/14/2023, 5:13 PM
Croniter 0.3.34
d

daniel

03/14/2023, 5:22 PM
I think I can reproduce a version of this problem with that version of croniter, but not with that cron string that you posted - just to confirm, are you certain that it's
0 2 * * *
and not something more complicated?
r

Robert Wade

03/14/2023, 5:23 PM
no it is exactly 0 2 * * *
d

daniel

03/14/2023, 5:24 PM
OK - I think there's a good shot that upgrading croniter to a more recent version will fix this problem, although I don't have a full understanding of why it would be hitting croniter in the first place yet
r

Robert Wade

03/14/2023, 5:25 PM
oh wait -- in our build i see: Collection cronitor>=0.3.34
but the next line shows: Downloading ...... croniter-1.3.8-py2.py3.non-any.whl
d

daniel

03/14/2023, 5:27 PM
If you run `pip freeze | grep croniter' that should give you the final answer
The full output of pip freeze might be useful if you're able to send that
r

Robert Wade

03/14/2023, 5:36 PM
i don't have ssh into our docker container, but i can share some output that occurs during the build that shows all the packages installed
Successfully installed Jinja2-3.1.2 Mako-1.2.4 MarkupSafe-2.1.2 PyYAML-6.0 Send2Trash-1.8.0 alembic-1.10.2 aniso8601-9.0.1 anyio-3.6.2 argon2-cffi-21.3.0 argon2-cffi-bindings-21.2.0 arrow-1.2.3 asttokens-2.2.1 backcall-0.2.0 backoff-2.2.1 bcrypt-4.0.1 beautifulsoup4-4.11.2 bleach-6.0.0 click-8.1.3 colorama-0.4.6 coloredlogs-14.0 comm-0.1.2 configparser-5.3.0 credstash-1.17.1 credstashfx-5.1.0 croniter-1.3.8 dagit-1.1.14 dagster-1.1.14 dagster-graphql-1.1.14 dagster-pandas-0.17.14 dagster-postgres-0.17.14 debugpy-1.6.6 decorator-5.1.1 defusedxml-0.7.1 distro-1.8.0 docstring-parser-0.15 ec2-metadata-2.11.0 executing-1.2.0 fastjsonschema-2.16.3 fqdn-1.5.1 gevent-22.10.2 gql-3.4.0 graphene-3.2.1 graphql-core-3.2.3 graphql-relay-3.2.0 greenlet-2.0.2 grpcio-1.47.5 grpcio-health-checking-1.43.0 h11-0.14.0 httptools-0.5.0 humanfriendly-10.0 ipykernel-6.21.3 ipython-8.11.0 ipython-genutils-0.2.0 ipywidgets-8.0.4 isoduration-20.11.0 jedi-0.18.2 jsonpath-ng-1.5.3 jsonpointer-2.3 jsonschema-4.17.3 jupyter-1.0.0 jupyter-client-8.0.3 jupyter-console-6.6.3 jupyter-core-5.2.0 jupyter-events-0.6.3 jupyter-server-2.4.0 jupyter-server-terminals-0.4.4 jupyterlab-pygments-0.2.2 jupyterlab-widgets-3.0.5 matplotlib-inline-0.1.6 mistune-2.0.5 nbclassic-0.5.3 nbclient-0.7.2 nbconvert-7.2.9 nbformat-5.7.3 nest-asyncio-1.5.6 notebook-6.5.3 notebook-shim-0.2.2 opensearch-py-2.1.1 pandocfilters-1.5.0 paramiko-2.12.0 parso-0.8.3 pendulum-2.1.2 pexpect-4.8.0 pickleshare-0.7.5 platformdirs-3.1.1 ply-3.11 prometheus-client-0.16.0 prompt-toolkit-3.0.38 protobuf-3.20.3 psutil-5.9.4 psycopg2-binary-2.9.5 ptyprocess-0.7.0 pure-eval-0.2.2 py4j-0.10.9.5 pyarrow-11.0.0 pycryptodome-3.17 pydantic-1.10.6 pygments-2.14.0 pyinfra-2.6.2 pynacl-1.5.0 pyrsistent-0.19.3 pyspark-3.3.1 pyspnego-0.8.0 pytest-mock-3.10.0 python-dotenv-1.0.0 python-json-logger-2.0.7 pytzdata-2020.1 pywinrm-0.4.3 pyzmq-25.0.0 qtconsole-5.4.0 qtpy-2.3.0 requests-2.28.2 requests-ntlm-1.2.0 requests-toolbelt-0.10.1 requests_aws4auth-1.2.1 rfc3339-validator-0.1.4 rfc3986-validator-0.1.1 smart-open-6.3.0 sniffio-1.3.0 soupsieve-2.4 sqlalchemy-1.4.46 stack-data-0.6.2 starlette-0.26.0.post1 tabulate-0.9.0 terminado-0.17.1 tinycss2-1.2.1 toposort-1.10 tornado-6.2 tqdm-4.65.0 traitlets-5.9.0 universal-pathlib-0.0.21 uri-template-1.2.0 uvicorn-0.21.0 uvloop-0.17.0 watchdog-2.3.1 watchfiles-0.18.1 wcwidth-0.2.6 webcolors-1.12 webencodings-0.5.1 websocket-client-1.5.1 websockets-10.4 widgetsnbextension-4.0.5 xmltodict-0.13.0 zope.event-4.6 zope.interface-5.5.2
d

daniel

03/14/2023, 6:01 PM
Is your dagster code and your daemon running in the same container / using the same package versions?
r

Robert Wade

03/14/2023, 6:01 PM
yes
d

daniel

03/14/2023, 6:05 PM
OK, I was able to reproduce something similar to this problem with an older version of croniter, but it sounds like you're sure that it was on that newer version when the problem was happening
If it’s possible to get the logs from that container that was running the schedules, that may be needed in order to get to the bottom of this
r

Robert Wade

03/14/2023, 6:33 PM
ok. i'm working on that
i was able to get a 500MB dump of logs from March 11 to March 14.
d

daniel

03/14/2023, 7:05 PM
Got it - if you have some way of sharing that with daniel@elementl.com I'd be happy to take a look
r

Robert Wade

03/14/2023, 7:28 PM
sent
d

daniel

03/14/2023, 8:16 PM
thanks - what's the name of the schedule that was misbehaving?
r

Robert Wade

03/14/2023, 8:18 PM
populate_alias_schedule
d

daniel

03/14/2023, 8:27 PM
Is there anything more that you can share about your deployment setup? You're running dagit and the daemon all in the same container, and using the default run launcher so that runs happen in that same container too?
r

Robert Wade

03/14/2023, 8:32 PM
i have been corrected. Dagit and the daemon are in the same fargate task, each in their own container
we are using the QueuedRunCoordinator
d

daniel

03/14/2023, 8:33 PM
OK, but you mentioned earlier that there are no separate tasks or containers that run your code
(not that what you're saying doesn't square with that - just confirming)
r

Robert Wade

03/14/2023, 8:34 PM
i believe that to be true