I have experienced some troubles when setting the sensor s c

Question

I have experienced some troubles when setting the sensor s cursor via the dagit UI Dagit shows the string I entered correctly but then the sensor always skips the runs as if there was some other characters in the sensor breaking the evaluation Could this be a bug in the UI like inserting a space or something like that

daniel · Answer

Hey Bernardo any chance you can share the code that reads the cursor What value are you setting it to in the Ui

Bernardo Cortez · Answer

Sure `class ADLSPartitionSensor ` `def init self context event stream listening table posterior job ` `self context = context` `self event stream = event stream` `self listening table = listening table` `self posterior job = posterior job` `def get date from folder self folder ` `members = folder name split ` `for path member in members ` `if data date part= in path member and path member in members 1 ` `partition date = path member split = 1 ` `last partition date = self context cursor if self context cursor else 0 ` `if partition date gt last partition date ` `formatted date = convert date str partition date from format= Y m d to format= Y m d ` `yield formatted date` `def run self ` `file system client = get file system client with connection string ` `partition dates to yield = date for folder in file system client get paths path=self listening table name recursive=False for date in self get date from folder folder if date ` ` yield temporally ordered RunRequests` `partition dates to yield sort ` `for date in partition dates to yield ` `if not self context cursor or date gt self context cursor ` `yield RunRequest ` `run key=get unique run key self event stream self posterior job name date ` `run config=read config job name=self posterior job name event stream=self event stream date=date ` ` ` `self context update cursor date ` Basically I put all partition dates of a certain table to `partition dates to yield` sort it and then iterate through it comparing each date with the cursor yielding a RunRequest if the date is higher than the cursor

Bernardo Cortez · Answer

Also I am trying to set the cursor with date in the format YYYYMMDD which is also what the sensor s code does when it runs automatically

Bernardo Cortez · Answer

< daniel> I have two new facts you might want to know sweat smile 1 starting with an empty cursor the sensor will issue the necessary runs and update it If I let that updated value be it does not work but if wrap it with quotation marks it works Somehow it seems that the default data type for the value written is integer not string as before 2 this has started happening on 2022 June 3rd Could there be a release that broke this

Bernardo Cortez · Answer

For example this works

Bernardo Cortez · Answer

I am updating the cursor with `self context update cursor date ` in which `date` is a string

daniel · Answer

Hey Bernardo does this only happen when the cursor is a numeric string like 20220805

daniel · Answer

If you add a ``` print CURSOR + str context cursor + TYPE + str type context cursor ``` at the start of your sensor function and check the daemon logs for the output it might give some clues about what s coming in When I set 20220805 as a sensor cursor in the dagit UI on 1 0 2 and print that at the beginning of a sensor function it outputs CURSOR 20220805 TYPE lt class str gt

daniel · Answer

which is what I would expect to happen

Bernardo Cortez · Answer

well my cursor is always a numeric string as it represents dates

Bernardo Cortez · Answer

when I test locally I get the same results you got When running in the cluster the prints don t show up in the logs I have access to I am trying to deploy a version in which I force the cursor to be string when reading it to see if it solves the problem

Bernardo Cortez · Answer

The fact that this starting happening on 2022 June 3rd rings a bell saying that it might be caused by some dagster s release don t you think

daniel · Answer

Very possible but I think getting those logs is the best way to sort this out they should be getting logged in the daemon

Bernardo Cortez · Answer

I tried that but they aren t In the daemon I only see INFO level messages and maybe prints get cut

daniel · Answer

Or sorry my mistake if you re running separate gRPC servers they would be in those servers Are you deployed on k8s

daniel · Answer

I m going to be unavailable until 9 4 due to parental leave but I m tagging in < prha> who knows a lot about sensor cursors and may be able to take a look at what s going on once you re able to pull those logs

Bernardo Cortez · Answer

I am trying to plot the cursor value and type wrapped in the SkipReason Congrats on your child and thank you

Bernardo Cortez · Answer

Where can I see SkipReasons in the dagit UI < prha>

prha · Answer

They should be available on the individual sensor page in a list of the recent ticks

Bernardo Cortez · Answer

Here

prha · Answer

Yes I think if you ve provided a SkipReason you can hover over the Skipped tag and view the message if one was provided

Bernardo Cortez · Answer

I only see this message although I had defined the following SkipReason

prha · Answer

Hi Bernardo What version of dagit are you running This is definitely a bug but I believe the fix went out in `0 14 13`

Bernardo Cortez · Answer

yeah I am using 0 14 6 The bug you refer to concerns only the fact that I cannot see the SkipReason or the sensor evaluation problem that started this thread < prha>

prha · Answer

The bug refers to the `SkipReason` not being visible w r t the underlying issue did you upgrade the version of dagster around the time that you started seeing the issue If you can share your logs from the sensor I might be able to help you debug the underlying issue

Bernardo Cortez · Answer

I have not upgraded dagster no

Bernardo Cortez · Answer

By upgrading dagster from 0 14 6 to 0 15 6 we managed to see SkipReason s message It seems that we were comparing string in different formats by accident YYYY MM DD vs YYYYMMDD For some reason it worked before and started crashing last June Perhaps because of some change in unicode s numeric value of the characters Thanks anyway

prha · Answer

Oh that s great Glad that you figured it out