I have experienced some troubles when setting the ...
# ask-community
b
I have experienced some troubles when setting the sensor's cursor via the dagit UI. Dagit shows the string I entered correctly, but, then, the sensor always skips the runs, as if there was some other characters in the sensor breaking the evaluation. Could this be a bug in the UI, like inserting a space or something like that?
d
Hey Bernardo - any chance you can share the code that reads the cursor? What value are you setting it to in the Ui?
b
Sure!
class ADLSPartitionSensor:
def __init__(self, context, event_stream, listening_table, posterior_job):
self.context = context
self.event_stream = event_stream
self.listening_table = listening_table
self.posterior_job = posterior_job
def _get_date_from_folder(self, folder):
members = folder.name.split('/')
for path_member in members:
if ("data_date_part=" in path_member) and (path_member in members[-1]):
partition_date = path_member.split('=')[1]
last_partition_date = self.context.cursor if self.context.cursor else '0'
if partition_date>last_partition_date:
formatted_date = convert_date_str(partition_date, from_format='%Y-%m-%d', to_format='%Y%m%d')
yield formatted_date
def run(self):
file_system_client = get_file_system_client_with_connection_string()
partition_dates_to_yield = [date for folder in file_system_client.get_paths(path=self.listening_table['name'], recursive=False) for date in self._get_date_from_folder(folder) if date]
# yield temporally ordered RunRequests
partition_dates_to_yield.sort()
for date in partition_dates_to_yield:
if not self.context.cursor or date>self.context.cursor:
yield RunRequest(
run_key=get_unique_run_key(self.event_stream, self.posterior_job.name, date),
run_config=read_config(job_name=self.posterior_job.name, event_stream=self.event_stream, date=date)
)
self.context.update_cursor(date)
Basically, I put all partition dates of a certain table to
partition_dates_to_yield
, sort it and then iterate through it comparing each date with the cursor, yielding a RunRequest if the date is higher than the cursor
Also, I am trying to set the cursor with date in the format YYYYMMDD, which is also what the sensor's code does when it runs automatically
@daniel I have two new facts you might want to know! 😅 1 - starting with an empty cursor, the sensor will issue the necessary runs and update it. If I let that updated value be, it does not work, but if wrap it with quotation marks " it works. Somehow, it seems that the default data type for the value written is integer, not string as before 2 - this has started happening on 2022 June, 3rd Could there be a release that broke this?
For example: this works
I am updating the cursor with
self.context.update_cursor(date)
, in which
date
is a string
d
Hey Bernardo - does this only happen when the cursor is a numeric string like 20220805?
If you add a
Copy code
print("CURSOR: " + str(context.cursor) + " , TYPE: " + str(type(context.cursor)))
at the start of your sensor function and check the daemon logs for the output, it might give some clues about what's coming in When I set 20220805 as a sensor cursor in the dagit UI (on 1.0.2) and print that at the beginning of a sensor function, it outputs: CURSOR: 20220805 , TYPE: <class 'str'>
which is what I would expect to happen
b
well, my cursor is always a numeric string, as it represents dates...
when I test locally, I get the same results you got. When running in the cluster, the prints don't show up in the logs I have access to. I am trying to deploy a version in which I force the cursor to be string when reading it, to see if it solves the problem
The fact that this starting happening on 2022 June, 3rd rings a bell saying that it might be caused by some dagster's release, don't you think?
d
Very possible but I think getting those logs is the best way to sort this out - they should be getting logged in the daemon
b
I tried that but they aren't. In the daemon I only see INFO level messages and maybe prints get cut
d
Or sorry, my mistake - if you're running separate gRPC servers they would be in those servers. Are you deployed on k8s?
I'm going to be unavailable until 9/4 due to parental leave, but I'm tagging in @prha who knows a lot about sensor cursors and may be able to take a look at what's going on once you're able to pull those logs
b
I am trying to plot the cursor value and type wrapped in the SkipReason. Congrats on your child and thank you!
Where can I see SkipReasons in the dagit UI? @prha
p
They should be available on the individual sensor page, in a list of the recent ticks
b
Here?
p
Yes, I think if you’ve provided a SkipReason, you can hover over the ‘Skipped’ tag and view the message (if one was provided).
b
I only see this message, although I had defined the following SkipReason
p
Hi Bernardo. What version of dagit are you running? This is definitely a bug, but I believe the fix went out in
0.14.13
.
b
yeah I am using 0.14.6... The bug you refer to concerns only the fact that I cannot see the SkipReason or the sensor evaluation problem that started this thread? @prha
p
The bug refers to the
SkipReason
not being visible. w.r.t. the underlying issue, did you upgrade the version of dagster around the time that you started seeing the issue? If you can share your logs from the sensor, I might be able to help you debug the underlying issue…
b
I have not upgraded dagster, no...
By upgrading dagster from 0.14.6 to 0.15.6, we managed to see SkipReason's message. It seems that we were comparing string in different formats, by accident (YYYY-MM-DD vs. YYYYMMDD). For some reason, it worked before and started crashing last June. Perhaps because of some change in unicode's numeric value of the characters. Thanks anyway!
p
Oh, that’s great! Glad that you figured it out!