https://dagster.io/ logo
Join Slack
Powered by
Hello everyone, I’m querying a table that only al...
# ask-community
j
Hello everyone, I’m querying a table that only allow data from the last 7 days to be queried. I’m wondering if there’s a way to block dagster partitions to be rebuild after that 7 days period? The reason is that if I rebuild something from more than 7 days I’ll receive an error or and empty string from the API, and I’ll update the records with that and I don’t want that at all. Is there a way to achieve this?
🤖 1
j
Have you seen this example in the documentation? It might be overkill for what you are trying to do, but seems like it is getting at the same problem. https://docs.dagster.io/concepts/partitions-schedules-sensors/asset-sensors#monitoring-daily-assets-to-materia[…]ly-asset-using-a-multi_asset_sensor
s
Hi Jacob, When you say “rebuild”, do you mean “rematerialized”? Or are you talking about changing the partitions definition itself?
Just checking in to see if you’ve made progress on this issue?
j
@sean I mean 
rematerialized
sorry for the confusion. And oh thank you I didn’t see those comments at all pop out in slack. I”ll look at the propose solution now. Thank you very much.
@Justin Taylor The documentation you sent me seems to be link to something else. Aka the materialization of downstream assets based of if a 
materialized
group of parent asset exist or not. My question was more. If the 
client
I’m using. Isn’t able to query more than X days in the past. is it possible to restrict users from 
rematerializing
assets that are 
x days
in the past from now. I know that I can restrict that in the 
client
itself. Like 
return
an 
error
in the partition I’m trying to rebuild is older than 
x days
. But I was wondering if that could be done in Dagster itself. Like a way to blury the materialization or something. Something that tell me, than this partition is 
completed
(green) or 
not
(red), but there’s no way for me to rebuild it by clicking on 
materialize
.
s
IIUC Jacob, a simple model of your situation is that you have a single partitioned asset, like this:
Copy code
@asset(partitions_def=DailyPartitionsDefinition(start_date=datetime(2020, 1, 1))
def my_asset(context: AssetExecutionContext):
    return client.query(context.partition_key)
And your question is whether it is possible to block any attempts to materialize certain partitions (those older than N days). Is that right?
j
Exactly.
s
That is not possible right now, though I see how this could be useful. If you want to see this feature, the best thing is to open a GH issue: https://github.com/dagster-io/dagster/issues
j
I know it’s possible in the asset itself. But was wondering if Dagster has any flag that block the rebuild and show it to the user. Any 
immutable
records that I get from dagster, shouldn’t be replayable IMO. But having the partitions setup so I know if I miss any is still very usefull. Thank @sean I’ll fill a feature request 🙂 Have a great day.
2 Views
Open in Slack
PreviousNext
Powered by