https://dagster.io/ logo
Join Slack
Powered by
<https://docs.dagster.io/concepts/assets/asset-aut...
# ask-community
g
https://docs.dagster.io/concepts/assets/asset-auto-execution what are the implications of auto? For the reconciliation sensor it was possible to specify the delay how often the sensor is executed (and shut down the containers otherwise) is something similar possible for auto-execution?
j
You were using the sensor evaluation interval to guarantee that assets wouldn’t be materialized too frequently? I’m curious what value you had it set to, and in what cases you might get too many runs without it.
(and shut down the containers otherwise)
Not sure what this is referring to
g
no it was set to 30 seconds 😉
My question was rather theoretical. I wanted to understand cloud costs of running too many code locations/containers
and given that the shut down period can then save cloud credits it would be great to undestand how this works using the automatic execution and shutdown
j
internally the reconciliation loop currently runs every 30s, but we’d like to treat that as more of an implementation detail than something you rely on
Internally we’ve been discussing a parameter on AutoMaterializePolicy that would specify a minimum time between materializations of a certain asset. I’m not sure that’s quite what you want here though
You’re thinking about setting up a sort of scheduled downtime for your agent and grpc servers?
g
not the agent only the rpcs
as long as we have in the 10s of code locations / RPCs this is fine
but if we have 100s or 1000s and most of these polling some sensor every 30 seconds this is bad (the shutdown feature no longer works). And I already expect that we will not have most sensors inside only a limited number of code locations.
That parameter could sort-of work. I basically want to ensure that the shutdown of the GRPCs can actually take place
j
g
yes
and it does not work if the code location is constantly up with a sensor polling every 30 seconds
j
So the new AutoMaterializePolicy will be an improvement over the previous sensor here- it doesn’t need to poll the grpc servers unless it’s launching a run for one of them
We store enough metadata on our side to determine what needs to be reconciled without reloading your code on your side
g
ah this is amazing!
🚀
Open in Slack
PreviousNext
Powered by