https://dagster.io/ logo
#ask-community
Title
# ask-community
d

Danny Steffy

03/21/2023, 5:21 PM
is it possible to run something on resource cleanup? For instance, we are trying to track database connections in order to properly manage load against a group of databases. When we're finished with the resource, we'd like to decrement a counter in a central database
j

jamie

03/21/2023, 6:12 PM
this doesn’t exist right now, but is something we’re looking into. Not 100% sure if this will meet your requirements, but could you use a context manager within the resource? For example, in the duckdb io manager (basically just a resource) we have a function that is a context manager and does some cleanup when the context is closed https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-duckdb/dagster_duckdb/io_manager.py#L132 in using this you would do something like
Copy code
with duckdb_resource.connect() as con:
    # do stuff

# do more stuff outside the with statement
and then after the
with
bock is exited, the section of the
connect
function after the
yield
is executed
d

Danny Steffy

03/21/2023, 6:32 PM
interesting, so I would define the resource as a class that has this
connect
method and that yields my actual database connection, and then after the yield, do my cleanup work? That would probably work for our usecase
I'm not too familiar with python, would this syntax be correct or am I missing anything?
Copy code
@resource
def sql_server_round_robin(init_context):
    def factory(scoring_pipe_label):
        @contextmanager
        def connect():
            cnxn = get_scoring_server(init_context, scoring_pipe_label)
            yield cnxn[1]
            cleanup_dispatch(cnxn[0])

    return factory
or could I define the "factory" part as the context manager and just get rid of the connect call altogether?
j

jamie

03/21/2023, 7:16 PM
You should be able to have the factory be the context manager and get rid of the nested function
👍 1
d

Danny Steffy

03/21/2023, 7:22 PM
thank you!
we have a retry on the op that uses this resource, and the behavior we're seeing is that the failed ops that are waiting for retry aren't cleaning up their connection status. Is there some way to force the context to close when an exception occurs?
j

jamie

03/23/2023, 3:12 PM
you could put a try finally in your context manager and then the finally will always be run, even if there’s a failure
Copy code
@contextmanager
def managed_resource(*args, **kwds):
    # Code to acquire resource, e.g.:
    resource = acquire_resource(*args, **kwds)
    try:
        yield resource
    finally:
        # Code to release resource, e.g.:
        release_resource(resource)
some helpful docs https://docs.python.org/3/library/contextlib.html#contextlib.contextmanager
d

Danny Steffy

03/23/2023, 3:19 PM
thanks!
2 Views