https://dagster.io/ logo
Powered by
Title
a

Arun Kumar

09/11/2021, 1:59 AM
Hi team, any examples on how to use context managers with resource functions for cleanup? Can I just yield from the resource function?
c

chris

09/11/2021, 2:13 AM
Kinda buried in docs, but yes: 
@resource
@contextmanager
def my_cm_resource(_):
    yield "foo"
Can also do try/finally syntax
a

Arun Kumar

09/11/2021, 2:18 AM
Thanks a lot Chris:thankyou:
I tried this approach and seeing some errors with multi solid pipelines. In the below example, it works for 
solid1
, but errors out while executing 
solid2
Am I missing something here? 
class ServiceClient:
    def __init__(self):
        self.msg = "Foo"


@resource
@contextmanager
def service_resource(_):
    yield ServiceClient()


@solid(required_resource_keys={"service_client"})
def solid1(context):
    with context.resources.service_client as service:
        <http://context.log.info|context.log.info>(service.msg)


@solid(required_resource_keys={"service_client"})
def solid2(context):
    with context.resources.service_client as service:
        <http://context.log.info|context.log.info>(service.msg)


@pipeline(mode_defs=[ModeDefinition(resource_defs={"service_client": service_resource})])
def hello_pipeline():
    solid1()
    solid2()
AttributeError: args
  File "/Users/abalasubramani/.pyenv/versions/3.8.0/lib/python3.8/site-packages/dagster/core/execution/plan/utils.py", line 44, in solid_execution_error_boundary
    yield
  File "/Users/abalasubramani/.pyenv/versions/3.8.0/lib/python3.8/site-packages/dagster/utils/__init__.py", line 383, in iterate_with_context
    next_output = next(iterator)
  File "/Users/abalasubramani/.pyenv/versions/3.8.0/lib/python3.8/site-packages/dagster/core/execution/plan/compute_generator.py", line 65, in _coerce_solid_compute_fn_to_iterator
    result = fn(context, **kwargs) if context_arg_provided else fn(**kwargs)
  File "pipeline.py", line 24, in solid2
    with context.resources.service_client as service:
  File "/Users/abalasubramani/.pyenv/versions/3.8.0/lib/python3.8/contextlib.py", line 111, in __enter__
    del self.args, self.kwds, self.func
c

chris

09/12/2021, 1:02 PM
The resource has already been opened up for you as a cm, so you can just use it directly
Before running any solids, dagster initialized all resources
a

Arun Kumar

09/12/2021, 10:18 PM
Thanks for the response. I tried that first, but I was getting the below error 
AttributeError: '_GeneratorContextManager' object has no attribute 'msg'
  File "/Users/abalasubramani/.pyenv/versions/3.8.0/lib/python3.8/site-packages/dagster/core/execution/plan/utils.py", line 44, in solid_execution_error_boundary
    yield
  File "/Users/abalasubramani/.pyenv/versions/3.8.0/lib/python3.8/site-packages/dagster/utils/__init__.py", line 383, in iterate_with_context
    next_output = next(iterator)
  File "/Users/abalasubramani/.pyenv/versions/3.8.0/lib/python3.8/site-packages/dagster/core/execution/plan/compute_generator.py", line 65, in _coerce_solid_compute_fn_to_iterator
    result = fn(context, **kwargs) if context_arg_provided else fn(**kwargs)
  File "pipeline.py", line 23, in solid2
    <http://context.log.info|context.log.info>(context.resources.service_client.msg)
class ServiceClient:
    def __init__(self):
        self.msg = "Foo"


@resource
@contextmanager
def service_resource(_):
    yield ServiceClient()


@solid(required_resource_keys={"service_client"})
def solid1(context):
    <http://context.log.info|context.log.info>(context.resources.service_client.msg)


@solid(required_resource_keys={"service_client"})
def solid2(context):
    <http://context.log.info|context.log.info>(context.resources.service_client.msg)


@pipeline(mode_defs=[ModeDefinition(resource_defs={"service_client": service_resource})])
def hello_pipeline():
    solid1()
    solid2()
c

chris

09/13/2021, 2:06 AM
Hm looks like this might be a bug. Will investigate this coming week.
:thankyou: 1
a

Arun Kumar

09/13/2021, 6:35 AM
Thanks. FYI, I am using version 0.12.6
Hi @chris, Just wanted to check if you got a chance to take a look at this one 🙂
c

chris

09/22/2021, 4:55 AM
Hey Arun, apologies. This drifted under my radar. Taking another look now.
Yea this is a bug with how we handle the context manager syntax. A fix will go out in an upcoming release.
a

Arun Kumar

09/22/2021, 5:38 AM
Thanks a lot Chris🙏
Hi @chris, sorry to bug you again. Did the fix get make it to today's release? I don't see it mentioned in the change log https://github.com/dagster-io/dagster/releases/tag/0.12.12
c

chris

09/25/2021, 1:04 AM
Hi Arun, no worries for the bugging. Apologies for my scant replies, currently out of office till mid next week. The fix is up for review, but did not land in this release. Will be in for next release.
👍 1
:thankyou: 1
a

Arun Kumar

09/25/2021, 9:53 AM
Thanks for the update 🙏
Hi @chris, a quick check 🙂 Did the above change already land? could not find it in the change log
c

chris

10/26/2021, 4:21 AM
Hey Arun, apologies for not keeping more up to date about this. https://github.com/dagster-io/dagster/pull/4892 is the change, has not landed yet. Now that 0.13.0 is out, should have time to get it in. But to be clear, not in a release yet
a

Arun Kumar

10/26/2021, 4:23 AM
Thanks Chris for the update. Not very urgent, but have some TODOs in our pipeline code which could be removed by this change 😄
c

chris

10/26/2021, 4:49 AM
Appreciate the understanding. Will keep you updated
:next-level-daggy: 1
Hey Arun, just wanted to follow up that the fix for this was merged in master ! You can use context manager decorator like so: 
from contextlib import contextmanager
from dagster import resource

@resource
@contextmanager
def my_resource():
    try:
        yield ...
    finally:
        ...
a

Arun Kumar

10/28/2021, 10:23 PM
Thanks a lot @chris :celebrate: Will try this out sometime next week
c

chris

10/28/2021, 10:23 PM
Of course. It should be in for the next release
:next-level-daggy: 1
#dagster-supportJoin Slack
Powered by