Hi folks Is there a means by which I can run `dagster dev` w dagster #ask-community

Hi folks. Is there a means by which I can run `dag...

Steven Murphy

08/08/2023, 12:13 PM

Hi folks. Is there a means by which I can run

dagster dev

with a debugger? My use case is that I'm testing the Airbyte integration with Dagster, using the

load_assets_from_airbyte_instance

function. If I run

dagster dev -f airbyte.py

(without the if main part below), the

connection_filter_print

function will print. If I try and run it just as a Python file, it does not print. I think can get away with doing print statements, though feels a bit old school. Would be nice to test this without having to fire up

dagster dev

each time.

Copy code

from dagster_airbyte import load_assets_from_airbyte_instance
def connection_filter_print(meta: Optional[AirbyteConnectionMetadata]) -> bool:
    print(f"meta - [{meta}]")
    return True


if __name__ == "__main__":
    airbyte_assets = load_assets_from_airbyte_instance(airbyte_instance,
                                                       connection_filter=connection_filter_print,
                                                       )

    defs = Definitions(assets=[airbyte_assets])

Zach

08/08/2023, 3:28 PM

I need to look into it more but I've seen someone use run configurations in Pycharm to run

dagster job execute ...

on a specific job with run configuration, which allowed them to use Pycharm's built-in debugger

👍 1

Zach

08/08/2023, 3:49 PM

Trying it out now, I think you'd have to have a version of your job configured which uses execute-in-process / the in-process executor and attach to that process. Alternatively you could just call your job from a python script / unit test and then it should be quite easy to use the debugger in Pycharm or VSCode

daggy love 1

Zach

08/08/2023, 3:53 PM

In your case I think you could do something like

Copy code

from <your_library> import airbyte_asset # this could be one or many assets
from dagster import materialize_to_memory

def test_asset():
  materialize_to_memory([airbyte_asset])

then if you run the test you should be able to sue the debugger with breakpoints in your asset. As a bonus you get an end-to-end test that you can add to your CI/CD test suite.

Tim Castillo

08/08/2023, 3:54 PM

If you use VS Code, this GitHub Discussion outlines how someone else had previously configured attaching a debugger to their dagster process

❤️ 1

Steven Murphy

08/08/2023, 4:50 PM

I thought I'd done similar to the VSCode example before which didn't work at the time. Anyway, just tried it again setting up an equivalent config in Pycharm and it worked perfect, I've now got my breakpoints being triggered. Thanks to your both for your suggestions

❤️ 1

alexander jacob

01/29/2024, 1:10 PM

Thank you zach

dagster job execute

works fine to debug with pycharm :-)

163 Views

Open in Slack

Previous Next