https://dagster.io/ logo
#ask-community
Title
# ask-community
s

Steven Murphy

08/08/2023, 12:13 PM
Hi folks. Is there a means by which I can run
dagster dev
with a debugger? My use case is that I'm testing the Airbyte integration with Dagster, using the
load_assets_from_airbyte_instance
function. If I run
dagster dev -f airbyte.py
(without the if main part below), the
connection_filter_print
function will print. If I try and run it just as a Python file, it does not print. I think can get away with doing print statements, though feels a bit old school. Would be nice to test this without having to fire up
dagster dev
each time.
Copy code
from dagster_airbyte import load_assets_from_airbyte_instance
def connection_filter_print(meta: Optional[AirbyteConnectionMetadata]) -> bool:
    print(f"meta - [{meta}]")
    return True


if __name__ == "__main__":
    airbyte_assets = load_assets_from_airbyte_instance(airbyte_instance,
                                                       connection_filter=connection_filter_print,
                                                       )

    defs = Definitions(assets=[airbyte_assets])
z

Zach

08/08/2023, 3:28 PM
I need to look into it more but I've seen someone use run configurations in Pycharm to run
dagster job execute ...
on a specific job with run configuration, which allowed them to use Pycharm's built-in debugger
👍 1
Trying it out now, I think you'd have to have a version of your job configured which uses execute-in-process / the in-process executor and attach to that process. Alternatively you could just call your job from a python script / unit test and then it should be quite easy to use the debugger in Pycharm or VSCode
daggy love 1
In your case I think you could do something like
Copy code
from <your_library> import airbyte_asset # this could be one or many assets
from dagster import materialize_to_memory

def test_asset():
  materialize_to_memory([airbyte_asset])
then if you run the test you should be able to sue the debugger with breakpoints in your asset. As a bonus you get an end-to-end test that you can add to your CI/CD test suite.
t

Tim Castillo

08/08/2023, 3:54 PM
If you use VS Code, this GitHub Discussion outlines how someone else had previously configured attaching a debugger to their dagster process
❤️ 1
s

Steven Murphy

08/08/2023, 4:50 PM
I thought I'd done similar to the VSCode example before which didn't work at the time. Anyway, just tried it again setting up an equivalent config in Pycharm and it worked perfect, I've now got my breakpoints being triggered. Thanks to your both for your suggestions
❤️ 1
a

alexander jacob

01/29/2024, 1:10 PM
Thank you zach
dagster job execute
works fine to debug with pycharm :-)
100 Views