I upgraded to 1 3 1 and made a new op based job just a singl dagster #ask-community

I upgraded to 1.3.1 and made a new op-based job (j...

Rasmus Bonnevie

04/21/2023, 2:10 PM

I upgraded to 1.3.1 and made a new op-based job (just a single op). It shows up nicely, resources are linked, (pythonic) config template displayed in "Info" tab, but again I cannot figure out how to actually run the job? I can see an older job I have has a launchpad tab, but this doesn't. Pressing "launch a run" in the Runs tab confusingly takes me back to the job listing, but this seems intended. Can there be an error somewhere preventing me from seeing the launch option? (also tried rebooting)

owen

04/21/2023, 8:53 PM

hi @Rasmus Bonnevie! mind sharing a screenshot of what you're seeing here? I wasn't able to replicate with this:

Copy code

from dagster import job, op


@op
def my_op():
    pass


@job
def my_job():
    my_op()

from here, I was able to navigate to the job page (from the left nav), then click on the "Launchpad" (next to "Overview"), then launch a run from there

Rasmus Bonnevie

04/24/2023, 9:12 PM

@owen sorry for the delayed reply! Was locked out of my remote environment. 😓 Just tried adding a new job

Copy code

@job()
def other_job():
    pass

and it shows up like in the image attached. What I just noticed is that it has 3 assets associated with it, so I guess Dagster assumes it to be an asset job - but why? All 3 associated assets are source assets.

owen

04/24/2023, 9:42 PM

ah I see -- because the job is empty (has no ops), the option to launch it is hidden (as launching it would do no work). if an op is added, then the Launchpad link will be in between "Overview" and "Runs". as for why the 3 assets are associated with it, is it possible that you had another job with the same name in the past, and that job produced those assets?

Rasmus Bonnevie

04/25/2023, 4:08 AM

@owen sorry, I wasn't clear I see - the issue occurred with a proper non-empty job, as illustrated by the newly attached screenshot, I just created the empty

other_job

to see if I could isolate or circumvent the problem. Both jobs have the same three source assets associated, so I don't see how it can be a naming thing? I have more than 3 source assets, so AFAICT the only distinguishing factor of the three is that they have an observe function.

Rasmus Bonnevie

04/25/2023, 4:38 AM

@owen I tried commenting out the observe function of the three source assets, and then it's all good and the launchpad appears in the jobs. But if I add the observe function back the jobs become asset jobs again. I can't figure out why they'd be associated. The source asset observe function is just taken from the docs more or less,

Copy code

def _observe():
        with open(full_path) as file:
            content = file.read()
        hash_sig = sha256()
        hash_sig.update(bytearray(content, "utf8"))
        return DataVersion(hash_sig.hexdigest())

Rasmus Bonnevie

04/25/2023, 11:23 AM

I checked, and the list of associated jobs update to include any additional observable source assets I add, so there is some implicit connection to observable source assets

Rasmus Bonnevie

04/27/2023, 8:14 AM

@owen I know you guys are busy especially prior to release, but this must be a bug right? This seems to reproduce it:

Copy code

from dagster import (Definitions, job,
                     observable_source_asset, op)


@observable_source_asset
def source():
    pass

@op
def do_something():
    pass

@job
def a_job():
    do_something()

defs = Definitions(assets=[source], jobs=[a_job])

owen

04/27/2023, 4:33 PM

ah sorry! this slipped out of my queue, and yikes this is definitely a bug. looking into this now

❤️ 1

2 Views

Open in Slack

Previous Next