https://dagster.io/ logo
#dagster-support
Title
# dagster-support
o

Oliver

06/23/2022, 11:23 PM
Hey, I'm seeing this when clicking materialise on a specific asset in a pipeline. If I go to the launchpad for the pipeline it tells me all config is present and correct. If I run the entire pipeline it also runs successfully. Google is not being kind on this one
🤖 1
z

Zach

06/23/2022, 11:43 PM
I'm seeing the same behavior and am similarly a bit confused... are you grouping these assets using
load_assets_from_modules
?
o

Oliver

06/23/2022, 11:47 PM
Phew, not only me! 😅 I have workspace.yml like this
Copy code
load_from:
  - python_package: tmp
and
tmp/__init__.py
has an
@repository
definition. some of my assets are defined as
@asset
some from
AssetsDefinition.from_graph
..now that you mention it, I think it's the
AssetsDefinition.from_graph
that are having this issue
i will check further in 20 mins and reprot back
definitely everything that pops this error is
AssetsDefinition.from_graph
workaround by removing config from these assets (with
asset.configured
) but that's not ideal
s

sandy

06/24/2022, 12:21 AM
hey Oliver - are you supplying a config value for
define_asset_job
? or manually specifying config in the launchpad at the time you launch the job?
o

Oliver

06/24/2022, 1:33 AM
I am using
build_asset_selection_job
Copy code
dev_asset_job = build_asset_selection_job("dev", assets=resourced_assets, source_assets=[],        config=yaml.safe_load(Path("tmp/dev.yaml").read_text()),
        resource_defs=dev_resource_defs,
)
s

sandy

06/24/2022, 3:20 PM
Ah, I see what's going on. This is because you've supplied the config for the job, but not for the assets themselves - that means it will apply when you run the job, but not when you materialize the assets independently from the job. You could do something like:
Copy code
@repository
def dev_repo():
    resource_config = yaml.safe_load(Path("tmp/dev.yaml").read_text())
    return [with_resources(assets, dev_resource_defs, resource_config), define_asset_job("dev", selection=AssetSelection.assets(assets))]
By the way, any particular reason to use
build_asset_selection_job
instead of
define_asset_job
? The former is an internal API that could change in a future release.
o

Oliver

06/26/2022, 2:27 AM
Ah ok thanks.
build_asset_selection_job
allows you to specify the executor while
define_asset_job
does not which is why I'm using it. Could switch that to be defined on the repo but it's a little awkward for the workflow im developing. I have one job that uses local multiprocessing executor for quick dev and testing. Then I have another job that uses a custom executor to run pipelines on infra in a cloud environment to test on more large scale datasets
I am only seeing this issue on assets defined from graphs though. assets defined using
@asset
run fine even without supplying config
s

sandy

06/27/2022, 3:46 PM
I have one job that uses local multiprocessing executor for quick dev and testing. Then I have another job that uses a custom executor to run pipelines on infra in a cloud environment to test on more large scale datasets
Got it - do you want these jobs to show up in Dagit at the same time? Our typical recommendation is to put these jobs in different dagster repos, so that you can target one with your production setup and another with your local dev setup.
o

Oliver

06/29/2022, 2:01 AM
oh I see, yup.. nice thanks 🙂
2 Views