https://dagster.io/ logo
#ask-community
Title
# ask-community
c

clay

03/23/2023, 4:22 PM
The issue seems to be that if I do:
Copy code
my_assets = load_assets_from_package_module(package_module=my_module, group_name="MY_GROUP")
I cannot then do
all_assets = [*my_assets]
to feed into the
Definitions
for the code location if
my_assets
mixes Source and Regular assets. However, if I do:
Copy code
my_regular_assets = load_assets_from_package_module(package_module=my_regular_module, group_name="MY_REGULAR_ASSET_GROUP")
and also
Copy code
my_source_asset = SourceAsset(key=AssetKey("my_source_asset_key"))
then I can do the following without a problem:
Copy code
all_assets = [*my_regular_assets, my_source_asset]
j

jamie

03/23/2023, 4:48 PM
hey @clay I’m able to pass both assets and source assets loaded via
load_assets_from_package_module
to the
assets
parameter of Definitions. Could you share the code for your Definitions so i can see if something else is going on?
c

clay

03/23/2023, 6:00 PM
Sure, sorry... had to catch a swarm of bees in my yard and put them in a hive. 🐝 I have a file for the assets related to a specific group, called
jira_kpis.py
. It's in a submodule. In it, I had:
Copy code
jira_tickets_source_asset = SourceAsset(
    key=AssetKey("ED_CRM_DASH_CRM_JIRA_TICKETS_CONSOLIDATED"),
)
Followed by:
Copy code
@asset(required_resource_keys={"snowflake"}, compute_kind="Python")
def calc_jira_kpis(context: OpExecutionContext, ED_CRM_DASH_CRM_JIRA_TICKETS_CONSOLIDATED: pd.DataFrame) -> None:
    """
    Calculate Jira-derived KPIs.
    """
    stuff
    return None
In the parent module
assets
(similar to one of the dagster examples) in the
__init__.py
file, I had:
Copy code
jira_derived_assets = load_assets_from_package_module(package_module=jira_kpis, group_name=JIRA_KPIS)
and some other irrelevant stuff The parent module to that is the main code location where I am defining stuff in
__init__.py
. In there, I had:
Copy code
all_assets = [*player_base_kpi_assets, *channel_kpi_assets, *jira_derived_assets]
and
Copy code
defs = Definitions(
    assets=all_assets,
    .... stuff
)
Presumably, my call to
load_assets_from_package_module
was picking up the
SourceAsset
. When I ran this in 1.1.18, it all worked just fine. Then I upgraded to 1.2.3 and the code location would not load. The error I saw when I did
docker-compose up
was the one I mentioned:
dagster._check.CheckError: Invariant failed. Description: Asset selection specified both regular assets and source assets. This is not currently supported. Selections must be all regular assets or all source assets.
When I moved the
SourceAsset
definition to the same file as the Definitions and then did:
Copy code
all_assets = [*player_base_kpi_assets, *channel_kpi_assets, *jira_derived_assets, jira_tickets_source_asset]
Followed by
Copy code
defs = Definitions(
    assets=all_assets,
    .... stuff
)
The error went away and everything worked as expected. It seemed that trying to unpack
*jira_derived_assets
when it contained both Regular and Source assets was the problem?
j

jamie

03/23/2023, 6:04 PM
ok. Are you making an
AssetSelection
object anywhere? also could you share the full stack trace?
c

clay

03/23/2023, 6:06 PM
Is that "create a post"? for a longer snippet?
j

jamie

03/23/2023, 6:06 PM
here’s my replication setup in
foo/__init__.py
Copy code
from dagster import asset, SourceAsset

baz_source = SourceAsset(key="baz")


@asset(
    op_tags={"hello": "world"}
)
def foo():
    return 1
in
bar/__init__.py
Copy code
from dagster import asset

@asset
def bar():
    return 1
in
__init__.py
Copy code
from dagster import Definitions, load_assets_from_package_module
from . import foo, bar



defs = Definitions(
    assets=[*load_assets_from_package_module(foo, group_name="foo"), *load_assets_from_package_module(bar, group_name="bar")]
)
and when i run the ui everything loads correctly and i see the three assets. might be worthwhile to try the minimal setup yourself just to make sure the error you’re seeing is from the
load_assets_from_package_modules
and not from something else
c

clay

03/23/2023, 6:07 PM
I'll just put the stack trace here
Copy code
kpis                | Traceback (most recent call last):
kpis                |   File "/usr/local/bin/dagster", line 8, in <module>
kpis                |     sys.exit(main())
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_cli/__init__.py", line 46, in main
kpis                |     cli(auto_envvar_prefix=ENV_PREFIX)  # pylint:disable=E1123
kpis                |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
kpis                |     return self.main(*args, **kwargs)
kpis                |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1055, in main
kpis                |     rv = self.invoke(ctx)
kpis                |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
kpis                |     return _process_result(sub_ctx.command.invoke(sub_ctx))
kpis                |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
kpis                |     return _process_result(sub_ctx.command.invoke(sub_ctx))
kpis                |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
kpis                |     return ctx.invoke(self.callback, **ctx.params)
kpis                |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 760, in invoke
kpis                |     return __callback(*args, **kwargs)
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_cli/api.py", line 724, in grpc_command
kpis                |     location_name=location_name,
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/server.py", line 942, in __init__
kpis                |     location_name=location_name,
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/server.py", line 269, in __init__
kpis                |     self._container_image,
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/server.py", line 120, in __init__
kpis                |     loadable_target_origin.attribute,
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/utils.py", line 47, in get_loadable_targets
kpis                |     else loadable_targets_from_python_module(module_name, working_directory)
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_core/workspace/autodiscovery.py", line 39, in loadable_targets_from_python_module
kpis                |     remove_from_path_fn=remove_from_path_fn,
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_core/code_pointer.py", line 135, in load_python_module
kpis                |     return importlib.import_module(module_name)
kpis                |   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
kpis                |     return _bootstrap._gcd_import(name[level:], package, level)
kpis                |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
kpis                |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
kpis                |   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
kpis                |   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
kpis                |   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
kpis                |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
kpis                |   File "/opt/dagster/app/kpis/kpis/__init__.py", line 97, in <module>
kpis                |     jobs=[player_base_kpi_job, channel_kpi_job, jira_kpi_update_job],
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/definitions_class.py", line 401, in __init__
kpis                |     loggers=loggers,
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/definitions_class.py", line 277, in _create_repository_using_definitions_args
kpis                |     _top_level_resources=resource_defs,
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/decorators/repository_decorator.py", line 133, in __call__
kpis                |     top_level_resources=self.top_level_resources,
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/repository_definition/repository_data.py", line 482, in from_list
kpis                |     top_level_resources=top_level_resources,
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/repository_definition/repository_data_builder.py", line 273, in build_caching_repository_data_from_list
kpis                |     asset_graph=asset_graph, default_executor_def=default_executor_def
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/unresolved_asset_job_definition.py", line 204, in resolve
kpis                |     selected_asset_keys = self.selection.resolve(asset_graph)
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/asset_selection.py", line 196, in resolve
kpis                |     "Asset selection specified both regular assets and source assets. This is not"
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_check/__init__.py", line 1684, in invariant
kpis                |     raise CheckError(f"Invariant failed. Description: {desc}")
kpis                | dagster._check.CheckError: Invariant failed. Description: Asset selection specified both regular assets and source assets. This is not currently supported. Selections must be all regular assets or all source assets.
👍 1
j

jamie

03/23/2023, 6:09 PM
how are the jobs
[player_base_kpi_job, channel_kpi_job, jira_kpi_update_job]
defined?
c

clay

03/23/2023, 6:11 PM
It was actually pretty simple. structure would have been. The highlighted
__init__.py
had Definitions and
all_assets = [*jira_derived_assets, ...]
. The
__init__.py
right above that, in the
assets
directory, had
jira_derived_assets = load_assets_from_package_module(package_module=jira_kpis, group_name=JIRA_KPIS)
and you can see the
jira_kpis.py
file.
ignore the
dagster-kpis
parent directory... the Code Location points to the
kpis
directory under it.
That's just there for git hygiene
I'll try to reproduce with a more simple example
By the way -- the asset I'm trying to set up as a SourceAsset is in a different code location that's running with it's own gRPC server and mounting its own volume, etc. Looks from your example like you have everything in the same code location. I don't know if that makes a difference?
The jobs
[player_base_kpi_job, channel_kpi_job, jira_kpi_update_job]
were defined all similarly, like:
Copy code
jira_kpi_update_job = define_asset_job(
    name="Jira_KPIs",
    description="Pulls Jira tables from Snowflake and prepares a table for KPI analysis. Code courtesy of Natasha Borders.",
    selection=AssetSelection.groups(JIRA_KPIS),
)
Could it be that using
AssetSelection.groups
in the job definition was causing the issue? Because the SourceAsset was not in that group when I moved it to the same file as Definitions
I bet that was the problem
j

jamie

03/23/2023, 6:35 PM
yeah i think that was the problem. you can remove the source assets from an AssetSelection. I think if you do
AssetSelection.groups(JIRA_KPIS) - AssetSelection.groups(JIRA_KPIS).roots()
that’ll do it
❤️ 1
c

clay

03/23/2023, 6:35 PM
Great! I'll give it a shot
That works, when I put everything back in the original structure. Thanks for helping figure that out.
j

jamie

03/23/2023, 7:50 PM
absolutely! happy to help
44 Views