https://dagster.io/ logo
Join Slack
Powered by
The issue seems to be that if I do: ```my_assets =...
# ask-community
c
The issue seems to be that if I do:
Copy code
my_assets = load_assets_from_package_module(package_module=my_module, group_name="MY_GROUP")
I cannot then do 
all_assets = [*my_assets]
to feed into the 
Definitions
for the code location if 
my_assets
mixes Source and Regular assets. However, if I do:
Copy code
my_regular_assets = load_assets_from_package_module(package_module=my_regular_module, group_name="MY_REGULAR_ASSET_GROUP")
and also
Copy code
my_source_asset = SourceAsset(key=AssetKey("my_source_asset_key"))
then I can do the following without a problem:
Copy code
all_assets = [*my_regular_assets, my_source_asset]
j
hey @clay I’m able to pass both assets and source assets loaded via 
load_assets_from_package_module
to the 
assets
parameter of Definitions. Could you share the code for your Definitions so i can see if something else is going on?
c
Sure, sorry... had to catch a swarm of bees in my yard and put them in a hive. 🐝 I have a file for the assets related to a specific group, called 
jira_kpis.py
. It's in a submodule. In it, I had:
Copy code
jira_tickets_source_asset = SourceAsset(
    key=AssetKey("ED_CRM_DASH_CRM_JIRA_TICKETS_CONSOLIDATED"),
)
Followed by:
Copy code
@asset(required_resource_keys={"snowflake"}, compute_kind="Python")
def calc_jira_kpis(context: OpExecutionContext, ED_CRM_DASH_CRM_JIRA_TICKETS_CONSOLIDATED: pd.DataFrame) -> None:
    """
    Calculate Jira-derived KPIs.
    """
    stuff
    return None
In the parent module 
assets
(similar to one of the dagster examples) in the 
__init__.py
file, I had:
Copy code
jira_derived_assets = load_assets_from_package_module(package_module=jira_kpis, group_name=JIRA_KPIS)
and some other irrelevant stuff The parent module to that is the main code location where I am defining stuff in 
__init__.py
. In there, I had:
Copy code
all_assets = [*player_base_kpi_assets, *channel_kpi_assets, *jira_derived_assets]
and
Copy code
defs = Definitions(
    assets=all_assets,
    .... stuff
)
Presumably, my call to 
load_assets_from_package_module
was picking up the 
SourceAsset
. When I ran this in 1.1.18, it all worked just fine. Then I upgraded to 1.2.3 and the code location would not load. The error I saw when I did 
docker-compose up
was the one I mentioned: 
dagster._check.CheckError: Invariant failed. Description: Asset selection specified both regular assets and source assets. This is not currently supported. Selections must be all regular assets or all source assets.
When I moved the 
SourceAsset
definition to the same file as the Definitions and then did:
Copy code
all_assets = [*player_base_kpi_assets, *channel_kpi_assets, *jira_derived_assets, jira_tickets_source_asset]
Followed by
Copy code
defs = Definitions(
    assets=all_assets,
    .... stuff
)
The error went away and everything worked as expected. It seemed that trying to unpack 
*jira_derived_assets
when it contained both Regular and Source assets was the problem?
j
ok. Are you making an 
AssetSelection
object anywhere? also could you share the full stack trace?
c
Is that "create a post"? for a longer snippet?
j
here’s my replication setup in 
foo/__init__.py
Copy code
from dagster import asset, SourceAsset

baz_source = SourceAsset(key="baz")


@asset(
    op_tags={"hello": "world"}
)
def foo():
    return 1
in 
bar/__init__.py
Copy code
from dagster import asset

@asset
def bar():
    return 1
in 
__init__.py
Copy code
from dagster import Definitions, load_assets_from_package_module
from . import foo, bar



defs = Definitions(
    assets=[*load_assets_from_package_module(foo, group_name="foo"), *load_assets_from_package_module(bar, group_name="bar")]
)
and when i run the ui everything loads correctly and i see the three assets. might be worthwhile to try the minimal setup yourself just to make sure the error you’re seeing is from the 
load_assets_from_package_modules
and not from something else
c
I'll just put the stack trace here
Copy code
kpis                | Traceback (most recent call last):
kpis                |   File "/usr/local/bin/dagster", line 8, in <module>
kpis                |     sys.exit(main())
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_cli/__init__.py", line 46, in main
kpis                |     cli(auto_envvar_prefix=ENV_PREFIX)  # pylint:disable=E1123
kpis                |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
kpis                |     return self.main(*args, **kwargs)
kpis                |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1055, in main
kpis                |     rv = self.invoke(ctx)
kpis                |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
kpis                |     return _process_result(sub_ctx.command.invoke(sub_ctx))
kpis                |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
kpis                |     return _process_result(sub_ctx.command.invoke(sub_ctx))
kpis                |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
kpis                |     return ctx.invoke(self.callback, **ctx.params)
kpis                |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 760, in invoke
kpis                |     return __callback(*args, **kwargs)
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_cli/api.py", line 724, in grpc_command
kpis                |     location_name=location_name,
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/server.py", line 942, in __init__
kpis                |     location_name=location_name,
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/server.py", line 269, in __init__
kpis                |     self._container_image,
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/server.py", line 120, in __init__
kpis                |     loadable_target_origin.attribute,
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/utils.py", line 47, in get_loadable_targets
kpis                |     else loadable_targets_from_python_module(module_name, working_directory)
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_core/workspace/autodiscovery.py", line 39, in loadable_targets_from_python_module
kpis                |     remove_from_path_fn=remove_from_path_fn,
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_core/code_pointer.py", line 135, in load_python_module
kpis                |     return importlib.import_module(module_name)
kpis                |   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
kpis                |     return _bootstrap._gcd_import(name[level:], package, level)
kpis                |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
kpis                |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
kpis                |   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
kpis                |   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
kpis                |   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
kpis                |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
kpis                |   File "/opt/dagster/app/kpis/kpis/__init__.py", line 97, in <module>
kpis                |     jobs=[player_base_kpi_job, channel_kpi_job, jira_kpi_update_job],
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/definitions_class.py", line 401, in __init__
kpis                |     loggers=loggers,
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/definitions_class.py", line 277, in _create_repository_using_definitions_args
kpis                |     _top_level_resources=resource_defs,
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/decorators/repository_decorator.py", line 133, in __call__
kpis                |     top_level_resources=self.top_level_resources,
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/repository_definition/repository_data.py", line 482, in from_list
kpis                |     top_level_resources=top_level_resources,
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/repository_definition/repository_data_builder.py", line 273, in build_caching_repository_data_from_list
kpis                |     asset_graph=asset_graph, default_executor_def=default_executor_def
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/unresolved_asset_job_definition.py", line 204, in resolve
kpis                |     selected_asset_keys = self.selection.resolve(asset_graph)
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/asset_selection.py", line 196, in resolve
kpis                |     "Asset selection specified both regular assets and source assets. This is not"
kpis                |   File "/usr/local/lib/python3.7/site-packages/dagster/_check/__init__.py", line 1684, in invariant
kpis                |     raise CheckError(f"Invariant failed. Description: {desc}")
kpis                | dagster._check.CheckError: Invariant failed. Description: Asset selection specified both regular assets and source assets. This is not currently supported. Selections must be all regular assets or all source assets.
👍 1
j
how are the jobs 
[player_base_kpi_job, channel_kpi_job, jira_kpi_update_job]
defined?
c
It was actually pretty simple. structure would have been. The highlighted 
__init__.py
had Definitions and 
all_assets = [*jira_derived_assets, ...]
. The 
__init__.py
right above that, in the 
assets
directory, had 
jira_derived_assets = load_assets_from_package_module(package_module=jira_kpis, group_name=JIRA_KPIS)
and you can see the 
jira_kpis.py
file.
ignore the 
dagster-kpis
parent directory... the Code Location points to the 
kpis
directory under it.
That's just there for git hygiene
I'll try to reproduce with a more simple example
By the way -- the asset I'm trying to set up as a SourceAsset is in a different code location that's running with it's own gRPC server and mounting its own volume, etc. Looks from your example like you have everything in the same code location. I don't know if that makes a difference?
The jobs 
[player_base_kpi_job, channel_kpi_job, jira_kpi_update_job]
were defined all similarly, like:
Copy code
jira_kpi_update_job = define_asset_job(
    name="Jira_KPIs",
    description="Pulls Jira tables from Snowflake and prepares a table for KPI analysis. Code courtesy of Natasha Borders.",
    selection=AssetSelection.groups(JIRA_KPIS),
)
Could it be that using 
AssetSelection.groups
in the job definition was causing the issue? Because the SourceAsset was not in that group when I moved it to the same file as Definitions
I bet that was the problem
j
yeah i think that was the problem. you can remove the source assets from an AssetSelection. I think if you do 
AssetSelection.groups(JIRA_KPIS) - AssetSelection.groups(JIRA_KPIS).roots()
that’ll do it
❤️ 1
c
Great! I'll give it a shot
That works, when I put everything back in the original structure. Thanks for helping figure that out.
j
absolutely! happy to help
106 Views
Open in Slack
PreviousNext
Powered by