https://dagster.io/ logo
#ask-community
Title
# ask-community
g

geoHeil

04/07/2023, 8:51 AM
What is different in dagster from 1.2.2 to 1.2.3? 1.2.2 works fine for me, whereas > 1.2.2 I get:
DagsterInvalidDefinitionError: Tried to build a partitioned job, but none of the selected assets are partitioned.
Is this related to: Jobs defined via
define_asset_job
now auto-infer their partitions definitions if not explicitly defined. ?
I have a job which is processing partitioned assets (sources) and then mashes them up into scd2. (unpartitioned)
This is the stacktrace:
Copy code
______________________________________________________________________________________________________________________________ ERROR collecting /test_project_loads.py ______________________________________________________________________________________________________________________________
/test_project_loads.py:1: in <module>
    from my_project import my_project
my_project/__init__.py:1: in <module>
    from .repository import my_project
my_project/repository.py:27: in <module>
    def my_project():
/lib/python3.10/site-packages/dagster/_core/definitions/decorators/repository_decorator.py:380: in repository
    return _Repository()(definitions_fn)
/lib/python3.10/site-packages/dagster/_core/definitions/decorators/repository_decorator.py:154: in __call__
    else CachingRepositoryData.from_list(
/lib/python3.10/site-packages/dagster/_core/definitions/repository_definition/repository_data.py:398: in from_list
    return build_caching_repository_data_from_list(
/lib/python3.10/site-packages/dagster/_core/definitions/repository_definition/repository_data_builder.py:254: in build_caching_repository_data_from_list
    resolved_job = unresolved_job_def.resolve(
/lib/python3.10/site-packages/dagster/_core/definitions/unresolved_asset_job_definition.py:198: in resolve
    raise DagsterInvalidDefinitionError(
E   dagster._core.errors.DagsterInvalidDefinitionError: Tried to build a partitioned job, but none of the selected assets are partitioned.
Unfortunately, for me it is totally unclear which particular job is the problem here. Also, it is unclear if this is working as intended - or perhaps a bug in the auto-infer partitions method.
Can I somehow assign it an unpartitioned definition? But I would have assumed by default it works this way
the self dependencies though
s

sandy

04/07/2023, 6:40 PM
Hi @geoHeil - I am going to create a PR that makes that error more informative
I have a job which is processing partitioned assets (sources) and then mashes them up into scd2. (unpartitioned)
Ah, and you want each run to process a single partition?
@claire - curious on your thoughts on how to handle this case. If we just removed this check inside UnresolvedAssetJobDefinition.resolve, are there any problems you'd anticipate?
Copy code
if len(asset_keys_by_partitions_def) == 0 and self.partitions_def:
            raise DagsterInvalidDefinitionError(
c

claire

04/07/2023, 6:58 PM
I think it's fine if we want to remove this check (we didn't have it < 1.2.3). I don't anticipate any issues, this would just allow for specifying a partitions def on a job when the selected assets aren't partitioned
s

sandy

04/07/2023, 7:54 PM
👍 - Claire - do you have bandwidth to take it out for the next release?
c

claire

04/07/2023, 8:15 PM
yep, I will take it out
g

geoHeil

04/07/2023, 9:12 PM
Great. Would it be possible to specify suitable partitions in my case? Without rewriting to the proper self referencing asset? I am not sure what you mean with a single partition. But the upstream is a csv full daily copy of some table. And I am stateful tracking changes in SCD2 using dml merge into commands. However this scd2 asset no longer is sensibly a partitioned asset. At least as far I see it