With Dagster deployed on Kubernetes, I am trying t...
# ask-community
With Dagster deployed on Kubernetes, I am trying to materialize a downstream TimeWindowPartition and I am getting this error, even though I know that the upstream assets exist (currently in s3) - Any idea what might cause this error? I was thinking it could be the IO manager, but not sure
Copy code
dagster._check.CheckError: Failure condition: Tried to access partition key for asset 'AssetKey(['alarm_watchdog_report_graph'])', but the number of input partitions != 1: 'TimeWindowPartitionsSubset([PartitionKeyRange(start='2023-03-27', end='2023-03-28')])'
dagster bot responded by community 1
🤖 1
Using the
So, now I am thinking the dagster_aws IO manager does not currently support getting TimeWindowPartionsSubset
I have also been struggling with this. In my case it's a dynamic partition:
Copy code
partitions_def = DynamicPartitionsDefinition(name='top_stories')

def top_stories(context):
    # omitted

def unique_top_stories(top_stories):
    # omitted
If I have exactly 1 partition I get that partition's value in
(was expecting some collection). If I have multiple partitions I get the same error as Aaron. I tried explicitly specifying an
but no change.
I think you're right about this being tied to the IO manager @Aaron T. Confirmed just now that if I use the default IO manager my assets work, but switching to the following (pointing at empty bucket) breaks.
Copy code
s3_pickle_io_manager.configured({ 's3_bucket': 'my_bucket_here', 's3_prefix': 'my_prefix_here'         })
Brand new to
and its code base but looking at the source code comparing the FS IO manager and S3 IO manager I see these inheritance chains:
-> ... and
-> ...
specifically says "handles partitioned assets", and comparing the following two methods seems to support the idea that the S3 IO manager just doesn't handle partitions. https://github.com/dagster-io/dagster/blob/8a83cd5d52e5be18dd0ab7a991cf48f594feedbd/python_modules/dagster/dagster/_core/storage/upath_io_manager.py#L187 https://github.com/dagster-io/dagster/blob/8a83cd5d52e5be18dd0ab7a991cf48f594feedb[…]thon_modules/libraries/dagster-aws/dagster_aws/s3/io_manager.py
This issue seems relevant: https://github.com/dagster-io/dagster/issues/10908 But crucially is missing the actual exception message, which is why I missed it before.
@Aaron T, in case this is useful to you: https://github.com/dagster-io/dagster/issues/13290 It includes a custom S3 IO manager that works for me.
thanks for the heads up! I will take a look