I've been scouring the docs but still not sure on the right pattern for m workflow. Feels like a mix-mash of partitioned assets, and ops. Workflow: • Step 1 -> download some data, transform, store -> These could be monthly partitioned assets • Step 2 -> perform calculations using the asset data: ◦ This would be a configured job, for a specific partition ◦ Several other paramaters input to the calculations ◦ Usually several different scenarios run for a given month, using different parameters. It feels like Step 1 would fit nicely as assets with monthly partitions, But Step 2 the job with ops needs to be able to access the values from the assets in step 1, for a specific partition. Does this design make sense?

@asset(partitions_def=MonthlyPartitionsDefinition(start_date='2018-02-01'))
def sales_data():
  """ Refreshed automatically every month """
  ...

@op(config_schema={'discount_rate': float, 'rep_date': str})
def sales_with_discount():
  # need to get sales data for a specific month here:
  ..

@graph
def graph_sales_discounted()
  return sales_with_discount

# This job might be run many times with different discount rate
job_discounts = graph_sales_discounted.to_job(config={'ops': {'sales_with_discount': {'config': { 'discount_rate': 0.3, 'rep_date': '2022-09-30'}}}})

I'm looking at doing something similar and, though I haven't gotten here yet, I plan to pass asset information into the Metadata and retrieve that from the upstream context. Have you explored whether you can pass the information you need that way?

so far i haven't found a way to expose asset values to ops. The more I read / search through old posts the more it seems discouraged. Surprising to me, as this feels like a typical use case for parameterized models: • set up input data • perform some calculations using input data and a set of parameters (which may change for different scenarios) There is probably a workaround to load files if you know where specific asset materializations are stored -> but this woudl be hacky and I don't think you'd maintain lineage (graph) between the input data and the calculations.

Hmm - lemme see if I can get test running. I'll check back in shortly

1.1.7

That's kinda what I was exploring. Assets are such a difficult way of thinking of these things for me. I can't tell if Dagster intends assets to be thought of as objects or instances (though I've seen them used both ways). 😕

I think, objects. If assets were instances then each instance of an asset would be treated separately. But, as an experiment, I ran two different dagit instances locally (different terminal sessions & ports), and materlized the same asset but with a different parameter. The differnt instances didn't know that they weren't the same asset., i.e. instance A showed re-materialized after instance B was materialized -> this was using a common DB for run storage

Oh that's interesting. I have seen folks define assets with a one-to-one relationship to DB tables too and that seems more instance-like, at least in practice.

hey, so I got something working with a custom IO Manager. Minimal POC is in my playground repo, job "downstream_job". • Upstream assets are multi-partitioned, monthly + static • Downstream are not partitioned For a downstream asset job I define an upstream partition key override, which gets picked up by the iomanager when loading input assets. Then the downstream knows to pick a specific partition of the upstream assets. I wanted downstream to have a different static partition but seems not supported in the config mapping -< not sure if I can work around this. https://github.com/pdavidsonFIA/dagster-poc/blob/9006f49d004c2313cba01647f670fd1c867018fc/dagster_poc/resources/__init__.py#L93 I did borrow from @Sean Lopp’s snow report for the asset factory, but didn't quite understand his logic in terms of selecting specific partitions: https://github.com/slopp/snowreport/blob/main/snowreport/assets/report_raw_json.py#L42

@Peter Davidson Thanks for this. I hadn't looked at it until now. In my instance I am using Dagster's predefined UPathIOManager which has handling of partitioned assets built-in. I mention this because it's possibly a good template for your io_manager, albeit it loads everything and then you choose from the dictionary