🙌

so here's what I'm thinking in terms of GraphQL API:

launchBackfillParams: {
  partitionsByAsset: [
    {
      assetKey: {path: ["a", "b", "c"]},
      partitions: {
        range: {
          start: "2022-02-01",
          end: "2022-02-04",
        }
      }
    }
  ]
}

the class in the codebase to add

partitionByAsset

to is

GrapheneLaunchBackfillParams

and here are the relevant graphql tests:

./python_modules/dagster-graphql/dagster_graphql_tests/graphql/test_asset_backfill.py

create_and_launch_partition_backfill

inside

./python_modules/dagster-graphql/dagster_graphql/implementation/execution/backfill.py

contains the logic for turning the backfill params into an

AssetBackfillData

object

and here's a Github discussion with a proposal for APIs for coalescing backfills: https://github.com/dagster-io/dagster/discussions/14829

Nice, I take it as a design decision on the "chunking" requirement. If everything goes smoothly, we can start the work next week!

@sandy However, we are not sure how the Dagster project RFC process work. So, may I ask does that actually mean a "go-ahead"? Also, I am wondering if there will be someone from Dagster also working on this or we are on our own.

However, we are not sure how the Dagster project RFC process work. So, may I ask does that actually mean a "go-ahead"?

Not all RFCs mean "go-ahead", but I shared this one internally and everyone is onboard, so I'd say go ahead. We still need to decide on the precise-name, and that might take a week or two, but we can start with

BackfillStrategy

or something and do a find/replace.

hi @sandy , I have the first PR ready to introduce BackfillPolicy to the core API layer, however, it seems like I don't have push access to the remote repo, should I go for a PR from my fork approach?

I went with the PR from my fork approach for now, and please let me know if we have another convention here. And here is the PR to add BackfillPolicy in the core API layer only. https://github.com/dagster-io/dagster/pull/15206

Hey @sandy, sorry to nag you again on the PR reviews. I have got another PR that adds custom partition range params support to the graphql backfill API ready here. https://github.com/dagster-io/dagster/pull/15442 Btw, the first PR that adds BackfillPolicy to the core API layer was updated to address your comments on the docstrings, can you take another quick look at it so we can get it approved and merged? https://github.com/dagster-io/dagster/pull/15206 Meanwhile, I am starting to work on the last piece of the puzzle - modifying the backfill "daemon" to pick up all those added metadata and params to launch runs accordingly.

Hi sandy and claire, apologize for the weekend ping, but the last PR for chunked backfill is ready. This should not rely on the graphql range param PR (the one in review) since we can do chunked backfill under the condition allPartitions=true with BackfillPolicies. https://github.com/dagster-io/dagster/pull/15474 Heads up, I will go on PTO next week from Monday to Friday, so please take your time to do the PR review, there is no rush there. I will be back the week after next week and I will address your comments on my PR once I am back. (Thank you claire again for reviewing my 2nd PR, I will address your comments a week later too!)

@claire @sandy we are getting this error when trying to deploy our user code in the latest version of dagster.

dagster._serdes.errors.SerializationError: Can only serialize whitelisted namedtuples, received BackfillPolicy(max_partitions_per_run=None).

Does it mean we have to put `@whitelist_for_serdes`decorator on the BackfillPolicy class? https://github.com/dagster-io/dagster/blob/da23cb0c09b2cd0437b0ea2519dae76d61710a6[…]on_modules/dagster/dagster/_core/definitions/backfill_policy.py Beside making another change to release it, is there other way to add this class to whitelist?

@claire it seems like some recent changes introduced are giving me this error for the chunked backfill. "chunked" here means running multiple partitions in a single run.

dagster._check.CheckError: Invariant failed. Description: Did not submit run requests for all expected partitions
  File "/usr/local/lib/python3.8/site-packages/dagster/_daemon/backfill.py", line 34, in execute_backfill_iteration
    yield from execute_asset_backfill_iteration(
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/execution/asset_backfill.py", line 669, in execute_asset_backfill_iteration
    for updated_asset_backfill_data in _submit_runs_and_update_backfill_in_chunks(
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/execution/asset_backfill.py", line 591, in _submit_runs_and_update_backfill_in_chunks
    check.invariant(
  File "/usr/local/lib/python3.8/site-packages/dagster/_check/__init__.py", line 1589, in invariant
    raise CheckError(f"Invariant failed. Description: {desc}")

looks like the run request got executed properly, but failed at this

check.invariant

after the runs are executed, and it is not properly returning updated AssetBackfillData