on a related note to the above, is there an easy way to run a backfill on a schedule? • I have a data source that is sensibly partitioned into days but is updated in a “lumpy” way, i.e. not daily but X days at a time, every X days, with a variable number X. • I would like to use asset sensors to then kick off downstream jobs upon materialization The obvious way to handle this (for me) would be to just attempt a backfill every day, but I don’t see an easy way to do that within the scheduler code. I’d like to avoid programmatically making calls to the dagster CLI, e.g. defining a cron job that just runs

dagster job backfill

from console every day, but if that’s the only way to do it then i guess I can do that

Right now, the two ways to do this is to use the CLI:

dagster job backfill

or to use our GraphQL API, hitting the

launchPartitionBackfill

mutation (how backfills are launched from dagit).

@prha makes sense. Looking into it though - what if it’s a partitioned asset and not a partitioned job?

You can create an explicit job of assets using

define_asset_job

. A job is a bound set of operations, tied to an environment. An asset materialization is the single instance of an asset execution, as part of a job run.

awesome, thanks!

hmm, yeah, we don’t expose that easily in any fashion except by manually fetching tags by job, or by querying our graphql endpoint for the field

so would it be best to keep my own separate table of successful partitions?

Yeah, I think querying the graphql endpoint might be your best bet.

alright i’ll give that an honest try and report back if i get stuck

Here’s the query schema we use to calculate it in dagit:

partitionSetOrError(
      repositorySelector: $repositorySelector
      partitionSetName: $partitionSetName
    ) {
      ... on PartitionSet {
        id
        name
        pipelineName
        partitionsOrError {
          ... on Partitions {
            results {
              name
            }
          }
        }
        partitionStatusesOrError {
          __typename
          ... on PartitionStatuses {
            results {
              id
              partitionName
              runStatus
              runDuration
            }
          }
        }
      }
    }
  }

nice thanks

I think

partitionsOrError

gives the full range of partitions, and

partitionStatusesOrError

give the status of any runs that exist for each partition. I think the difference of those two (when selecting partition names) will give you the set of missing partitions.

is there an easy way to like pry into the dagit UI to see what query is being run

Yes, in the user settings tab (the gear in the upper right corner), you can enable “Debug console logging”

nice, i got it working. Thanks so much for your help!