Is it possible to define an asset without an input? I know you can do so with Ops by creating one in the form of:

@Op(ins={"start": In(Nothing)})

but

start

is not a key that is available in

ins

of an

asset

. I have a pipeline that is like:

ingest_dataset1 -> ingest_dataset2 -> start_processing_data

in the form of:

asset -> asset -> op.

The assets do not depend on each other, I just want one to run after the other. I could also use:

op -> op -> op

but it is to my understanding that an Op should be used when performing some type of transformation or logic whereas an asset is signifies something which is stored to a persistent storage.

This minimal example works:

from dagster import asset, AssetIn


@asset
def asset_one():
    pass


@asset(ins={"start": AssetIn("asset_one")})
def asset_two(start):
    pass

Hey Vinnie, I tried out multi assets and it works, sort of. My datasets are ingested and pushed to S3 as one Op so they no longer depend on each other. However, my next node in the graph (which is an Op) launches a cluster so I can run PySpark jobs. I want the cluster to start after the datasets have been ingested. The Op is now complaining that it receives multiple outputs for a single input. Is it possible to set the op to not expect any inputs, no matter how many outputs are generated upstream? The error I get:

dagster._core.errors.DagsterInvalidDefinitionError: In @graph ingestion_pipeline, received a tuple of multiple outputs for input "start" (at position 0) in op invocation launch_emr_cluster. Must pass individual output, available from tuple: ('customer_transactions', 'customer_information')

My Op:

@op(

ins={"start": In(Nothing)},

)

def launch_emr_cluster(context: OpExecutionContext) -> str:

...

It’s not a case I’ve encountered yet, but I imagine the solution would be analog to my first example, something along the lines of

@op(ins={"customer_transactions": AssetIn("customer_transactions"), "customer_information": AssetIn("customer_information")}
def my_op(context, customer_transactions, customer_information):
  ...

I also have to say, I’m not entirely sure if ops can depend on assets. The Dagster team is probably waking up around now and I’m sure they can help though 🙂

Yes I'd imagine that this would work but in my case I want the Op that launches a cluster to be Asset independent but is executed after my multi_asset function has finished. Thanks for the suggestion! 🙂 I'll let you know if I figure out a way to solve this

you might also think about using an AssetSensor to trigger the op when the assets have been materialized. this would break up the graph though, but can be helpful for managing this kind of dependency where you only need an order-based dependency between assets and ops.

You can also use

non_argument_deps

https://docs.dagster.io/concepts/assets/software-defined-assets#non-argument-dependencies to define order dependencies without data dependencies between assets

Thank you! I will look into both of them 🙂