NateV03/31/2023, 9:10 PM
to feed assets directly into the job. Following this pattern, which I think was only recently made possible: https://docs.dagster.io/concepts/ops-jobs-graphs/graphs#loading-an-asset-as-an-input. But I think if fails because of Serverless dagster's ephemeral storage. The Job fails if the asset isn't there. So can I force the job to materialize the asset if its not materialized yet? Or does this pattern of feeding assets into jobs not work in serverless dagster because of that ephemeral storage? (I guess more generally, am I using things the wrong way by having an asset just fetch some data and return it? Will this only work if I have set up external services for permanent storage (like azure data lake 2)?
sandy03/31/2023, 9:53 PM
NateV04/01/2023, 1:17 AM
that returns some value
, I can't actually call that function to get
somewhere, and any later calls to
are actually going to fetch
put it. I suppose this is the nature of data-pipeline type systems to work this way (to which I'm clearly new)?