https://dagster.io/ logo
Title
j

jorge.arada

08/18/2021, 5:08 PM
it is possible to create an AssetMaterialization in a create_databricks_job_solid that represent a table processed in a databricks spark_jar_task ?
s

sandy

08/18/2021, 11:04 PM
not currently, but I believe it would make sense to add that. for now, if you want to do that, I would recommend rolling your own implementation of
create_databricks_job_solid
that yields an AssetMaterialization. you can copy the code from here: https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-databricks/dagster_databricks/solids.py#L12 if you have interest in submitting a pull request, we'd also be happy to review
j

jorge.arada

08/19/2021, 8:33 AM
Thanks
The implementations details for an AssetMaterialization are tricky. what will you yield? there is no return from the databricks api about wat was processed (table name or partitions afected)