https://dagster.io/ logo
Join Slack
Powered by
I'm relatively new to Dagster but I'm enjoying the...
# ask-community
s
I'm relatively new to Dagster but I'm enjoying the experience of using it so far. I'm playing with the new 
@asset
decorator since it nicely aligns with my use case. How does one add assets (or the ability to materialize them) or jobs built with 
AssetGroup.build_job
to a repository? At a slightly higher level, Software Defined Assets are really cool, but it does feel like they aren't quite fully integrated. Should I simply avoid them for the time being?
c
Hi Sean! Today we actually released a major release (0.15.0) that formalizes assets to be fully stable, so I'd recommend you use them whenever you are working with data assets. As of this release, you can directly add assets and asset jobs to repositories:
Copy code
@asset 
def my_asset():
  pass

@repository
def my_repo():
  return [my_asset]
🦜 1
s
I should ask questions more often here if they all result in new major releases that answer them! Thanks, Claire.
c
released it just for you blob grin
d
Hi @claire, sorry to piggy-back on this question, but I'm struggling a bit to upgrade to 0.15.0. Previously, I was using 
AssetGroup.build_job()
, but I understand this is now deprecated in favour of 
define_asset_job()
. However, I can't execute the job created by 
define_asset_job()
directly because it seems to only become defined when attached to a repository? Is there no way to programmatically run jobs anymore without defining a repository of jobs upfront?
Okay, I just answered my own question by actually thinking about it a bit more. There's a 
UnresolvedAssetJobDefinition.resolve()
function that I can use directly by just passing the list of assets, so I can bypass having a repository. Including this here in case anyone else runs into this same question!
c
@dpad there's also a build_assets_job function that accepts a list of assets
❤️ 1
d
Thanks a lot!
I'm now wondering why 
execute_in_process
defaults to 
fs_io_manager
instead of 
mem_io_manager
like is stated on the IO Managers page... 😛
@claire Just for your awareness, it seems that 
with_resources()
adds the "fs_io_manager" to the resource. This doesn't match the "default_job_io_manager" so it doesn't get swapped automatically in 
_swap_default_io_man()
. I don't know whether this is intended but thought I'd let you know. I guess for now I just need to specify the IO manager manually.
(The IO Managers documentation page says that 
execute_in_process()
should swap automatically to "mem_io_manager", but this doesn't happen if you build a job using 
with_resources()
.)
c
Hi Dan. This is actually expected behavior--calling 
execute_in_process
with 
fs_io_manager
allows materializations to be persisted and viewable in tests. We do however have another method that allows you to materialize to memory for testing purposes, if that is your preferred behavior: https://docs.dagster.io/_apidocs/execution#dagster.materialize_to_memory
👍 1
d
Thanks again! I guess with this function I need to pass the list of assets directly, rather than specifying a selection (like "*MyAsset" to automatically have MyAsset and all its dependencies)?
I think manually specifying the io_manager in my case is fine (more explicit anyway), so no problem there. Thanks again for all your help and for the 0.15 release!
c
Yep, exactly, 
materialize_to_memory
accepts a list of assets definitions. No problem, happy to help!
Open in Slack
PreviousNext
Powered by