https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
g

George Pearse

06/23/2022, 10:26 AM
In the Software Defined Asset world, assets will always end up in some type of storage e.g. Snowflake, Postgres and come from some type, e.g. S3, Mongo etc. Dagster would be in a good position to integrate with something like https://diagrams.mingrammer.com/ to provide little icons for what type of data is upstream and downstream. At that point I could just send screenshot of a job to management to act as my data architecture diagram. Could associate an icon with an io manager, or SDA pseudo code:
Copy code
s3_vectors = SourceAsset(
    key=AssetKey('s3_vectors'),
    description="bunch o' vectors",
    metadata={'format': 'npy files'},
    data_storage_type='s3' # <--- this addition
)
Can't decide if this goes counter to separating code + storage, or goes along perfectly with it. Tldr: I want little elephant faces on the definitions of the tables in my budget data warehouse
🤖 1
❤️ 1
s

Stephen Bailey

06/23/2022, 12:28 PM
This would be a cool Streamlit/Hex app. Kind of like dbt docs but at the platform level
j

josh

06/23/2022, 3:51 PM
This is a really cool idea. If I understand correctly, you'd basically want to be able to define a storage icon for every asset, then maybe the compute kind tags should actually get displayed on the lines between assets. 🤔 I threw together a super quick mockup. Thoughts?
🎉 3
D 1
❤️ 4
g

George Pearse

06/23/2022, 4:23 PM
@josh spot on, not sure how it'd work on the engineering side. One of dagster's positives is the ability to change the storage easily so it may have to be attached to IO manager's or similar, but @sandy would know what to do / if it's reasonable
ty spinny 1
b

Ben Gatewood

06/23/2022, 10:56 PM
I'd actually love this too
ty spinny 2
r

rex

06/24/2022, 1:51 AM
Could have some open source repo that contains all these mappings from compute kind tag to icon (similar to Buildkite’s https://github.com/buildkite/emojis), so that mappings could be crowdsourced
g

George Pearse

06/24/2022, 9:19 AM
@rex I'm sure there'd be a way to just make use of mingrammer's definitions
s

sandy

06/24/2022, 11:02 PM
btw I filed an issue to track this: https://github.com/dagster-io/dagster/issues/8616
👍 3