https://dagster.io/ logo
Powered by
#random
Title
s

Stefan Adelbert

12/07/2021, 12:06 AM
View Image Data in 
dagit
 Some of my ops will use a resource that wraps a headless selenium webdriver. I'd like to take screenshots of the pages that the headless webdriver is interacting with and expose them via 
dagit
so that someone following up on a failed job can see what the headless webdriver was seeing. Does anyone have any suggestions on how to achieve something like this? I'd imagined that I could create a failure sensor that would get a screenshot from the webdriver resource and "put" it somewhere.
n

Noah K

12/07/2021, 1:02 AM
Copy code
yield AssetMaterialization(
        asset_key=AssetKey(["screenshots", asset["intermediary_bucket"], output_key]),
        metadata_entries=[
            EventMetadataEntry.md(
                f"![](<https://storage.cloud.google.com/{asset['intermediary_bucket']}/{output_key}>)",
                label="image",
            ),
        ],
    )
I put in a request for a raw HTML type but not sure if that ever happened
Doesn't seem to have
s

Stefan Adelbert

12/07/2021, 1:43 AM
@Noah K Hmm, yes, I like that. Thanks for the suggestion. I'll need to have a think about when and how this 
AssetMaterialization
would be yielded. I would ideally like the capability to be generic such that the it could be applied to any job and if that job fails, the relevant resources would have their statused recorded, including screenshots for selenium resources. And that reminds me that I noticed being able to record and retrieve debug info for a job. I'll look into that too.
m

max

12/07/2021, 1:47 AM
cc @sandy
m

mrdavidlaing

12/07/2021, 9:04 PM
I'd find an HTML and/or iframe option useful too. Would this be a difficult thing to contribute?
👍 1
g

George Pearse

12/08/2021, 12:51 PM
Would also be interested
👍 1
s

sandy

12/09/2021, 9:00 PM
Wow - sounds like hefty demand for this functionality. I think the biggest thing that has made us hesitant to add it is security. Arbitrary HTML might be vulnerable to exploits. Thoughts on this? Does it strike you as a concern?
n

Noah K

12/09/2021, 9:04 PM
@sandy The only place this HTML is coming from is my op code, I think if someone has access to change that then XSS injection in dagit is the least of my worries 😄
s

Stefan Adelbert

12/09/2021, 10:25 PM
My solution for now is to add an 
EventMetadata.url
to the 
AssetMaterialization
which allows me to click through to view a screenshot in a GCP Cloud Storage Bucket. The link requires authorization, which is actually a nice side effect. I guess the link could take me to a static HTML page with embedded images, if I wanted more than a simple image. This'll do me.
👍 2
s

sandy

12/10/2021, 12:17 AM
@Noah K - fair. I don't think this is something that we're going to prioritize now, but if someone is interested in contributing it, would be open to discussing
n

Noah K

12/10/2021, 12:17 AM
Heh, I tried writing it like a year ago but the code on the JS side was ... complicated >_<
Maybe after we do our big Dagster upgrade I can take another swing at it 😄
(we're still stuck on 0.9 because of that custom thread safety patch that I think never made it upstream)
Powered by