so maybe you could try merging master back into your branch

also this is beautiful zoom out 💖

hey guys when you get a chance - I put up a new branch with some more experimental changes to the UI (bengotow/experimental-ui) that I would love your thoughts on. It 1) removes the left nav (which takes up a good bit of space) in favor of searchable breadcrumbs you can activate with the “s” and “p” keys (also “esc” deselects the current solid). It adds two distinct zoom levels to the graph, which autocenters on solids that you click and zooms out to the overview when you deselect solids, and adds a right sidebar that alternates between showing the pipeline description / info and the selected solid’s info. (Right now the “reference” of all the solids / context is gone, but we’d need to put context back here somewhere.)

the code is super rough and not PR ready yet, but I think it’s enough we can try it out and see if it’s a good approach for developers trying to understand / debug their pipelines

(also the framerate isn’t quite as rough as the video when i’m not recording, but it /is/ a bit janky on very large graphs because hiding the in/out titles triggers a single React re-render half way through the zoom-out animation. I think we can definitely fix it but probably not the highest priority thing)

this is really promising stuff

gimme a few mins to play with it

ok i have some feedback

will make each piece of feedback a message so we can do a thread for each one if we want

1) Zooming is a bit “twitchy” from a product experience. I really like the fixed two levels of zoom but I think we need to make the user feel very in control. Right now it zooms and unzooms when I don’t want it to. Examples: - When I single click a solid from the zoom out view I don’t really want it to zoom. I just want the detail pane to remember what the full name is or something, - When I pan and drag it when zoomed in it zooms out. I would suggest having clear rules about zooming. I think this would be both be clearer to the user and easier to implement. I think a good start is to have double clicking just unconditionally toggle zoom level. Very simple and very clear

2) I think the right hand pane will have to be hideable and/or resizable. Think laptop screen only and you want to click on a solid and read a long description (e.g. a SQL query). You want to be able to get that on your whole screen and then get rid of it when you are back in DAG land

3) Bug: Escape key not de-selecting solids

4) When you are typing in the solid typehead “p” activates the pipeline typeahead

hey hey this is great stuff - hmm, ahh you know what, I fixed 3/4 and didn’t push up my changes. will do that now

5) Typeheads should match on arbitrary substring, not just the leading characters

6) We should rethink the input and output colors

7) Let’s increase the text size on zoom out

8: if there’s a single output and it is named result, let’s only display the type of the oututp

<!channel> heads up I pushed a new version of dagster. https://github.com/dagster-io/dagster/releases/tag/v0.2.3 Driving factor to release this is a bug in the command line interface in 0.2.2 (#207) Other changes in this release: - CLI interface has changed slightly. Whenever dagit or dagster needs to specify a function to load a repo or a pipeline, use the -n/--fn-name flag combo. Before this was split out into to different use cases in the dagster CLI tool. - We now have the ability to reuse a single solid definition multiple times within the same pipeline using the SolidInstance API. See the corresponding tutorial section for more details. h/t to @freiksenet for this work! - Documentation continues to improve.

hey @freiksenet were you looking in to the performance of the DAG graph at all? I noticed this AM that it’s much faster in firefox, but that firefox is clipping foreignobjects to their bounds while Chrome isn’t

I have a hunch the best thing to do might be to convert to basic svg (since the drawing isn’t that complicated anyway), but wanted to get your thoughts - wasn’t sure if you’d started making changes

I just pushed v0.2.5 to pypi. @Curtis this includes the command line flag we added for watch bug. @Ben Gotow and @freiksenet this includes the type browser. Excellent work. Version bump to 0.2.5 (#227) - Added the Type Explorer in Dagit. You can now browse all the types declared in a pipeline. - Added the --watch/--no-watch flag to dagit. This allows you to turn off watching in cases where there are two many files below the current working directory.

👋

👋 @max

welcome

Hi all! Want to see the biggest dagster dag yet?

😮

To your credit the scrolling was still shockingly performant.

oh man so I think I finally figured out why the hot-reloading isn’t really working… (even in the no-errors case). It looks like

importlib.reload(self.module)

reloads /just/ the file specified by module, not the tree of imports. So if

repository.py

just imports a bunch of

define_pipeline

methods and returns a repo, none of it will really be reloaded. In my initial testing I was changing this main file, and tonight hot-reloading was just totally busted, but I was modifying the pipeline functions