it looks like doing this correctly involves shimming

import

and building a dependency graph as files are loaded, and then reloading the file that has changed, followed by all of the files that imported that file, etc. until you’ve hit the whole tree. Thankfully it seems like there’s a pip module to do it: https://pypi.org/project/reloader/… gonna give it a shot

Hey @schrockn that error-handling PR is ready for another review. I switched to

reloader

and it seems to have taken care of the reloading issues, but I did end up monkey-patching it so it only reloads user code via some sorta gross module path checks. not sure if there’s a better approach to this in python: https://github.com/dagster-io/dagster/commit/db35b2b8544edf15f0e596ea200c96b86a8fc64e.

it’s faster, but it also turned out reloading the entire tree caused problems because numpy doesn’t support reloading at all (and probably other modules too…). The

reloader

module only supports an explicit blacklist which seemed insufficient

Eek ok

Will be back on at 4:30 or so

In transit

ahh hmm python 2.7 support…

Well we may have to punt on this issue

I have another issue in the repo

Which references another reloading library

AiokReloader or something

hmm yeah - seems like reloader is working nicely, but only supports python 3+

it’s odd the pypi.org pages don’t mention which python versions a module supports given how huge the rift is between Python 2 and 3…

Ok well let’s just gate the feature on python 3 only

And call it a day

You can check the version at runtime

👍 will do

@Ben Gotow pro tip if you want to debug 2.7 issues locally

just run ‘tox’ from dagster/python_modules/dagster

oh cool - yeah I didn’t realize even the install of the reloader package would fail on 2.x, should have read the error more carefully before I gated it

should be good now

@Ben Gotow diff approved. have some non-trivial followup questions that would be good to address in the final rewview

*review

most substantive change is the after_load_fn eliminating

elimination

i think it is more confusing than the coercion within DynamicObject

yeah I think so too… just didn’t want to change the factory methods under test

I think the trick is that if the dynamic object is generic, the factory methods that return one really should be also, and you should put the dynamic object into a repository container to get a repository out of one

<!channel> Version 0.2.7 has been pushed to pypi. https://github.com/dagster-io/dagster/releases/tag/v.0.2.7 Release notes: Version 0.2.7 Release Notes Most notable improvements in this release are bunch of improvements to dagit, most notably hot reloading and the in-browser rendering of python error. Also the ability to scaffold configs from the command line is the first fruit of the rearchitecting of the config system. - Dagster improvements: - Added scaffold_config command which generates the template of a yaml file needed to drive the execution of a particular pipeline - Added the ability to automatically serialize intermediate inputs as they flow between solids. Consider this alpha quality. It is currently hard-coded to write out to /tmp/dagster/runs/<<run_id>> - Dagit improvements: - Hot-Reloading and in-browser rendering of python errors. - Scrolling and performance improvements - Keyboard short cuts to navigate between solids using arrow keys - In-app previews of notebooks for dagstermill solids