19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
announcements
dagster-airbyte
dagster-airflow
dagster-bigquery
dagster-cloud
dagster-cube
dagster-dask
dagster-dbt
dagster-de
dagster-ecs
dagster-feedback
dagster-kubernetes
dagster-noteable
dagster-releases
dagster-serverless
dagster-showcase
dagster-snowflake
dagster-support
dagster-wandb
dagstereo
data-platform-design
events
faq-read-me-before-posting
gigs-freelance
github-discussions
introductions
jobs
random
tools
豆瓣酱帮
Powered by Linen announcements
- b
Ben Gotow
12/05/2018, 4:54 PMhaha 😬 - f
freiksenet
12/05/2018, 4:55 PMat initial load I mean - b
Ben Gotow
12/05/2018, 4:55 PMhmm does the subscription emit new log messages? we can probably keep the last 10,000 in the client or something like that - b
Ben Gotow
12/05/2018, 4:55 PMahh yeah i see - f
freiksenet
12/05/2018, 4:55 PMthen we can do infinite scrolling to any direction - f
freiksenet
12/05/2018, 4:56 PMwe can filter logs and not cache them - f
freiksenet
12/05/2018, 4:56 PMeg not show raw logs before we need - f
freiksenet
12/05/2018, 4:56 PMthen they won't take up memory💯 1 - s
schrockn
12/05/2018, 4:56 PMYeah we should page them in and not store all of them - s
schrockn
12/05/2018, 4:56 PMFor sure - b
Ben Gotow
12/05/2018, 4:56 PMmmm let’s do all the filtering on the client side for now though - f
freiksenet
12/05/2018, 4:57 PMbut first I need example pipeline where it's an issue - b
Ben Gotow
12/05/2018, 4:57 PMyeah - b
Ben Gotow
12/05/2018, 4:57 PMI think in the common case of <20,000 log messages or so we can do all the filtering in JS, keep all of them / load all of them etc and it won’t be a huge issue - f
freiksenet
12/05/2018, 4:57 PMyeah - b
Ben Gotow
12/05/2018, 4:58 PMmight as well just do that for now until we have a good example case of it really failing and needing a cursor - f
freiksenet
12/05/2018, 4:58 PMDom is gonna be slower much faster - b
Ben Gotow
12/05/2018, 4:58 PMI wanna see some Chrome “Aw Snap” screenshots - b
Ben Gotow
12/05/2018, 4:58 PMoh for sure - we def. won’t give all those to React - b
Ben Gotow
12/05/2018, 4:59 PMbut I think we can load and filter them and do a scrolling display of that much data client side - we really just have to watch out for the load / parse time and the heap overhead of the log messages - b
Ben Gotow
12/05/2018, 4:59 PMand I think we can get away with a lot before we crash the browser tab and truly need something better 😛 - t
Taylor
12/06/2018, 10:11 PM👋 Hi - I had to pass on this feedback - I was just in a handoff meeting with a client and dagit drew some "oohs and ahs" and some "this looks really nice". So, thank you!🙌 3 - s
schrockn
12/06/2018, 10:12 PM👍 - b
Ben Gotow
12/06/2018, 10:19 PMCool added a comment to that PR @schrockn, go ahead and merge if it looks good - s
schrockn
12/06/2018, 10:19 PMda - b
Ben Gotow
12/06/2018, 10:19 PMwish there was an official supported way to draw those dots… oh well - b
Ben Gotow
12/06/2018, 10:20 PMdefinitely an improvement either way - s
schrockn
12/06/2018, 10:21 PM👍 - m
max
12/06/2018, 11:41 PM@freiksenet would it be helpful to have a pipeline you could run that would spew out a million log messages? - b
Ben Gotow
12/06/2018, 11:41 PMoh for sure 👍
Title
b
Ben Gotow
12/06/2018, 11:41 PMoh for sure 👍
View count: 1