https://dagster.io/ logo
Join the conversationJoin Slack
Channels
announcements
dagster-airbyte
dagster-airflow
dagster-bigquery
dagster-cloud
dagster-cube
dagster-dask
dagster-dbt
dagster-de
dagster-ecs
dagster-feedback
dagster-kubernetes
dagster-noteable
dagster-releases
dagster-serverless
dagster-showcase
dagster-snowflake
dagster-support
dagster-wandb
dagstereo
data-platform-design
events
faq-read-me-before-posting
gigs-freelance
github-discussions
introductions
jobs
random
tools
豆瓣酱帮
Powered by Linen
announcements
  • b

    Ben Gotow

    12/05/2018, 4:54 PM
    haha 😬
  • f

    freiksenet

    12/05/2018, 4:55 PM
    at initial load I mean
  • b

    Ben Gotow

    12/05/2018, 4:55 PM
    hmm does the subscription emit new log messages? we can probably keep the last 10,000 in the client or something like that
  • b

    Ben Gotow

    12/05/2018, 4:55 PM
    ahh yeah i see
  • f

    freiksenet

    12/05/2018, 4:55 PM
    then we can do infinite scrolling to any direction
  • f

    freiksenet

    12/05/2018, 4:56 PM
    we can filter logs and not cache them
  • f

    freiksenet

    12/05/2018, 4:56 PM
    eg not show raw logs before we need
  • f

    freiksenet

    12/05/2018, 4:56 PM
    then they won't take up memory
    💯 1
  • s

    schrockn

    12/05/2018, 4:56 PM
    Yeah we should page them in and not store all of them
  • s

    schrockn

    12/05/2018, 4:56 PM
    For sure
  • b

    Ben Gotow

    12/05/2018, 4:56 PM
    mmm let’s do all the filtering on the client side for now though
  • f

    freiksenet

    12/05/2018, 4:57 PM
    but first I need example pipeline where it's an issue
  • b

    Ben Gotow

    12/05/2018, 4:57 PM
    yeah
  • b

    Ben Gotow

    12/05/2018, 4:57 PM
    I think in the common case of <20,000 log messages or so we can do all the filtering in JS, keep all of them / load all of them etc and it won’t be a huge issue
  • f

    freiksenet

    12/05/2018, 4:57 PM
    yeah
  • b

    Ben Gotow

    12/05/2018, 4:58 PM
    might as well just do that for now until we have a good example case of it really failing and needing a cursor
  • f

    freiksenet

    12/05/2018, 4:58 PM
    Dom is gonna be slower much faster
  • b

    Ben Gotow

    12/05/2018, 4:58 PM
    I wanna see some Chrome “Aw Snap” screenshots
  • b

    Ben Gotow

    12/05/2018, 4:58 PM
    oh for sure - we def. won’t give all those to React
  • b

    Ben Gotow

    12/05/2018, 4:59 PM
    but I think we can load and filter them and do a scrolling display of that much data client side - we really just have to watch out for the load / parse time and the heap overhead of the log messages
  • b

    Ben Gotow

    12/05/2018, 4:59 PM
    and I think we can get away with a lot before we crash the browser tab and truly need something better 😛
  • t

    Taylor

    12/06/2018, 10:11 PM
    👋 Hi - I had to pass on this feedback - I was just in a handoff meeting with a client and dagit drew some "oohs and ahs" and some "this looks really nice". So, thank you!
    🙌 3
  • s

    schrockn

    12/06/2018, 10:12 PM
    👍
  • b

    Ben Gotow

    12/06/2018, 10:19 PM
    Cool added a comment to that PR @schrockn, go ahead and merge if it looks good
  • s

    schrockn

    12/06/2018, 10:19 PM
    da
  • b

    Ben Gotow

    12/06/2018, 10:19 PM
    wish there was an official supported way to draw those dots… oh well
  • b

    Ben Gotow

    12/06/2018, 10:20 PM
    definitely an improvement either way
  • s

    schrockn

    12/06/2018, 10:21 PM
    👍
  • m

    max

    12/06/2018, 11:41 PM
    @freiksenet would it be helpful to have a pipeline you could run that would spew out a million log messages?
  • b

    Ben Gotow

    12/06/2018, 11:41 PM
    oh for sure 👍
Powered by Linen
Title
b

Ben Gotow

12/06/2018, 11:41 PM
oh for sure 👍
View count: 1