https://dagster.io/ logo
Powered by
#dagster-airflow
  • s

    schrockn

    04/03/2019, 5:24 PM
    @nate at what point in airflow does it become a real problem for the scheduler to create the dags
  • s

    schrockn

    04/03/2019, 5:24 PM
    in terms of execution time
  • s

    schrockn

    04/03/2019, 5:26 PM
    this could be a problem potentially because the user controls pipeline creation
  • s

    schrockn

    04/03/2019, 5:26 PM
    for example the SC folks have high cost pipeline creation
  • s

    schrockn

    04/03/2019, 5:26 PM
    because they eagerly all the SQL they execute
  • s

    schrockn

    04/03/2019, 5:26 PM
    which is on the order of 100s of files
  • s

    schrockn

    04/03/2019, 5:26 PM
    and a lot of data
  • s

    schrockn

    04/03/2019, 5:27 PM
    yeah not concerned about cpu time
  • s

    schrockn

    04/03/2019, 5:28 PM
    they have 100s of sql queries
  • s

    schrockn

    04/03/2019, 5:28 PM
    each in their own file
  • s

    schrockn

    04/03/2019, 5:29 PM
    there may be some other source of performance issues there
  • s

    schrockn

    04/03/2019, 5:29 PM
    probably has never been run through a profiler
  • s

    schrockn

    04/03/2019, 5:29 PM
    nor has our codebase
  • n

    nate

    04/03/2019, 5:29 PM
    yah our code was basically
    Copy code
    for sql_file in sql_filenames:
    op = SQLOperator(sql_file)
    op.set_upstream(start_op)
    🦜 1
  • n

    nate

    04/03/2019, 5:30 PM
    for O(100s) of SQL files
  • m

    max

    04/03/2019, 9:18 PM
    fixed the issues with black on #1112 above
    s
    • 2
    • 3
  • s

    schrockn

    04/03/2019, 10:48 PM
    @max just to be clear in https://github.com/dagster-io/dagster/pull/1112#pullrequestreview-222489850 we are copy pasta’ing six with_metaclass because we don’t want to count on dagster bringing in six
  • m

    max

    04/03/2019, 11:01 PM
    that dagster_plugin.py file actually still doesn't depend on dagster
  • m

    max

    04/03/2019, 11:01 PM
    the DagsterPythonOperator hides its imports from dagster
  • s

    schrockn

    04/03/2019, 11:01 PM
    oh cool
  • s

    schrockn

    04/03/2019, 11:01 PM
    kk
  • m

    max

    04/03/2019, 11:01 PM
    not sure if this is good at all, but it supports the scaffold/plugin approach
  • m

    max

    04/03/2019, 11:01 PM
    maybe just for today 🙂
  • s

    schrockn

    04/03/2019, 11:02 PM
    👍
  • s

    schrockn

    04/03/2019, 11:02 PM
    gtg
  • m

    max

    04/12/2019, 9:07 PM
    otoh, airflow just released 1.10.3, and a CVE for all previous versions 😑, -- 1.10.3 is also compatible with py37
  • m

    max

    04/12/2019, 9:07 PM
    otoh, i am sure people aren't quick to upgrade so i don't want to have a hard requirement
    👍 1
  • m

    max

    04/12/2019, 9:07 PM
  • s

    schrockn

    04/12/2019, 9:10 PM
    yeah we can’t enforce this
  • s

    schrockn

    04/12/2019, 9:10 PM
    given the difficulty of upgrading airflow
123456Latest