https://dagster.io/ logo
#announcements
Title
# announcements
a

Adonis

12/17/2020, 3:24 PM
Hello, I've been using dagster for a while (great job guys by the way!), however I'm hitting a roadblock: • The backfill using the CLI appears to work properly • The backfill using dagit doesn't do anything (no log, no action after clicking the button) Has anyone here already experienced this issue? Does anyone can tell me how to have debug/verbose mode enabled when using dagit? Many thanks! Tested on 0.9.21 and 0.9.22.post0 Previously the backfill with the UI on 0.9.5 worked properly
s

sandy

12/17/2020, 3:48 PM
Hi - does the button spin after you press it?
a

Adonis

12/17/2020, 3:52 PM
@sandy Hi, well it spins and then goes back to its previous state:
s

sandy

12/17/2020, 4:23 PM
Very strange. You can go to <dagit url>/flags to turn on debug logging in the browser console
a

Adonis

12/17/2020, 4:43 PM
Was hoping for an error, but I don't see anything outrageous in the console
d

daniel

12/17/2020, 4:46 PM
Sorry this is giving you trouble - is there any way you'd be able to post code to reproduce this locally? (Filed an issue to improve our verbose logging to be more helpful in cases like this in the future: https://github.com/dagster-io/dagster/issues/3438)
a

Adonis

12/17/2020, 4:49 PM
@daniel I'll investigate more on my end, and if I can't find anything I'll reduce my code to the bare minimum for a reproducible case - This feature would indeed be very nice!
d

daniel

12/17/2020, 4:55 PM
@Adonis another thing that @dish on our team suggested that might work - if you're able to use chrome devtools to capture the Websocket request and response that goes out when you press the launch button and send that our way, that would also be very useful for us to see what's happening
a

Adonis

12/17/2020, 5:02 PM
@daniel Here it is:
As a side note, my switch from 0.9.5 to 0.9.22post0 might be too brutal (although I'm not using any of the features mentioned in the breaking changes) with the same setup. I might to rebuild my setup to be sure
d

daniel

12/17/2020, 5:07 PM
Ah, OK, so I do see an InvalidStepError in there, which we just added better error handling for a couple of days ago actually (but that wouldn't be in 0.9.22 yet). Two things are surprising to me: • that the same command is passing in the CLI, I'd expect that to also give you some kind of InvalidStepError • That you're not seeing some kind of catchall red box that says "An unexpected error occurred"
a

Adonis

12/17/2020, 5:09 PM
For the screenshot I was actually using
0.9.21
...
Copy code
$pip freeze | grep dagster
dagster==0.9.21
dagster-cron==0.9.21
dagster-graphql==0.9.21
dagster-pandas==0.9.21
dagster-postgres==0.9.21
d

daniel

12/17/2020, 5:16 PM
OK, I think the red box might be in 0.9.22pre0. So that could solve that mystery
The mystery of why you're not getting an InvalidStepError in the CLI backfill is still there though.
a

Adonis

12/17/2020, 5:21 PM
Yeah I'm obtaining this through the CLI directly:
Copy code
$ dagster pipeline backfill -p total_groene_mient_etl_gm -w ./dagster/workspace.yaml  --location scp_data_app.repos.groene_mient --partitions 2020-04-01
$ PYTHONPATH=. dagster pipeline backfill -p total_groene_mient_etl_gm -w ./dagster/workspace.yaml  --location scp_data_app.repos.groene_mient --partitions 2020-04-01

     Pipeline: total_groene_mient_etl_gm
Partition set: monthly_total_groene_mient_etl_gm
   Partitions: 
                2020-04-01

Do you want to proceed with the backfill (1 partitions)? [y/N]: y
Launching runs... 
Launched backfill job `lpugyymr`

$ dagster - DEBUG - total_groene_mient_etl_gm - 5b832854-a8c9-4631-922d-c4399a16f276 - 13471 - ENGINE_EVENT - Starting initialization of resources [asset_store, measurement_metadata_repo, measurement_publisher, measurement_repo].
2020-12-17 18:19:25 - dagster - DEBUG - total_groene_mient_etl_gm - 5b832854-a8c9-4631-922d-c4399a16f276 - 13471 - ENGINE_EVENT - Finished initialization of resources [asset_store, measurement_metadata_repo, measurement_publisher, measurement_repo].
2020-12-17 18:19:25 - dagster - DEBUG - total_groene_mient_etl_gm - 5b832854-a8c9-4631-922d-c4399a16f276 - 13471 - PIPELINE_START - Started execution of pipeline "total_groene_mient_etl_gm".
2020-12-17 18:19:27 - dagster - DEBUG - total_groene_mient_etl_gm - 5b832854-a8c9-4631-922d-c4399a16f276 - 13471 - ENGINE_EVENT - Executing steps using multiprocess executor: parent process (pid: 13471)
2020-12-17 18:19:27 - dagster - DEBUG - total_groene_mient_etl_gm - 5b832854-a8c9-4631-922d-c4399a16f276 - 13471 - total.extract_total_house_energy_demand.compute - ENGINE_EVENT - Launching subprocess for total.extract_total_house_energy_demand.compute
2020-12-17 18:19:27 - dagster - DEBUG - total_groene_mient_etl_gm - 5b832854-a8c9-4631-922d-c4399a16f276 - 13471 - total.extract_total_house_energy_supply.compute - ENGINE_EVENT - Launching subprocess for total.extract_total_house_energy_supply.compute
2020-12-17 18:19:27 - dagster - DEBUG - total_groene_mient_etl_gm - 5b832854-a8c9-4631-922d-c4399a16f276 - 13471 - total.extract_total_house_heating_consumption.compute - ENGINE_EVENT - Launching subprocess for total.extract_total_house_heating_consumption.compute
And I can see them properly in the UI as well
d

daniel

12/17/2020, 5:22 PM
and, sorry, I just verified, I think the red box change is coming in a couple weeks in 0.10.0, not 0.9.21 or 0.9.22pre0. So that explains that.
@Adonis would it be possible to get the full JSON request immediately before the InvalidStepError line in that screenshot earlier?
a

Adonis

12/18/2020, 10:05 AM
@daniel Sorry for the delay, here is the JSON from the WS before an InvalidStepError:
Copy code
{
  "id": "33",
  "type": "start",
  "payload": {
    "variables": {
      "backfillParams": {
        "selector": {
          "partitionSetName": "monthly_total_schoonschip_etl_sc",
          "repositorySelector": {
            "repositoryName": "schoonschip",
            "repositoryLocationName": "scp_data_app.repos.schoonschip"
          }
        },
        "partitionNames": [
          "2020-11-01"
        ],
        "reexecutionSteps": [
          "total.compute",
          "extract_total_house_battery_consumption.compute",
          "transform_total_house_battery_consumption.compute",
          "load_total_house_battery_consumption.compute",
          "extract_total_house_battery_production.compute",
          "transform_total_house_battery_production.compute",
          "load_total_house_battery_production.compute",
          "extract_total_house_battery_soc.compute",
          "transform_total_house_battery_soc.compute",
          "load_total_house_battery_soc.compute",
          "extract_total_house_heating_consumption.compute",
          "transform_total_house_heating_consumption.compute",
          "load_total_house_heating_consumption.compute",
          "extract_total_house_pv_production.compute",
          "transform_total_house_pv_production.compute",
          "load_total_house_pv_production.compute"
        ],
        "fromFailure": false,
        "tags": []
      }
    },
    "extensions": {},
    "operationName": "LaunchPartitionBackfill",
    "query": "mutation LaunchPartitionBackfill($backfillParams: PartitionBackfillParams!) {\n  launchPartitionBackfill(backfillParams: $backfillParams) {\n    __typename\n    ... on PartitionBackfillSuccess {\n      backfillId\n      __typename\n    }\n    ... on PartitionSetNotFoundError {\n      message\n      __typename\n    }\n    ... on PythonError {\n      message\n      stack\n      __typename\n    }\n  }\n}\n"
  }
}
d

daniel

12/18/2020, 1:17 PM
Thanks! For some reason it doesn’t seem to like one of those steps that’s being passed in the reexecutionSteps key (which is almost certainly a bug). If it’s possible to post the code for the pipeline I think we should be able to get to the bottom of this
a

Adonis

12/18/2020, 2:56 PM
@daniel Fair enough, though as I mentioned before, I need to strip down the code to a small reproducible version of this issue, since it was made generic to serve for several of our use cases. It would be quite unreadable if I simply copy and pasted it
d

daniel

12/18/2020, 3:00 PM
Sure -it's possible we'd be able to find the bug just from inspecting the original code (even if it's big or we can't execute it locally). But if you're willing to strip it down for us that's great too - either option works!
(to be clear, the bug in dagster, not the bug in your code 🙂 )
7 Views