Robert Lawson

02/15/2023, 4:04 PM
Is there a way to get dagster to materialize everything from the command line, or the module in python? I want to set up integration testing and one would simply be to run through every single materialization (like I went to the web UI and clicked "materalize all" on every group) and make sure nothing fails
Incidentally it looks like I could just pull it from the web UI but that feels sloppy
curl '<http://localhost:8080/graphql>' -X POST -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'Referer: <http://localhost:8080/instance/assets>' -H 'content-type: application/json' -H 'Origin: <http://localhost:8080>' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-origin' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache' --data-raw '{"operationName":"AssetCatalogTableQuery","variables":{},"query":"query AssetCatalogTableQuery {\n  assetsOrError {\n    __typename\n    ... on AssetConnection {\n      nodes {\n        id\n        ...AssetTableFragment\n        __typename\n      }\n      __typename\n    }\n    ...PythonErrorFragment\n  }\n}\n\nfragment PythonErrorFragment on PythonError {\n  __typename\n  message\n  stack\n  causes {\n    message\n    stack\n    __typename\n  }\n}\n\nfragment AssetTableFragment on Asset {\n  __typename\n  id\n  key {\n    path\n    __typename\n  }\n  definition {\n    id\n    ...AssetTableDefinitionFragment\n    __typename\n  }\n}\n\nfragment AssetTableDefinitionFragment on AssetNode {\n  id\n  groupName\n  partitionDefinition\n  description\n  repository {\n    id\n    name\n    location {\n      id\n      name\n      __typename\n    }\n    __typename\n  }\n  __typename\n}\n"}'
works well enough
Well I think I can do this with
dagster job launch -j __ASSET_JOB  -w workspace.yml --location
though I'm not sure what the best way to get a success or failure is - use
dagster run list
and grab the most recent one I guess