Manny Schneck

12/02/2021, 8:19 AM
Can we use dagster in anger now that we have a pandas model of the backend? Let's find out, a twitter thread.
👌 1
I can't share the code but yeah.
I can.
This is entirely hacking at the toplevel of the ipython repl.
I have an idempotent script/makefile.
And it's just lifting cleanly into dagster.
Copy code
2021-12-02 02:43:16 - dagster - INFO - doPrice - 77da679c-9ab6-4237-b7de-699bceebe25e - price - 0xDEADBEEF: 3.2069958317065073
Can't share more than that because $JOB, but that's all the wirign working.
Copy code
(venv) [2/12/21 2:43:16] ➜  analysis git:(dagster) ✗ dagster job execute -f scripts/
This is ripping out some intermediate state from an excel worksheet and pulling all kinds of stuff in from dev to mix and match.
And it runs in the orchestrator.
This lift might be completely smooth....
Copying values from the old script into unit job-integration tests and just slicing up the old script into steps.
And I don't even know dagster!
I just used ops.
Copy code
Stuff like this in my dag job.