# ask-community

Sashikanth Dareddy

06/14/2022, 8:14 PM
Hi all, I'm currently working on a project which uses Great Expectations with Dagster as the Orchestrator - I was hoping you can give me some pointers on the following: 1. In our use case - a user can tell us which table in a database they would like to monitor with great expectations - to this end they will provide a no. of inputs such as the table name, database connection strings, at what periodicity they want to monitor (this is just a cron expression to define the schedule). 2. It is possible that two users can monitor the same table but at different frequencies eg: one user wants to monitor on a monthly basis and the other could want to monitor on a daily basis. These two requests will be treated as independent on our end. 3. The way I have it in my head is - each monitoring request from a user is given a unique identifier (UUID) and should be set up as a separate job (or a DAG) - here's where I'm stuck. 4. I cannot for the life of me figure out how to submit a job programmatically to Dagster. For example: with airflow I could dynamically generate a dag and add it to the dags folder. Is something like that (or a different approach) possible in Dagster?. 5. Further context: The user provides the monitoring request using a UI and in the background we have FastAPI acting as a backend which needs to ping across the received monitoring request across to Dagster as a job. Would the gRPC server is what we need to interact with here? 6. Is there a feature in Dagster that is similar to "catchup" feature in Airflow. The need for this feature arises in this situation: Say a user submits a request today to monitor a table on a monthly basis and this table has been around since Jan-2021. In airflow setting the catchup=True we can effectively backfill data as if we had this monthly monitoring setup since Jan-2021 on that table. Is something like this possible with Dagster? Many thanks


06/14/2022, 9:25 PM
Hi Sashikanth. To answer your question partially (4): you can programmatically create graphs/jobs by doing something like this:
In order to launch jobs programmatically, you can use the GraphQL API to launch runs by providing job name and run config: You can also do this through the command line.
There is existing functionality to run backfills in Dagster. You can either do this manually through the Dagit UI, or you can do this programmatically through the backfill CLI:

Sashikanth Dareddy

06/15/2022, 6:07 AM
@claire many thanks I will review the links