hey I have the following setup dagster on k8s same pod execu

Question

hey I have the following setup dagster on k8s same pod executes the scheduled pipelines too cron is running in the pod `schedule storage postgres` my problem scheduled jobs don t start when they supposed to when I run `dagster schedule up location Business Intelligence ` I get this every time ```Errors Resolved Schedule daily answer position pipeline is set to be running but the scheduler is not running the schedule Schedule daily events rounds pipeline is set to be running but the scheduler is not running the schedule``` I renamed the repo several times now when I run `dagster schedule debug` I can see schedules from those repos with `status RUNNING` what can be the problem

sashank · Answer

Hey < szalai1> you can check for runtime errors with the schedule using the `dagster schedule logs` command

sashank · Answer

Try running `dagster schedule logs location Business Intelligence daily answer position pipeline`

sashank · Answer

Also if you have dagit running try clicking on the schedule name within dagit to go to the schedule specific view You may see helpful error messages there

szalai1 · Answer

on Dagit I cannot see previous runs

szalai1 · Answer

when I check the logs the log files are empty

szalai1 · Answer

it seems like it never tried to execute the pipelines

szalai1 · Answer

if I run the script scheduled in cron manually the run appears on Dagit

Titouan · Answer

If you use the helm package to deploy Dagster on your k8s cluster you need to change the command in the h `elm dagster templates deployment dagit yaml` and add the scheduler commands

szalai1 · Answer

i did the `schedule up` but not the ` start all`

szalai1 · Answer

thanks

sashank · Answer

The ` start all` shouldn t really matter since you have manually started the schedule through the UI

szalai1 · Answer

I manually started from the pod

szalai1 · Answer

I run the script genereted by dagster

szalai1 · Answer

with `crontab l` I listed the scheduled jobs

sashank · Answer

Were the jobs listed there

sashank · Answer

Could you post the output of your `dagster schedule debug` command here It should show us everything including the cron tab

szalai1 · Answer

```root dagster dagit 6f6c6c64bd 85jpw opt dagster app dagster schedule debug Errors Run `dagster schedule up` to resolve ============================================= Schedule daily answer position pipeline is set to be running but the scheduler is not running the schedule Schedule daily events rounds pipeline is set to be running but the scheduler is not running the schedule Scheduler Configuration ======================= Scheduler module dagster cron cron scheduler class SystemCronScheduler config Scheduler Info ============== Running Cron Jobs 0 1 opt dagster dagster home schedules scripts cb7af3fe21f2fc954ea60dad4502658594ede1fa sh gt opt dagster dagster home schedules logs cb7af3fe21f2fc954ea60dad4502658594ede1fa scheduler log 2 gt amp 1 dagster schedule cb7af3fe21f2fc954ea60dad4502658594ede1fa daily opt dagster dagster home schedules scripts f9b01b67ce89b111f67ea06bf192c062e5698471 sh gt opt dagster dagster home schedules logs f9b01b67ce89b111f67ea06bf192c062e5698471 scheduler log 2 gt amp 1 dagster schedule f9b01b67ce89b111f67ea06bf192c062e5698471 daily opt dagster dagster home schedules scripts b03f3aa7aa968b38665e82bfb4fecbbf405c280a sh gt opt dagster dagster home schedules logs b03f3aa7aa968b38665e82bfb4fecbbf405c280a scheduler log 2 gt amp 1 dagster schedule b03f3aa7aa968b38665e82bfb4fecbbf405c280a daily opt dagster dagster home schedules scripts 3f340cb3cd5b02d635e20de3012be5aec6d28fe4 sh gt opt dagster dagster home schedules logs 3f340cb3cd5b02d635e20de3012be5aec6d28fe4 scheduler log 2 gt amp 1 dagster schedule 3f340cb3cd5b02d635e20de3012be5aec6d28fe4 Scheduler Storage Info ====================== daily answer position pipeline cron schedule 0 0 python path usr local bin python repository origin id 94883ff7995e2aeb02a9caf1f8ba503b654dd122 repository pointer opt dagster app src repositories py kafka topic loader repository schedule origin id d5c6079c7dfb3d375c01e431aad6a52c0d24f978 status STOPPED daily events rounds pipeline cron schedule 0 0 python path usr local bin python repository origin id 94883ff7995e2aeb02a9caf1f8ba503b654dd122 repository pointer opt dagster app src repositories py kafka topic loader repository schedule origin id 60590970adb2fa159911597e625a5cdffeab4e78 status STOPPED daily answer position pipeline cron schedule 0 0 python path usr local bin python repository origin id c2a9de0a59b1f05956c1688d4c3a7dce90d949ea repository pointer opt dagster app src repositories py bi repository schedule origin id e785e863d275b277ec030e30e446e60a68412c11 status RUNNING daily events rounds pipeline cron schedule 0 0 python path usr local bin python repository origin id c2a9de0a59b1f05956c1688d4c3a7dce90d949ea repository pointer opt dagster app src repositories py bi repository schedule origin id 4ac68c41140226dc41b6382b02e37117baba6d5f status RUNNING daily answer position pipeline cron schedule 0 0 python path usr local bin python repository origin id cd0eff9914141ed9ca53ba32b71349d7d82a9903 repository pointer opt dagster app src repositories py business intelligence repository schedule origin id f9b01b67ce89b111f67ea06bf192c062e5698471 status RUNNING daily events rounds pipeline cron schedule 0 0 python path usr local bin python repository origin id cd0eff9914141ed9ca53ba32b71349d7d82a9903 repository pointer opt dagster app src repositories py business intelligence repository schedule origin id b03f3aa7aa968b38665e82bfb4fecbbf405c280a status RUNNING daily maker organisation changes cron schedule 0 0 python path usr local bin python repository origin id cd0eff9914141ed9ca53ba32b71349d7d82a9903 repository pointer opt dagster app src repositories py business intelligence repository schedule origin id 3f340cb3cd5b02d635e20de3012be5aec6d28fe4 status RUNNING daily aggregate model monitoring pipeline cron schedule 0 1 python path usr local bin python repository origin id 1837b1588a140ee6a1f478dc3d0bece0c3c083d4 repository pointer opt dagster app src model repo py ds models repository schedule origin id cb7af3fe21f2fc954ea60dad4502658594ede1fa status RUNNING```

szalai1 · Answer

I renamed `bi repository` repo to `business intelligence repository` but I still can see the schedules in there ^

sashank · Answer

Ah the rename is problematic This is because the location name also changes and the scheduler does not know to delete the schedules from the previous location when running `up` with the new location To prevent this again I would recommend manually setting a location name in your `dagster yaml` that stays consistent For now the first thing I would do is run `dagster schedule wipe` then `dagster schedule up` again to get us into a clean state

sashank · Answer

Then I would start the schedule you want to debug perhaps set it to run soon or on a short interval so we can make sure it runs successfully and once you know the scheduled time is passed then check the logs It should have some output since we redirect all output from the script command to the log file

szalai1 · Answer

thank you < sashank> I ll do as you advised

sashank · Answer

If there is no output that would probably mean that cron service itself is not running We can check if cron is running by adding a test cron job perhaps echo something to a file and check if it runs ``` echo test gt testfile txt```

sashank · Answer

Let me know if you end up verifying cron is running but there is no output in the log files or if `dagster schedule debug` shows and issues after you run `dagster schedule up` and `dagster schedule up`

szalai1 · Answer

< sashank> my findings cron starts the job `scheduled log` is empty there are files like this `20200723T070117 f9b01b67ce89b111f67ea06bf192c062e5698471 result` though and an error message `You have attempted to fetch the environment variable DAGSTER PG PASSWORD which is not set `

szalai1 · Answer

which makes sense as cron just starts the script but nothing sets the env vars before that

szalai1 · Answer

schedule config from my `dagster yaml` ```schedule storage module dagster postgres schedule storage class PostgresScheduleStorage config postgres db username dagster password env DAGSTER PG PASSWORD hostname dagster postgresql db name dagster port 5432```

sashank · Answer

Since cron runs in a separate mostly empty environment it doesn t have the same env vars as your default shell However each schedule decorator takes an environment vars argument that lets you copy over the required env vars to the shell script that is invoked by the scheduler

sashank · Answer

Here s an example of specifying which env vars you need <https github com dagster io dagster blob master python modules dagster test dagster test test project test pipelines repo py L206 L227>

szalai1 · Answer

this is brilliant thank you very much < sashank> I m supper excited about dagster thanks for the great tool and the support Take care

sashank · Answer

That s great to hear Please let me know if you still run into issues after specifying the env vars or if we can help in any other way

Paul Wyatt · Answer

< Travis Cline> < aqm> this is what s causing our current problem

szalai1 · Answer

another gotcha env vars are put into the generated script as without quotes which means that if you have special characters e g ` ` then it will make generated script fail with syntax error I solved the problem by escaping the string `f os environ get key `