Is there a way to make sure no dagster runs for a job overlap? I have one set of credentials for some third party API, and concurrent logins will invalidate existing ones. So when job runs overlap, they can screw each other up.

Yes, Dagster provides mechanisms to ensure that runs for a job do not overlap. You can achieve this by setting concurrency limits for your jobs. Here are a few methods you can use: 1. Tag Concurrency Limits: You can assign a unique tag to your job and then set up a concurrency limit for that tag in your Dagster instance's configuration. This will ensure that only one run of the job with that tag can execute at a time. Here's an example of how you might configure this in your `dagster.yaml`:

yaml    run_coordinator:      module: dagster.core.run_coordinator      class: QueuedRunCoordinator      config:        tag_concurrency_limits:          - key: "job"            value: "my_unique_job_tag"            limit: 1

And in your job definition, you would include the tag:

python    @job(tags={"job": "my_unique_job_tag"})    def my_job():        ...

2. Custom Sensor Logic: If you are using sensors to trigger your job runs, you can write custom logic in your sensor to check if there are any runs of the job currently in progress or queued. If a run is detected, the sensor can skip triggering a new run. Here's a rough example of how you might implement this:

python    @sensor(job=my_job)    def my_sensor(context):        runs = context.instance.get_runs(            filters=RunsFilter(                job_name=my_job.name,                statuses=[DagsterRunStatus.STARTED, DagsterRunStatus.QUEUED],            ),            limit=1,        )        if runs:            # Skip if there's already a run in progress or queued            return SkipReason(f"{my_job.name} is already running or queued.")        else:            # Trigger a new run if no runs are in progress or queued            return RunRequest(run_key="unique_run_key")

3. Queued Run Coordinator: The QueuedRunCoordinator can be configured to manage run queuing and execution, including setting concurrency limits. This is similar to the tag concurrency limits method but managed through the run coordinator. Please note that the exact implementation details may vary based on your specific Dagster version and setup. The provided examples are based on the documentation and should be adapted to fit your environment. If you need further assistance with setting this up, please refer to the Dagster documentation or consider reaching out to the community for more detailed guidance.