Cris
07/02/2020, 3:53 PMalex
07/02/2020, 3:59 PMcelery
is likely your best bet. You can control number of workers and even set up different queues and tag solids to submit themselves in to those queues.alex
07/02/2020, 4:00 PMalex
07/02/2020, 4:03 PMresource
that uses the local filesystem to coordinateCris
07/02/2020, 4:06 PMCris
07/02/2020, 4:07 PMalex
07/02/2020, 4:09 PMCris
07/02/2020, 4:11 PMalex
07/02/2020, 4:11 PMCris
07/02/2020, 4:11 PMalex
07/02/2020, 4:13 PMshould_execute
function you can override on the schedule, and in there you can query the instance to see if there is run still going. A related example can be seen here https://github.com/dagster-io/dagster/blob/master/examples/legacy_examples/dagster_examples/schedules.pyCris
07/02/2020, 4:13 PMCris
07/02/2020, 4:14 PMCris
07/02/2020, 4:14 PMDanny
07/02/2020, 4:25 PMalex
07/02/2020, 4:27 PMSemaphore
if you want to allow N simultaneous things to go http://semanchuk.com/philip/posix_ipc/
otherwise there are lots of file lock libraries, portalocker
is one i saw repeatedlyalex
07/02/2020, 4:32 PM@resource
def my_db():
with Semaphore(name='/some/unique/path', initial_value=MAX_CONCURRENT_CONNECTIONS) as sem:
yield make_database_connection()
Cris
07/02/2020, 4:49 PM