Each controller object only supports only one type of worker
configuration which you set in advance. However, different controllers
may have different types of workers, and
controller groups to coordinate among these different worker types. With
third-party launcher subclasses from other packages, this mechanism will
allow you to e.g. send some tasks to GPU-capable or high-memory workers
while other tasks go to low-spec workers.
We demonstrate with a controller of fully persistent workers which always stay running and a controller of semi-persistent workers which terminate after completing four tasks. We create controller objects with names.
crew uses a different TCP port for each controller you
run, so please do not create hundreds of controllers. Please see the
subsection on ports in the README.
We put these controller objects into a new controller group object.
This controller group has a global
connect() method to
initialize both controllers.
You can choose which worker pool to receive tasks.
The controller group also supports global methods for
These methods operate on all controllers at once by default, but the
controllers argument allows you to select a subset of
controllers to act on. Below in
launcher column of the output indicates which controller
ran the task.
group$wait(controllers = "semi-persistent") group$pop() #> # A tibble: 1 × 12 #> name command result seconds seed algorithm error trace warnings #> <chr> <chr> <list> <dbl> <int> <chr> <chr> <chr> <chr> #> 1 my task NA <dbl> 0 NA NA NA NA NA #> # ℹ 3 more variables: launcher <chr>, worker <int>, instance <chr>
method provides functional programming, and the
argument lets you choose the controller to submit the tasks.
group$map( command = a + b + c + d, iterate = list( a = c(1, 3), b = c(2, 4) ), data = list(c = 5), globals = list(d = 6), controller = "persistent" ) #> # A tibble: 2 × 12 #> name command result seconds seed algorithm error trace warnings #> <chr> <chr> <list> <dbl> <int> <chr> <chr> <chr> <chr> #> 1 1 NA <dbl > 0 NA NA NA NA NA #> 2 2 NA <dbl > 0 NA NA NA NA NA #> # ℹ 3 more variables: launcher <chr>, worker <int>, instance <chr>
The controller group has a
summary() method which
aggregates the summaries of one or more controllers.
When you are finished, please call
terminate() with no
arguments to terminate all controllers in the controller group.