vignettes/nonmem-parallel.Rmd
nonmem-parallel.RmdThis file shows some helpful patterns for using bbr with
NONMEM, mostly relating to running models with parallel execution,
either multiple cores on a local machine (head node, laptop, etc.) or by
executing on a grid.
library(bbr)
MODEL_DIR <- system.file("model", "nonmem" , "pk-parallel", package = "bbr")
DATA_DIR <- system.file("extdata" , package = "bbr")If you would like to set the parallelization for a specific model,
you can do this by setting parallel = TRUE and
threads=SOME_INTEGER where SOME_INTEGER is the
number of cores that you would like the model to use. This can be done
in two ways:
You can pass these settings directly to
submit_model(.bbi_args). This will make the model execute
in parallel for this submission.
mod <- read_model(file.path(MODEL_DIR, 200))
submit_model(
mod,
.bbi_args = list(parallel = TRUE, threads = 4)
)You can attach them to the model object. This will make the model execute in parallel for all future submissions.
mod <- add_bbi_args(mod, list(parallel = TRUE, threads = 4))Note that anything passed to submit_model(.bbi_args)
will override any attached bbi_args of the same name.
The bbi.yaml file is created when
bbi_init() is originally called. It usually lives in the
same folder as your control streams. This file contains “global”
settings for running bbi.
You can set a default number of threads (the number
of cores bbi will use for each model) in this file. This
will be used whenever you call submit_model(), but can be
overriden by passing
submit_model(..., .bbi_args = list(threads = SOME_INTEGER)).
It defaults to parallel: false so you need
to change to parallel: true or you will need to
pass .bbi_args = list(parallel=TRUE) to each of your
submit_model() calls to make them respect the
threads argument.
You can also pass through .bbi_args to
bbi_init() to set the global defaults when you first create
the bbi.yaml
bbi_init(
.dir = MODEL_DIR,
.nonmem_dir = "/opt/NONMEM",
.nonmem_version = "nm75",
.bbi_args = list(
parallel = TRUE,
threads = 4
)
)Note that anything passed to submit_model(.bbi_args) and
bbi_args attached to model objects will override default
global settings of the same name.
For small runs, use a 4-core or 8-core head node and just do
submit_model(..., .mode = "local").
This will execute the model on your head node, means you will not have to wait for worker nodes to spin up.
This is especially nice for new models that are fairly simple and you just want a quick result back.
submit_model(
mod,
.mode = "local",
.bbi_args = list(parallel = TRUE, threads = 4, overwrite = TRUE) # not needed if set in bbi.yaml
)Running:
bbi nonmem run local /data/home/sethg/bbr/inst/model/nonmem/pk-parallel/100.ctl --overwrite --threads=4 --parallel
In /data/home/sethg/bbr/inst/model/nonmem/pk-parallel
Process finished.You can also set options("bbr.bbi_exe_mode" = "local")
to default to local execution for all models, though this will reset
with each new R session.
You can pass use submit_model(..., .wait = FALSE) to get
you console back immediately.
This is only relevant for
.mode = "local" because grid jobs give you your
console as soon as the job is submitted to the grid (which
typically takes less than a second).
mod <- read_model(file.path(MODEL_DIR, 200))
proc <- submit_model(
mod,
.mode = "local",
.wait = FALSE, # this is what gets you your console back
.bbi_args = list(parallel = TRUE, threads = 8) # not needed if set in bbi.yaml
)There are a few ways to check on your model while it runs:
mod
── Status ──────────────────────────────────────────────────────────────────────
• Incomplete Run
── Absolute Model Path ─────────────────────────────────────────────────────────
• /data/home/sethg/bbr/inst/model/nonmem/pk-parallel/200
── YAML & Model Files ──────────────────────────────────────────────────────────
• /data/home/sethg/bbr/inst/model/nonmem/pk-parallel/200.yaml
• /data/home/sethg/bbr/inst/model/nonmem/pk-parallel/200.ctl
── Description ─────────────────────────────────────────────────────────────────
• original acop model
── Tags ────────────────────────────────────────────────────────────────────────
• acop tag
• other tag
── BBI Args ────────────────────────────────────────────────────────────────────
• overwrite: TRUE
• threads: 4bbr::tail_*() helpers to check on the
.lst or OUTPUT files.
tail_lst(mod)Mon Sep 27 12:37:49 EDT 2021
$PROBLEM From bbr: see 107.yaml for details
...
WARNINGS AND ERRORS (IF ANY) FOR PROBLEM 1
(WARNING 2) NM-TRAN INFERS THAT THE DATA ARE POPULATION.
tail_output(mod, .head = 0, .tail = 10)...
1.1744E-01 -2.8718E+02 -1.4260E+02 -1.2769E+02 3.1651E+02
0ITERATION NO.: 1 OBJECTIVE VALUE: 31860.4697608811 NO. OF FUNC. EVALS.: 15
CUMULATIVE NO. OF FUNC. EVALS.: 26
NPARAMETR: 4.9843E-01 3.6746E+00 1.0057E+00 4.4079E+00 1.9381E+00 9.9679E-01 9.9702E-01 4.9976E-01 2.0001E-01 9.9998E-03
1.0000E-02 2.0002E-01 1.0003E-02 2.0001E-01 4.9994E-02
PARAMETER: 9.9686E-02 1.0499E-01 1.0057E-01 1.1020E-01 9.6903E-02 9.9679E-02 9.9702E-02 9.9952E-02 1.0001E-01 9.9996E-02
1.0000E-01 1.0005E-01 1.0003E-01 1.0002E-01 9.9941E-02
GRADIENT: 2.9997E+03 -1.2157E+04 -3.9326E+03 5.4698E+04 2.2187E+04 1.5492E+03 1.5490E+03 2.5233E+02 -2.5887E+02 2.7454E+00
3.2905E+01 3.4014E+01 -9.9727E+01 -1.5446E+02 2.2149E+02tail -f in
the terminal to “follow” the files as they change.
lst_path <- build_path_from_model(mod, ".lst")
cat(paste("tail -f", lst_path))tail -f /data/home/sethg/bbr/inst/model/nonmem/pk-parallel/200/200.lstprocessx object including
in the bbi_process object.
submit_model() to a variable (proc,
above)processx docs for details on available methods.
proc$process$is_alive()TRUEYou can submit a dummy job to the grid to get it to bring up a
worker, while you finish preparing your run. For SGE, this is done with
qsub.
system("echo 'sleep 10' | qsub")Unable to run job: warning: sethg's job is not allowed to run in any queue
Your job 1 ("STDIN") has been submitted
Exiting.This message is just telling you that there are no worker nodes currently up, but one will begin to spin up now.
If you have a long-running model, it is often nice to launch it locally first, just to see if it starts.
DATASET ERROR or some silly syntax
error.
mod <- read_model(file.path(MODEL_DIR, 300))
proc <- submit_model(
mod,
.mode = "local",
.wait = FALSE # this is what gets you your console back
)
tail_output(mod, .tail = 15)License Registered to: Metrum Research Group
Expiration Date: 14 JUL 2022
Current Date: 27 SEP 2021
...
IWRS=IWRESI
IPRD=IPREDI
IRS=IRESI
MONITORING OF SEARCH:
0ITERATION NO.: 0 OBJECTIVE VALUE: 32098.8565339901 NO. OF FUNC. EVALS.: 11
CUMULATIVE NO. OF FUNC. EVALS.: 11
NPARAMETR: 5.0000E-01 3.5000E+00 1.0000E+00 4.0000E+00 2.0000E+00 1.0000E+00 1.0000E+00 5.0000E-01 2.0000E-01 1.0000E-02
1.0000E-02 2.0000E-01 1.0000E-02 2.0000E-01 5.0000E-02
PARAMETER: 1.0000E-01 1.0000E-01 1.0000E-01 1.0000E-01 1.0000E-01 1.0000E-01 1.0000E-01 1.0000E-01 1.0000E-01 1.0000E-01
1.0000E-01 1.0000E-01 1.0000E-01 1.0000E-01 1.0000E-01
GRADIENT: 1.6936E+03 -2.6923E+04 -3.0654E+03 -5.5039E+04 1.6715E+04 1.7314E+03 1.6062E+03 2.6154E+02 -7.7264E+01 2.0742E+01
1.1744E-01 -2.8718E+02 -1.4260E+02 -1.2769E+02 3.1651E+02There are several ways to kill the process:
killall nonmem in the terminal (or with
system) to kill all NONMEM jobs running
locally.
system("killall nonmem")kill() method of the
processx object (see above about how to get to this
object).
proc$process$kill()Then relaunch the model on the grid, using
.bbi_args = list(overwrite = TRUE) to overwrite the
incomplete results.
submit_model(
mod,
.bbi_args = list(
overwrite = TRUE,
parallel = TRUE,
threads = 8
)
)Running:
bbi nonmem run sge /data/home/sethg/bbr/inst/model/nonmem/pk-parallel/300.ctl --overwrite --threads=8 --parallel
In /data/home/sethg/bbr/inst/model/nonmem/pk-parallel
Process finished.Note: .mode = "sge" (submit to the grid) is the default
for submit_model() so there is no need to pass it as an
argument.
You can monitor your grid jobs with tail_lst() or
tail_output() as above, or use qstat -f in the
terminal, or with the Grafana dashboard (if you are on Metworx).
system("qstat -f")############################################################################
- PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS
############################################################################
7 0.55500 Run_300 sethg qw 09/27/2021 12:53:10 8 You can use submit_models() to submit a batch of models
to the grid.
Each model will be a separate grid job
The grid will “auto-scale”; bringing up new workers to handle all of the jobs
This is also a very easy way to use nested parallelism:
.bbi_args = list(parallel = TRUE, threads = SOME_INTEGER)
then each model will use SOME_INTEGER cores on its
worker node.
mods <- map(c(200, 300, 400), ~ read_model(file.path(MODEL_DIR, .x)))
submit_models(
mods,
.bbi_args = list(
overwrite = TRUE,
parallel = TRUE,
threads = 8
)
)
system("qstat -f")############################################################################
- PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS
############################################################################
8 0.55500 Run_200 sethg qw 09/27/2021 12:56:18 8
9 0.55500 Run_300 sethg qw 09/27/2021 12:56:18 8
10 0.55500 Run_400 sethg qw 09/27/2021 12:56:18 8 qdel in the
terminal (or with system() from R).
qdel SOME_NUMBER kills the job specified by it’s “job
id” (the first column in the qstat -f output)qdel -u YOUR_USERNAME kills all jobs launched
by you
this_user <- Sys.getenv("USER")
system(paste("qdel -u", this_user))For a big model, try running a few iterations with different numbers
of threads. This way you can see the gains from adding more threads and
decide how many to use as you continue to work on this model. This can
be done via test_threads(), which will do the following
things:
delete_models() (shown
below).MAX_EVAL or
NITER in your control stream so that it finishes fairly
quickly. Typically, something between 10 (the default) and 100 is good,
depending on how long each iteration takes. Ideally, you want something
that will run for 3-5 minutes total to give you the most accurate
estimates.
# running it
mod <- read_model(file.path(MODEL_DIR, 200))
mods_test <- test_threads(mod, .threads = c(2, 4, 8))Submitting 3 models with 3 unique configurations.We can see that 3 models have been created and submitted to the grid.
system("qstat -f")############################################################################
- PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS
############################################################################
1 0.00000 Run_200_2_ sethg qw 06/28/2022 15:52:36 2
2 0.00000 Run_200_4_ sethg qw 06/28/2022 15:52:36 4
3 0.00000 Run_200_8_ sethg qw 06/28/2022 15:52:36 8 Once run, you can use check_run_times() to inspect the
run times for each of the thread options you are benchmarking. By
default, your R console with be frozen until the model runs have
finished. You can pass .wait = FALSE to disable this, in
which case you will get NA in the rows corresponding to
models that have not yet finished.
check_run_times(mods_test)Waiting for 3 model(s) to finish...
3 model(s) have finished
# A tibble: 3 × 5
run threads estimation_time
<chr> <int> <dbl>
1 200_2_threads 2 14.04
2 200_4_threads 4 9.59
3 200_8_threads 8 5.64This tibble has one row for each test run and a column showing the
time for each (in seconds). By default it returns the estimation time,
but you can pass .return_times
As outlined above, you can easily delete these dummy models via
delete_models(). The default .tags argument is
“test threads”, meaning only model objects with that tag will be
deleted. In addition, the default behavior is to prompt you if you’re
sure you want to delete the associated model files.
delete_models(mods_test)Are you sure you want to remove 3 models with the following tags?: `test threads` (Yes/no/cancel) y
Removed 3 models with the following tags:
- test threads