vignettes/using-summary-log.Rmd
using-summary-log.RmdThis vignette demonstrates how to use summary_log() to
extract model diagnostics like the objective function value, condition
number, and parameter counts.
If you are new to bbr, the “Getting
Started with bbr” vignette will take you through some basic
scenarios for modeling with NONMEM using bbr, introducing
you to its standard workflow and functionality.
There is a lot of information in the bbi_summary_log_df
tibble that is output from summary_log(). However, it is
important to note that all of this, and quite a bit more, is contained
in the bbi_nonmem_summary object output from
model_summary(). If you are trying to dig deep into the
outputs of a small number of models, see the Summarize
section of the “Getting Started” vignette for an introduction to
that functionality.
summary_log() is more useful for getting a slightly
higher-level view of a larger batch of models; potentially all the
models in a given project, or something like a large group of
bootstrapped runs.
There is some initial set up necessary for using bbr.
Please refer to the “Getting
Started” vignette, mentioned above, if you have not done this yet.
Once this is done, load the library.
As mentioned above, the bbi_summary_log_df tibble
contains a lot of information, which is a subset of what is contained in
the bbi_nonmem_summary object returned from
model_summary().
MODEL_DIR <- system.file("model", "nonmem", "complex", package = "bbr")
sum1 <-
read_model(file.path(MODEL_DIR, "iovmm")) %>%
model_summary()
names(sum1)
#> [1] "absolute_model_path" "run_details" "run_heuristics"
#> [4] "parameters_data" "parameter_names" "ofv"
#> [7] "condition_number" "shrinkage_details" "success"For example, the run_details section alone contains
wealth of information about this model run:
str(sum1$run_details)
#> List of 16
#> $ version : chr "7.4.4"
#> $ run_start : num NA
#> $ run_end : chr "Tue Aug 25 16:50:27 EDT 2020"
#> $ estimation_time : num 132
#> $ postprocess_time : num 1.04
#> $ cpu_time : num 134
#> $ function_evaluations : int 380
#> $ significant_digits : int 3
#> $ problem_text : chr "10 mixture model and IOV on CL"
#> $ mod_file : num NA
#> $ estimation_method : chr "First Order Conditional Estimation"
#> $ data_set : chr "../MixSim.csv"
#> $ number_of_subjects : int 300
#> $ number_of_obs : int 12600
#> $ number_of_data_records: int 13500
#> $ output_files_used : chr [1:5] "iovmm.lst" "iovmm.cpu" "iovmm.ext" "iovmm.grd" ...Much of this is very useful, but it’s also a bit intimidating, and it can take some work to unpack it all and find the bits and pieces you’re looking for.
The summary_log() function is designed to extract some
of the most relevant diagnostics and model outputs from a batch of model
summaries and organize them into a more easily digestible tibble. Like
run_log() and config_log(), it takes two
arguments:
.base_dir – Directory to look for models in..recurse – Logical indicating whether to search
recursively in subdirectories. This is TRUE by
default.
sum_df <- summary_log(MODEL_DIR)
names(sum_df)
#> [1] "absolute_model_path" "run"
#> [3] "bbi_summary" "needed_fail_flags"
#> [5] "error_msg" "problem_text"
#> [7] "estimation_method" "number_of_subjects"
#> [9] "number_of_obs" "ofv"
#> [11] "param_count" "condition_number"
#> [13] "any_heuristics" "covariance_step_aborted"
#> [15] "large_condition_number" "eigenvalue_issues"
#> [17] "correlations_not_ok" "parameter_near_boundary"
#> [19] "hessian_reset" "has_final_zero_gradient"
#> [21] "minimization_terminated" "eta_pval_significant"
#> [23] "prderr"The specific columns returned are described below, though there is
also a list of them, with brief definitions, in the summary_log()
docs that can be accessed any time with ?summary_log()
in the console.
The first column is absolute_model_path which contains
an absolute path that unambiguously identifies each model. This serves
as the primary key for the tibble. The second column is simply the
basename of this path, which is just a convenience for
printing and viewing.
The third column contains the bbi_nonmem_summary object,
discussed above, for each model. This can be extracted and manipulated
if you would like more detailed data from it.
The error_msg and needed_fail_flags columns
describe whether bbi had any trouble parsing the model
outputs. These won’t be discussed in detail here. Refer to the summary_log()
docs for more information.
The next batch of columns contain the core diagnostics and model
outputs. As mentioned above, descriptions of what each column contains
can be found in the summary_log()
docs.
sum_df %>%
collapse_to_string(estimation_method) %>%
select(
run,
ofv,
param_count,
estimation_method,
problem_text,
number_of_subjects,
number_of_obs,
condition_number
)
#> # A tibble: 6 × 8
#> run ofv param_count estimation_method problem_text number_of_subjects
#> <chr> <dbl> <int> <chr> <chr> <int>
#> 1 1001 3843. 15 MCMC Bayesian An… Run# 1001.1 240
#> 2 acop-fa… 2675. 7 First Order Cond… PK model 1 … 40
#> 3 acop-iov 44159. 9 First Order Cond… PK model 1 … 39
#> 4 acop-on… NA NA NA PK model 1 … 40
#> 5 example… -10839. 21 Stochastic Appro… RUN# exampl… 400
#> 6 iovmm 14722. 11 First Order Cond… 10 mixture … 300
#> # ℹ 2 more variables: number_of_obs <int>, condition_number <dbl>The run_heuristics element of the
bbi_nonmem_summary object contains a number of logical
values indicating whether particular heuristic issues were found in the
model. Note that these are not necessarily errors with the
model run, but are closer to warning flags that should possibly be
investigated. Each heuristic is described in more detail in the summary_log()
docs.
Note that all heuristics will be FALSE by
default (and never NA) and will only be
TRUE if they are explicitly triggered. For
example, large_condition_number will be FALSE
even in the case when a condition number was not calculated at all.
All of the heuristic flags are pivoted out to their own columns in
the bbi_summary_log_df tibble. It’s useful to note that,
except for needed_fail_flags (discussed above), these are
the only logical columns in the tibble and can therefore be
easily selected with tidyselect::where(is.logical).
(Note: where() only became available in
tidyselect (>= 1.1.0), released May 2020.)
#> # A tibble: 6 × 13
#> run needed_fail_flags any_heuristics covariance_step_aborted
#> <chr> <lgl> <lgl> <lgl>
#> 1 1001 TRUE TRUE FALSE
#> 2 acop-fake-bayes FALSE TRUE FALSE
#> 3 acop-iov FALSE TRUE FALSE
#> 4 acop-onlysim FALSE FALSE FALSE
#> 5 example2_saemimp FALSE FALSE FALSE
#> 6 iovmm FALSE TRUE FALSE
#> # ℹ 9 more variables: large_condition_number <lgl>, eigenvalue_issues <lgl>,
#> # correlations_not_ok <lgl>, parameter_near_boundary <lgl>,
#> # hessian_reset <lgl>, has_final_zero_gradient <lgl>,
#> # minimization_terminated <lgl>, eta_pval_significant <lgl>, prderr <lgl>Notice that there is also an any_heuristics column, which can easily be used to filter to only runs that had at least one heuristic flag triggered.
Just like config_log()
has add_config(), you can also use
add_summary() to join all of these columns onto an existing
bbi_run_log_df (the tibble output from
run_log()). This can be useful if you have a run log that
you have previously filtered on something like the tags
or based_on
columns, and you would like to append some simple diagnostics.
# filter to two specific runs
log_df <- run_log(MODEL_DIR, .include = c("acop-iov", "iovmm"))
# add summary columns
log_df <- log_df %>% add_summary()
log_df %>% select(run, ofv, param_count, any_heuristics)
#> # A tibble: 2 × 4
#> run ofv param_count any_heuristics
#> <chr> <dbl> <int> <lgl>
#> 1 acop-iov 44159. 9 TRUE
#> 2 iovmm 14722. 11 TRUE