Basic Usage of mrgvalprep and mrgvalidate

Introduction

The mrgvalprep package contains helper functions for gathering, formatting, and preprocessing input data for mrgvalidate. The mrgvalidate package (as of the 2.0.0 release) is used to generate 7 specific documents that are necessary for the software validation process at Metrum Research Group. Those documents are:

  • release-notes.docx
  • validation-plan.docx
  • testing-plan.docx
  • requirements-specification.docx
  • traceability-matrix.docx
  • testing-results.docx
  • validation-summary.docx

This vignette demonstrates some of the primary functionality for using mrgvalprep to validate an R package and build validation docs from User Stories and Requirements defined in YAML files.

library(mrgvalprep)
library(mrgvalidate)

mrgvalidate inputs

mrgvalidate requires two primary inputs:

  • A directory of test outputs (.csv and .json files, described below)
  • A tibble of user stories and, optionally, technical requirements

These inputs must be in the format described in ?mrgvalidate::input_formats. The purpose of mrgvalprep is to take test outputs and stories from a variety of formats and transform them into the format that mrgvalidate expects.

Below we show an example of how to do this using User Stories and Requirements defined in YAML files. Note that there are several other formats that are supported for defining Stories and Requirements. However, these are no longer used at MetrumRG. You can find examples of those in the “Legacy Input Formats” vignette.

End-to-end code

This example shows how to validate software that uses R’s testthat framework. The full code (replicated in each section below) is included here for ease of reference. See below for details about each of these functions and their inputs.


# run automated test suite
devtools::test(reporter = testthat::ListReporter) %>% # run testthat tests suite
  mrgvalprep::parse_testthat_list_reporter() %>%      # parse test results to tibble
  readr::write_csv("test_results/package_tests.csv")  # write tibble for .csv file

# capture system information for test suite run
get_sys_info(
  out_path = "test_results/package_tests.json",
  env_vars = c("METWORX_AMI_VERSION"),    # optionally pull with Sys.getenv() and include in .json
  sys_info = TRUE                         # optionally write results of Sys.info() to .json
)

# pull stories and requirements from YAML files in package repo
spec_df <- mrgvalprep::read_spec_yaml(
  "inst/validation/stories.yaml",
  "inst/validation/requirements.yaml"
)

# create validation documents
mrgvalidate::create_package_docs(
  "fake.package",                   # name of the package you are validating
  "vFake",                          # version number for this validation (to be printed in docs)
  language = "R",                   # can be either "R" or "Go" 
  "github.com/user/package",        # URL of associated git repository 
  spec_df,                          # tibble of stories and requirements
  "release_notes.md",               # path to a Markdown file with release notes
  auto_test_dir = "auto_tests",     # dir containing .csv and .json files for automated test outputs
  output_dir = "some_dir_for_docs"  # dir to write validation docs into
)

Test outputs

This section shows how to include the results of automated tests in your validation documents.

The format required for automated test outputs is described in ?mrgvalidate::input_formats.

Note that mrgvalidate can accept test outputs from an arbitrary number of test suites for a single set of validation documents. If you have more than one relevant test suite, simply repeat the steps below for each suite and include the .csv and .json files for all of them in the same directory.

testthat outputs

Currently, mrgvalprep has formatters for R’s testthat package (parse_testthat_list_reporter()), as well as Go’s go test --json output (parse_golang_test_json()). Both functions return a tibble that conforms to ?mrgvalidate::input_formats.

To create test outputs from an R package using testthat, navigate to your package directory and run to following.

devtools::test(reporter = testthat::ListReporter) %>% # run testthat tests suite
  mrgvalprep::parse_testthat_list_reporter() %>%      # parse test results to tibble
  readr::write_csv("test_results/package_tests.csv")  # write tibble for .csv file

This will run your full test suite, format the output appropriately, and then write the formatted output to a .csv for mrgvalidate::create_package_docs() to consume.

Test Id’s

For these tests to be mapped to the Stories described below, they will need to contain Test Id’s. This is simply a string that uniquely identifies each test. These Id’s will look like ABC-XYZ-001 and contain the following three pieces, separated by a hypen (-):

  • Any number of capital letters (typically identifies the test suite)
  • Any number of capital letters (typically identifies some category, for example CFG for “configuration”)
  • Any number of integers, optionally zero-padded for readability

These must be included, with square brackets, in the Test Name of any test that will be mapped to a Story. For example:

test_that("feature X works correctly [MRGV-CFG-001]", {
  # test code
})

In this case, MRGV-CFG-001 will automatically be parsed out by parse_testthat_list_reporter() and will be matched against any Stories or Requirements containing MRGV-CFG-001 in the TestIds column described below.

Note that more than one test can share the same test ID, though you will need to pass roll_up_ids = TRUE to parse_testthat_list_reporter() See the documentation for the relevant parsing function above for more details.

System information

It is typically necessary to capture some system information about where and when the tests were run. This is passed to mrgvalidate::create_package_docs() via a .json file that can be easily created with get_sys_info().

get_sys_info(
  out_path = "test_results/package_tests.json",
  env_vars = c("METWORX_AMI_VERSION"),    # optionally pull with Sys.getenv() and include in .json
  sys_info = TRUE                         # optionally write results of Sys.info() to .json
)

Note: mrgvalidate::create_package_docs() will expect to find a .csv and .json with the same basename for each test suite, so be sure that what you pass to create_sys_info(out_path) matches the .csv of test ouputs that you wrote in the previous section.

Stories and Requirements from YAML files

The validation process typically requires defining User Stories that capture the promised functionality of the software, and mapping them to tests that verify this functionality. The easist way to do this with mrgvalidate is by keeping your Stories (and optionally accompanying Technical Requirements) in YAML files, version controlled in the same repository as the actual code, and read in directly using read_spec_yaml().

The format of these files is described in the read_spec_yaml() documentation. At a high-level, the Stories file should have entries formatted like so:

STORY-ID-001:
  name: Story name
  description: >
    Longer story description
  requirements:
  - REQ-ID-001
  - REQ-ID-002
  ProductRisk: risk
Requirements

While Stories can be mapped directly to tests, it is often useful to instead map them to associated Technical Requirements, which are then mapped to specific tests. To do this, include a second YAML file, with entries formatted like so:

REQ-ID-001:
  description: >
    Description
  tests:
  - TEST-ID-001
  - TEST-ID-001

These Requirements will be included, bullet-pointed, under the relevant Stories in the resulting requirements-specification.docx document.

Calling read_spec_yaml() will pull the data in your files into a tibble that can be passed directly to mrgvalidate::create_package_docs().

# pull stories and requirements from YAML files in package repo
spec_df <- mrgvalprep::read_spec_yaml(
  "inst/validation/stories.yaml",
  "inst/validation/requirements.yaml"
)

Generating the docs

Now that you have test outputs and a tibble of Stories and Requirements, you are ready to generate your validation documents. This is done with mrgvalidate::create_package_docs().

You will have .csv and .json files of automated test outputs, from your validate_tests() call above, in the auto_tests directory, as well as .md capturing your manual tests in the man_tests directory. Simply pass these paths, along with your tibble of Stories, into mrgvalidate::create_package_docs().

mrgvalidate::create_package_docs(
  "fake.package",                   # name of the package you are validating
  "vFake",                          # version number for this validation (to be printed in docs)
  language = "R",                   # can be either "R" or "Go" 
  "github.com/user/package",        # URL of associated git repository 
  spec_df,                          # tibble of stories and requirements
  "release_notes.md",               # path to a Markdown file with release notes
  auto_test_dir = "auto_tests",     # dir containing .csv and .json files for automated test outputs
  output_dir = "some_dir_for_docs"  # dir to write validation docs into
)

This will write out the seven .docx files, mentioned above, into the directory passed to output_dir.

If these documents look correct after human inspection, you’re done!

Formatting note: It is possible to pass .docx “style reference templates” via the style_dir argument to mrgvalidate::create_package_docs(). This can be used to control the formatting, as well as to add a custom header and footer to all of your documents.