getting-started.RmdA regulated environment has to answer four questions about every R package it uses, and answer them with evidence rather than assertion:
assess_risk()
iq_record()
oq_record()
env_manifest()
valpack answers all four from the library you already
have, and packs the answers into a directory you can hand to QA.
assess_risk() runs the riskmetric pipeline
against the installed copy of each package and flattens the
result into one row per package.
The default assessment set queries CRAN over the network for downloads, reverse dependencies and check results. For a fast, offline pass, hand it a subset:
quick <- riskmetric::all_assessments()[c(
"assess_has_news", "assess_has_vignettes", "assess_has_examples",
"assess_exported_namespace", "assess_has_maintainer"
)]
risk <- assess_risk(c("whisker", "jsonlite"), assessments = quick)
risk[, c("package", "version", "score", "has_news", "has_vignettes")]
#>
#> ── valpack risk assessment ─────────────────────────────────────────────────────
#> ℹ 2 packages assessed | 2 scored | riskmetric 0.2.7
#> ℹ 5 assessments: assess_has_news, assess_has_vignettes, assess_has_examples, assess_exported_namespace, assess_has_maintainer
#> ! Scores are comparable only to scores from this same assessment set.
#> ℹ score range 0.104 - 0.264 (higher is better)
#> # A tibble: 2 × 5
#> package version score has_news has_vignettes
#> <chr> <chr> <dbl> <dbl> <dbl>
#> 1 whisker 0.4.1 0.264 1 0
#> 2 jsonlite 2.0.0 0.104 1 1score is on 0-1 and higher is better.
One caveat worth writing into your SOP: riskmetric weights its roll-up
across the assessment set it is handed, so the same package at the same
version scores differently under different sets. Measured here:
tibble is 0.1480 on the five assessments above and 0.3478
on all nineteen — but whisker scores higher on two
assessments (0.5000) than on all nineteen (0.4600), so “a subset
deflates the score” is not a safe rule either. The two numbers are
simply not on the same scale. The set used is recorded on the result so
a mismatch is detectable:
attr(risk, "assessments")
#> [1] "assess_has_news" "assess_has_vignettes"
#> [3] "assess_has_examples" "assess_exported_namespace"
#> [5] "assess_has_maintainer"A package riskmetric cannot reference does not take the run down with it:
assess_risk(c("jsonlite", "not.a.package"), assessments = quick)[, c("package", "score", "error")]
#> ! Cannot reference not.a.package: there is no package called 'not.a.package'
#>
#> ── valpack risk assessment ─────────────────────────────────────────────────────
#> ℹ 2 packages assessed | 1 scored | riskmetric 0.2.7
#> ℹ 5 assessments: assess_has_news, assess_has_vignettes, assess_has_examples, assess_exported_namespace, assess_has_maintainer
#> ! Scores are comparable only to scores from this same assessment set.
#> ℹ score range 0.104 - 0.104 (higher is better)
#> ! not.a.package: there is no package called 'not.a.package'
#> # A tibble: 2 × 3
#> package score error
#> <chr> <dbl> <chr>
#> 1 jsonlite 0.104 NA
#> 2 not.a.package NA there is no package called 'not.a.package'Everything below is read off disk. description_md5 is
the checksum of the installed DESCRIPTION, so a later run
can prove the installation has not been touched.
iq <- iq_record(c("whisker", "jsonlite"))
iq
#>
#> ── valpack installation qualification ──────────────────────────────────────────
#> ℹ R version 4.5.2 (2025-10-31) | x86_64-pc-linux-gnu | Linux 7.0.0-14-generic
#> ℹ Libraries: /tmp/RtmpZtJJJ6/temp_libpathfb7902312fb71, /usr/local/lib/R/site-library, /usr/lib/R/site-library, /usr/lib/R/library
#> ℹ 2 packages: CRAN=2
#> ℹ 1/2 installed with --install-tests (an OQ can only run test suites for those)
#> ℹ Recorded at 2026-08-03 04:47:42 UTC
#> # A tibble: 2 × 10
#> package version library built_r source repository description_md5 priority
#> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
#> 1 whisker 0.4.1 /usr/loca… 4.5.2 CRAN https://c… 4c5fdfe31d8664… NA
#> 2 jsonlite 2.0.0 /usr/loca… 4.5.2 CRAN https://c… 281ddbf8581f58… NA
#> # ℹ 2 more variables: needs_compilation <chr>, installed_tests <lgl>iq_record() with no arguments records the whole library
— that is the form you want for a real IQ.
This is the R replacement for the SAS practice of archiving
PROC SETINIT and the installed-component list: same
purpose, but per package and checksummed.
oq_record() extracts the examples from each installed
help page and runs them in a clean R subprocess with the package
attached — what R CMD check does, including the
cleanEx() reset between topics — but wraps each topic in
tryCatch() so one failure does not hide the rest.
oq <- oq_record("whisker", types = c("examples", "tests"))
#> ℹ Running examples for whisker
#> ℹ Running installed tests for whisker
oq$summary
#> # A tibble: 2 × 7
#> package type pass fail skip harness elapsed
#> <chr> <chr> <int> <int> <int> <int> <dbl>
#> 1 whisker examples 4 0 5 0 0.02
#> 2 whisker tests 5 2 0 0 3.02A topic with no runnable example (everything inside
\dontrun{}, say) is recorded as skip. It is
never quietly counted as a pass:
table(oq$results$status)
#>
#> fail pass skip
#> 2 9 5There is a fourth status, harness, and it is the one to
read carefully. It means the harness could not produce a
verdict — the unit was refused (it needs the console’s
.Last.value, or the package refuses to run while the
isolating tryCatch has handlers on the stack), cut short at
timeout, or never reached because the subprocess ended
early. message says which. A harness row is
missing coverage: it is not a pass and it is not a package defect, and
letting either of those absorb it is exactly how a validation tool fails
a package that is fine.
The per-package budget is an operator setting, not a property of any package, so it is recorded alongside the result:
oq$session$timeout
#> [1] 600And when something genuinely fails, you get the text that proves it, not a flag:
fails <- subset(oq$results, status == "fail")
if (nrow(fails)) cat(substr(fails$message[1], 1, 400))
#> Test failed with 1 failure and 0 successes.
#> Test passed with 1 success 😀.
#> Test passed with 1 success 🌈.
#> ── Failure: Sections ───────────────────────────────────────────────────────────
#> Expected "[\n I got interpolated.\n |data|\n\n{{}}|#section|\n {{data}}\n |data|\n|/section|\n]\n" to equal "[\n I got interpolated.\n |data|\n\n {{data}}\n I gotypes = "tests" only finds a suite if the package was
installed with --install-tests. When it was not, that is
recorded as a skip with the reason — the absence of tests
is itself a finding for your risk file. On the library this vignette was
built against, 7 of 580 packages had an installed suite;
iq_record() records that per package as
installed_tests.
oq_record() reports only on the types you
asked for, and records the ask:
oq$session$types
#> [1] "examples" "tests"That matters more than it looks. If you take the default
("examples"), there will be no tests row
anywhere in the output and nothing else in the summary will tell you.
valpack will not invent a row for a type you did not
request — that would be reporting on something never attempted — so
$session$types is the field a reviewer reads to know the
scope.
oq_record() was run across the whole installed library
on 2026-08-02 (R 4.5.2, x86_64-pc-linux-gnu,
timeout = 300): 580 packages, examples
only, 14,554 pass / 140 fail / 8,765 skip, 120.1 minutes. Read
that narrowly — 37.4% of all units were skips, 32 packages had no
executed unit at all, and no test suite was run because the library was
not installed with --install-tests.
Those are the pre-fix numbers, kept as measured. All 140 failures were adjudicated on 2026-08-03 by re-executing every one: 92 (65.7%) were artefacts of this harness, 20 were the machine’s environment, 17 were one package hitting the operator’s timeout, and 7 (5.0%) were genuine package defects. The harness classes are fixed in this version, each with a regression test; on the five packages the adjudication named (Tplyr, mgcv, huxtable, Hmisc, psych) the fixes take 33 recorded failures to 0. No corrected full re-run has been published, so the 140 stands as the published figure.
Two limits no code in this package can close, and both belong in a
validation report: riskmetric is a hard
Imports of valpack and an examples-only OQ
qualifies it on zero executed units (108 topics, none with a
runnable example) — it is covered only by valpack’s own test suite; and
a pass is reachability, not correctness — no expected
output is compared. The package README carries the full scope statement
and the failure taxonomy.
json <- env_manifest(pkgs = c("whisker", "jsonlite"))
json
#>
#> ── valpack environment manifest ────────────────────────────────────────────────
#> ℹ R version 4.5.2 (2025-10-31) | x86_64-pc-linux-gnu | Linux 7.0.0-14-generic
#> ℹ 2 packages | 1167 bytes of JSON
#> ℹ Generated 2026-08-03 04:47:47 UTC by valpack 0.0.0.9002
cat(substr(unclass(json), 1, 400))
#> {
#> "manifest_version": 1,
#> "generated_by": "valpack 0.0.0.9002",
#> "r_version": "R version 4.5.2 (2025-10-31)",
#> "platform": "x86_64-pc-linux-gnu",
#> "os": "Linux 7.0.0-14-generic",
#> "libraries": ["/tmp/RtmpZtJJJ6/temp_libpathfb7902312fb71", "/usr/local/lib/R/site-library", "/usr/lib/R/site-library", "/usr/lib/R/library"],
#> "generated_at": "2026-08-03 04:47:47 UTC",
#> "packages_with_installedThere is deliberately no renv dependency. The manifest
describes what is on disk, which is what an auditor asks about; a
lockfile describes intent.
path <- file.path(tempdir(), "evidence")
files <- evidence_bundle(iq, oq, risk, path = path)
#> ✔ Evidence bundle written to /tmp/RtmpA62ICG/evidence
basename(files)
#> [1] "iq.html" "oq.html" "risk.html" "manifest.json"
#> [5] "index.html"index.html states the headline numbers and links the
other four. Plain HTML and JSON, no rendering toolchain, so it opens
from a file share on a locked-down validated desktop.