Generate the validation evidence for an R environment, on your own machine.
Open-source clinical R packages are free. The evidence that they are fit for regulated use is not — that is what Atorus OpenVal and Posit Package Manager’s validation tier are sold for. valpack produces the same class of artefact locally, from the library you actually have installed:
installed library -> IQ record + OQ record + risk assessment + manifest -> evidence bundle| Artefact | Function | What it proves |
|---|---|---|
| Risk assessment | assess_risk() |
Each package’s riskmetric score and every underlying metric |
| Installation qualification | iq_record() |
What is installed: version, library, built-under R, provenance, DESCRIPTION checksum |
| Operational qualification | oq_record() |
The package was exercised and here is what happened, per unit, with the real failure text |
| Environment manifest | env_manifest() |
A JSON snapshot another site can diff against theirs |
| Evidence bundle | evidence_bundle() |
All four plus a linked index.html for the reviewer |
Nothing is inferred and nothing is assumed to pass: every OQ row is the observed outcome of running real code in a clean R subprocess.
install.packages(
"https://clincoder.cloud/valpack/valpack_0.0.0.9001.tar.gz",
repos = NULL, type = "source"
)
library(valpack)
pkgs <- c("whisker", "jsonlite")
# What is installed, and is it still the file we qualified?
iq <- iq_record(pkgs)
iq$packages[, c("package", "version", "source", "description_md5")]
# Exercise it. `types = "tests"` also runs the suite when one is installed.
oq <- oq_record("whisker", types = c("examples", "tests"))
oq$summary
subset(oq$results, status == "fail")[, c("unit", "message")]
# Risk. The default assessments query CRAN; this subset is offline and fast.
quick <- riskmetric::all_assessments()[c(
"assess_has_news", "assess_has_vignettes", "assess_has_examples",
"assess_exported_namespace", "assess_has_maintainer"
)]
risk <- assess_risk(pkgs, assessments = quick)
risk[, c("package", "version", "score")]
# Hand the reviewer one directory.
evidence_bundle(iq, oq, risk, path = "evidence")
#> v Evidence bundle written to .../evidenceReal output from the run above on this machine (whisker 0.4.1, R 4.5.2):
# A tibble: 2 x 7
package type pass fail skip harness elapsed
<chr> <chr> <int> <int> <int> <int> <dbl>
1 whisker examples 4 0 5 0 0.02
2 whisker tests 5 2 0 0 2.74Those two failures are genuine: whisker 0.4.1 fails two mustache-spec test files, and oq$results$message carries the testthat diff that proves it. A validation tool that reported 7/7 there would be worthless.
oq_record() has been run across the whole installed library. That is the scale claim; here it is stated precisely, with the limits attached.
These are the numbers of the pre-fix 2026-08-02 run, kept as measured because they are cited elsewhere. All 140 of its failures were adjudicated on 2026-08-03 by re-executing every one of them: 92 (65.7%) were artefacts of valpack’s own harness, 20 were this machine’s environment, 17 were one package hitting the operator’s 300 s budget, and 7 (5.0%) were genuine package defects. The harness classes are fixed in this version — see “The harness was the largest defect” below — but no corrected full re-run has been published, so nothing here supersedes the figures in this table.
2026-08-02 23:25 UTC · R 4.5.2 · x86_64-pc-linux-gnu · Ubuntu 26.04 LTS · 580 packages · per-package timeout 300 s · 120.1 min of subprocess wall time.
| Units passed | 14,554 |
| Units failed | 140 (0.95% of the 14,694 executed) |
| Units skipped | 8,765 (37.4% of all 23,459 units) |
| Packages with at least one failing unit | 41 |
| Packages with zero executed units | 32 |
types = "examples"; there is not one tests row in it. Asking for tests would have changed little: only 7 of the 580 installed packages ship an installed tests/ directory, because the library was not installed with --install-tests. So “OQ across the library” means “every package’s help-page examples were executed” — materially weaker than it sounds. iq_record() now records installed_tests per package and env_manifest() reports the library-wide count, so the fact ships with the evidence instead of living in a README.\dontrun{}), or one that guards itself against an absent suggested package. A skip is honest, but it is not qualification.AsioHeaders, BH, clipr, devtools, extrafontdb, filehash, filelock, foghorn, fontBitstreamVera, fontLiberation, pharmaversesdtm, pharmaverseraw and 20 more — header-only, data-only and meta-packages. Full list in the bundle’s zero_coverage_packages.csv.riskmetric — valpack’s own hard Imports — was qualified on zero executed units. 108 topics: 63 have no \examples section at all and 45 are entirely \dontrun/\donttest. assess_risk() and every evidence_bundle() risk page run through riskmetric::pkg_ref() → pkg_assess() → pkg_score(), so the tool that produces this evidence has a hard dependency this OQ says nothing about. It is exercised by valpack’s own testthat suite (tests/testthat/test-risk.R, test-bundle.R), which is not part of any types = "examples" run. State that in the validation report; no code change here can close it.Adjudication of all 140 failures found 92 caused by the harness itself. valpack refuses to use tools::testInstalledPackage() because it reports false failures, and had then reproduced the same class of defect in its own runner. Each class below is fixed, and each fix has a regression test that fails without it (tests/testthat/test-oq.R).
| Class | False failures | Cause | Fix |
|---|---|---|---|
| H3 | 52 | No cleanEx() between topics, so one topic’s objects shadowed the next topic’s data (all 6 Tplyr, 13 mgcv, 7 huxtable) |
The global environment is emptied, attached packages detached, RNG reseeded, warn and the working directory restored between topics — what R CMD check does |
| H1 | 29 | An example rebound a base function the runner itself called (DHARMa) |
All harness state lives in an environment parented at baseenv(), and the driving loop runs inside it |
| H4 | 7 |
source() defaults print.eval to FALSE, so lazily-drawn objects were never drawn (Hmisc, lattice) |
print.eval = TRUE on a real pdf() device, as the REPL and R CMD check do |
| H7 | 2 |
progressr refuses to run while condition handlers are on the stack, and the isolating tryCatch is one |
Recorded harness, never fail
|
| H5 | 1 |
options(keep.source = TRUE) set inside a file source() has already parsed (utils::removeSource) |
keep.source = TRUE set before parsing |
| H6 | 1 | Example needs the REPL’s .Last.value (MASS::renumerate) |
Detected and refused as harness — no batch harness can close this one
|
| H2 | 17 | One slow robmixglm topic hit the 300 s per-package budget and condemned its 16 siblings |
Results are checkpointed after every unit; the timeout is recorded in $session$timeout; cut-short units are harness, never fail
|
| H8 | 14 | Packages phrasing their own missing-Suggests guard in wording the skip regex did not match |
Regex widened to the wrapped phrasings |
Measured on the packages the adjudication named, before and after, same machine, same day, timeout = 300:
| Package | fail before | fail after |
|---|---|---|
| Tplyr | 6 | 0 |
| mgcv | 13 | 0 |
| huxtable | 7 | 0 |
| Hmisc | 3 | 0 |
| psych | 4 | 0 |
| total | 33 | 0 |
Passes on those five went from 524 to 557; skips were unchanged at 173.
The other named packages, measured the same way (before = the 2026-08-03 re-run under the previous harness): DHARMa 5 → 0, lattice 2 → 0, latticeExtra 1 → 0, rsvg 1 → 0, utils 1 → 0, tidycmprsk 1 → 0 (now a skip), metafor 3 → 0 (now skips), MASS 1 → 0 fail + 1 harness, progressr 2 → 0 fail + 2 harness, and robmixglm 17 fail → 3 pass, 0 fail, 14 harness — the budget cut it short, which is not a property of the package and is no longer recorded as one.
status now has four values, and the fourth exists because a validation tool must never let “we could not run this” read as either:
| Status | Meaning |
|---|---|
pass |
Ran to completion without erroring |
fail |
Ran and errored — the only status that points at the package |
skip |
Nothing to run (\dontrun{}), or the example guarded itself against an absent suggested package, as CRAN does with _R_CHECK_FORCE_SUGGESTS_=false
|
harness |
The harness produced no verdict: refused (needs the REPL, or the package refuses condition handlers), cut short at timeout, or never reached because the subprocess ended early. message says which. Missing coverage, not a defect |
A bare failure count is not reviewable. evidence_bundle() groups every failing unit by the cause its message evidences:
| Cause | Failures | Packages |
|---|---|---|
| error in example | 78 | 29 |
| subprocess did not complete | 46 | 2 |
| missing suggested package | 13 | 9 |
| absent external resource | 2 | 2 |
| network | 1 | 1 |
subprocess did not complete (46) is missing coverage, not defect — the units were never judged, which is why such units now carry status harness rather than fail. 17 are robmixglm hitting the 300 s per-package timeout. The other 29 are DHARMa 0.4.7, and they were valpack’s own bug: a DHARMa example bound c in the global environment and the harness, which resolved c there too, died. Fixed in this version (the runner’s own names resolve from baseenv()) and covered by a regression test. A targeted re-run of DHARMa alone under the fix records 24 pass / 5 fail / 23 skip instead of 0 / 29 / 23. The 140 above is what the 2026-08-02 run measured and has not been restated — a corrected full re-run would report fewer.missing suggested package (13, measured 14 on adjudication) are false failures too. The subprocess turns “there is no package called X” into a skip, as CRAN does with _R_CHECK_FORCE_SUGGESTS_=false, but these packages phrase their own guard differently (“Please install the ggdist package…”, The package "aod" (>= 1.3.3) is required.) and were recorded as failures. Now reclassified: the regex covers the wrapped phrasings, with a regression test. The residual risk — a genuine defect whose message says “Please install X” — is real but smaller than booking 14 environment gaps as package failures.error in example (78) meant the example errored as run here — in Rd order, sharing one global environment. Adjudication re-ran every one: 52 of them were the missing cleanEx() and pass in isolation, 7 were the undrawn-plot bug, 20 were this machine’s environment, and 7 were genuine package defects (5 of them one cardx/cards version mismatch). The order-dependence is gone; the environment ones remain environment ones.Bundle: /root/evidence/valpack_full_oq/bundle/ — index.html, oq.html (taxonomy), failure_taxonomy.csv, zero_coverage_packages.csv. Re-publish with Rscript data-raw/publish_full_oq.R.
score is riskmetric’s roll-up on 0-1, higher is better. Scores from different assessment sets are not comparable, in either direction. Measured here (R 4.5.2, riskmetric 0.2.7): tibble scores 0.1480 on a five-metric subset and 0.3478 on all nineteen — but whisker scores higher on two metrics (0.5000) than on all nineteen (0.4600), so “a subset deflates the score” is not a safe rule either. The tibble carries an assessments attribute, print() shows it, and the bundle’s risk page states it, so a mismatch is detectable rather than a matter of memory.oq_record() reports only on the types it was asked for and records that request in $session$types. It does not emit a tests row when tests were never requested — reporting on something that was not attempted would be fabrication — so $session$types is what tells a reviewer the scope.valpack records NA rather than inventing one.renv dependency: the manifest describes what is on disk, not what a lockfile says should be.