oq_record.RdExercises installed packages and records what actually happened. Nothing is assumed to pass: every result in the returned table is the observed outcome of running real code.
Character vector of installed package names.
Library paths to search.
Which evidence to collect: any of "examples", "tests".
Defaults to "examples" only, because installed test suites can run for
minutes.
Optional integer cap on the number of help topics exercised
per package. NULL (default) runs all of them.
Seconds allowed per package subprocess before it is killed.
The budget is an operator setting, not a property of the package, so it is
recorded in $session$timeout and the units it cut short are recorded
harness, never fail. Results are checkpointed after every unit, so a
kill costs the unit that was running and the ones after it, not the whole
package.
A valpack_oq object.
Ignored.
An object of class valpack_oq: a list with results (a tibble of
package, type, unit, status, message, elapsed), summary (per
package pass/fail/skip/harness counts) and session, whose types and
timeout elements record what was requested and under what budget.
Two kinds of evidence are collected:
"examples" — the examples shipped in the package's own help pages are
extracted from the installed Rd database (tools::Rd_db(), tools::Rd2ex())
and each topic is run, one at a time, in a separate R process with the
package attached, exactly as R CMD check would — including the
cleanEx() reset between topics (see below).
"tests" — the package's own test suite, copied out of the installed
library and run file by file in the same way. This only exists when the
package was installed with --install-tests; when it was not, that is
recorded as a skip, never as a pass.
A validation tool must never let "we could not run this" read as a pass or as a package failure. Every unit therefore carries one of four statuses:
pass — the unit ran to completion without erroring. It says the code
path is reachable; it says nothing about whether the numbers were right.
fail — the unit ran and errored, with the real error text in
message. This is the only status that points at the package.
skip — there was nothing to run, or the unit refused to run for a
reason outside the package: no runnable example code (everything inside
\dontrun{}), or an example guarding itself against an absent suggested
package, which CRAN also treats as a skip via _R_CHECK_FORCE_SUGGESTS_=false.
harness — the harness could not produce a verdict. The unit was
refused, cut short, or never reached, and message says which. Four things
land here: an example needing the console REPL's .Last.value; a package
that refuses to run while the harness's tryCatch has condition handlers
on the stack (progressr); a unit still running when the subprocess hit
timeout; and units the subprocess never reached because it ended early.
A harness row is missing coverage, and it is neither a pass nor a defect.
R CMD check concatenates a package's examples into one script and calls
cleanEx() between topics. oq_record() does the same: the global
environment is emptied, anything an example attached is detached, the RNG is
reset (RNGkind("default", ...), set.seed(1)), options(warn = 1) is
restored and the working directory is put back. Without that reset one
topic's objects shadow a dataset of the same name in the next — the largest
single source of false failures in the 2026-08-02 run (52 of 140).
tools::testInstalledPackage() is deliberately not used: it runs test
files under --vanilla without attaching the package, so suites that rely on
R CMD check having attached it report false failures, and it stops at the
first failing file instead of reporting the rest.
This is the R analogue of re-running a validated SAS macro's own test suite during OQ and filing the log.
types records itself in $session$types, and the evidence bundle prints
it, because "0 failures" means nothing until you know what was attempted.
The default is "examples" only: if you never ask for "tests" there is no
tests row anywhere in the output, and the absence is otherwise invisible.
oq_record() deliberately does not emit a tests row for a type the
caller never requested — that would be reporting on something that was not
attempted — so $session$types is the field that closes the gap. When
"tests" is requested and the package has no installed suite, a skip row
with the reason is emitted; it is never a pass.
oq_record() was run across the whole installed library on 2026-08-02
(R 4.5.2, x86_64-pc-linux-gnu), with timeout = 300: 580 packages,
examples only, 14,554 pass / 140 fail / 8,765 skip, 120.1 minutes of
subprocess wall time. That is the pre-fix run and its numbers are cited
elsewhere; they are not restated here.
All 140 of its failures were adjudicated on 2026-08-03 by re-executing every
one of them. 92 (65.7%) were artefacts of this harness, not of any
package; 20 were this machine's environment, 17 were one package hitting
the operator's 300 s budget, and 7 (5.0%) were genuine package defects,
with 4 reproducible but undetermined. The harness classes that produced the
92 are fixed in this version and each carries a regression test; see NEWS.md.
A corrected full re-run has not been published, so no number here supersedes
the 2026-08-02 figures.
Read the original claim narrowly:
It qualifies examples, not test suites. The run used the default
types = "examples". Asking for "tests" would not have helped much:
only 7 of the 580 installed packages ship an installed tests/ directory,
because the library was not installed with --install-tests.
37.4% of all units were skips — an example with no runnable code, or one that guards itself against an absent suggested package.
32 of the 580 packages had zero executed units, so nothing about them
was qualified at all (AsioHeaders, BH, clipr, devtools,
pharmaversesdtm, pharmaverseraw among them — header-only, data-only
and meta-packages with no runnable examples).
The published bundle for that run, including the failure taxonomy, is described in the package README.
riskmetric is a hard Imports of valpack and the 2026-08-02 run
qualified it on zero executed units — 108 topics, 63 with no \examples
section and 45 whose examples are entirely \dontrun/\donttest. The tool
that produces this evidence therefore has an unqualified hard dependency.
It is covered by valpack's own testthat suite, which is not part of
any types = "examples" run. Say so in the validation report.
No source()-based harness can reproduce the console REPL. An example
that reads .Last.value is recorded harness, not fail.
A pass is reachability, not correctness. No expected output is compared; nothing is checked against a reference.
# `whisker` is tiny: a handful of help topics, runs in about a second.
oq <- oq_record("whisker", max_topics = 3)
#> ℹ Running examples for whisker
oq
#>
#> ── valpack operational qualification ───────────────────────────────────────────
#> ℹ R version 4.5.2 (2025-10-31) | x86_64-pc-linux-gnu
#> ℹ evidence requested: examples
#> ℹ per-package timeout: 600s
#> ✔ pass: 1
#> ✔ fail: 0
#> ℹ skip: 2 | harness (not judged): 0 | total elapsed 0s
#> # A tibble: 1 × 7
#> package type pass fail skip harness elapsed
#> <chr> <chr> <int> <int> <int> <int> <dbl>
#> 1 whisker examples 1 0 2 0 0.01
oq$summary
#> # A tibble: 1 × 7
#> package type pass fail skip harness elapsed
#> <chr> <chr> <int> <int> <int> <int> <dbl>
#> 1 whisker examples 1 0 2 0 0.01
subset(oq$results, status == "fail")$message
#> character(0)