All 140 failures of the 2026-08-02 full-library run were adjudicated on 2026-08-03 by re-executing every one of them, several under four harness configurations. 92 (65.7%) were artefacts of this harness and not of any package. 20 were the machine’s environment, 17 were one package hitting the operator’s 300 s budget, and 7 (5.0%) were genuine package defects (4 more reproducible but undetermined). This package refuses to use tools::testInstalledPackage() precisely because it reports false failures, and had reproduced the same defect in its own runner. Every fix below has a regression test in tests/testthat/test-oq.R that fails without it.
Tplyr failures were four earlier topics assigning to mtcars; 13 mgcv and 7 huxtable failures were the same shape. oq_record() now performs the cleanEx() reset R CMD check performs between topics: the global environment is emptied, anything an example attached is detached, the RNG is reset (RNGkind("default", ...), set.seed(1)), options(warn = 1) is restored and the working directory is put back. OQ results for a multi-topic package are no longer order-dependent.
source() defaults print.eval to echo, i.e. FALSE, so a trellis or ggplot object was evaluated but never rendered and topics that needed the drawing to have happened failed (Hmisc’s Key(), lattice’s trellis.focus()). Now print.eval = TRUE on a real pdf() device, as R CMD check uses — not pdf(NULL).options(keep.source = TRUE) inside an example came too late — 1 false failure. source() parses the whole file before evaluating any of it; R CMD check feeds the script to the REPL, which parses one expression at a time. The driver now sets keep.source = TRUE before parsing (utils::removeSource).Suggests guards in non-standard wording — 14 false failures. The auto-skip regex matched only R’s own “there is no package called X”. It now also matches the wrapped phrasings that 10 packages use (“Please install the ‘BiasedUrn’ package…”, The package "aod" (>= 1.3.3) is required.), which is how CRAN treats them under _R_CHECK_FORCE_SUGGESTS_=false.robmixglm siblings condemned by one slow topic.baseenv() in the general case, not just for c. All harness state now lives in an environment parented at baseenv() and the driving loop runs inside it, so no name an example binds in the global environment can reach any part of the runner.status gains harness: the harness produced no verdict for the unit. It is never a pass and never a package failure, and message says which of four things happened — the example needs the console REPL’s .Last.value; the package refuses to run while the isolating tryCatch has condition handlers on the stack (progressr); the unit was still running when the subprocess hit timeout; or the subprocess ended before the unit was reached. summary gains a harness column, print() reports it, and evidence_bundle() states the count and what it means on both the OQ page and the index.
$session$timeout records the per-package budget the run used, print() and the bundle show it, and data-raw/full_library_oq.R reads VALPACK_OQ_TIMEOUT (default 300) and writes it into full_oq.rds. A budget that is not in the evidence cannot be adjudicated.
riskmetric 0.2.7 is a hard Imports of valpack and the full-library run qualified it on zero executed units — 108 topics, 63 with no \examples section and 45 entirely \dontrun/\donttest. assess_risk() and the bundle’s risk page run through it. The tool producing this evidence therefore has an unqualified hard dependency. It is covered by valpack’s own testthat suite, which is not part of any types = "examples" run. No code change here can close this; it must be stated in the validation report, and it now is — in ?oq_record, the README, the vignette and the bundle index.source()-based harness. An example that reads .Last.value (MASS::renumerate) is detected and recorded harness, not fail. This is a refusal, not a fix.tryCatch (progressr, 2 units). Recorded harness. Closing it would mean one subprocess per topic.Suggests regex carries a residual risk: a genuine defect whose message happens to read “Please install the X package” would be recorded as a skip. Judged smaller than booking 14 environment gaps as package failures, but it is a real trade and it is a deliberate one.oq_record() has now been run across the whole installed library — the thing v1 said had never happened. 2026-08-02 23:25 UTC, R 4.5.2, x86_64-pc-linux-gnu, 580 packages, 120.1 min: 14,554 pass / 140 fail / 8,765 skip. The claim is documented in ?oq_record, the README “Scope” section, and the published bundle — not only in HANDOVER.md.
What that run does not cover, stated in all three places:
types = "examples"; results.csv contains no tests row at all. That is not an accident of recording: only 7 of the 580 installed packages ship an installed tests/ directory, because the library was not installed with --install-tests.oq_record()’s subprocess driver evaluated its own helper code with the global environment on the search path, so an example that bound a name like c broke the harness and every remaining unit of that package was reported as a failure. DHARMa 0.4.7 did exactly that in the full-library run and produced 29 false failures — 21% of all 140. The runner’s free variables now resolve from baseenv(). A targeted re-run of DHARMa under the fix records 24 pass / 5 fail / 23 skip instead of 0 / 29 / 23. The published 2026-08-02 numbers are not restated; they are what was measured.oq_record() records the requested evidence types in $session$types, and print() and evidence_bundle() state them. A run that collected examples only now says so on the face of the evidence. It still does not emit a tests row for a type that was never requested — that would be reporting on something not attempted.iq_record() gains an installed_tests column: whether each package was installed with --install-tests. env_manifest() reports the library-wide count as packages_with_installed_tests. That single fact bounds what any OQ over the library can claim.evidence_bundle() writes a failure taxonomy — every failing unit grouped by the cause its message evidences (missing suggested package, absent external resource, network, subprocess did not complete, error in example). Classification is conservative: anything ambiguous stays in error in example, the bucket that reflects worst on the library. The index also states the skip share, the count of packages with no executed unit, and the --install-tests count.assess_risk() attaches an assessments attribute naming the set used, and print() and the bundle’s risk page state it, so a subset score cannot be compared to a full score by accident. Asserted in a test, not only in prose.data-raw/publish_full_oq.R publishes the stored run as a bundle without re-running it.cAIC4 required for this function to work.”), so they are recorded as failures rather than skips. Not fixed: widening the regex risks reclassifying a genuine defect as a skip, and changing the rule would make later numbers incomparable to the published run.error in example (78 failures) is not adjudicated. It means the example errored as run — in Rd order, sharing one global environment, as R CMD check does — not that the package is defective. Deciding that needs a re-run of the single topic in isolation, which valpack does not do.robmixglm) hitting the 300 s timeout. Those units were never judged.tools::testInstalledPackage() remains deliberately unused: it runs test files without attaching the package under test and reports false failures.Initial development version.
assess_risk() — per-package risk assessment via the riskmetric pipeline (pkg_ref() -> pkg_assess() -> pkg_score()), returned as a tidy tibble with one column per metric. Packages riskmetric cannot reference get an NA row and the reason, instead of aborting the run.iq_record() — installation qualification: version, library path, R version built under, provenance and a DESCRIPTION MD5 for every installed package, plus session context.oq_record() — operational qualification: runs each help topic’s examples, and the package’s installed test suite when there is one, in a clean R subprocess with the package attached, and records the observed pass / fail / skip with the real failure text.env_manifest() — the installed environment as JSON, no renv dependency.evidence_bundle() — the four artefacts plus a linked index.html.print() methods for valpack_risk, valpack_iq, valpack_oq and valpack_manifest.