Test Constancy and Assess Logistic Trend-Filtering Prediction

Assess whether an order-2 logistic trend filter detects a nonconstant binary probability, and separately estimate predictive improvement over a constant. This is an additional assessment, not an option that changes the fitted objective or supplies confidence bands.

Usage

assess.logistic.trend.filtering(
  object,
  permutations = 999L,
  inner.folds = 5L,
  outer.folds = 5L,
  seed = 1L,
  predictive = TRUE,
  warn = TRUE,
  additional.tests = character()
)

# S3 method for class 'logistic.trend.filtering.assessment'
print(x, ...)

Arguments

object: A logistic.trend.filtering.fit with independent binary observations, unit weights, no covariates and no offset. Predictor choice, penalty path, ridge penalties and solver settings must have been specified independently of the outcomes. Backend and explicit penalties are reused.
permutations: Number of random outcome permutations, at least one. The default 999 gives a minimum possible p-value of 0.001. Small values in examples illustrate usage only.
inner.folds: Number of balanced, outcome-independent tuning folds. Used for the global test and within each outer training set.
outer.folds: Number of outer folds for predictive assessment.
seed: Nonnegative integer random seed. The caller's RNG state is restored. Realized folds, permutation seeds and RNG kind are returned.
predictive: Whether to run nested CV in addition to the global test.
warn: Warn if the observed fit, any permutation, or an outer fit fails.
additional.tests: Character vector selecting optional permutation tests: "absolute.deviation", "total.variation", or both. Default character() tests likelihood improvement only. Both curve summaries are always returned for the observed fit. Specify the tests before inspecting outcomes.
x: A logistic.trend.filtering.assessment to print.
...: Reserved for generic compatibility.

Value

A logistic.trend.filtering.assessment list containing statistic, p.value, p.value.bounds, status, null.probability, the newly tuned observed fit, permutation statistics and diagnostic records, optional nested-CV prediction, realized folds, seeds and solver settings. curve.summaries is a named vector with absolute.deviation and total.variation; curve.summary.details records their conventions and any calculation failure. tests has one row per requested test, including likelihood, with statistics, status, completed/failed permutation counts, exceedances, comparison tolerances, raw p-values, lower/upper failure bounds and p.value.holm. permutation.statistics contains a replicate column and one column per tested statistic. The original permutations table retains its likelihood-only schema. prediction$scores contains model losses; prediction$improvement contains baseline-minus-smoother gains; prediction$predictions and prediction$diagnostics retain row-level and fold-level evidence.

Details

The null is a common unknown Bernoulli probability at all observed predictor locations. The constant estimate is the sample event rate, not a uniformly weighted integral over the continuous predictor domain. Independent observations must be exchangeable under this null. Repeated participants, serial dependence, confounding adjustment and grouped binomial counts require other null procedures and are not supported here. Independence cannot be established from a fitted object.

The input object's selected penalty and fold labels are not reused: those labels may depend on outcomes. A new outcome-independent fold assignment selects a penalty from the supplied path, including a full-data refit. Every permutation repeats this complete selection procedure on the same folds with the same solver settings. The statistic is the nonnegative part of twice the ordinary binomial log-likelihood improvement over the constant. It is not a penalized-objective difference and is not compared with a chi-squared distribution. Lambda order determines selection ties. Changing the path after inspecting the assessment invalidates the stated test unless that additional selection is included in the randomization.

Absolute deviation is mean(abs(p.hat(x) - mean(y))), evaluated at all observed predictor locations, including repeats. It averages over sampled observations, not uniformly over the predictor domain. A value of 0.04 means an average absolute departure of four percentage points. Total variation is the sum of absolute probability changes along the fitted curve over [min(x), max(x)]. It includes every knot and interior turning point of the piecewise quadratic logit, up to floating-point precision; it is not a coarse-grid approximation or the trend-filtering penalty. For a monotone curve it equals the endpoint probability difference in absolute value; an oscillating curve can have total variation above one. These are descriptive summaries of the newly tuned fit, not uncertainty intervals or bias-corrected population effect estimates.

Optional tests use the same selected fits, folds and permutations as the likelihood test, with large values indicating departure from constancy. No additional model fits or random draws are needed. Each statistic has its own inclusive comparison tolerance and failure bounds. tests reports individual p-values and Holm-adjusted p-values across likelihood and the requested additional tests. Holm adjustment remains valid under dependence; it does not adjust across other predictors or outcomes. If any requested test is incomplete, all Holm values are unavailable. Choosing the smallest unadjusted p-value after inspecting several tests does not preserve the stated test level. The legacy scalar fields always describe the unadjusted likelihood test, regardless of additional.tests.

Outcomes are shuffled uniformly with the predictor fixed, preserving the total event count. With B successful permutations the Monte Carlo p-value is (1 + sum(permuted.statistic >= observed.statistic))/(B + 1). Under exchangeability this is a valid conditional randomization test, subject to Monte Carlo discreteness. For numerical ties, comparisons allow a conservative tolerance of 1e-10 * max(1, abs(observed.statistic)). A failed candidate is ineligible under the fitter's usual CV rule. If no candidate is eligible, that replicate fails. Failed replicates are never dropped or retried: the point p-value is NA and p.value.bounds gives the smallest/largest possible Monte Carlo p-value counting each missing statistic as a nonexceedance/exceedance. These bounds are not a confidence interval. An observed-fit failure leaves both bounds unavailable.

Predictive assessment uses separate outer folds. Each outer training set inherits the input fit's cv.penalty.scale. Under "training_mass", its lambda grid and ridge penalties are multiplied by the outer/full sample-size ratio, and its inner folds apply their own training ratio. Observed and permuted full-data refits use the same unscaled reference grid. Older saved fits without this field retain "none" for reproduction. Each outer training set tunes the path using only its inner folds and predicts its outer test set. The constant predictor is that outer training set's event rate. Both use identical test observations. Pooled mean natural-log loss and Brier score are reported; improvements are constant loss minus trend-filtering loss, so positive values favor the smoother. These assess training on the outer training-set size, not the full sample. No standard error is computed from overlapping CV folds. Incomplete smoother predictions leave its overall scores and improvements unavailable rather than scoring a selected subset.

Computation is sequential. A complete run uses approximately (permutations + 1) * (inner.folds + 1) native path fits for the test, plus outer.folds * (inner.folds + 1) for prediction. Per-replicate diagnostics are retained, but full permuted fitted objects are not. Rejecting constancy is evidence of association, not necessarily nonlinearity, predictive usefulness or a causal effect.

References

Ojala and Garriga (2010). Permutation Tests for Studying Classifier Performance. https://www.jmlr.org/papers/v11/ojala10a.html

Phipson and Smyth (2010). Permutation p-values should never be zero. doi:10.2202/1544-6115.1585

Examples