Local embeddings versus subsets of a global embedding

Selecting points is not fitting them again

A restriction of a population embedding keeps the population coordinates for selected observations. A local embedding fits new coordinates using only distances between those selected observations. The rows can be identical while the optimization problem is different.

This distinction matters when inspecting a dominant-taxon neighborhood in VME. A population fit spends its three dimensions representing the whole dataset. Its arrangement of one neighborhood can reflect compromises needed to represent relationships with observations outside that neighborhood.

Here we compare the two views on a synthetic neighborhood with known geometry, then on the Li-maximal subset of the same 300 bundled compositions used in What does a 3D embedding preserve?. We use all three normalization/distance combinations from that article. No selection is made from the appearance of an embedding.

The comparison must use the same target pairs

Let $S$ be a fixed selection and $\delta_{ij}$ the chosen original-space target. The full population objective includes within-S, outside-S, and cross-boundary pairs. The local objective includes only pairs with both endpoints in S. For comparison, evaluate both coordinate sets on exactly those within-S pairs:

$E_S(Y)=100\sqrt{\frac{1}{\binom{|S|}{2}} \sum_{i<j,\ i,j\in S} \left(\frac{\|y_i-y_j\|_2-\delta_{ij}}{\delta_{ij}}\right)^2}.$

All targets here are positive. We use the same inverse-squared weighting for fitting and the same RMS relative-error definition for scoring. We do not compare the population-wide error directly with the local error as if they covered the same pairs. The restricted coordinates are not rescaled before evaluation. Each local result is evaluated at the scale returned by its solver.

For exact global minimizers, the best local objective cannot exceed the restricted population objective: the restricted coordinates themselves are a feasible local solution. A finite numerical run is not an exact global minimizer. We therefore report several starts and retain a fit initialized from the restriction as a useful numerical check.

A known planar neighborhood inside a larger population

The synthetic population has 96 observations in five dimensions. Twenty-four lie equally spaced on a radius-0.18 circle in the first two coordinates; the other 72 are a seeded five-dimensional Gaussian cloud shifted by two units in the third coordinate. These numbers define an illustrative geometry, not a microbiome model.

The selected circle admits an exact three-dimensional representation: use its first two coordinates and a zero third coordinate. The surrounding population does not generally admit an exact 3D representation of all its Euclidean distances. We fit the full population with three random starts, then refit the circle starting from its restricted population coordinates.

knitr::kable(data.frame(View=names(synthetic.score),RMS_percent=unname(synthetic.score)),
  digits=4,row.names=FALSE,col.names=c('Within-circle view','RMS relative error (%)'))

Within-circle view	RMS relative error (%)
Restriction	0.4832
Local refit	0.0065

The independently known planar coordinates have error below $10^{-10}$ percent. The restricted fit has 0.4832% error and the local refit has 0.0065%. These are measured outcomes under the stated budget, not an assertion that every local refit always improves. Both views below use the same coordinate limits and sphere radius.

Circle retained from the population fit

Circle: restriction of the 96-observation fit

Circle fitted locally

Circle: local refit of the same 24 observations

A fixed Li neighborhood in bundled microbiome data

We reuse the exact 300 distinct compositions and all 178 taxon coordinates from the fidelity article. The selection contains observations where Lactobacillus iners is a most abundant taxon, including exact dominance ties. This gives 97 compositions and 4,656 within-selection pairs. It is a rank-based Li neighborhood, not a merged reference-state union. The original valencia2k sampling design is described in the linf dataset guide.

For each target family, the population view is the best of the same three random starts used in the first article. Local fitting uses random-start seeds 11, 29 and 47 plus a restriction-initialized run with seed 59. Every run uses full-pair SGD, inverse-squared weights, 300 passes, and the Zheng schedule with schedule.epsilon=0.01, checking every ten passes. We report achieved results, not convergence certificates. The local result shown is the best of these four starts under the within-selection objective.

knitr::kable(comparison,digits=2,row.names=FALSE,
  col.names=c('Method','All population pairs (%)','Restriction (%)','Best local (%)','Warm start (%)'))

Method	All population pairs (%)	Restriction (%)	Best local (%)	Warm start (%)
L1 / Euclidean	12.91	9.93	8.93	8.96
Max / Euclidean	14.79	11.00	9.03	9.03
Max / L-infinity	19.66	15.16	11.34	11.34

In this example, local refitting lowers within-selection error for all three target families. The reductions are 1, 1.97, 3.82 percentage points, respectively. The size of the improvement depends on the selection and geometry; this example does not guarantee the same outcome elsewhere.

Only Restriction, Best local, and Warm start share the same pair set within a row. The population column is included to show why it is an inappropriate baseline for claiming local improvement. Columns are evaluated in each method’s own target geometry, not against one universal metric.

local.seed.table<-rbind(local.random$scores[,1:3],do.call(rbind,lapply(names(warm),function(m)
  data.frame(Method=m,Seed=59,RMS_percent=study_score(warm[[m]]$coords,local.targets[[m]])[1]))))
knitr::kable(local.seed.table,digits=2,row.names=FALSE,
  col.names=c('Method','Seed (59 starts from restriction)','Local RMS error (%)'))

Method	Seed (59 starts from restriction)	Local RMS error (%)
L1 / Euclidean	11	9.06
L1 / Euclidean	29	8.93
L1 / Euclidean	47	8.93
Max / Euclidean	11	9.07
Max / Euclidean	29	9.20
Max / Euclidean	47	9.13
Max / L-infinity	11	11.35
Max / L-infinity	29	11.37
Max / L-infinity	47	11.34
L1 / Euclidean	59	8.96
Max / Euclidean	59	9.03
Max / L-infinity	59	11.34

The restriction-initialized runs also document whether the numerical procedure can improve the existing coordinates without depending on a lucky random initialization. More epochs or different schedules would be separate experiments, not silently substituted results.

Inspect one comparison with consistent display choices

We illustrate max normalization with Euclidean distance, chosen as the middle case before comparing errors. The tables above cover all three methods. The population view highlights the selected Li neighborhood; the two following views contain exactly those selected observations.

Full 300-composition population fit; purple spheres identify the Li selection used below.

For the selected views, colors identify the most abundant non-Li taxon, with first-column tie handling. We translate and orthogonally align the local coordinates to the restriction, without changing scale or pair distances. Both selected views use identical limits, camera, colors and sphere radius. The population overview is framed separately because it shows more rows.

Restriction of the population embedding

Max / Euclidean: restriction of the population fit

Local embedding of the same selection

Max / Euclidean: independently fitted local coordinates

A displaced point between these two images has not moved biologically. The coordinate systems result from different optimization problems. Orthogonal alignment removes arbitrary rotation/reflection and translation; remaining differences reflect fitted deformation. A shared camera alone cannot align independently fitted coordinate systems.

What changes, and what should stay fixed?

A useful interface names the scope explicitly: population fit, restriction of a population fit, or local fit of the selection. Selection membership, normalization, metric, weighting and color definitions should be recoverable from the figure or its methods. Background points cannot be appended to a local fit using their old population coordinates as if they shared the same frame; placing them requires a defined alignment or out-of-sample procedure.

For longitudinal data, a trajectory within one jointly fitted coordinate system can be interpreted geometrically under its chosen targets. Connecting points from separately fitted visit or participant layouts would instead mix coordinate systems. Neither this example nor low local stress establishes a biological transition or participant-level uncertainty.

Reproduce this comparison

Run make local-global-vignette in a grip source checkout. The shared embedding-study.R contains target construction, fitting, independent scoring, cache identity and distance-preserving alignment. The renderer records selected rows, implementation fingerprints, settings, scores and session information. It reuses compatible population fits from the fidelity article. These are small, exact all-pairs teaching examples, not newly computed VME dataset views.