Lab or Field? A Method-Comparison Study of Lighthouse and CrUX Measurements on the Turkish Web

1. Introduction

Web performance is measured from two basic sources: controlled, synthetic laboratory tests (e.g., Lighthouse) and field data collected from real users (e.g., the Chrome UX Report, CrUX). A laboratory test provides a reproducible diagnosis from a single throttled run, whereas field data report the 75th percentile (p75) of the real distribution of devices, networks and geographies. The two are related but not identical quantities.

In practice many teams base decisions on laboratory scores alone; yet how well the laboratory represents real experience is an empirical question. This study asks: on the Turkish web, are laboratory Lighthouse measurements a valid proxy for real-user field experience? We answer it by centring a single feature — Core Web Vitals — and applying method-comparison statistics.

2. Related Work

Core Web Vitals (LCP, FCP, CLS, INP) are established indicators of user experience; Google's guidance positions field data as the 'ground truth' and the laboratory as a diagnostic tool. To test the agreement of two measurement methods, the Bland & Altman (1986) approach (bias plus limits of agreement) from the clinical-measurement literature and, for categorical agreement, Cohen's (1960) kappa are standard; kappa is interpreted with the Landis & Koch (1977) benchmarks. The contribution of this study is to apply this method-comparison framework to lab–field CWV agreement across a large population of Turkish sites.

3. Method

The sample comprises 7,403 Turkish sites audited by the 1st.com.tr analysis engine that had both laboratory and CrUX field data (field data were available for 53.7% of all 13,775 audited sites). Laboratory values were taken from Lighthouse's default mobile-throttled run; field values from CrUX's 28-day p75 distribution. For each site, LCP, FCP (milliseconds) and CLS (unitless) laboratory–field pairs were matched.

An important distinction: the laboratory measures a single synthetic run, whereas the field measures the p75 of the real-user distribution. The comparison is therefore not 'do they measure the same thing' but 'is the laboratory a useful proxy for the field'.

4. Statistical Analysis

Three complementary methods were used. (1) Association: Pearson (p1–p99 winsorized to limit outlier influence) and distribution-free Spearman correlations, each with a 95% confidence interval (Fisher z) and two-tailed p. (2) Agreement: Bland-Altman analysis — bias (mean lab − field difference) and 95% limits of agreement (bias ± 1.96·SD); the median absolute difference was also reported as an outlier-robust measure. (3) Class agreement: using Core Web Vitals thresholds (LCP 2,500/4,000 ms; FCP 1,800/3,000 ms; CLS 0.10/0.25), each measurement was classified 'good/needs-improvement/poor' and Cohen's kappa computed (interpreted per Landis & Koch 1977). Because significance is near-guaranteed at large n, interpretation rests on effect/agreement magnitude.

5. Results — Laboratory and Field Values

The laboratory reported loading metrics strikingly higher (more pessimistic) than the field (Table 1). For LCP the laboratory median is roughly five times the field median; for FCP the gap is smaller but marked. For CLS the two sources gave the closest values.

6. Results — Correlation

The relationship between laboratory and field values is weak for loading metrics: Spearman ρ = 0.26 for LCP and ρ = 0.17 for FCP. Only layout stability (CLS) showed a moderate relationship (ρ = 0.41). The overall Lighthouse performance score correlated with field LCP and INP in the expected direction but weakly (r ≈ −0.25): a higher laboratory score is only loosely associated with better real-user values (Table 2).

7. Results — Bland-Altman Agreement

The Bland-Altman analysis reveals wide, unusable disagreement for loading metrics (Table 3). For LCP the laboratory exceeds the field by 12,254 ms on average (bias); the 95% limits of agreement span from −18.4 s to +42.9 s — so the true field LCP cannot, in practice, be inferred from a single laboratory LCP. Even the outlier-robust median absolute difference is 8,384 ms for LCP. For CLS the bias is small (+0.034) but the limits of agreement remain wide.

8. Results — Class Agreement (Cohen's Kappa)

For the Core Web Vitals 'good/needs-improvement/poor' classification, agreement between laboratory and field does not exceed chance for loading metrics (Table 4). For LCP, κ = 0.00: the laboratory classification predicts the field classification no better than chance. The most striking result is the asymmetry: 85.6% of sites fell into a worse class in the laboratory than in the field; while only 3.1% of sites appeared 'good' for LCP in the laboratory, 59.6% were 'good' in the field. FCP is similar (κ = 0.01). Only CLS shows moderate agreement (κ = 0.33; 72.8% of sites in the same class).

9. Discussion

The findings paint a consistent picture: on the Turkish web, default laboratory Lighthouse loading metrics are a poor proxy for real-user experience. The weak correlation (LCP ρ = 0.26), the unusable Bland-Altman limits of agreement, and the chance-level kappa (LCP κ = 0.00) all point the same way. The laboratory's systematic pessimism is explained by the default mobile-throttled run, cold cache, and the fact that a single run is not the p75.

Layout stability (CLS) is the exception: laboratory and field agree moderately (κ = 0.33), because CLS is largely a structural property of the document, independent of network conditions. The practical implication is clear: performance decisions and rankings should rest on field (CrUX) data; the laboratory should be used to diagnose regressions locally. Ranking the field by the laboratory's overall score (r ≈ −0.25) would mis-position most sites.

10. Limitations

(i) The laboratory measures a single synthetic run while the field measures the p75; this difference in estimand naturally explains part of the observed disagreement — but from a practitioner's standpoint the question remains whether the laboratory is a useful proxy, so the finding stands. (ii) Field data are available only for the higher-traffic 53.7% subset (selection bias). (iii) The laboratory used the default configuration; different throttling settings may yield different bias. (iv) The study is observational and cross-sectional. (v) Because of outliers, the primary interpretation rests on robust measures (Spearman, median difference, kappa).

11. Ethics and Data Use

All measurements rest on automated audits of public pages and on public CrUX aggregates; no personal data are collected and results are aggregated. The study is published open-access under CC BY 4.0; data are shared free for academic purposes on request.

12. Conclusion

Across 7,403 Turkish sites, agreement between laboratory and field Core Web Vitals declines from weak to negligible for loading metrics and rises only to moderate for layout stability. Laboratory loading scores are not a reliable proxy for real-user experience and are systematically pessimistic. We recommend practitioners leave the decision to the field and the diagnosis to the laboratory. Future work will examine how different throttling configurations and device classes affect agreement.