Oak Grove Analysis

Overview

What was done

  • We compared students taught by trained teachers versus untrained teachers.
  • For CAASPP, we examined math achievement using prior-year CAASPP scale score as a baseline adjustment where available (ANCOVA).
  • For i-Ready, we examined student within-year growth and prioritized diagnostic gain for the headline summary.
  • Because few schools contained both trained and untrained exposure in the analytic samples, school fixed effects are not emphasized in the main results.

Analysis map

Research question Panel definition Outcomes Controls
Do students taught by trained teachers show better math achievement than students taught by untrained teachers? 2022–23 CAASPP math compared for trained vs untrained; adjusts for prior-year (2021–22) CAASPP math scale score. CAASPP Math scale score (2022–23) with 2021–22 as baseline (ANCOVA) Prior-year CAASPP scale score; Grade; Sex (if available); Hispanic/Latino; English learner; Special education; Economic disadvantage; Reporting ethnicity indicators; Teacher gender; Teacher experience; Baseline overall scale score
In 2023–24, is achievement higher for students taught that year by a teacher trained in 2022–23? 2023–24 CAASPP math compared for current-exposure trained vs untrained; main spec is cross-sectional (prior-year CAASPP available for context). CAASPP Math scale score (2023–24) Grade; Hispanic/Latino; English learner; Special education; Economic disadvantage; Reporting ethnicity indicators, Teacher gender; Teacher experience; Baseline overall scale score
In 2023–24, is achievement higher for students who had a trained teacher in 2022–23 (carryover)? 2023–24 CAASPP math compared for delayed-exposure trained vs untrained; main spec is cross-sectional (prior-year CAASPP available for context). CAASPP Math scale score (2023–24) Grade; Hispanic/Latino; English learner; Special education; Economic disadvantage; Reporting ethnicity indicators; Teacher gender; Teacher experience; Baseline overall scale score
Do students taught by trained teachers show more growth in math ability than students taught by untrained teachers? Within 2022–23, compares students taught by trained vs untrained teachers; growth uses baseline vs most-recent diagnostic within the year. Diagnostic gain; Annual Typical Growth; Annual Stretch Growth; % progress to Typical; % progress to Stretch; Mid-on-grade-level; Domain (Number & Operations, Algebra, Measurement & Data, Geometry) Grade group; Sex; Race; Hispanic/Latino; English learner; Special education; Teacher gender; Teacher experience; Baseline overall scale score (i-Ready)
In 2023–24, does being taught that year by a teacher trained in 2022–23 relate to greater growth? Within 2023–24, compares students taught in 2023–24 by previously-trained teachers vs students taught by untrained teachers; growth uses baseline vs most-recent YTD diagnostic. Diagnostic gain; Annual Typical Growth; Annual Stretch Growth; % progress to Typical; % progress to Stretch; Mid-on-grade-level; Domain (Number & Operations, Algebra, Measurement & Data, Geometry) Grade; Sex; Race; Hispanic/Latino; English learner; Special education; Teacher gender; Teacher experience; Baseline overall scale score (i-Ready)
In 2023–24, do students previously taught by a trained teacher in 2022–23 show greater growth (carryover effect)? Within 2023–24, compares students who had trained teachers in 2022–23 vs students who did not; growth uses baseline vs most-recent YTD diagnostic in 2023–24. Diagnostic gain; Annual Typical Growth; Annual Stretch Growth; % progress to Typical; % progress to Stretch; Mid-on-grade-level; Domain (Number & Operations, Algebra, Measurement & Data, Geometry) Grade; Sex; Race; Hispanic/Latino; English learner; Special education; Teacher gender; Teacher experience; Baseline overall scale score (i-Ready)

Primary results

Analysis Estimate CI low CI high p N
CAASPP 2022-23 Math Achievement (exposure: taught by teacher trained in 2022-23) 12.619 -8.412 33.650 0.2570 1670
CAASPP 2023-24 Math Achievement (exposure: taught in 2023–24 by a teacher who had been trained in 2022–23) 0.930 -37.418 39.279 0.9630 1513
CAASPP 2023-24 Math Achievement (exposure: taught by a trained teacher in 2022-23) -1.528 -21.059 18.003 0.8800 3105
i-Ready 2022-23 student within year growth (exposure: taught by teacher trained in 2022-23) -3.390 -7.144 0.364 0.0786 3771
i-Ready 2023-24 student within year growth (exposure: taught in 2023–24 by a teacher who had been trained in 2022–23) -2.197 -6.894 2.501 0.3620 2376
i-Ready 2023-24 student within year growth (exposure: taught by a trained teacher in 2022-23) 3.327 -2.909 9.563 0.2980 2540

Main takeaways

  • For CAASPP 2022–23 math, students taught by trained teachers scored higher by 12.619 points on average than those with untrained teachers (estimate = 12.619, 95% CI = −8.412 to 33.650, p = 0.2568, N = 1670).
  • For CAASPP 2023–24 math (taught in 2023–24 by a teacher who had been trained in 2022–23), students taught by trained teachers scored 0.931 points higher on average (estimate = 0.931, 95% CI = −37.418 to 39.279, p = 0.9627, N = 1513).
  • For CAASPP 2023–24 math (taught by a teacher trained in 2022–23), students taught by trained teachers scored 1.528 points lower on average (estimate = −1.528, 95% CI = −21.059 to 18.003, p = 0.88, N = 3105).
  • For i‑Ready 2022–23 within‑year growth, students taught by trained teachers showed 3.390 points lower growth on average (estimate = −3.390, 95% CI = −7.144 to 0.364, p = 0.0786, N = 3771).
  • For i‑Ready 2023–24 within‑year growth (taught in 2023–24 by a teacher who had been trained in 2022–23), students taught by trained teachers showed 2.197 points lower growth on average (estimate = −2.197, 95% CI = −6.894 to 2.501, p = 0.3618, N = 2376).
  • For i‑Ready 2023–24 within‑year growth (taught by a teacher trained in 2022–23), students taught by trained teachers showed 3.327 points higher growth on average (estimate = 3.327, 95% CI = −2.909 to 9.563, p = 0.2978, N = 2540).

Across these six comparisons, the point estimates go in both directions and all 95% confidence intervals include zero, and none of the p-values are below 0.05, so there is no clear evidence of a consistent effect of having a trained teacher on these outcomes. Effect sizes are generally small relative to the width of the confidence intervals, and sample sizes range from about 1,513 to 3,771, so estimates are imprecise. Future work could focus on increasing precision or examining subgroups or implementation fidelity to explain the mixed patterns.

Results by analysis run

Achievement (CAASPP)

A1. CAASPP 2022-23 Math Achievement (exposure: taught by teacher trained in 2022-23)

This analysis compared students whose 2022–23 teacher was trained in 2022–23 versus untrained teachers. The outcome is the CAASPP Math scale score; 136 students were in the trained group and 2,662 in the untrained group. The ANCOVA estimate (trained minus untrained) was +12.62 points (95% CI −8.41 to 33.65), p = 0.2568, indicating substantial uncertainty. Standard errors in the primary specification were clustered by school.

Full primary model results

Term Estimate Std. Error CI low CI high p
Intercept 490.988 45.743 401.332 580.645 <0.001
Trained teacher (current) 12.619 10.730 -8.412 33.650 0.25700
Prior-year score 0.799 0.020 0.760 0.838 <0.001
Grade: 4 24.343 5.930 12.721 35.965 <0.001
Grade: 5 13.475 5.877 1.956 24.995 0.03570
Student sex: Male 2.798 2.427 -1.959 7.556 0.26600
Hispanic/Latino: Yes 13.166 12.055 -10.461 36.793 0.29100
English learner: Yes -8.116 3.730 -15.428 -0.805 0.04490
Special education: Yes 1.239 8.184 -14.801 17.278 0.88200
Economic disadvantage: Yes -5.564 2.890 -11.228 0.100 0.07220
American Indian/Alaska Native 14.810 12.564 -9.814 39.435 0.25600
Asian 39.812 12.621 15.074 64.549 0.00614
Black/African American -5.001 16.128 -36.612 26.611 0.76100
Filipino 26.813 12.120 3.058 50.568 0.04180
Native Hawaiian/Pacific Islander 26.797 12.306 2.676 50.917 0.04480
Two or more / not reported 30.992 14.275 3.012 58.972 0.04530

Note: N = 1,670; R² = 0.732; Adj. R² = 0.730; Residual df = NA

A2. CAASPP 2023-24 Math Achievement (exposure: taught in 2023–24 by a teacher who had been trained in 2022–23)

This analysis compares 2023–24 CAASPP math scores for students taught in 2023–24 by a teacher who had been trained in 2022–23 versus students taught that year by a teacher who had not been trained. The outcome is the CAASPP Math scale score. There were 98 students in the trained group and 1,417 students in the untrained group. The estimated difference (trained − untrained) is 0.9305, with a 95% confidence interval of −37.4178 to 39.2788 (p = 0.9627), indicating a very small point estimate with wide uncertainty; standard errors are clustered by school.

Full primary model results

Term Estimate Std. Error CI low CI high p
Intercept 2414.236 14.776 2385.275 2443.198 <0.001
Trained teacher (current) 0.930 19.565 -37.418 39.279 0.96300
Grade: 4 46.238 11.134 24.416 68.060 <0.001
Grade: 5 71.345 8.195 55.283 87.406 <0.001
Grade: 6 87.148 15.579 56.613 117.683 <0.001
Hispanic/Latino: Yes 21.847 11.018 0.252 43.442 0.06600
English learner: Yes -65.628 5.764 -76.925 -54.331 <0.001
Special education: Yes -81.599 9.447 -100.114 -63.084 <0.001
Economic disadvantage: Yes -25.936 4.891 -35.522 -16.351 <0.001
American Indian/Alaska Native -19.633 18.525 -55.943 16.677 0.30600
Asian 103.522 13.405 77.248 129.796 <0.001
Black/African American 41.535 46.277 -49.168 132.238 0.38400
Filipino 57.252 15.083 27.689 86.816 0.00176
Native Hawaiian/Pacific Islander 63.542 11.999 40.024 87.060 <0.001
Two or more / not reported 64.367 15.720 33.555 95.179 <0.001

Note: N = 1,513; R² = 0.421; Adj. R² = 0.415; Residual df = NA

A3. CAASPP 2023-24 Math Achievement (exposure: taught by a trained teacher in 2022-23)

This analysis compares 2023–24 CAASPP math outcomes for students who had a trained teacher in 2022–23 versus those who did not. The outcome is the CAASPP Math scale score; 175 students were in the trained group and 2,932 in the untrained group. The estimated difference (trained minus untrained) is -1.53 (95% CI: -21.06 to 18.00), p = 0.88, based on n = 3,105 in the modeled sample, indicating a small negative point estimate with wide uncertainty. Standard errors in the primary specification are clustered by school.

Full primary model results

Term Estimate Std. Error CI low CI high p
Intercept 2415.924 13.586 2389.296 2442.553 <0.001
Trained teacher (current) -1.528 9.965 -21.059 18.003 0.88000
Grade: 4 46.932 6.443 34.304 59.559 <0.001
Grade: 5 64.477 7.159 50.446 78.508 <0.001
Grade: 6 87.682 7.842 72.311 103.053 <0.001
Hispanic/Latino: Yes 17.879 11.865 -5.377 41.135 0.15100
English learner: Yes -56.578 5.730 -67.810 -45.346 <0.001
Special education: Yes -87.201 11.031 -108.822 -65.580 <0.001
Economic disadvantage: Yes -29.362 3.831 -36.870 -21.854 <0.001
American Indian/Alaska Native 79.576 43.463 -5.611 164.763 0.08580
Asian 104.528 12.655 79.723 129.332 <0.001
Black/African American -9.413 17.102 -42.932 24.106 0.59000
Filipino 54.922 14.217 27.056 82.788 0.00138
Native Hawaiian/Pacific Islander 65.494 12.364 41.260 89.728 <0.001
Two or more / not reported 76.911 14.180 49.118 104.704 <0.001

Note: N = 3,105; R² = 0.417; Adj. R² = 0.414; Residual df = NA

Growth (i-Ready)

G1. i-Ready 2022-23 student within year growth (exposure: taught by teacher trained in 2022-23)

This analysis compares, within 2022–23, students whose 2022–23 teacher was trained to students whose 2022–23 teacher was untrained, with growth measured from baseline to the most recent i-Ready diagnostic within the year. The outcome is Diagnostic gain (i-Ready); 199 students were in the trained-teacher group and 3,573 in the untrained-teacher group. The estimated difference in diagnostic gain (trained vs untrained) is −3.3902 (95% CI −7.1444 to 0.3641; p = 0.0786), with standard errors clustered by teacher.

Full primary model results

Term Estimate Std. Error CI low CI high p
Intercept 72.903 9.083 55.100 90.706 <0.001
Trained teacher (current) -3.390 1.915 -7.144 0.364 0.07860
Grade group: 3 0.125 1.528 -2.869 3.119 0.93500
Grade group: 4–5 -0.707 1.495 -3.637 2.224 0.63700
Student sex: male 1.255 0.516 0.243 2.267 0.01610
Student sex: Unknown 6.489 1.344 3.855 9.123 <0.001
Student race: Asian -0.297 6.341 -12.726 12.131 0.96300
Student race: Black or African American -5.269 6.633 -18.270 7.733 0.42800
Student race: Native Hawaiian or Other Pacific Islander -3.748 7.196 -17.853 10.357 0.60300
Student race: Two or More Races -1.487 6.456 -14.141 11.167 0.81800
Student race: White -2.899 6.375 -15.394 9.596 0.65000
Student race: Unknown -5.891 6.343 -18.324 6.542 0.35400
English learner: Y -0.967 0.791 -2.517 0.584 0.22400
Special education: Y -2.891 1.085 -5.018 -0.763 0.00850
Teacher gender: Male -1.019 2.256 -5.441 3.403 0.65200
Teacher gender: Unknown -3.356 1.202 -5.713 -0.999 0.00587
Baseline overall scale score -0.111 0.014 -0.138 -0.084 <0.001

Note: N = 3,771; R² = 0.083; Adj. R² = 0.079; Residual df = NA

Other outcomes
Outcome Estimate CI low CI high p N
Annual Typical Growth (i-Ready) -0.345 -1.116 0.426 0.38100 3771
Annual Stretch Growth (i-Ready) -0.220 -0.725 0.285 0.39400 3771
% progress to Typical Growth (i-Ready) -11.891 -30.323 6.542 0.20800 3771
% progress to Stretch Growth (i-Ready) -9.075 -20.125 1.974 0.10900 3771
Mid-on-grade-level scale score (i-Ready) -0.937 -3.600 1.726 0.49100 3771
Number & Operations domain (gain score) -4.859 -9.385 -0.334 0.03680 3771
Number & Operations domain (scale score) -3.447 -8.313 1.420 0.16700 3771
Algebra & Algebraic Thinking domain (gain score) -4.838 -7.757 -1.918 0.00141 3771
Algebra & Algebraic Thinking domain (scale score) -3.486 -6.579 -0.393 0.02850 3771
Measurement & Data domain (gain score) -1.214 -7.326 4.897 0.69700 3771
Measurement & Data domain (scale score) 0.271 -5.747 6.289 0.93000 3771
Geometry domain (gain score) -6.638 -12.536 -0.739 0.02880 3771
Geometry domain (scale score) -4.515 -10.681 1.650 0.15300 3771

G2. i-Ready 2023-24 student within year growth (exposure: taught in 2023–24 by a teacher who had been trained in 2022–23)

This within‑year comparison contrasts students taught in 2023–24 by a teacher who had been trained in 2022–23 with students taught by untrained teachers; the outcome is Diagnostic gain (i-Ready). There were 155 students in the treated group and 2,223 students in the untreated group. The estimated difference is −2.1966 (treated minus control), with a confidence interval from −6.894 to 2.5008 and p = 0.3618, indicating appreciable uncertainty around the point estimate. Standard errors are clustered by teacher.

Full primary model results

Term Estimate Std. Error CI low CI high p
Intercept 83.279 8.959 65.719 100.839 <0.001
Trained teacher (current) -2.197 2.397 -6.894 2.501 0.36200
Grade group: 2 1.571 2.815 -3.946 7.087 0.57800
Grade group: 3 4.429 2.863 -1.182 10.040 0.12500
Grade group: 4 5.155 3.108 -0.936 11.246 0.10100
Grade group: 5 3.546 3.041 -2.415 9.507 0.24700
Grade group: 6 12.382 3.974 4.594 20.170 0.00244
Student sex: male 1.718 0.710 0.325 3.110 0.01750
Student sex: Unknown -1.255 2.685 -6.519 4.008 0.64100
Student race: Asian 7.129 4.444 -1.581 15.839 0.11200
Student race: Black or African American 0.612 4.724 -8.647 9.871 0.89700
Student race: Native Hawaiian or Other Pacific Islander -3.459 6.532 -16.261 9.343 0.59800
Student race: Two or More Races 0.563 4.416 -8.092 9.218 0.89900
Student race: White 3.278 4.422 -5.390 11.945 0.46000
Student race: Unknown -15.402 4.677 -24.569 -6.234 0.00140
Hispanic/Latino: Y 16.380 1.807 12.838 19.923 <0.001
English learner: Y -3.221 1.219 -5.611 -0.831 0.00968
Special education: Y -4.255 1.516 -7.225 -1.284 0.00608
Teacher gender: Male 1.441 2.128 -2.730 5.613 0.50000
Teacher gender: Unknown -2.780 1.530 -5.778 0.218 0.07240
Baseline overall scale score -0.154 0.019 -0.192 -0.116 <0.001

Note: N = 2,376; R² = 0.105; Adj. R² = 0.098; Residual df = NA

Other outcomes
Outcome Estimate CI low CI high p N
Annual Typical Growth (i-Ready) -0.264 -0.525 -0.004 0.0498 2376
Annual Stretch Growth (i-Ready) -0.150 -0.655 0.354 0.5610 2376
% progress to Typical Growth (i-Ready) -12.696 -39.278 13.885 0.3520 2376
% progress to Stretch Growth (i-Ready) -6.836 -21.390 7.718 0.3600 2376
Mid-on-grade-level scale score (i-Ready) 0.000 0.000 0.000 1.0000 2376
Number & Operations domain (gain score) -1.916 -8.553 4.721 0.5730 2376
Number & Operations domain (scale score) -1.677 -8.166 4.813 0.6140 2376
Algebra & Algebraic Thinking domain (gain score) -2.180 -8.574 4.214 0.5060 2376
Algebra & Algebraic Thinking domain (scale score) -1.452 -7.815 4.910 0.6560 2376
Measurement & Data domain (gain score) -4.385 -9.137 0.368 0.0738 2376
Measurement & Data domain (scale score) -1.896 -6.796 3.004 0.4500 2376
Geometry domain (gain score) -4.498 -11.618 2.623 0.2190 2376
Geometry domain (scale score) -3.757 -11.248 3.734 0.3280 2376

G3. i-Ready 2023-24 student within year growth (exposure: taught by a trained teacher in 2022-23)

This analysis compares students who had a trained teacher in 2022–23 versus those who did not, using within‑year 2023–24 diagnostic growth (baseline to most‑recent YTD) as the outcome. The outcome highlighted is Diagnostic gain (i‑Ready); 85 students were in the trained (treated) group and 2,457 students were in the untrained (control) group. The estimated difference in diagnostic gain for the trained group is 3.3273 (95% CI: −2.9088 to 9.5634), p = 0.2978, indicating a positive but imprecise estimate. Primary specification clusters standard errors by teacher.

Full primary model results

Term Estimate Std. Error CI low CI high p
Intercept 72.759 10.527 52.125 93.392 <0.001
Trained teacher (current) 3.327 3.182 -2.909 9.563 0.29800
Grade group: 2 -15.406 3.477 -22.221 -8.592 <0.001
Grade group: 3 6.265 1.730 2.874 9.657 <0.001
Grade group: 4 6.441 1.787 2.938 9.944 <0.001
Grade group: 5 4.991 1.613 1.829 8.152 0.00248
Grade group: 6 9.490 1.886 5.794 13.185 <0.001
Student sex: male 2.522 0.724 1.102 3.942 <0.001
Student race: Asian 1.117 7.277 -13.145 15.379 0.87800
Student race: Black or African American -6.431 7.406 -20.947 8.085 0.38700
Student race: Native Hawaiian or Other Pacific Islander -8.574 8.177 -24.601 7.453 0.29700
Student race: Two or More Races -3.302 7.187 -17.389 10.786 0.64700
Student race: White -2.972 7.244 -17.171 11.226 0.68200
Student race: Unknown -4.800 7.211 -18.934 9.334 0.50700
English learner: Y -1.075 1.080 -3.191 1.041 0.32100
Special education: Y -0.374 1.961 -4.219 3.470 0.84900
Teacher gender: Male 1.987 2.792 -3.485 7.459 0.47800
Teacher gender: Unknown -0.314 1.475 -3.205 2.576 0.83200
Baseline overall scale score -0.123 0.018 -0.159 -0.087 <0.001

Note: N = 2,540; R² = 0.083; Adj. R² = 0.076; Residual df = NA

Other outcomes
Outcome Estimate CI low CI high p N
Annual Typical Growth (i-Ready) -0.262 -0.536 0.011 0.0629 2540
Annual Stretch Growth (i-Ready) -0.214 -0.762 0.335 0.4470 2540
% progress to Typical Growth (i-Ready) 13.775 -15.966 43.517 0.3660 2540
% progress to Stretch Growth (i-Ready) 9.242 -9.964 28.449 0.3480 2540
Mid-on-grade-level scale score (i-Ready) 0.000 0.000 0.000 1.0000 2540
Number & Operations domain (gain score) 4.997 -2.201 12.195 0.1760 2540
Number & Operations domain (scale score) 4.622 -2.484 11.728 0.2050 2540
Algebra & Algebraic Thinking domain (gain score) 6.648 -1.297 14.594 0.1040 2540
Algebra & Algebraic Thinking domain (scale score) 6.051 -2.499 14.601 0.1680 2540
Measurement & Data domain (gain score) 1.476 -6.089 9.042 0.7030 2540
Measurement & Data domain (scale score) 2.301 -5.320 9.921 0.5550 2540
Geometry domain (gain score) -1.457 -8.169 5.256 0.6710 2540
Geometry domain (scale score) -1.096 -8.846 6.654 0.7820 2540