Oak Grove Analysis

Overview

What was done

  • We compared students taught by trained teachers versus untrained teachers.
  • For CAASPP, we examined math achievement using prior-year CAASPP scale score as a baseline adjustment where available (ANCOVA).
  • For i-Ready, we examined student within-year growth and prioritized diagnostic gain for the headline summary.
  • Because few schools contained both trained and untrained exposure in the analytic samples, school fixed effects are not emphasized in the main results.

Analysis map

Research question Panel definition Outcomes Controls
Do students taught by trained teachers show better math achievement than students taught by untrained teachers? 2022–23 CAASPP math compared for trained vs untrained; adjusts for prior-year (2021–22) CAASPP math scale score. CAASPP Math scale score (2022–23) with 2021–22 as baseline (ANCOVA) Prior-year CAASPP scale score; Grade; Sex (if available); Hispanic/Latino; English learner; Special education; Economic disadvantage; Reporting ethnicity indicators; Teacher gender; Teacher experience; Baseline overall scale score
In 2023–24, is achievement higher for students taught that year by a teacher trained in 2022–23? 2023–24 CAASPP math compared for current-exposure trained vs untrained; main spec is cross-sectional (prior-year CAASPP available for context). CAASPP Math scale score (2023–24) Grade; Hispanic/Latino; English learner; Special education; Economic disadvantage; Reporting ethnicity indicators, Teacher gender; Teacher experience; Baseline overall scale score
In 2023–24, is achievement higher for students who had a trained teacher in 2022–23 (carryover)? 2023–24 CAASPP math compared for delayed-exposure trained vs untrained; main spec is cross-sectional (prior-year CAASPP available for context). CAASPP Math scale score (2023–24) Grade; Hispanic/Latino; English learner; Special education; Economic disadvantage; Reporting ethnicity indicators; Teacher gender; Teacher experience; Baseline overall scale score
Do students taught by trained teachers show more growth in math ability than students taught by untrained teachers? Within 2022–23, compares students taught by trained vs untrained teachers; growth uses baseline vs most-recent diagnostic within the year. Diagnostic gain; Annual Typical Growth; Annual Stretch Growth; % progress to Typical; % progress to Stretch; Mid-on-grade-level; Domain (Number & Operations, Algebra, Measurement & Data, Geometry) Grade group; Sex; Race; Hispanic/Latino; English learner; Special education; Teacher gender; Teacher experience; Baseline overall scale score (i-Ready)
In 2023–24, does being taught that year by a teacher trained in 2022–23 relate to greater growth? Within 2023–24, compares students taught in 2023–24 by previously-trained teachers vs students taught by untrained teachers; growth uses baseline vs most-recent YTD diagnostic. Diagnostic gain; Annual Typical Growth; Annual Stretch Growth; % progress to Typical; % progress to Stretch; Mid-on-grade-level; Domain (Number & Operations, Algebra, Measurement & Data, Geometry) Grade; Sex; Race; Hispanic/Latino; English learner; Special education; Teacher gender; Teacher experience; Baseline overall scale score (i-Ready)
In 2023–24, do students previously taught by a trained teacher in 2022–23 show greater growth (carryover effect)? Within 2023–24, compares students who had trained teachers in 2022–23 vs students who did not; growth uses baseline vs most-recent YTD diagnostic in 2023–24. Diagnostic gain; Annual Typical Growth; Annual Stretch Growth; % progress to Typical; % progress to Stretch; Mid-on-grade-level; Domain (Number & Operations, Algebra, Measurement & Data, Geometry) Grade; Sex; Race; Hispanic/Latino; English learner; Special education; Teacher gender; Teacher experience; Baseline overall scale score (i-Ready)

Primary results

Analysis Estimate CI low CI high p N
CAASPP 2022-23 Math Achievement (exposure: taught by teacher trained in 2022-23) 12.619 -8.412 33.650 0.2570 1670
CAASPP 2023-24 Math Achievement (exposure: taught in 2023–24 by a teacher who had been trained in 2022–23) 0.930 -37.418 39.279 0.9630 1513
CAASPP 2023-24 Math Achievement (exposure: taught by a trained teacher in 2022-23) -1.528 -21.059 18.003 0.8800 3105
i-Ready 2022-23 student within year growth (exposure: taught by teacher trained in 2022-23) -3.390 -7.144 0.364 0.0786 3771
i-Ready 2023-24 student within year growth (exposure: taught in 2023–24 by a teacher who had been trained in 2022–23) -2.197 -6.894 2.501 0.3620 2376
i-Ready 2023-24 student within year growth (exposure: taught by a trained teacher in 2022-23) 3.327 -2.909 9.563 0.2980 2540

Main takeaways

  • CAASPP 2022-23 Math Achievement: Students taught by teachers trained in 2022–23 scored higher than students with untrained teachers by an estimated 12.619 (95% CI −8.412 to 33.650), p = 0.2568, N = 1670.
  • CAASPP 2023-24 Math Achievement (taught in 2023–24 by a teacher who had been trained in 2022–23): Students taught by trained teachers performed about the same as those with untrained teachers, estimated difference 0.931 (95% CI −37.418 to 39.279), p = 0.9627, N = 1513.
  • CAASPP 2023-24 Math Achievement (taught by a trained teacher in 2022-23): Students taught by trained teachers scored slightly lower than those with untrained teachers, estimated −1.528 (95% CI −21.059 to 18.003), p = 0.88, N = 3105.
  • i-Ready 2022-23 within-year growth: Students taught by teachers trained in 2022-23 showed lower within-year growth by an estimated −3.390 (95% CI −7.144 to 0.364), p = 0.0786, N = 3771.
  • i-Ready 2023-24 within-year growth (taught in 2023–24 by a teacher who had been trained in 2022–23): Students taught by trained teachers showed lower growth by an estimated −2.197 (95% CI −6.894 to 2.501), p = 0.3618, N = 2376.
  • i-Ready 2023-24 within-year growth (taught by a trained teacher in 2022-23): Students taught by trained teachers showed higher growth by an estimated 3.327 (95% CI −2.909 to 9.563), p = 0.2978, N = 2540.

Synthesis: Across the six comparisons, point estimates vary in direction and are generally small relative to their uncertainty; confidence intervals include both positive and negative values and none of the p-values indicate clear evidence of an effect. In short, the data do not provide consistent or precise evidence that the training produced clear improvements in these outcomes.

Results by analysis run

Achievement (CAASPP)

A1. CAASPP 2022-23 Math Achievement (exposure: taught by teacher trained in 2022-23)

Comparing students whose 2022–23 teacher was trained versus untrained (exposure defined as “Trained in 2022–23”), the outcome examined was the CAASPP Math scale score. There were 136 students taught by trained teachers and 2,662 taught by untrained teachers. The ANCOVA estimate indicates a positive difference of 12.62 scale-score points (95% CI: −8.41 to 33.65), p = 0.2568, reflecting considerable uncertainty. The primary specification clusters standard errors by school.

Full primary model results

Term Estimate Std. Error CI low CI high p
Intercept 490.988 45.743 401.332 580.645 <0.001
Trained teacher (current) 12.619 10.730 -8.412 33.650 0.25700
Prior-year score 0.799 0.020 0.760 0.838 <0.001
Grade: 4 24.343 5.930 12.721 35.965 <0.001
Grade: 5 13.475 5.877 1.956 24.995 0.03570
Student sex: Male 2.798 2.427 -1.959 7.556 0.26600
Hispanic/Latino: Yes 13.166 12.055 -10.461 36.793 0.29100
English learner: Yes -8.116 3.730 -15.428 -0.805 0.04490
Special education: Yes 1.239 8.184 -14.801 17.278 0.88200
Economic disadvantage: Yes -5.564 2.890 -11.228 0.100 0.07220
American Indian/Alaska Native 14.810 12.564 -9.814 39.435 0.25600
Asian 39.812 12.621 15.074 64.549 0.00614
Black/African American -5.001 16.128 -36.612 26.611 0.76100
Filipino 26.813 12.120 3.058 50.568 0.04180
Native Hawaiian/Pacific Islander 26.797 12.306 2.676 50.917 0.04480
Two or more / not reported 30.992 14.275 3.012 58.972 0.04530

Note: N = 1,670; R² = 0.732; Adj. R² = 0.730; Residual df = NA

A2. CAASPP 2023-24 Math Achievement (exposure: taught in 2023–24 by a teacher who had been trained in 2022–23)

This analysis compares students in 2023–24 whose 2023–24 teacher had been trained in 2022–23 to students whose 2023–24 teacher had not been trained, with the outcome CAASPP Math scale score. There were 98 students in the treated group and 1417 students in the untreated group. The estimated difference in CAASPP Math scale score was 0.9305 points (95% CI: −37.4178 to 39.2788; n = 1513), p = 0.9627, indicating a very imprecise and non-significant estimate. Standard errors were clustered by school, and school fixed effects were not emphasized due to limited within-school treated/control overlap.

Full primary model results

Term Estimate Std. Error CI low CI high p
Intercept 2414.236 14.776 2385.275 2443.198 <0.001
Trained teacher (current) 0.930 19.565 -37.418 39.279 0.96300
Grade: 4 46.238 11.134 24.416 68.060 <0.001
Grade: 5 71.345 8.195 55.283 87.406 <0.001
Grade: 6 87.148 15.579 56.613 117.683 <0.001
Hispanic/Latino: Yes 21.847 11.018 0.252 43.442 0.06600
English learner: Yes -65.628 5.764 -76.925 -54.331 <0.001
Special education: Yes -81.599 9.447 -100.114 -63.084 <0.001
Economic disadvantage: Yes -25.936 4.891 -35.522 -16.351 <0.001
American Indian/Alaska Native -19.633 18.525 -55.943 16.677 0.30600
Asian 103.522 13.405 77.248 129.796 <0.001
Black/African American 41.535 46.277 -49.168 132.238 0.38400
Filipino 57.252 15.083 27.689 86.816 0.00176
Native Hawaiian/Pacific Islander 63.542 11.999 40.024 87.060 <0.001
Two or more / not reported 64.367 15.720 33.555 95.179 <0.001

Note: N = 1,513; R² = 0.421; Adj. R² = 0.415; Residual df = NA

A3. CAASPP 2023-24 Math Achievement (exposure: taught by a trained teacher in 2022-23)

This analysis compares students who had a teacher trained in 2022–23 to those who did not, focusing on 2023–24 CAASPP Math scale scores. There were 175 students in the trained group and 2,932 in the untrained group. The estimated difference (trained minus untrained) is -1.53 points, with a 95% confidence interval of -21.06 to 18.00 and p = 0.88, indicating substantial uncertainty. The primary specification clusters standard errors by school.

Full primary model results

Term Estimate Std. Error CI low CI high p
Intercept 2415.924 13.586 2389.296 2442.553 <0.001
Trained teacher (current) -1.528 9.965 -21.059 18.003 0.88000
Grade: 4 46.932 6.443 34.304 59.559 <0.001
Grade: 5 64.477 7.159 50.446 78.508 <0.001
Grade: 6 87.682 7.842 72.311 103.053 <0.001
Hispanic/Latino: Yes 17.879 11.865 -5.377 41.135 0.15100
English learner: Yes -56.578 5.730 -67.810 -45.346 <0.001
Special education: Yes -87.201 11.031 -108.822 -65.580 <0.001
Economic disadvantage: Yes -29.362 3.831 -36.870 -21.854 <0.001
American Indian/Alaska Native 79.576 43.463 -5.611 164.763 0.08580
Asian 104.528 12.655 79.723 129.332 <0.001
Black/African American -9.413 17.102 -42.932 24.106 0.59000
Filipino 54.922 14.217 27.056 82.788 0.00138
Native Hawaiian/Pacific Islander 65.494 12.364 41.260 89.728 <0.001
Two or more / not reported 76.911 14.180 49.118 104.704 <0.001

Note: N = 3,105; R² = 0.417; Adj. R² = 0.414; Residual df = NA

Growth (i-Ready)

G1. i-Ready 2022-23 student within year growth (exposure: taught by teacher trained in 2022-23)

Within 2022–23, this analysis compares students whose 2022–23 teacher was trained in 2022–23 to students whose teacher was untrained. The outcome highlighted is Diagnostic gain (i-Ready); 199 students were in the treated group and 3,573 in the untreated group. The estimated difference in diagnostic gain is −3.39 (confidence interval −7.1444 to 0.3641; p = 0.0786), a negative point estimate with uncertainty that overlaps values near zero. The primary specification clusters standard errors by teacher.

Full primary model results

Term Estimate Std. Error CI low CI high p
Intercept 72.903 9.083 55.100 90.706 <0.001
Trained teacher (current) -3.390 1.915 -7.144 0.364 0.07860
Grade group: 3 0.125 1.528 -2.869 3.119 0.93500
Grade group: 4–5 -0.707 1.495 -3.637 2.224 0.63700
Student sex: male 1.255 0.516 0.243 2.267 0.01610
Student sex: Unknown 6.489 1.344 3.855 9.123 <0.001
Student race: Asian -0.297 6.341 -12.726 12.131 0.96300
Student race: Black or African American -5.269 6.633 -18.270 7.733 0.42800
Student race: Native Hawaiian or Other Pacific Islander -3.748 7.196 -17.853 10.357 0.60300
Student race: Two or More Races -1.487 6.456 -14.141 11.167 0.81800
Student race: White -2.899 6.375 -15.394 9.596 0.65000
Student race: Unknown -5.891 6.343 -18.324 6.542 0.35400
English learner: Y -0.967 0.791 -2.517 0.584 0.22400
Special education: Y -2.891 1.085 -5.018 -0.763 0.00850
Teacher gender: Male -1.019 2.256 -5.441 3.403 0.65200
Teacher gender: Unknown -3.356 1.202 -5.713 -0.999 0.00587
Baseline overall scale score -0.111 0.014 -0.138 -0.084 <0.001

Note: N = 3,771; R² = 0.083; Adj. R² = 0.079; Residual df = NA

Other outcomes
Outcome Estimate CI low CI high p N
Annual Typical Growth (i-Ready) -0.345 -1.116 0.426 0.38100 3771
Annual Stretch Growth (i-Ready) -0.220 -0.725 0.285 0.39400 3771
% progress to Typical Growth (i-Ready) -11.891 -30.323 6.542 0.20800 3771
% progress to Stretch Growth (i-Ready) -9.075 -20.125 1.974 0.10900 3771
Mid-on-grade-level scale score (i-Ready) -0.937 -3.600 1.726 0.49100 3771
Number & Operations domain (gain score) -4.859 -9.385 -0.334 0.03680 3771
Number & Operations domain (scale score) -3.447 -8.313 1.420 0.16700 3771
Algebra & Algebraic Thinking domain (gain score) -4.838 -7.757 -1.918 0.00141 3771
Algebra & Algebraic Thinking domain (scale score) -3.486 -6.579 -0.393 0.02850 3771
Measurement & Data domain (gain score) -1.214 -7.326 4.897 0.69700 3771
Measurement & Data domain (scale score) 0.271 -5.747 6.289 0.93000 3771
Geometry domain (gain score) -6.638 -12.536 -0.739 0.02880 3771
Geometry domain (scale score) -4.515 -10.681 1.650 0.15300 3771

G2. i-Ready 2023-24 student within year growth (exposure: taught in 2023–24 by a teacher who had been trained in 2022–23)

This analysis compares students in 2023–24 who were taught that year by teachers trained in 2022–23 versus students taught by untrained teachers, measuring within-year growth from baseline to the most recent YTD diagnostic. The outcome is Diagnostic gain (i-Ready); 155 students were in the treated condition and 2,223 in the untreated condition. The estimated difference is -2.1966 (treated minus control), with a 95% CI of [-6.894, 2.5008] and p = 0.3618, indicating uncertainty around the negative point estimate. Standard errors are clustered by teacher (primary specification).

Full primary model results

Term Estimate Std. Error CI low CI high p
Intercept 83.279 8.959 65.719 100.839 <0.001
Trained teacher (current) -2.197 2.397 -6.894 2.501 0.36200
Grade group: 2 1.571 2.815 -3.946 7.087 0.57800
Grade group: 3 4.429 2.863 -1.182 10.040 0.12500
Grade group: 4 5.155 3.108 -0.936 11.246 0.10100
Grade group: 5 3.546 3.041 -2.415 9.507 0.24700
Grade group: 6 12.382 3.974 4.594 20.170 0.00244
Student sex: male 1.718 0.710 0.325 3.110 0.01750
Student sex: Unknown -1.255 2.685 -6.519 4.008 0.64100
Student race: Asian 7.129 4.444 -1.581 15.839 0.11200
Student race: Black or African American 0.612 4.724 -8.647 9.871 0.89700
Student race: Native Hawaiian or Other Pacific Islander -3.459 6.532 -16.261 9.343 0.59800
Student race: Two or More Races 0.563 4.416 -8.092 9.218 0.89900
Student race: White 3.278 4.422 -5.390 11.945 0.46000
Student race: Unknown -15.402 4.677 -24.569 -6.234 0.00140
Hispanic/Latino: Y 16.380 1.807 12.838 19.923 <0.001
English learner: Y -3.221 1.219 -5.611 -0.831 0.00968
Special education: Y -4.255 1.516 -7.225 -1.284 0.00608
Teacher gender: Male 1.441 2.128 -2.730 5.613 0.50000
Teacher gender: Unknown -2.780 1.530 -5.778 0.218 0.07240
Baseline overall scale score -0.154 0.019 -0.192 -0.116 <0.001

Note: N = 2,376; R² = 0.105; Adj. R² = 0.098; Residual df = NA

Other outcomes
Outcome Estimate CI low CI high p N
Annual Typical Growth (i-Ready) -0.264 -0.525 -0.004 0.0498 2376
Annual Stretch Growth (i-Ready) -0.150 -0.655 0.354 0.5610 2376
% progress to Typical Growth (i-Ready) -12.696 -39.278 13.885 0.3520 2376
% progress to Stretch Growth (i-Ready) -6.836 -21.390 7.718 0.3600 2376
Mid-on-grade-level scale score (i-Ready) 0.000 0.000 0.000 1.0000 2376
Number & Operations domain (gain score) -1.916 -8.553 4.721 0.5730 2376
Number & Operations domain (scale score) -1.677 -8.166 4.813 0.6140 2376
Algebra & Algebraic Thinking domain (gain score) -2.180 -8.574 4.214 0.5060 2376
Algebra & Algebraic Thinking domain (scale score) -1.452 -7.815 4.910 0.6560 2376
Measurement & Data domain (gain score) -4.385 -9.137 0.368 0.0738 2376
Measurement & Data domain (scale score) -1.896 -6.796 3.004 0.4500 2376
Geometry domain (gain score) -4.498 -11.618 2.623 0.2190 2376
Geometry domain (scale score) -3.757 -11.248 3.734 0.3280 2376

G3. i-Ready 2023-24 student within year growth (exposure: taught by a trained teacher in 2022-23)

This analysis compares students who had a teacher trained in 2022–23 to students who did not, evaluating within-year growth in 2023–24. The outcome is Diagnostic gain (i-Ready). Eighty-five students were in the treated group and 2,457 in the untreated group. The estimated difference in diagnostic gain was 3.3273 (95% CI: −2.9088 to 9.5634; p = 0.2978), with standard errors clustered by teacher.

Full primary model results

Term Estimate Std. Error CI low CI high p
Intercept 72.759 10.527 52.125 93.392 <0.001
Trained teacher (current) 3.327 3.182 -2.909 9.563 0.29800
Grade group: 2 -15.406 3.477 -22.221 -8.592 <0.001
Grade group: 3 6.265 1.730 2.874 9.657 <0.001
Grade group: 4 6.441 1.787 2.938 9.944 <0.001
Grade group: 5 4.991 1.613 1.829 8.152 0.00248
Grade group: 6 9.490 1.886 5.794 13.185 <0.001
Student sex: male 2.522 0.724 1.102 3.942 <0.001
Student race: Asian 1.117 7.277 -13.145 15.379 0.87800
Student race: Black or African American -6.431 7.406 -20.947 8.085 0.38700
Student race: Native Hawaiian or Other Pacific Islander -8.574 8.177 -24.601 7.453 0.29700
Student race: Two or More Races -3.302 7.187 -17.389 10.786 0.64700
Student race: White -2.972 7.244 -17.171 11.226 0.68200
Student race: Unknown -4.800 7.211 -18.934 9.334 0.50700
English learner: Y -1.075 1.080 -3.191 1.041 0.32100
Special education: Y -0.374 1.961 -4.219 3.470 0.84900
Teacher gender: Male 1.987 2.792 -3.485 7.459 0.47800
Teacher gender: Unknown -0.314 1.475 -3.205 2.576 0.83200
Baseline overall scale score -0.123 0.018 -0.159 -0.087 <0.001

Note: N = 2,540; R² = 0.083; Adj. R² = 0.076; Residual df = NA

Other outcomes
Outcome Estimate CI low CI high p N
Annual Typical Growth (i-Ready) -0.262 -0.536 0.011 0.0629 2540
Annual Stretch Growth (i-Ready) -0.214 -0.762 0.335 0.4470 2540
% progress to Typical Growth (i-Ready) 13.775 -15.966 43.517 0.3660 2540
% progress to Stretch Growth (i-Ready) 9.242 -9.964 28.449 0.3480 2540
Mid-on-grade-level scale score (i-Ready) 0.000 0.000 0.000 1.0000 2540
Number & Operations domain (gain score) 4.997 -2.201 12.195 0.1760 2540
Number & Operations domain (scale score) 4.622 -2.484 11.728 0.2050 2540
Algebra & Algebraic Thinking domain (gain score) 6.648 -1.297 14.594 0.1040 2540
Algebra & Algebraic Thinking domain (scale score) 6.051 -2.499 14.601 0.1680 2540
Measurement & Data domain (gain score) 1.476 -6.089 9.042 0.7030 2540
Measurement & Data domain (scale score) 2.301 -5.320 9.921 0.5550 2540
Geometry domain (gain score) -1.457 -8.169 5.256 0.6710 2540
Geometry domain (scale score) -1.096 -8.846 6.654 0.7820 2540