Statistical Analysis of Arizona’s

School Accountability Model “AZ LEARNS”

 

October 25, 2005

 

Statistical analysis of Arizona’s school accountability model (AZ LEARNS) was conducted using data from schools in the Tucson Unified School District (TUSD).

The purpose of this investigation was two-fold, first simply to determine whether the AZ LEARNS model could be described accurately by a simply linear model, and second to determine the relative importance of the components or variables that make-up the model. Generalizing the results found in this investigation to schools outside TUSD should be done with caution.

First, correlations were calculated between all the major components of the AZ LEARNS model.  Calculations were performed separately for elementary, middle and high schools with the resulting coefficients shown in style="mso-bidi-font-weight: normal" Tables 1 through 3 .  It should be noted in Table 2 that the z-score has very little influence on a school’s label (Profile) for middle schools compared to elementary and high schools.  The z-score is a standard score resulting from the linear transformation of the percent of students scoring in the “exceeds” category on the AIMS relative to state norms.  The z-score was only applied to schools scoring enough points to be on the border between a “Performing Plus” and “Highly Performing” or “Highly Performing” and “Excelling” label.  Since most TUSD middle schools were well below the total points necessary for a Highly Performing label the z-score rarely entered into the equation.  It should also be noted that the “Measure of Academic Progress” (MAP) demonstrated little if any relation to the remaining AZ LEARNS indicators.  A small although significant correlation was obtained between the total points and MAP for elementary schools, however this was likely due to the fact that the total points includes MAP points.  To test this assumption MAP points were removed from the total points before correlating MAP with total points.  The resulting correlation yielded a coefficient of r = -.11, supporting the contention that the MAP component of AZ LEARNS shows little relation to the remaining AZ LEARNS indicators.  The MAP component of AZ LEARNS will be discussed later in greater detail.  Finally, the correlations between “Percent Tested” and “Adequate Yearly Progress” (AYP) are small and moreover negative for elementary and high schools in spite of the fact that AYP includes percent tested.

Tables 4 through 6 show the results of a multiple regression analysis using total points as the dependent variable and the components that are used to calculate total points as predictors.  The purpose of this analysis was to determine the relative importance of the variables that sum to a school’s total points.  One would expect an extremely high multiple correlation (R) assuming that all the components in the model have been included.  As can be seen in Tables 5 and 6 the multiple correlations were extremely high to perfect as expected.  The relative contribution to the total points can be determined by inspection of the beta weights, the larger the magnitude of the betas the greater the contribution of that variable.  When the multiple-R is perfect (1.0) or near perfect the percentage of total variance explained can be calculated by summing the betas to form the denominator and using the individual betas as the numerator.  For middle schools it was determined that 64% of the variance was due to AIMS scores, 26% due to MAP, and 10% contributed by AYP.  Intuitively, one might expect that the variable that contributes the most points would also be the variable making the greatest relative contribution.  This is not always the case, as can be seen in Table 4, AYP slightly outweighs MAP even though MAP points range from 0 to 8 and AYP can only assume a value of 1 or 0.  The AYP score although smaller in absolute magnitude was more closely related to a schools overall points status.  The greater contribution of AYP may be partially attributed to the fact that AYP includes such measures as attendance rate, percent tested, and AIMS performance, but more likely it was due simply to the fact that MAP has been demonstrated to be a poor indicator of academic performance.  It should also be noted that the multiple correlation for elementary schools was somewhat lower than one might expect.  Only 65% of the variance in total points could be explained by the measures on which it was based.  The variance that was unaccounted for (35%) must be attributed to error, variables not in the equation, or both.  Analysis of the relative contributions of the predictor variables was not possible for high schools due to the perfect correlation and the resulting inability to invert the variance/covariance matrix.

Tables 7 through 9 show the results of a multiple regression analysis using the School Profile as the dependent measure and the major components that make up the profile as predictor variables.  Achievement Profiles were converted to numeric values with an “Underperforming” label resulting in a value of 1, and an “Excelling” label resulting in a value of 5.  The Tables show a similar pattern to that found when using total points as the dependent variable (Tables 4 through 6).  Only 61% of the variance in a school’s profile could be explained by the major contributing components for elementary schools, leading to the conclusion that some factor or factors that are influencing a school’s profile have not been identified in this model.  The multiple correlations for middle and high schools on the other hand are substantially higher indicating that the vast majority of variance has been accounted for.  One additional component included in the high school model was omitted from this analysis.  High schools were awarded 1 point for graduation rate, and 1 point for drop-out rate resulting in a maximum of 2 points.  Since all high schools in TUSD obtained the maximum number of points it contributed nothing to the analysis.

Tables 10 through 12 show the correlations between a school’s demographics and AZ LEARNS indicators.  The analysis also included the percent of students meeting State Standards on the AIMS, attendance, Stress Index^1, and Effectiveness Index².  The correlations between demographic variables and academic performance show that schools under less demographic stress tend to outperform schools with greater stress.  Once more, the stress level appears to have the greatest negative impact for high schools, followed by middle schools and elementary schools.  The negative correlations found between MAP and AIMS performance, especially for middle schools was troubling.  It should be remembered that a similar lack of relation was found between MAP and the remaining AZ LEARNS indicators (Tables 1 & 2).  Currently, calculation of MAP relies on a comparison between a student’s performance last year on the Stanford 9 to this years performance on the AIMS.  If a student maintained or increased their percentile rank they are said to have made “One years Growth” (OYG).  The percent of students making one years growth are compared to a table that indicates the points earned for the school (MAP).  Intuitively, one would expect that the higher a schools performance this year the greater the likelihood that they also made growth.  Not only does the data not support this logic it suggests the opposite.  Barring an error in calculation of MAP the fact that a students relative standing on one test was compared to that of a completely different test may at least partially explain the error introduced to the model by the inclusion of MAP.  Another possible explanation is that schools were allotted points based on last years MAP if it was higher.  Last years MAP was based on a stanine comparison of Stanford 9 scores that year (2003-2004) to the previous years Stanford 9 scores (2002-2003).  Thus a school may have been awarded points based on the previous year’s growth which may or may not be related to this year’s growth.  If a school demonstrated significant growth last year it may be more likely that the school would show less growth if any this year.  This situation could result in small or perhaps even negative coefficients like the ones obtained.  The fact that MAP was not as detrimental to elementary schools may be a result of better alignment between the Stanford 9 and AIMS at lower grade levels, or MAP points were more often awarded based on current growth compared to last years growth.  The differential frequency of MAP years by grade level was not investigated.  A more detailed look at the data was accomplished by forming a scatter plot of z-scores (% exceeding) and MAP scores (Figure 1).  As can be seen all middle and elementary schools received either a 4 or 6 for MAP and once again these values appeared to be unrelated to a schools current academic performance.  MAP was not calculated for high schools; instead high schools received a maximum of 2 points for high graduation rate and low drop-out rate (Grad/Drop).  Since all TUSD high schools received the maximum number of points statistics could not be calculated for this indicator.  The correlations between the AZ LEARNS profile and the Effectiveness Index should also be noted, especially for middle schools.  The complete lack of correlation between the Profile and the Effectiveness Index (an alternate method for assessing schools) for middle schools was troublesome.  Although the methodology is quite different one would expect at least a moderate correlation given the two methods measure similar constructs.

Tables 13 through 15 show the results of multiple regression analyses predicting a school’s label using demographic factors alone.  For elementary schools the best single demographic predictor of a school’s label was SES based on a weighted free and reduced lunch count, followed by attendance.  For middle schools the best single predictor was attendance rate followed by mobility rate.  SES was the best single predictor for high schools followed by attendance rate and mobility rate.  The multiple correlation for elementary schools using demographic predictors alone was R = .60 indicating that 31 percent of the variance in a schools label could be explained by a schools demographics.  The multiple correlations for middle and high schools were R = .86 and R = .98 respectively.  For high schools 92 percent of the variance in a schools label could be attributed to the schools demographics.  Past research has shown that measures of academic growth are much less related to demographic variables compared to measures of academic status.  Since three new grades were administered the AIMS this year (4, 6, 7) it was not possible to calculate a measure of growth for these grades.  For grades that were tested for the first year a measure of status alone was used as an indicator of academic performance.  Not all grade levels were equally affected by the addition of these three new test grades.  Middle schools were impacted the most followed by elementary schools, and high schools were not affected at all by the addition of new test grades.  Elementary schools had eight sources of academic growth in the AZ LEARNS model.  The eight sources of growth come from the two grades that tested the previous year (3 and 5) by three subject areas (reading, writing and mathematics) equaling six sources plus AYP and MAP for a total of eight.  Middle schools on the other hand had only five sources of academic growth.  Middles schools had only one grade that had been tested the previous year by three subject areas plus AYP and MAP.  Although high schools were not directly impacted by the addition of new test grades they only had four sources of growth in the model.  High schools have one grade by three subject areas plus AYP.  MAP was not calculated for high schools; instead points are awarded based on graduation rate and drop-out rate.  The differences in the model across grade levels in terms of the number of sources of growth would logically explain the differences in the magnitudes of the multiple regression coefficients.  The multiple-R between demographic variables and school labels was smallest for elementary schools because elementary school labels were more heavily weighted by growth compared to high schools whose labels were more reflective of status alone.

Tables 16 through 23 show the results of correlation and multiple regression comparisons of status versus growth for elementary, middle and high schools.  As can be seen in Table 16, status correlates more highly with AZ LEARNS components than growth across all subjects for elementary schools.  The multiple regression results for elementary schools indicate that 43% of the variance in a schools profile is attributable to status on AIMS, while only 4% was attributable to growth.  For middle schools 61% of the variance accounted for is explained by status, for high schools 71% of the variance in the school profile is explained by status alone.  Again, the problem with MAP is readily apparent in the correlation tables.  Growth on AIMS correlates higher with MAP compared to status which is reasonable given both variables are measures of academic growth.  The fact that the correlations between MAP and AIMS growth are so low especially for middles schools was again troublesome.

Table 24 shows the results of profile labels broken out by grade level.  The results show that elementary schools received a higher percentage of “Performing Plus” and “Highly Performing” labels compared to middle and high schools.

Summary and Conclusions

First, it was demonstrated that multiple regression could accurately describe the AZ LEARNS model provided all sources of variance were taken into account (Tables 5, 6, 8, & 9).  Secondly, by accounting for the unique contribution of each variable or indicator it was possible to determine an indicators relative contribution to a school’s label.  By calculating correlations between all the indicators of the AZ LEARNS model it was possible to draw some conclusions regarding how well the indicators fit together, and how they might influence one another.  It was shown that the z-score (percent of students exceeding on AIMS) had no relation to the AIMS performance score for TUSD middle schools (Table 2).  The possibility that TUSD middle schools scored too low to have the z-score become a factor was suggested as one possible explanation.  It was also demonstrated that MAP was not only unrelated to AIMS performance, it was actually negatively correlated with AIMS performance for both elementary and middle schools.  Alignment differences between the Stanford 9 and AIMS were cited as one possible explanation for the lack of relation between MAP and other measures of academic achievement.  A second possible explanation for the lack of relationship between MAP and other measures of academic growth was the fact that schools may have been awarded points based on last years MAP.  Although inclusion of MAP in the multiple regression model of AZ LEARNS had no detrimental effect on prediction (Tables 7 & 8) it may have had a detrimental effect in the AZ LEARNS model.  Multiple regression uses only the unique variance contributed by each independent variable that increases the multiple correlation (R).  If a variable contributes nothing to the model it is either dropped or given so little weight that its inclusion has little or no impact on the correlation.  In AZ LEARNS however the variables are not weighted, they are simply added into the model (MAP), and as a result may have negatively impacted the models validity.  Finally, it was shown that the AZ LEARNS label was heavily weighted by a schools current academic status, especially for upper grades.  The influence of academic status on AZ LEARNS and its corresponding relationship to demographic stress was also discussed.

¹        The Stress Index is the weighted linear combination of demographic factors that have been demonstrated to negatively impact student achievement.  The stress factors include: 1) Student Attendance Rate, 2) Suspension Rate, 3) Percentage of students classified as English Language Learners (ELL), 4) Socio-Economic-Status (SES) based on weighted free and reduced lunch counts, 5) Student Mobility Rate, and 6) Percentage of students in Exceptional Education. 

 

²        The Effectiveness Index used in this study was a statistical byproduct of the Stress Index calculation.  Stress factors were weighted according to their relative contribution to explaining variance in academic achievement (AIMS).  Once weights had been calculated actual (observed) stress values were inserted into the equation allowing prediction of AIMS performance based on the amount of stress for individual schools.  Subtracting the predicted score from the observed score yields a “residual”.  Analysis of residuals is a commonly used “value added” method for determining school effectiveness.  If a school has a positive residual (observed > predicted) the school is said to be performing better than expected given the population the school serves, and it is considered an effective school.  Conversely, (observed < expected) would be indicative of an ineffective school.  If such a method is used for “high stakes” labeling of schools it is suggested that errors in prediction be taken into account to form confidence bands.  

Correlations between AZ LEARNS Components

Table 1

Elementary Schools

Variables	Total Points	Profile	AYP	%Tested	Z-Score	AIMS	MAP
Total Points	1.00
Profile	.83	1.00
AYP	.43	.36	1.00
%Tested	.02	.10	-.14	1.00
Z-Score	.50	.65	.29	.13	1.00
AIMS	.75	.67	.26	.18	.51	1.00
MAP	.36	.20	.04	.03	-.19	.21	1.00

 

Table 2

Middle Schools

                                                                    

Variables	Total Points	Profile	AYP	%Tested	Z-Score	AIMS	MAP
Total Points	1.00
Profile	.91	1.00
AYP	.42	.36	1.00
%Tested	.47	.53	.42	1.00
Z-Score	.08	.22	-.08	-.33	1.00
AIMS	.90	.86	.20	.55	-.06	1.00
MAP	-.16	-.25	.05	-.47	.36	-.56	1.00

N = 18

Table 3

High Schools

                                                                                

Variables	Total Points	Profile	AYP	%Tested	Z-Score	AIMS	Grad/Drop
Total Points	1.00
Profile	.91	1.00
AYP	.51	.59	1.00
%Tested	-.40	-.34	-.56	1.00
Z-Score	.43	.63	.30	.35	1.00
AIMS	.99	.88	.38	-.34	.41	1.00
Grad/Drop	--	--	--	--	--	--	1.00

N = 11

Multiple Regression Analysis of Total Points in AZ LEARNS

Table 4

Elementary schools

Predictor		St. Err.		St. Err.
Variables	BETA	of BETA	B	of B	t(69)	p-level
Intercept			5.476920	1.034096	5.296335	.000001
AYP	.256065	.072014	2.493690	.701309	3.555768	.000686
AIMS	.637898	.073623	.609794	.070379	8.664397	.000000
MAP	.216108	.071140	.461457	.151907	3.037767	.003363

R = .82           R² = .65

Table 5

Middle schools

 

R = .999         R² = .999

 

Table 6

High schools

Cannot invert matrix;  All variance explained in Points

R = 1.00         R² = 1.00

Multiple Regression Analysis of AZ LEARNS School Profile

Table 7

Elementary Schools

Predictor		St. Err.		St. Err.
Variables	BETA	of BETA	B	of B	t(68)	p-level
Intercept			.030133	.461041	.065359	.948080
AYP	.121847	.077671	.480804	.306485	1.568771	.121343
Z_Score	.469504	.093029	.605799	.120035	5.046837	.000004
AIMS	.357910	.092139	.138633	.035689	3.884455	.000235
MAP	.212236	.080333	.183629	.069505	2.641949	.010220

R= .80                        R²= .61

Table 8

Middle Schools

Predictor		St. Err.		St. Err.
Variables	BETA	of BETA	B	of B	t(13)	p-level
Intercept			-2.72425	.893864	-3.04772	.009341
AYP	.182044	.103217	.46689	.264723	1.76369	.101255
Z_Score	.230155	.108025	.77663	.364520	2.13056	.052795
AIMS	.935374	.125735	.36918	.049626	7.43924	.000005
MAP	.177951	.132467	.17445	.129861	1.34336	.202136

R= .94                        R²= .84

Table 9

High Schools

Predictor		St. Err.		St. Err.
Variables	BETA	of BETA	B	of B	t(7)	p-level
Intercept			-1.37832	.667573	-2.06467	.077827
AYP	.252062	.120736	.67029	.321063	2.08771	.075224
Z_Score	.276374	.122614	.16947	.075186	2.25403	.058847
AIMS	.671469	.126539	.30899	.058230	5.30642	.001116

R= .96                        R²= .88

Correlations between School Demographics and
AZ LEARNS Components

Table 10

Elementary Schools

Variables	Profile	AYP	%Tested	Z-Score	AIMS	MAP
Reading %meet	.65	.34	.20	.79	.54	-.20
Math %meet	.65	.43	.18	.74	.58	-.14
Writing %meet	.59	.28	.27	.72	.51	-.19
Total %meet	.65	.36	.22	.78	.56	-.18
Attendance	.13	-.01	.45	.13	.16	-.11
% Suspensions	-.26	-.40	.21	-.18	-.11	.13
% SPED	.14	.02	.13	-.00	-.07	.05
% ELL	-.32	-.27	-.09	-.55	-.26	.23
SES	-.48	-.27	-.04	-.68	-.48	.14
Mobility Rate	-.34	-.20	-.16	-.52	-.45	.13
Effectiveness	.38	.30	-.28	.21	.07	.08
Stress Index	-.36	-.24	.09	-.56	-.39	.11

N = 73

Table 11

Middle Schools

Variables	Profile	AYP	%Tested	Z-Score	AIMS	MAP
Reading %meet	.74	.27	.64	-.17	.89	-.67
Math %meet	.74	.24	.64	-.26	.88	-.65
Writing %meet	.78	.25	.61	-.28	.91	-.59
Total %meet	.77	.26	.65	-.24	.91	-.66
Attendance	.81	.26	.55	-.01	.84	-.45
% Suspensions	-.60	-.30	-.42	-.09	-.38	-.03
% SPED	-.50	-.47	-.58	.45	-.54	.24
% ELL	-.61	-.38	-.62	.42	-.76	.55
SES	-.72	-.29	-.75	.38	-.85	.66
Mobility Rate	-.70	-.35	-.68	.28	-.77	.59
Effectiveness	-.01	-.28	-.47	.47	.14	-.01
Stress Index	-.75	-.33	-.75	.36	-.85	.64

N = 18

Correlations between School Demographics
and AZ LEARNS Components (continued)

Table 12

High Schools

Variables	AZ LEARNS Components
	Profile	AYP	%Tested	Z-Score	AIMS	Grad/Drop
Reading %meet	.85	.36	-.08	.61	.92	--
Math %meet	.92	.49	-.13	.73	.90	--
Writing %meet	.87	.44	-.14	.69	.89	--
Total %meet	.89	.43	-.12	.68	.92	--
Attendance	.52	.37	-.07	.59	.50	--
% Suspensions	-.58	-.11	-.21	-.63	-.63	--
% SPED	-.47	.09	-.28	-.40	-.70	--
% ELL	-.88	-.51	.31	-.53	-.83	--
SES	-.93	-.61	.36	-.57	-.81	--
Mobility Rate	-.83	-.60	.26	-.73	-.66	--
Effectiveness	.50	.05	.15	.49	.72	--
Stress Index	-.92	-.53	.22	-.66	-.85	--

N=11

Figure 1

 

Scatter Plot of Z-score & MAP score

Pairings for Elementary and Middle Schools

 

R = -.16

 

 

Predicting AZ LEARNS Labels based on
Demographic Variables Alone

Table 13

Elementary Schools

Predictor Variables		St. Err.		St. Err.
	BETA	of BETA	B	of B	t(66)	p-level
Intercept			1.974521	.834857	2.36510	.020973
Attendance	.247405	.105243	.022474	.009560	2.35081	.021729
% Suspensions	-.223584	.101033	-.024884	.011245	-2.21298	.030362
% SPED	.145045	.107036	.022802	.016827	1.35511	.180003
% ELL	.234346	.170999	.021483	.015676	1.37045	.175188
SES	-.641315	.202214	-.042992	.013556	-3.17147	.002302
Mobility Rate	-.065482	.157607	-.005900	.014201	-.41547	.679143

R= .60                        R²= .31

 

           

Predicting AZ LEARNS Labels based
on Demographic Variables Alone (continued)

Table 14

Middle Schools

Predictor Variables		St. Err.		St. Err.
	BETA	of BETA	B	of B	t(11)	p-level
Intercept			-32.4002	19.46186	-1.66481	.124145
Attendance	.650662	.341056	.3957	.20740	1.90779	.082850
% Suspensions	-.094135	.249453	-.0012	.00325	-.37737	.713080
% SPED	-.200393	.220623	-.0876	.09640	-.90830	.383186
% ELL	.015038	.418209	.0015	.04033	.03596	.971960
SES	.178320	.520196	.0138	.04020	.34279	.738214
Mobility Rate	-.260423	.299711	-.0213	.02446	-.86891	.403455

R= .86                        R²= .59

Table 15

High Schools

Predictor Variables		St. Err.		St. Err.
	BETA	of BETA	B	of B	t(4)	p-level
Intercept			45.69135	13.34334	3.42428	.026675
Attendance	-.64222	.212807	-.40990	.13582	-3.0178	.039246
% Suspensions	-.05133	.310576	-.00159	.00965	-.16530	.876729
% SPED	-.25592	.235813	-.02764	.02546	-1.0852	.338826
% ELL	.10200	.290568	.02069	.05893	.35104	.743264
SES