SOS Research Paper - Save Our Schools: Reforms to the New York City public education system have drawn praise from the Federal Minister of Education, Julia Gillard. She says that she is “impressed” by the reforms introduced by New York’s Schools Chancellor, Joel Klein, and that they are “working” and have produced “remarkable outcomes”. She says that there has been continual improvement in student achievement in New York City under Klein. However, these claims are refuted by an analysis of national and state test results for New York City. They show no general improvement in average student achievement or reductions in achievement gaps since reforms to the public education system were implemented by Klein.
SOS Research Paper - Save Our Schools
http://www.soscanberra.com/
Two aspects of student achievement are considered in this paper: average test scores and gaps in average test scores between different groups of students.
Average scores in reading and mathematics in Grades 4 and 8 in New York City have mostly stagnated since 2003, with virtually no improvements for Black, Hispanic and low income students.
1.National tests show:
2.State tests show:
1.National tests show:
2.State tests show:
Reforms to the New York City public education system have drawn praise from the Federal Minister of Education, Julia Gillard. She says that she is “impressed” by the reforms introduced by New York’s Schools Chancellor, Joel Klein, and that they are “working” and have produced “remarkable outcomes”. She says that there has been continual improvement in student achievement in New York City under Klein.
However, these claims are refuted by an analysis of national and state test results for New York City. They show no general improvement in average student achievement or reductions in achievement gaps since reforms to the public education system were implemented by Klein.
National tests in reading and mathematics show that average student achievement in New York City schools has mostly stagnated since Klein took charge while state tests show a mixture of small increases and declines.
The National Assessment of Education Progress (NAEP) tests conducted by the US Department of Education show no statistically significant change in average student scores for reading in Grades 4 and 8 between 2003 and 2007 in New York City [Lutkus, Grigg & Donahue 2007; Lutkus, Grigg & Dion 2007].1 They show a small improvement in Grade 4 mathematics but no improvement in Grade 8.
Despite Klein’s claims, there was no general improvement for disadvantaged students. There was no improvement in average reading scores for low income, Black and Hispanic students in either Grade 4 or 8. There were small improvements in average mathematics scores in Grade 4 for low income, Black and Hispanic students. In Grade 8 mathematics there was no improvement for Black and Hispanic students, but a slight improvement for low income students.
Scores for New York City students on the New York State Department of Education tests are just as unconvincing about improved achievement as the national tests. Average scores for English Language Arts across Grades 3-8 show a mixture of large and small increases and declines between 2003 and 2008 [NYC 2008a]. For example, average Grade 3 scores increased significantly while average Grade 8 scores declined substantially. Small increases occurred in Grades 4 and 5 while Grade 5 and 6 scores declined slightly.
Large improvements in average scores occurred in mathematics in Grades 3, 4 and 5 between 2003 and 2007 while there was a large decline in Grade 8 [NYC 2008b]. There was a small decline in Grade 7 and no change for Grade 6.
The pattern of increases and decreases in scaled scores for 2003-2007/8 is not dissimilar to that prior to 2003. Between 1999 and 2003, there were instances of large improvements in average scores as well as small improvements. Consequently, it is difficult to claim any substantial improvement in state test scores during 2003-2007/8 compared to the previous period.
It should be noted that there are issues about the comparability of average scaled scores on the state tests over time (see below). Also, in contrast to the NAEP tests, statistical uncertainty intervals are not reported for state tests so it is difficult to distinguish what are real improvements or declines except in those few instances of very large changes.
National and state tests also show that the achievement gaps in average scores between Black and White students, between Hispanic and White and between low income and other students have remained as large as ever under Klein’s reign.
This evidence contradicts the recent testimony to the US Congress by Mayor Bloomberg and Chancellor Klein claiming to have reduced New York City’s achievement gap [Bloomberg 2008; Klein 2008]. Indeed, Bloomberg claimed the achievement gap has been reduced by half. It contradicts claims made by the New York City Department of Education [NYC 2008c]. As one testing expert who advises the New York State Department of Education and the US Department of Education says:
This is not strong evidence that the gap is closing…The only thing you can say is that they're relatively flat, that the gap is relatively stable. [cited in Green 2008]
The NAEP reports show that there was no reduction in the gaps between Blacks and Whites, Hispanics and Whites and between low income and other students in reading and mathematics in Grades 4 and 8 between 2003 and 2007 [Lutkus, Grigg & Donahue 2007; Lutkus, Grigg & Dion 2007].2 While there were small reductions and increases in the point score gaps none were statistically significant.
The gap between all White students and low income, Black and Hispanic students in New York City remains very large. The average score of 4th grade White students in reading in 2007 was equal to or better than 61% of students nationwide while the average scores of low income, Black and Hispanic students were equal to or better than only 35%, 33% and 30% of students nationwide, respectively. In 8th grade reading the corresponding figures were 58% for White students compared to 30%, 25% and 26% for low income, Black and Hispanic students respectively. Similar gaps exist for mathematics in grades 4 and 8.
Jennings [2008a, 2008b] found that, in 2007, 77% of White 4th grade students in New York City scored above the average NAEP reading score of Black students in the City and 79% of White 8th grade students were above the average score for Blacks. In mathematics, 80% of White 8th grade students performed above the average score for Black students. The gap was similar for Grade 4 students and there were similar gaps between Hispanic and White students in reading and mathematics in grades 4 and 8.
Achievement gaps can also be considered for the same cohort of students over time. Students in grade 4 in 2003 were in grade 8 in 2007, so the achievement gaps can be compared for a similar group of students at the two points in time to see if any progress has been made in reducing them.
The NAEP reports show that the achievement gaps between Black and White students, Hispanic and White students and low income and other students who were in 8th grade in 2007 were not statistically different from those for the same cohort in 4th grade in 2003 [Lutkus, Grigg & Donahue 2007; Lutkus, Grigg & Dion 2007].
State test data on achievement gaps tends to support the national data. A recent study by Jennings & Dorn [2008; see also Jennings 2008c] found small changes in the achievement gaps in New York City on state test scores between 2003 and 2008. In most cases, the gaps in average test scores increased between Blacks and Whites and between Hispanics and Whites.
The achievement gap between Blacks and Whites in reading increased in Grade 4 but declined in Grade 8 and increased in mathematics in both Grades 4 and 8. For example, the achievement gap between Blacks and Whites in 4th grade increased by 13% in reading and 7% in mathematics. The gap for 8th grade mathematics increased by 4%, but the reduction in the gap for 8th grade reading of 12% was due to a fall in the average score for Whites.
The achievement gap between Hispanics and Whites increased in Grade 4 reading and mathematics but narrowed in Grade 8 reading and mathematics. In 4th grade, the gaps increased by 6% in reading and by 5% in mathematics. In 8th grade, the gaps narrowed by 3% in reading and by 6% in mathematics.
Klein and his Department resort to several artifices to claim that student achievement has increased in New York City since he took over the public school system. These are:
First, Klein and the New York City Department of Education often compare the 2007/08 results with those in 2002, which pre-dates the changes he implemented to the New York City public school system [Bloomberg & Klein 2008; NYC 2008a, 2008b, 2008c]. Klein became head of the New York City schools in August 2002, but his reforms were not implemented until September 2003. Thus, the appropriate starting point for comparisons on the impact of these reforms is the NAEP tests conducted in early 2003, not those conducted in early 2002. The 2002 tests were conducted some 18 months before Klein’s reforms began to be implemented.
Using 2002 as the starting point for comparing changes in student test scores instead of 2003 exaggerates improvements in student achievement. The NAEP and state test results show significant improvement between 2002 and 2003. By comparing the results for the later years with those of 2002, Bloomberg and Klein incorrectly include improvements that occurred before he made any changes to the New York City public school system and create the perception that their reforms are working.
The second artifice is not to report statistical margins of error or uncertainty intervals within which the average scores and percentages can be considered to reliably lie. Reporting margins of error is necessary in order to indicate whether a change in test scores is likely to indicate a real change.
When Klein and his Department cite NAEP results they do so without reference to the reported measurement errors or statistical confidence limits. They compare changes in average scores or percentages without regard to whether they are statistically distinguishable. The New York City Education Department has re-produced the results of the NAEP tests in a special publication [NYC 2007]. It claims significant improvements in student achievement, in contrast to the actual NAEP reports. However, it fails to report measurement errors and fails to take account of the fact that several of the results for different grades in 2007 are statistically indistinguishable from 2005 and 2003. Instead, the changes are presented as improvements without qualification.
The National Center on Education Statistics has reportedly criticised this practice, saying that Klein’s conclusions about progress on student achievement are “incomplete” and “do not take into account whether the changes or differences are statistically significant” [Green 2008]. Klein’s response was that confidence limits do not matter and that statistical significance is “playing something of a game”.
Third, Klein and his Department generally measure achievement gaps by figures on the proportion of students achieving certain standards, often called ‘proficiency rates’ [Bloomberg & Klein 2008; NYC 2007, 2008a, 2008b, 2008c]. While the percentage of students achieving different standards can convey useful information it is a misleading and inaccurate way to measure achievement gaps because it does not distinguish between students who achieve just above the standard from those achieving well above the standard. Trend comparisons are very dependent on the choice of scores for the cut-off points of different standards [Thiessen et.al. 2008: 3-4].
Large increases in proficiency rates can occur even while achievement gaps are increasing. For example, a number of low income students who have been slightly below a given standard may improve sufficiently to score slightly above a standard, thus significantly increasing the percentage of these students above standard. However, the gap between the average scores of low income and other students may increase if the average scores of other students increase by more than for the low income students.
The proficiency gap is closing even as the achievement gap stays essentially the same because each gap represents a different kind of improvement. Proficiency rates detect movements across the proficiency bar, rising when students who had been below it learn enough knowledge and skills to reach the standard, but registering no change if students who were already meeting the standard surge even further above it. The achievement gap, on the other hand, is sensitive to changes both above and below the proficiency bar. [Green 2008]
A comparison of average scores of different student group is a better indication of the achievement gap than differences in the proportions of students meeting a given standard(s). It reflects changes in the achievement of students across the spectrum and not just those who moved above or below the proficiency standard.
In any case, Klein’s claims are largely contradicted by the NAEP reports which show no improvement in the proportion of all students at or above the “Basic” standard for reading in Grades 4 and 8 and no improvement in the percentage of students at or above the “Basic” standard in Grade 8 mathematics [Lutkus, Grigg & Donahue 2007; Lutkus, Grigg & Dion 2007]. However, there was a significant improvement in the percentage of Grade 4 students at or above the “Basic” mathematics standard.
There was no improvement in the percentage of Hispanic and low income students above the “Basic” standard in Grade 4 reading, but a small improvement for Black students. There was no improvement in the percentage of Black, Hispanic and low income students above “Basic” in Grade 8 reading and mathematics. However, there was a large increase in the percentage of Black, Hispanic and low income students above “Basic” in Grade 4 mathematics.
A further issue is that Klein and other officials frequently base their claims for improvement in student achievement on state test results. There are several reasons to consider the state data to be less reliable than the national test data.
State tests are prone to “test score inflation” because of the high stakes attached to them as measures of school performance and teacher accountability [Koretz 2008]. The NAEP results are not used to make high-stakes decisions regarding the performance of individual educators or schools and there is also less room for schools to manipulate results by excluding some students, cheating or other means [Thiessen et.al. 2008].
A number of studies have found considerable discrepancies between student performance trends on state assessments and those on the NAEP [for example Jacob 2007, NCES 2007; Thiessen et.al. 2008]. There are significant discrepancies in student achievement in New York City on the state and national tests [Medina 2007, 2008; Stern 2008].
In contrast to the NAEP, the published results of New York state tests do not report measurement and sampling errors, so that it is not possible to determine whether results are statistically different over given periods or between different groups of students.
Another problem with the state test data is that it is not strictly comparable between 2003 and 2007/8. In 2006, the State Department of Education changed the way it calculated the scale scores so that they are calculated differently for 2007/8 and the years prior to 2006. This makes it difficult to determine with any confidence whether the absolute scores of any particular groups of students increased or declined over the period [VerBruggen 2008].
Despite the claims of the New York City Schools Chancellor, Joel Klein, there has been little to no progress in student achievement in New York City schools since 2003 when Klein’s reforms began to be implemented. In particular:
Klein and his Department have resorted to several artifices to claim that student achievement has increased and that achievement gaps have been reduced.
First, they often use the 2002 results as the comparison benchmark instead of 2003. The 2002 tests were conducted some 18 months prior to the implementation of the Klein reforms. The reason this benchmark is used is because there were significant increases in student achievement from 2002 to 2003, so this comparison exaggerates the impact of the reforms.
Second, they fail to report the margins of statistical error on test results. Reporting margins of error is necessary in order to indicate whether a change in test scores is likely to indicate a real change.
Third, they use proficiency gaps rather than achievement gaps to indicate changes in the gaps between groups of students. Proficiency gaps can give a false perception of differences between groups of students because they can be declining while gaps in average scores are increasing.
Fourth, they tend to cite state test data which is less reliable than the results of national tests. There are significant discrepancies between state and national test results.
Sources: Lutkus, Grigg & Donahue 2007; Lutkus, Grigg & Dion 2007
Troy Buswell and the State Government have appointed a former Howard Government industrial relations advocate to review Western Australian workplace laws and will soon announce changes that could affect up to 300,000 Western Australian workers.
Public schools are a foundation stone of our society. They are places of learning, of inspiration and opportunity where every Australian child can get the education they deserve. It's the key to a strong economy, a skilled workforce and a socially cohesive society.
Read More
Public education should be free and open to all to ensure non-selective and non-discriminatory access to education. Every child has a right to be educated, to develop as a human being. This capability should not be denied to anyone, either by design or neglect.
