The following article was published in the 2002 Yearbook of the Urban Learning, Teaching, and Research Special Interest Group of the American Educational Research Association, pp. 44-53.

_____________________________________________________________________

 

The Reading Achievement of Economically-Disadvantaged Children

in Urban Schools Using Open Court vs. Comparably Disadvantaged Children

in Urban Schools Using Non-Scripted Reading Programs

 

Margaret Moustafa and Robert E. Land

California State University Los Angeles

 

This study compares average SAT 9 reading scores of second through fifth grade English-only children in schools using Open Court with the scores of comparable schools using non-scripted programs in one very large urban school district. Findings show that schools using Open Court are significantly more likely to be in the bottom quartile than comparable schools using non-scripted programs.

  

In 1996 the reading / language arts teacher specialists on California’s Instructional Resources Evaluation Panel recommended that Open Court’s Collections for Young Scholars (1995), a program for teaching children to read, not be placed on the state’s textbook adoption list (Holland, 1996). Nevertheless, the California Board of Education overruled the Panel and placed it on the California textbook adoption list. By January 2000, Open Court, as it is commonly referred to, had grown from being used in one in every hundred to one in every eight elementary schools in California (Helfand, 2000). Since that time it has been adopted in many urban school districts in California and around the nation. When it was adopted district-wide in the nation’s second largest school district the spring of 2000, it won the largest reading textbook adoption every recorded (McGraw-Hill, 2000, p.11). 

 

Open Court’s dramatic growth in market share was aided by reports that a National Institute of Child Health and Development (NICHD) study, done by a group of researchers lead by Barbara Foorman of the University of Texas—Houston Medical Center, had found that economically disadvantaged children have better reading achievement with Open Court than economically disadvantaged children with whole language reading instruction programs. The research was presented to the California State Assembly Education Committee in 1996 and used to promote laws before it was peer reviewed (Taylor, 1998).

 

Whose views were more valid: those of the reading / language arts teacher specialists on the 1996 California Instructional Resources Evaluation panel or those of the NICHD Houston researchers? In this paper we analyze the NICHD Houston Study and other studies on the effectiveness of Open Court and report on our own study. Because the term whole language is confused with whole word by non-specialists, in this paper we will use the term contemporary reading instruction rather than the term whole language reading instruction. By contemporary reading instruction we mean reading instruction that uses text with language familiar to children and teaches children to use their knowledge of the world and the language of the text as well as the letter-sound system of the text to make sense of text.

 

The NICHD Houston Study

 

In 1994-95 Foorman, Francis, Fletcher, Schatschneider, & Mehta studied the effect of various types of reading programs on the reading achievement of low-achieving, economically-disadvantaged, first and second grade children in a school district in Texas. The programs they studied were: (1) the pre-publication version of Open Court’s Collections for Young Scholars, (2) the researchers’ adaptation of Hiebert, Colt, Catto and Gray’s 1992 program, (3) the researchers’ adaptation of contemporary reading instruction, and (4) the ongoing contemporary reading instruction in place in the district before the study began. They collected data on children in various classrooms, each with one of the four programs.

 

Foorman and her colleagues found the children in the classrooms with Open Court improved in word reading more than the children in the other classrooms. Consequently, they concluded that the “[r]esults show advantages for reading instructional programs that emphasize explicit instruction in the alphabetic principle for at-risk children” (Foorman, Francis, Fletcher, Schatschneider, & Mehta, 1998, p. 37).

 

            There are many problems with Foorman et al.’s research. Some of the most important problems are:

 

1)      The version presented to the California State Assembly Education Committee May 8, 1996 and the version published after peer review in the Journal of Educational Psychology in 1998 used considerably different data (Taylor, 1998, 333-338). Taylor, Anderson, Au, and Raphael (2000) suggest the changes may have been the consequence of peer review.

 

2)      In both versions, the sample favored Open Court (Taylor, 1998, pp. 333-338; Foorman et al., 1998, Table 1). In the published version most of classrooms with Open Court were in schools where 40-43% of the children were on the federal lunch program. In contrast, most of the classrooms with the adapted Hiebert et al. program were in schools where 64-65% of the children were on the federal lunch program; most of the classrooms with the adapted contemporary reading instruction were in schools where 50-64% of the children were in the federal lunch program; and, most of the classrooms with the ongoing contemporary instruction were in a school where 71% of the children were on the federal lunch program (Foorman et al., 1998, Table 1). The school that had 71% of the children on the federal lunch program also had the lowest achievement scores on the statewide test in Grade 3 (Foorman et al., 1998, p. 39).

 

 

3)      In both versions, the children receiving Open Court instruction had higher average pre-test scores on word reading and phonological processing than the children in each of the three comparison groups at both grade levels (Taylor, 1998, pp. 327-338; Foorman et al., 1998, pp. 43-44).  We analyzed the data provided by Taylor (1998, pp. 327-338). In the prepublication version two-tailed t-tests show differences were statistically significant at the .05 level or less for word reading. In the published version pre-test differences are less pronounced, but still favor the Open Court group in every case.

 

4)      In both versions the data were incorrectly aggregated. The data on the children who had been in the district’s classrooms with contemporary reading instruction when they were in first grade but in the Open Court classrooms when they were in second grade were averaged with the data on the children who were in the classrooms with Open Court (Foorman et al., 1998, Tables 3 & 4), making it impossible to distinguish the effect of each program.

 

5)      In both versions the researchers equated pronouncing print words with reading. That is, they assumed that if a child can pronounce a print word the child is “reading,” regardless of whether the child is making sense of the print or not.

 

            At the end of the six-month study, despite their lower pre-test scores and their greater economic disadvantage, the children in the classrooms with the ongoing contemporary reading instruction answered comprehension questions about text they had read silently better than the children in the Open Court classrooms (Foorman et al., 1998, Table 5). If one defines reading as making sense of print, one would have to conclude from the NICHD Houston study that contemporary reading instruction is more effective than traditional, parts-to-whole reading instruction. This reanalysis of Foorman et al.’s findings is consistent with independent research that has found that all children, but especially lower-achieving children, achieve more with contemporary reading instruction than with traditional reading instruction (e.g., Anderson, Wilkinson, and Mason, 1991; Cantrell, 1999; Eldredge, Reutzel, and Hollingsworth, 1996; Mullis, Campbell, and Farstrup, 1993; Reutzel and Cooter, 1990; Sacks and Mergendollar, 1997).

 

  Despite its flaws, the NICHD Houston research was used to justify policy and legislation that excluded contemporary approaches to reading instruction and favored the purchase of Open Court, especially in schools serving poor families (Coles, 2000; Taylor, 1998; Taylor et al., 2000). Teachers who are required to use only Open Court for reading instruction consistently report that instruction in writing, math, science, and social studies has declined dramatically, if they are taught at all.

 

Newspaper Studies

 

In June of 1999 Debra Saunders of the San Francisco Chronicle wrote about Sacramento City Unified School District’s almost exclusive use of Open Court. She reported that since signing on with Open Court and receiving a grant from the David and Lucile Packard Foundation for teaching coaches, “…[SAT 9 reading] scores for the Sacramento City Unified School District rose from the 35th percentile nationally in reading for first-graders [in 1997, before Open Court was used] to the 54th percentile [in 1998, the first year Open Court was used] to the 62nd percentile [in 1999, the second year Open Court was used]” (Saunders, 1999).

 

Similarly, in August of the same year, Duke Helfand of the Los Angeles Times wrote about Sacramento’s success with Open Court as measured by the SAT 9. He reported “The primary grades led the way, with second-graders making the largest gains: 15 percentile points in reading compared with 4 points among their counterparts across the state” (Helfand, 1999, p. A1). 

 

In their analyses of Open Court’s success in Sacramento, Saunders and Helfand compared Sacramento’s scores in one grade one year to scores in the same grade the next year. However, when we follow the children from one grade one year to the next grade the next year we see a different picture.

 

If we follow Sacramento’s children from one grade to the next we see that while Sacramento’s children averaged in the 35th percentile in first grade in 1997, they averaged in the 35th percentile in second grade in 1998 and in the 37th percentile in third grade in 1999. While Sacramento’s children averaged in the 54th percentile in first grade in 1998, they averaged in the 50th percentile in second grade in 1999 and in the 42nd percentile in third grade in 2000. Finally, while Sacramento’s children averaged in the 62nd percentile in first grade in 1999, they averaged in the 52nd percentile in second grade in 2000 and in the 43rd percentile in third grade in 2001.

 

Similarly, if we follow California’s children from one grade to the next we see that the state’s children averaged in the 39th percentile in second grade in 1998 and in the 40th percentile in 1999. Hence, between 1998 and 1999 children in Sacramento who were in second grade in 1998 and in third grade in 1999 gained only one percentile point more than the state’s children. Parenthetically, we note that regardless of the fact Sacramento’s average second grade scores were higher than the state’s in 1999, 2000, and 2001, this early advantage has never translated into higher scores in subsequent grades. In every year SAT 9 had been administered state wide, 1998, 1999, 2000, and 2001, Sacramento’s third, fourth, and fifth grade scores have been lower than the state’s.

 

The Treadway Study

 

After we made a working draft of our study available to others via the web, Jerry Treadway (2000), an author of Open Court, did a study on the effectiveness of Open Court and made his study available on the University of Texas—Houston, Center for Academic and Reading Skills (CARS) website. In his study, Treadway compared the average 1999 SAT 9 reading scores of 10 schools that use Open Court with those of 10 other schools in the same district we studied and concluded that the average SAT 9 reading scores of the 10 schools using Open Court was higher. However, there are methodological flaws in Treadway’s study:

 

·        Treadway selected 20 of over 400 schools in the district, without explaining his selection criteria, making it impossible to know why he chose the particular 20 schools.

 

·         The study was not a comparison of Open Court vs. non-scripted programs. District records show that two of Tredway’s 10 comparison schools were using Open Court and three were using Success for All, another scripted program.

 

·        The study used meaningless data. It used the SAT 9 reading scores of all students, English-only and limited-English proficient, without reference to how long the later group had been in English speaking schools. We elaborate on this problem below.

 

Our Study

 

Our research asked two questions:

 

(1)   Does Open Court foster higher reading achievement than non-scripted reading programs initially among economically disadvantaged children?

 

(2      Does Open Court foster higher reading achievement than non-scripted reading programs generally among economically disadvantaged children?

 

Traditionally scripted and non-scripted instructional programs are distinguished by the amount of professional judgment teachers are allowed to exercise in teaching. Traditional scripted programs provide teachers with a script for what they are to say verbatim during instruction. Non-scripted programs describe activities, provide examples and expect teachers to the choose activities that they judge to be most helpful to particular groups of students in their care.

 

The teachers’ manual of the 1995 edition of Open Court examined in this study looks like that of a non-scripted program. However, in every district-wide adoption we are aware of in California, the state in which our study took place, teachers are required to complete every activity described in the teachers’ manual with the entire class, whether it is appropriate or not, and to do it at a prescribed pace (i.e., so many lessons within so many days), whether it is appropriate or not. Michael Bazeley (2000) of the San Jose Mercury said “As one of the most strictly orchestrated programs on the market, Open Court is helping bury a notion central to public education for decades—that the classroom revolves around the teacher as a trained professional in control of what happens and when. With Open Court, teachers are told what to do from the minute class starts.” Debra Anderluh (1998) of the Sacramento Bee wrote that Open Court “lasts two to three hours a day, with half that time spent in whole group instruction. The detailed scripting, combined with the over sight of roving coaches, means that from school to school, teachers are presenting basically the same lesson at a given grade level in a given week.” We have observed teachers being on the same lesson in a given week from district to district. Hence, as currently being implemented, we see the 1995 and 2000 editions of Open Court as a new form of scripted instruction.

 

Method

 

We used the 1999 average SAT 9 reading achievement scores of English-only children to measure the success of Open Court vs. non-scripted programs.

 

Using SAT 9 scores to compare the success of one reading program with the success of other reading programs is problematic. SAT 9 is not a criterion-referenced test but a norm-referenced test. Norm-referenced tests are constructed so that 50% of the test takers will score in the bottom 50%, regardless of their mastery of the subject. Furthermore, norm referenced tests rarely include items that most children get right (Popham, 1999).

 

 In California where our study took place, another problem in using the SAT 9 to assess the effectiveness of reading instruction is that the SAT 9 norms were developed with a population where 1.8 percent of the children were limited English proficient. Over 25% of California’s K-12 children are limited English proficient with higher percentages in the lower grades. Hence, schools with large numbers of limited English proficient children will compare unfavorably to SAT 9’s norming group as well as to other California schools with lower numbers of limited English proficient children. Moreover, the scores of limited English proficient children, as presently reported, are meaningless. Immigration occurs at all ages. It takes a minimum of five to seven years for non-native speakers of English to achieve in English at levels equivalent to native speakers (Collier, 1989). It’s impossible to discern progress—or lack of progress—from data that combines the scores of children who have been in U.S. schools a few years with children who have been in U.S. schools many years.

 

 There are other problems with using SAT 9 scores to assess the effectiveness of reading programs. SAT 9 scores do not tell us if the teachers at each school are fully credentialed or not, if they are teaching the programs as the programs were designed to be taught or not, if they are providing instruction above and beyond the programs or not, if they are spending significantly more time in reading instruction or not, or if they are receiving significantly more assistance from aides, volunteers, or tutors or not. SAT 9 scores also do not tell us if the children at each school are economically disadvantaged or not, if they are the same children that were at that school the year before or not, if they had been at that school the entire year or not, or if significant numbers of children were excluded from taking the test by being placed in Special Education or not.

 

Nevertheless, given today’s high stakes testing where administrators, teachers and school staffs are rewarded or punished on the basis of their SAT 9 scores, an assessment of how successful various programs are in enabling students to do well on the SAT 9 reading assessment may be of interest to educators caught up in high-stakes testing.

 

 We chose the 1999 SAT 9 reading data because, at the time we began our work, it was the first year where the state reported English-only children’s scores disaggregated from those of limited English proficient children. Also, 1999 was the second year the SAT 9 was administered statewide. Presumably, classroom teachers who see the test when they administer it to their students probably had not yet figured out by 1999 that the questions on the SAT 9 are the same year after year and would have been less prepared to teach to particular test items than they were in subsequent years.

 

Working with the 1999 average SAT 9 reading scores of English-only children and the percent of children on free / reduced-price meals obtained from the California Department of Education’s web site, we investigated the effectiveness of Open Court schools against non-scripted programs in schools serving similar economic groups in one very large urban school district in California. The non-scripted programs used across schools in the district in 1998-99 were Invitations to Literacy (Houghton Mifflin Co.), Literacy Places (Scholastic, Inc.), Signatures (Harcourt-Brace), and Spotlight on Literacy (McGraw-Hill).

 

We limited our study to schools on the traditional calendar in the district. We further limited our study to schools that either (1) used Open Court not in combination with another program or (2) used one of the non-scripted programs used across the Los Angeles Unified School District in 1998-99 not in combination with other programs.

 

Finally, our study was limited to second through fifth grade scores. Second grade is the earliest grade in which scores are publicly reported and fifth grade is the highest elementary grade in the district we studied.

 

 We found 159 elementary schools in the district that met the research criteria for inclusion in the study. Of these, 21 used Open Court. Nine of the Open Court schools had been using Open Court over ten years. All the Open Court schools served populations where 50% or more of the children were on free / reduced-price meals. Hence, we further limited the schools in the study to schools serving populations where 50% or more of the children were on free / reduced price meals. This left 153 schools in the study. Table 1 shows the number of schools in the study by type of program and economic disadvantage of the children.

 

Table 1

Numbers of Schools by

Type of Reading Program and Percent Children Receiving Free / Reduced-Price Meals

 

 

Children Receiving Free / Reduced-Price Meals

Reading Program

50-59%

60-69%

70-79%

80-89%

90-100%

  Non-scripted programs

12

10

43

24

41

  Open Court           

  1

  1

  4

  4

  2

  Long-term Open Court

 

 

 

 

  9

 

            Among 9 of the schools using non-scripted programs with 95–100% of the children on free / reduced-price meals, some grades had too few English-only children for the scores to be reported by the state.

 

            To investigate whether Open Court fosters higher reading achievement initially among economically disadvantaged children, we compared the average second grade scores of the schools in the study. In this comparison we sorted the scores into one of three groups: those below the 25th percentile, those at the 25th to 49th percentile, and those at the 50th percentile or higher. Four of the 153 grades in this comparison had too few English-only children for the scores to be reported.

 

 To investigate whether Open Court fosters higher reading achievement generally among economically disadvantaged children, we compared the average scores of all the grades, second through fifth grade, of long-term Open Court schools with the scores of schools using the non-scripted programs that had the same percent of children on free / reduced-price meals. The long-term Open Court schools in the district we studied serve populations where 97-100% of the children receive free / reduced-price meals. Therefore, we compared the scores of the long-term Open Court schools with the scores of schools using non-scripted programs where 97-100% of the children received free / reduced-price meals. As shown in Table 2, the groups were comparably disadvantaged. The average long-term Open Court school in this group had 98.3 children receiving free / reduced-price meals. The average school using a non-scripted program had 98.5 children receiving free / reduced-price meals.

 

Table 2

Numbers of Schools with 97-100% of the Children on Free / Reduced-Price Meals by

Type of Reading Program

 

  

Children Receiving Free / Reduced-Price Meals

Reading Program

 97%

       98%

99%

100%

  Non-Scripted Programs

4

7

5

5

  Long-Term Open Court

4

1

1

3

 

Only one school in the all-grades comparison (a school using a non-scripted program) had average scores at 50% or higher. Hence, in this comparison we sorted the scores of each grade into one of two groups: those with average SAT 9 reading scores in the bottom quartile on the SAT 9 and those with average scores above the bottom quartile. Nine of the 120 grades in this comparison had too few English-only children for their scores to be reported.

 

The fifth-grade score of one long-term Open Court school was omitted in the all-grade comparisons because the score was an outlier. While the second grade children in this school averaged in the 28th percentile and the third and fourth grade children averaged in the 17th percentile, the fifth grade children averaged in the 62nd percentile, a difference far greater than in any other school.

 

Findings

 

            Second grade comparison. Table 3 shows the number of schools with average second grade English-only SAT 9 reading scores in the second, third, and fourth quartile. The percentage of children in each quartile is about the same for schools using Open Court and the non-scripted programs. A 2x3 Chi2 shows no significant difference in the average scores of the schools using Open Court and the average scores of the non-scripted programs (p=.545).

 

Table 3

Numbers of Schools with Average Second Grade English-Only SAT 9 Reading Scores

in the Second, Third, and Fourth Quartile

 

  

Average SAT 9 Reading Scores

Reading Program

0-24%

25-49%

50% and above

  Non-Scripted Programs

16 (13%)

94 (73%)

18 (14%)

  Open Court                   

  4 (19%)

13 (62%)

  4 (19%)

Chi2 (df2) = 1.214, p = .545

 

Among just the schools where 97-100% of the children are on free / reduced-price meals, the average SAT 9 reading scores for second grade English-only children was 33 among the 18 schools using non-scripted programs and 31 among the 9 long-term Open Court schools. Again, the difference is not statistically significant: t(df25)=-507, p =.617.

 

            All-grades comparison. As shown in Table 4, in schools where 97-100% of the children were receiving free / reduced-price meals, grades in schools using the non-scripted programs scored above the bottom quartile almost twice as often as grades in long-term Open Court schools. While 72% (54 out of 75) of the grades in the schools using non-scripted programs scored above the bottom quartile, only 43% (15 out of 35) of the grades in the long-term Open Court schools scored above the bottom quartile. A 2x2 Chi2 shows that this difference is statistically significant at the .01 level.

 

Table 4

Numbers of Grades with Average English-Only SAT 9 Reading Scores

In and Above the Bottom Quartile

In Schools Where 97-100% of the Children Receive Free / Reduced-Price Meals

 

 

Average SAT 9 Reading Scores

Reading Programs

In bottom quartile

Above bottom quartile

   Non-Scripted Programs

21 (28%)

54 (72%)

   Long Term Open Court

20 (57%)

15 (44%)

Chi2 (df1) = 8.669, p<.01

 

Discussion.

 

We found no evidence that Open Court fosters higher early reading achievement among economically disadvantaged children. At the second grade level, the earliest grade level reported publicly, we found no significant difference in the average SAT 9 reading scores between schools using Open Court and schools using the non-scripted programs serving similarly disadvantaged children.

 

We also found no evidence that Open Court fosters higher reading achievement generally among economically disadvantaged children. When we looked at the reading scores of all the elementary grades that are publicly reported (second through fifth grade), we found schools that had used Open Court 10 or more years were significantly more likely to be in the bottom quartile of the SAT 9 than schools using non-scripted programs serving similarly disadvantaged children.

 

In other words, we found no support for the assertion that implementing Open Court leads to higher SAT 9 reading scores. We further found no justification in sacrificing instruction in other curricular areas to implement Open Court.

 

The findings of this study are consistent with the findings of our reanalysis of children’s reading achievement in Open Court vs. the contemporary reading instruction in the NICHD Houston study in Texas. Altogether, the outcomes in the two school districts suggest that Open Court limits what children are able to achieve in reading relative to what they are able to achieve via many other programs. The outcomes support the professional judgment of the reading / language arts teacher specialists on California’s 1996 Instructional Resources Evaluation Panel who recommended Open Court not be placed on California’s textbook adoption list.

 

Acknowledgements

 

We are indebted to Stephen Krashen and Denise Ross for comments that enhanced this study.

 

References

 

Anderson, R.C., Wilkinson, A.G. and Mason, J.M. (1991). A microanalysis of small-group, guided reading lesson: Effects of an emphasis on global story meaning Reading Research Quarterly, 26, 417-441.

Anderluh, C. (1998). Building with words: City schools hope phonics boosts reading. Sacramento Bee, Feb 1.

Bazeley, M. (2000). Method seems to lift scores. San Jose Mercury News. May 29.

Cantrell, S.C. (1999). Effective teaching and literacy learning: A look inside primary classrooms. The Reading Teacher, 52 (4), 370-378.

Coles, G. (2000). Misreading Reading: The bad science that hurts children. Portsmouth, NH: Heinemann.

Collier, V.P. (1989). How long? A synthesis of research on academic achievement in a second language. TESOL Quarterly 23 (3): 509-531.

Eldredge, J.L., Reutzel, D.R. and Hollingsworth, P.M. (1996). Comparing the effectiveness of two oral reading practices: Round-robin reading and the shared book experience. Journal of Literacy Research, 28 (2), 201-225.

Foorman, B.R., Francis, D.J., Fletcher, J.M., Schatschneider, C. and Mehta, P. (1998). The role of instruction in learning to read: Preventing reading failure in at-risk children. Journal of Educational Psychology, 90 (1), 37-55.

Helfand, D. (2000). Decades later, frustrated father is phonics guru. Los Angeles Times, Orange County Edition, Jan. 10, A1 & A16.

Helfand, D. (1999). Sacramento gets high marks in school reform. Los Angeles Times, Orange County Edition, Aug. 22, A1 & A42.

Holland, B. (1996). No More Wright Group or Rigby?? An open letter sent by the 1996 President of the California Reading Association’s Inland Empire Reading Council to California Reading Association Council members and concerned educators.

McGraw-Hill (2000). This is the company…: Annual report 2000. New York: The McGraw-Hill Companies.

Mullis, I., Campbell, J., and Farstrup, A. (1993). NAEP 1992 Reading Report Card for the Nation and the States. Washington, D.C.: National Center for Education Statistics.

Popham, W.J. (1999). Why standardized tests don’t measure educational quality. Educational Leadership, March, 9-15.

Reutzel, D.R. and Cooter, R.B. (1990) Whole language: Comparative effects on first-grade reading achievement. Journal of Educational Research 83, 252-257.

Sacks, C.H. and Mergendoller, J.R. (1997). The relationship between teachers’ theoretical orientation toward reading and student outcomes in kindergarten children with different initial reading abilities. American Educational Research Journal 34 (4), 721-739.

Saunders, D. (1999). Eureka! School reform that works. San Francisco Chronicle. June 29.

Taylor, D. (1998). Beginning to Read and the Spin Doctors of Science. Urbana, IL: National Council of Teachers of English.

Taylor, B., Anderson, R.C., Au, K.H., and Raphael, T.E. (2000). Discretion in the translation of reading research to policy. Educational Researcher, 29, (6) 16-26.

Treadway, J. (2000). The research base of Open Court and its translation into instructional policy in California. Houston, TX: The Center for Academic and Reading Skills (CARS), <http://cars.uth.tmc.edu/debate/treadway.htm>