Well, now. Apparently that last post struck some sort of nerve, as it resulted in a record one day traffic spike at Iowahawk (don't like to brag, but it was up there in the multi-dozens). It also resulted in an email tsunami. The correspondence included a few critiques, some thoughtful, some stupid. Let's dispense with the stupid one first.
What qualifies you as a statistician?
I slept at a Holiday Inn Express last night.
I was amused by a couple of emails asking for my credentials, and/or pointing out Paul Krugman's sinecure at Princeton - as if this had any bearing on the validity of either of our conclusions. I may be a random internet drunk, but I am occasional cogent enough to recognize the ol' appeal-to-authority gambit. Unlike Mr. Krugman I am happy to cite/link to my source data, both in the previous post and in this one. It's a courtesy I learned at my beloved alma mater, Ottumwa Body & Fender (Southeast Iowa's finest dent repair institution). But hey, if credentials and oak-framed vellum degrees are your bag, let me share this email with you:
Dear Mr. Burger:
I edit educationnext.org. I have a blog on the site. I would like to do a blog that will depends heavily on your material,quoting you at length, as I also think Krugman is a nobel prize winning fraud and because your data are intrinsically interesting... I will link the piece to your site, obviously.
Paul E. Peterson
Henry Lee Shattuck Professor of Government
Director, Program on Education Policy and Governance
Harvard University
Senior Fellow, Hoover Institution
Degrees-y enough for you? Despite getting my name wrong, I accepted Prof. Peterson's request and encouraged him to go at my results hammer-and-tongs. His comments are here.
As for Mr. Krugman, I'll only note the remarks of his former ombudsman at the New York Times, Daniel Okrent:
"Op-Ed columnist Paul Krugman has the disturbing habit of shaping, slicing and selectively citing numbers in a fashion that pleases his acolytes but leaves him open to substantive assaults."
No shit, Sherlock. Now to the numerical issues brought up by readers.
The same site (NAEP) shows Wisconsin overall test scores are better than Texas overall test scores. (a) How in the world could Texas beat Wisconsin for every ethnic group and still be worse overall? (b) Why aren't you focusing on the most meaningful number, the overall average?
I am dumbfounded (derrrr...) by the number of people still confused by this. The short version, as I pointed out, is that Texas and Wisconsin have very different ethnic populations, and that the component groups have different means. Several emailers noted that this is a fairly common phenomenon in statistics known as Simpson's Paradox. Let's take a baseball example; suppose you were a manager and you were evaluating two hitters:
Hitter A:
Against right-handed pitchers: 300 at-bats, 90 hits (.300 average)
Against left-handed pitchers: 200 at-bats, 50 hits (.250 average)
Total: 500 at-bats, 140 hits (.280 average)
Hitter B:
Against right-handed pitchers: 100 at-bats, 32 hits (.320 average)
Against left-handed pitchers: 300 at-bats, 78 hits (.260 average)
Total: 400 at-bats, 110 hits (.275 average)
Hitter B has a higher batting average against both righties and lefties, but Hitter A has a higher overall average by dint of facing a different mix of pitchers. Now comes the question: it's the bottom of the 9th, two out, and you need a base hit. Who would you insert as a pinch hitter, A or B? The detailed data suggests Hitter B, irrespective of whether the pitcher was right- or left-handed. The overall average, in this context, is worse than meaningless - it leads you to exactly the wrong conclusion.
You're cherry-picking data to support a pre-conceived hypothesis. Why did you focus on NAEP, which is one of a number of data sets, and ignore statewide SAT/ACT scores and dropout rates which lead to an opposite conclusion?
Let's parse this one, because it's a more substantive criticism. First, why did I focus on the NAEP data? Because it is the most comprehensive, comparable and detailed data available for US education performance. It is administered at the same time, nationwide, to common age cohorts, and is designed to measure public school performance against defined standards in reading, math and science. NAEP testing is administered and compiled by the U.S. government, so it is flatly inane to insinuate it has a right wing ideological bias against Wisconsin. It's also the prefered data for academic researchers. If you want to argue there is a better yardstick for comparing state-to-state educational achievement, please feel free to do so - but only if your proposed alternative is more comprehensive, detailed and unbiased.
Now, with regard to SAT/ACT:
Let me first state my conclusion: the SAT/ACT is a garbage measure for comparing state-level public school achievement, even after controlling for ethnicity. Aha! He's dismissing it because it contradicts his conclusion! No, and here are the various reasons why.
1. There is no such thing as an "ACT/SAT" or an "SAT/ACT." There is an ACT, and there is an SAT*. Traditionally the ACT is the exam used by universities and colleges in the Midwest and Great Plains, the SAT on the coasts and Texas. The geographic lines have blurred over time, but in general which test a student takes depends on where they intend to go to college. In the Midwest, where I grew up, you took the ACT if you planned to go in-state or to a neighboring state. The very few people who took the SAT did so because they planned to go to a highly selective college on the East or West coast. That Midwest SAT self-selection is a big reason why states like Iowa and Wisconsin are chronically the top of the "average SAT" rankings - because only a few percent of the state's high school graduates, and usually the top achievers, take it.
*There is also the BAT (Bondo Aptitude Test) for prospective students at my beloved alma mater, Ottumwa Body & Fender.
2. As others have noted participation rates for SAT and ACT vary widely by state, and there is a negative correlation between a state's test participation rate and its average score. Case in point: Maine is the worst scoring state on the SAT. Whuh-uhhh? Lily-white Maine? The principal reason is that SAT testing is compulsory for Maine HS graduates, college-bound or not. In most all other states SAT is optional; in Iowa (ranked #1) only 3% of graduates took it. Similarly there are a number of states where ACT is compulsory (CO, IL, KY, MI, TN, WY) and they are, unsurprisingly, near the bottom.
The point here is that college board exams are sometimes optional tests taken by college-bound seniors, sometimes a standard requirement for anyone graduating high school. This also begs the question - what about all those high school graduates who aren't immediately going to college, and haven't been board tested? Are they going to a job? The military? How well are they prepared educationally for those careers? Public elementary and secondary schools educate all kinds of students, only about half of whom go on immediately to 4 year colleges. The ACT and SAT only give a weak indicator for a portion of those public school graduates.
3. Even ignoring those previous issues, statewide ACT and SAT scores include both public and private high school graduates. As much as I'm sure Wisconsin's teachers unions would love to claim credit for the performance of the state's private school graduates, my interest was in comparing the performance of public school education between Texas and collective bargainin' Wisconsin.
Again, here is another opportunity for Simpson's Paradox. The proportion of private school enrollment varies considerably by state; about 14% in Wisconsin, about 7% in Texas. Wisconsin's is among the highest, while Texas is among the lowest. A cynic might wonder what it is about Wisconsin public schools that compels parents there to dig deep into their pockets to send their own kids elsewhere. Knowing a little about the two states' cultures, I instead believe it's largely a Catholic thing. For example, Austin TX and Milwaukee WI are about the same population size; Milwaukee has at least 8 Catholic high schools with over 5000 enrollment, where Austin has 4 small boutique Catholic high schools with less that 800 enrolled. But the point is, a lot of the kids taking the ACT and SAT in Wisconsin are from private schools. I can't find a separately reported number, but some studies suggest that private schools have lower dropout rates and that private school graduates are much more likely to matriculate to 4-year college (presumably requiring an SAT or ACT test). Taken together, if the national pattern holds in Wisconsin it suggests that more than 20% of the state's ACT and SAT takers are private school graduates (by contrast, in Texas it would be around 10%). Oh, and by the way -- private school graduates outperform public school graduates on the SAT.
So let's take that all together: if 20%+ of Wisconsin ACT/SAT takers are private school graduates, and private school graduates are relatively high performers, to what degree can unionized Wisconsin public schools take credit for a happy "state average" result? This is precisely why I used NEAP statistics: common definitions, universality, public school focus, etc. I would be excited to see a comparison between Texas ACT/SAT scores, only if you were to control for (1) participation rate, (2) ethnicity and other socioeconomic factors, and (3) attendance at private vs. public school. If you can show that Wisconsin public schools consistently outperform Texas public schools after controlling for those differences, then you've contradicted my results.
And for those thinking I'm still evading the ACT/SAT question, may I present the ethnically-controlled 2010 ACT test results (h/t reader Larry Tunnell):
Average ACT Composite Score 2010
White students: Wisconsin 23.5, Texas 23.3 (national 23.1)
Black students: Texas 17.6, Wisconsin 16.9 (national 17.5)
Hispanic students: Wisconsin 19.8, Texas 18.7 (national 19.4)
Ignoring all the factors I previously mentioned, there are two things at odds with my NEAP results: white Wisconsin students slightly outperform white Texas students, and that the performance of Hispanic Wisconsin students and Texas students is reversed. But participation rates are much different in the 2 states, with only 62% of Wisconsin Hispanic students taking the ACT core plus, versus 83% in Texas. Keep in mind these results are (1) for all students, public and private, where private school is much more prevalent in Wisconsin; (2) only for the ACT; (3) among a self-selected group of students planning to enroll in 4-year college. Because of these confounds, the results are virtually meaningless as a measure of Texas public education versus Wisconsin public education - even the result for black students, which is completely consistent with my NAEP comparison.
Now, onto dropout rates.
I had a few people email accusing me of "ignoring" the hideous dropout rate in Texas, reported by Krugman as over 38%. In no case have I seen a source citation for this number, which appear to have sprung, like the Goddess Venus, fully formed from a mythical clamshell.
A couple of notes on this. First, it appears that different states have traditionally used different definitions of "dropout." Some are more indicative of simple attrition - for example if High School X has an incoming freshman class of 200 students, and only 130 of those freshman graduated with the class 4 years later, it had a 35% "dropout rate" - regardless of the reason. Other states were more dilligent in tracking where the lost students went (GED, transfer to private schools, work, jail, etc.) before counting them as true "dropouts." As a result traditional state-specific dropout rates were hard to compare. That's why in my update to the last post I focused exclusively on the commonly-defined NCES dropout tables. You may quibble with their definition, but it is consistent and applied the same across different states. As an aside, reader Dr. William Borland (Principal Research Engineer, Georgia Institute of Technology, lah-tee-dah) points out that 2010 state-specific public high school dropout rates are now available- and bolster my case.
2010 Public High School Event Dropout Rates
White students: Wisconsin 1.4%, Texas 1.8% (national average 2.8%)
Black students: Texas 6.3%, Wisconsin 7.8% (national average 6.7%)
Hispanic students: Texas 5.3%, Wisconsin 5.4% (national average 6.0%)
While no dropout event is good, Texas is hardly the outlier national shame claimed by Krugman. In fact, it has below national average dropout rates for all 3 ethnic groups considered, consistently in both 2007 and 2010 measures. Among white students, Wisconsin had the second lowest state event dropout rate (NJ #1), where Texas was tied for 7th. Among black students, Wisconsin was #39, Texas tied for #24. Among Hispanic students, Wisconsin was tied for #21, Texas was tied for #17.
Again these are based on a one-year calendar, based on continued enrollment from one October to the next. I will note there is a difference in kind between the NCES definition of "event dropout rate" (leaving school during a given year) and "graduation rate" (a common class cohort from freshman year to senior year). Using the common NCES definition for the 2007-8 public school senior class, Texas had a 73.1% graduation rate, where Wisconsin's was 89.6%.
So Wisconsin ranked #1 on the average graduation rate measure, where Texas came in #35. Not great for Texas, but I would note this overall number does not control for ethnicity (I have been unable to find the relevant graduation rate breakdown). As an aside I would also note that even without such controls, Texas still outranks lavishly unionized California and NY. The curious thing here is the seeming parodox between event dropout rates and graduation rates. Even with compounding and taking into account population demographics, the overall graduation rate gap between Texas and Wisconsin seems much larger that what would be suggested by the year-to-year event dropout rate (remember, both are based on NCES standard definitions). My hunch - and it's just a hunch - is that the effect of family mobility is at play here. In order to track dropout rates, researchers have to track enrollment status of individual students over time. This also involves issues of privacy, and/or losing track of students when they move. Texas is a key entry point for immigrants from Mexico and Central America, many of whom later move on to other U.S. states. Once an immigrant student enters the Texas public school system, it is one thing to track their enrollment status from year to year; it is another thing to track their enrollment status from freshman to senior year, especially if they have moved with their family to, say, Nebraska. It's unclear to what degree this plays a role in Texas' dropout vs. graduation rates, but if it does the shorter-period event dropout rates are a superior measure of true student retention.
Why the focus on Texas vs. Wisconsin? What about other non-union / non-collective bargaining states?
Because I'm lazy. And because I have dilligent readers like Michael Pollard who volunteer to do that for me. The short recap to Michael's NAEP results is that after controlling for ethnicity, compared to the running-dog Gang of Five non-collective bargaining states (TX, VA, SC, NC, GA), Wisconsin is a (1) middling performer for white students; (2) below middling for Hispanic students, and (3) an absolute disaster for black students.
Hey, it's been a fun two days based on a simple 30-minute study of educational statistics. As regards the effect of teacher collective bargaining on student learning, I wouldn't call what I did conclusive; just pointing out the fallacy of aggregate statistical comparisons. For a definitive study of the effect, I would point to Caroline Hoxby's (Harvard/ MIT /Stanford, lah tee dah) 1996 QJE paper, which statistically controls for additional variables. Her main conclusions: collective bargaining increases the input provided to schools (spending, construction and the like), but actual decreases school output (test scores and the like). If you don't like Greek letters, here's Hoxby discussing the effect on YouTube.
Anyhoo, class dismissed. I'm off to pound out a dented quarterpanel.
