Yet another set of Data Gymnastics

I just read an article that revealed how the US News and World Report’s best high schools in the country (at getting kids ready for college) are identified. As usual, I was beside myself at the naivety of the data gymnastics involved.

The researchers measured whether a school performed better than average on the state exam, including disadvantaged groups compared to other disadvantaged groups at other schools. Then they take the weighted average of the AP participation rate and performance, as well as the ratio of students passing the AP exams over the number of graduating seniors, do a little index magic and outcomes the praised number.

This, of course, has very little to do with college readiness. The ratio of passed AP exams to the total graduating class directly measures only the number of students signing up for AP classes with the confidence and money to take the exam.

While there is reason to champion the AP program as well as the AP exam, it is only a standardized exam that measures content knowledge with (as every multiple choice exam) mild treatment to analytical ability, reading comprehension, and form writing.

The AP exam cannot measure a students’ habits in the absence of supervision, nor can it measure a students ability to solve comprehensive problems that cannot be contained within a few paragraphs of text. It cannot measure knowledge of research procedure, or of self-monitored study-skills. While a commitment within the structure of high school and parent led homes can be inferred, true motivation nor an intrinsic love for learning cannot.

The AP exam is heralded simply because its hard data, but college readiness lies in much more qualitative assessments. With some serious effort on the part of policy makers and administrators, perhaps we can start touching on that too one day.