Braille Monitor                                      October 2016

(back) (contents) (next)

Educational Assessments, Math Innovations, and Real Accessibility:Progress at Pearson

by Jon Twing

Jon TwingFrom the Editor: Getting a fair shake in education means the opportunity to learn, and often that opportunity is conditioned on how the blind do on standardized tests. Sometimes the blind have been forbidden from taking them, the argument being that any accommodations made to the test would invalidate it. Accommodations which have been opposed by the industry include putting the test in Braille, making it available so it can be read using a screen reader, or increasing print size so it can be viewed by those who rely primarily on remaining vision to read and write.

One of the largest companies involved in creating and administering standardized tests is Pearson, so the progress we have made with them is significant. In his introduction President Riccobono said that Pearson has taken accessibility so seriously that they have not only done what we have asked of them but have worked hard to use their knowledge of test taking to enhance the accessibility experience. Here is what Jon Twing has to say about his job, the value of testing, and the commitment he has to accessibility:

Thank you, Mr. President, thank you, Federation. I am responsible for assessments at Pearson, so naturally I live in Iowa City, Iowa, the Mecca of assessments. I'm glad to be here today to talk a little bit about some of our activities. I'm a trained test-builder. I'm something the field calls a psychometrician. People often ask me, "How do you choose such an occupation?" And I have a little secret for you, with no disrespect intended to my good friend at H&R Block, but we all start out as accountants. Those of us with a sense of humor become psychometricians [laughter].

I've been building assessments for over thirty-five years of all types and shapes, both domestically and internationally. So I have to be honest with you: if you don't enjoy assessments, it's probably my fault. There's not an assessment in North America that I haven't touched. I'm happy to talk to you about any concerns you might have, just look up Jan McSorley or Sam Dooley or even Cricket, and they'll put you in touch with me. I'm going to be here the rest of the day; we can chitchat about that.

I am delighted to talk about some of the innovation we make. I'm going to point out that I'm going to talk about assessments and the strides and changes and innovation we've driven into assessments. I'm happy to talk about other aspects of Pearson whose accessibility track record is not that great, but I am the assessments guy, so I plan to talk about assessments. We're already seeing, however, generalization of our efforts to make our tools accessible in our assessments going to the other parts of Pearson, so I'm optimistic that we've turned the corner, and we're making that trek. I'll leave that for you to decide once you've heard my talk.

Being a subject matter expert in assessments means that most of my life has been focused on what researchers call variance decomposition, or specifically something known as construct-irrelevant variance. Now I know that sounds really important, but all that really means is that when we build measures, a piece of our measure is error, and we want to get it out of our measure, and a piece of our measure is true, and we want to keep it in our measure. That's almost by definition what psychometricians do when they build measures. Now before your eyes glaze over and you fall asleep and you dream about lunch and you really do think I'm an accountant, let me try to explain this whole issue of construct-irrelevant variance in a very specific and relevant, concrete example.

In the early 1970s AT&T was notorious for discriminatory hiring practices. They often hid behind a rare exception to Title VII of the 1964 Civil Rights Act called "Bonafide Occupational Qualification." Basically this exception claimed that if a job could only be performed by one gender, for example lifting a ladder to the top of a panel van, then accommodating that job for the other gender would be outside normal operations and not required [growls and grumbles]. Well I think the EEOC took your point of view, because they essentially went to court in what is now known as the landmark AT&T sex discrimination case and rejected this claim outright. The essence of their victory was claiming that the placement of a ladder on top of a panel van was irrelevant to the skills needed to perform that job. If you reaccommodated that by placing that ladder down on the van, anybody could have access to it.

If you think about the simplicity of this and the power of that one simple example, now we understand a little bit better about construct-relevant versus construct-irrelevant variance. The placement of the ladder had nothing to do with the job skills, and yet it was being used as a barrier for entry. AT&T—not because they were evil people—did the same thing a lot of people did: they didn't think about how the tool was going to be used by the end user. Arguably, according to the EEOC, they did hide behind that bad decision and tried to defend themselves, but ultimately that case was provoked, and accommodations were allowed, and the panel vans were changed.

In measurement and in testing, such accommodations are equally obvious. For example, providing a Braille version of a test for a blind person is a reasonable accommodation because it is obvious that the loss of sight has nothing to do with the cognitive skill in answering the question [applause]. Similarly, then, extended time for a Braille user is a good accommodation because it takes longer to use that tool. Supporting graphics for text items should not become an undue burden to test-takers, particularly those with varied visual needs. As such, accommodating tables and graphics in assessments has become quite routine. However, what is not so obvious are the rules that govern the accommodations or the evidence for just what constitutes undue burden. For example, in the Americans with Disabilities Act of the 1990s, federal law now required testing accommodations for cognitive disabilities such as dyslexia as well as other physical disabilities. But such accommodations might actually impact the validity of the measures. The court has been very wishy-washy on where the line is drawn on this issue of cognitive accommodations.

Now you might say that this is all well and good, but what have you done for me lately? Aren't these things that the assessment industry has been doing and should be doing as we evolve to make goods and services more and more accessible? I agree they are, but don't forget that it was in my lifetime where we used to have the nonstandard administration flag—a flag on a student's report card that any student that would engage in assessment in any nonstandard fashion would get—a flag that my friends at the Arc of Texas would call "the scarlet letter of assessments in the day." Just like when the Bell Lab engineers in Murray Hill, New Jersey, put the ladder on top of the van, they were not thinking of the final solution—how it would be used by the end users, just like how my predecessors in assessments did not anticipate the many diverse needs of their users when they came up with this idea of no exceptions. It reminds me of a wonderful book that I've read that for all intents and purposes is a primer on what is known as user-centered design. It's called The Psychology of Everyday Things by Don Norman. I'd like to quote that for a second: "Humans, I've discovered, do not always behave clumsily. Humans do not always err. But they do when the things they use are badly conceived and poorly designed. Nonetheless, we see human error blamed for all of society's pitfalls . . . While we all blame ourselves, the real culprit—faulty design—goes undetected." Accessibility is all about good user-centered design and making products and services fit for purpose for everyone. We can no longer ratchet something onto an assessment at the end and call it accessible [applause].

Now under NCLB [No Child Left Behind]—and we could have a debate about the value of NCLB, but we'd have to agree that a lot of light and focus was spent on students with disabilities. I recall testifying at a Senate education hearing in Texas. I was making the usual argument about how we're the constructing on-grade-level multiple choice assessments aligned to curriculum standards, but I was doing so for what was known as the 1 percent population at the time. These would be our most significantly cognitively-impaired students. The purpose of the law was to engage with them more than life skills, to teach them subject matter knowledge. I felt proud that we had done such a good job building scaffolding for the teachers, having teacher support and training, having peer tutoring in online interactive qualifications for teachers. I remember thinking how well the rubrics were crafted that helped the teachers decide how much independence the students would have in engaging these tasks.

When I was done, a teacher from one of the schools in Texas, who never took her eyes off me—it was a dagger stare straight at me—wanted to talk about how invalid, unreliable, and useless the assessments were because everybody knew that these children with these cognitive disabilities could not learn, and they certainly could not learn on grade level.

Now what struck home for me wasn't the obvious. It wasn't the bias or tyranny of low expectations that this teacher held or even the notion that this teacher was maintaining a status quo in a school district. What struck home for me was that the very product I thought was good failed to meet the criteria because it wasn't really used and implemented to improve instruction in the classroom.

If I could make that teacher see what those assessments might have done for those children in her classroom had they been challenged to reach that level, it would be an interesting thought experiment to see today. In many ways the current evolution of accessible assessments is the same. Building a Braille test, following the principles of universal design, ensuring that existing technologies work is just not good enough. We started behind in making assessments accessible, and we need to do more to catch-up and to close the gap, particularly as we evolve into more and more of a digital age.

We're making some progress in this regard, as President Riccobono reflected. We need to go faster, but we are making some progress. First and foremost, we hired a dedicated group under Jan McSorley to pay attention to nothing but accessibility inside assessments [applause]. And what she did, which was quite controversial to my management, was she immediately engaged the National Federation of the Blind [applause]. We're working toward a better understanding of what access barriers are faced by people with disabilities. For example, we've sent our developers, our content authors, our user experience professionals into the classroom to observe students with disabilities and to talk with them and their teachers about how to improve accessibility of our products. What a brilliant idea; too bad it was an afterthought—you'd think you might talk to the people who are using your products before you design your products.

Hiring subject matter experts who understand the best design principles with the flexibility needed to respond to individual needs of people with disabilities—again you'd think that would be a no-brainer, but we're doing that in assessments now.

Investing in innovation by tackling the long-standing problem of limited access, particularly in math for students who are blind, we're developing an Accessible Equation Editor that can be used on assessments and in the classroom. This equation editor can do now what has never been done in the history of math: it dynamically translates printed math in the Nemeth code and the Nemeth code into printed math, does it accurately and in real time. I do hope you got a chance to engage with the Accessible Equation Editor in the vendor hall or during the user studies at the convention. If not, no worries: we'd be happy to set up a demo site. We'd be happy to give you a tutorial. Again, probably the right person to contact is Jan McSorley. That's <>, and she'll be happy to set up a time to walk you through the equation editor.

But here's a simple story I want to tell that came from the convention this week that's both moving and powerful, though it leaves me sad. Sometime while at the Convention this week, Sam Dooley (one of Jan's team) was demonstrating the Accessible Equation Editor. As Sam was showing the editor, a seven-year-old blind girl walked up to the table with her parents. They said that she was a Braille reader and that she was learning Nemeth Braille, which is the Braille code for math. Sam introduced himself and asked, "Would you like to type some math on the computer?"

She asked, "Are you going to be my math teacher?" For those of us who've taught children of that age, that's a penetrating question, isn't it?

Sam said, "Tell me if you can read what I type," and Sam typed the number 1 on his laptop. "One," she said, as her face started to brighten. Sam typed the plus sign, then the number 2. "One plus two," she exclaimed with pride. Then, "One plus two plus three," as Sam continued to type.

At this point Sam said, "Now let's make it harder," as Sam typed the equal sign.

"Equal sign," she said with anticipation, starting to see what was coming next.

Sam asked her, "Do you know what one plus two plus three is?"

"Mmmm, three plus two is five, plus one is six," she replied.

Sam said, "That's correct. Do you know how to type six in Braille?" She put her tiny fingers on the keyboard on the Braille keys and searched around to find where they should go. Then the sounds of dots two, three, and five went down, and the numeral 6 appeared on the screen in front of her parents.

"That's right," Sam exclaimed, with her parents looking on.

"Mommy, mommy," she said, "You can read my math." [applause, cheers] She cried, and she jumped up and down as only a seven-year-old can when they discover how big the world is.

This story is very simple, it's moving, yet it's powerful. Answering a simple math question should not be a big deal. It was simple, though not easy to build a mathematics accessible equation editor. The power was really in the empowerment the young girl felt, perhaps the same empowerment we felt when we learned to read independently. My enthusiasm to enjoy the story, however, is tempered. I simply can't help but think, that's as it should be. That shouldn't be an exciting day in a child's life to do what everybody else is doing [applause]. This is perhaps where my passion lies, though being a former accountant it's hard to claim I have passion. But by utilizing simple user-centered design principles to build accessibility into our products and services, such that all students will benefit from their use, and not added as an afterthought—that is my goal. So as we evolve into the future, we'll continue to evolve our thinking, we'll continue to work with the NFB, and we hope we can generalize such small successes to the rest of the realm of the great Pearson enterprise. Thank you.

Media Share

Facebook Share

(back) (contents) (next)