Tennessee’s testing vendor, Pearson, testified on Tuesday to House Education Instruction Committee members. Pearson supplies Tennessee schools with both the state’s annual TNReady test and the recently adopted universal reading screener, AimsWeb Plus. The company took over the multi-million dollar state contract to create and administer the annual standardized test in 2020 after several years of missteps by then-vendor Queststar.
Pearson representative Amanda Deaner walked legislators through the process of the state’s assessment – from creation to administration. In describing how tests are crafted, Deaner painted a picture that is lengthy, deliberate, and deeply influenced by professional educators. Tennessee teachers are involved in the writing and reviewing of questions, and per Deaner, ultimately the creation of the cut scores used to define proficiency levels.
In her presentation, the Pearson representative explained what happened after the state standards review is completed. At that point, the company begins soliciting questions based on the revised standards. After collecting questions, a panel that includes educators reviews them. Deaner told legislators that over 500 Tennessee teachers were used to check the questions for a recent assessment.
This review includes whether a question is potentially biased and whether all students have the cultural experiences to answer the question correctly. The panel also looks to if the question is appropriate. A few years ago, students were required to take TCAP shortly after tornados struck Tennessee. Questions that referenced weather crises were deemed inappropriate and crossed from the exams.
After the test is given for the first time, teachers are brought in to evaluate and review. Through this process, it is established how many correct answers are needed to reach each level. Those levels stay in place until the next standards review, with minor modifications made each year to ensure that performance levels align with the test.
After teachers set the proficiency levels, they are submitted to the Tennessee Department of Education (TDOE) for review and passed on to the State Board of Education (SBE) for final approval.
The Tennessee Star contacted state superintends to verify Deaner’s claims that educators set the cut scores, their response was “yes and no.”
“Educators are brought in the summer after the first administration of a new assessment,” one superintendent who wished to remain anonymous told The Star. “The questions are sorted from least to most difficult based on how students actually performed. Educators are then asked to place a bookmark where they think an approaching student would start having difficulty. The same is done for the below. However, it doesn’t stop there. The psychometricians from the testing company take everyone’s cut scores and provide impact results to the group (e.g., based on your cut scores, 70% of students would score” below” or “approaching”). They guide the group to set the cuts higher or lower.”
Another superintendent concurred, adding, “I would describe the process as more of a political process than a technical one.”
It was further noted that the key is how the proficient student described at the beginning of the process. It was shared that in the early 2000s, educators were instructed to base expectations on the D student. In the latest standard setting, the group was asked to consider the B student who showed mastery of most standards.
Legislators have long fought for more transparency with TCAP. State Representative Mark Cochran (R-Englewood) voiced these concerns, telling Ms. Deaner, “As far as analyzing previous TCAP tests, it is like we’ve asked for the Kennedy assignation file, or the nuclear codes, so tell me why is that? Is there ever a time when I can look at the 2020 TCAP test and view the whole thing? Is that even possible?”
Deaner answered that it isn’t possible, but not because anybody doesn’t want to. Tennessee’s last change in standards left the answer bank – where approved assessment questions are held – depleted. Until that bank is built up, test security must be closely guarded. Cochran asked when that might be and was told three to four more years.
A bill, SB0421/HB0896, extending the standard review and adoption period from 73 months to 97 months passed in the State Senate this week and is scheduled for the House floor soon.
Tennessee standards are exclusive to Tennessee. When asked to compare Tennessee standards to national standards, Deaner told legislators that she was not privy to studies that compare the two. She did share that when Pearson’s content teams looked to other state answer banks for possible questions to be used on TNReady, they didn’t find great alignment with other states. They found many common areas where a large percentage overlapped heavily, and items could meet some Tennessee Standards.
Committee Vice-Chair Representative Kevin Raper (R-Cleveland) drew further attention to TNReady being state normed instead of nationally normed. He argued that comparing Tennessee scores to national scores is of limited value. Raper told The Star in a phone interview, “I think that the data we report to the Federal government should be from nationally normed assessments, so it’s more of an apples-to-apples comparison.”
The recently mandated Universal Literacy Screeners are nationally normed. State Representative Mark White (R-Memphis) questioned Deaner on whether a student who does well on benchmark tests will do well on TCAP testing. She responded, “I don’t have data that says exactly how those two things would compare, but I think you can say it would inform their success on TCAP.”
Later, State Representative Scott Cepicky (R-Culleoka) asked Deaner directly, “Can the universal screeners supplant TNReady?’
She answered, “No, because you are measuring different information.” She explained that the two are different assessments.
Over the past several years, Tennessee has faced numerous challenges in administering the state’s standardized test. Despite claims from Commissioner Penny Schwinn that the process has been improved under her watch, problems persisted through 2022 when testing scores for third, fourth, and fifth grade were incorrect for several schools and had to be rescored. The tests were not rescored in time to be included in final grades.
State Speaker of the House Cameron Sexton (R-Crossville) recently proposed legislation that would divorce Tennessee from federal education funding. Sexton said, “I don’t think the TCAP test measures much of anything, and I think teachers would tell you that you’re teaching to a test.”
– – –
TC Weber is a reporter at The Tennessee Star and The Star News Network. He also writes the blog Dad Gone Wild. Follow TC on Twitter. Email tips to [email protected]. He’s the proud parent of two public school children and the spouse of a public school teacher.
