A Rotten AAPPL
How could I avoid another punny title about this exam? Seriously though, the AAPPL (Form A at least) is rotten. I had my 6 Spanish IV students re-test with Form B, two weeks after doing Form A. Here are the comparative results on Interpretive Listening (IL) and Interpretive Reading (IR). Form A >> Form B
IR IL N4 >> I5 I2 >> I5 N4 >> I1 N3 >> I4 N4 >> I5 N4 >> I5 I1 >> I3 I3 >> I5 N4 >> I4 I3 >> A1 I1 >> A1 I1 >> N4
This is an average discrepancy in proficiency scores of 3.25 sub-levels!!
Other teachers have since followed suit after hearing about my students' wildly divergent scores. Their results have been similar. So, a problem has been identified. Time to come up with a solution. In the meantime, LTI/ACTFL need to immediately close Form A. Maybe Form B as well. I think a serious audit of the AAPPL and the other testing platforms is prudent. After all, our students are increasingly captive patrons of this oligopoly of testing companies, as they are understandably encouraged by teachers and parents to seek "official" recognition of their abilities. For some states like Minnesota and Illinois, the Seal of Biliteracy can equate to thousands of dollars worth of Foreign Language credits at colleges and universities. For thousands of other kids, these scores are a validation of all the time and effort they've invested toward their goal to become bilingual.
Can you imagine the damage it must do to a kid who has been taking Spanish for 4 years and is told they are only Novice High? We don't have to imagine very hard. As one Iowa teacher sadly recounts: "I had some serious disheartened kids and it undermined their confidence big time!!" This is a travesty. Most likely they ARE more proficient than they were told by this test. How many more kids were sent this same message?
Back to the AAPPL specifically. My decision to retest with this same group two weeks later using the other Form was prompted by a hypothesis of fellow Spanish teacher Jason Noble, after he and others voiced disappointment and confusion (on a private social media forum) in our students' scores who had taken the Form A. Meanwhile, Jason's level 4 students, who instead took the Form B, scored much closer to how Jason would have expected, perhaps even higher. But can we be sure that it is the Form A that is most flawed and not the Form B? Well, I think we can be pretty sure. Below is the email I sent to LTI over 3 weeks ago after my students' Reading and Listening scores came back.
--------------- Original Message ---------------
From: Jim Tripp [jim.tripp@mflmm.k12.ia.us]
Sent: 4/3/2019 5:18 PM
To: support@languagetesting.com
Subject: Contesting recent AAPPL scores I recently got back Interpretive (IR and IL) scores for my group of sixSpanish 4 students. I've had them for the last 3 years, while they spent their first year with a different yet very capable teacher. Only 1 of these students scored Intermediate-something on both the listening and reading sections. The other 5 students scored as straggling the N4/I-low. Three years ago I offered this same assessment (AAPPL - form A) to a group of my Spanish 1 and 2 students (at a different school). Several students passed with Intermediate-something on all four components, thereby earning MN's World Language Prof Certificate. Nearly all (20-some Spanish 1 and 2 students) scored very similarly on the Interpretive elements of the exam to how my Spanish IV students scored yesterday. This does not make any sense to me. I did not do anything above and beyond what I typically do in order to prep either of these groups (this year or three years ago) on particular language/tasks that are on the exam, other than a run through of the demo the day before. And I have not altered my instruction so much to warrant such a difference in Interpretive performance from my students. I cannot imagine poorer-prepared/less-proficient students being the explanation for this difference from 3 years ago. I would like to understand this curious situation. For now I am having a hard time accepting the validity of the results, or at least the consistency and integrity of the exam itself. Sincerely, Jim Tripp
And here is the response I got back from LTI, over 2 weeks ago:
Hi Jim, Thank you for contacting LTI in regards to your concerns about the AAPPL IR and IL tests in Spanish. We have forwarded this matter to ACTFL for follow-up, and we will get back to you with their response as soon as we hear back from them. Thank you for your patience in this matter.
I've even followed up since then, with no further response. When I saw in my inbox yesterday the AAPPL Alert - LTI News Brief Vol. 4 2019, I was hopeful there would be an announcement about all of this. Disappointingly, there was nothing of the sort. And of course we've been promptly billed for any and all tests taken. My patience -- especially given that the school year is drawing to a close -- is running short. So what should LTI/ACTFL be doing? Well, a lot more than they currently are. As I said above, they first need to immediately shut down Form A. They should be contacting all schools who've taken Form A since its last redesign (or earlier??) and communicate the issue. These schools and students deserve to be reimbursed for their purchase of the test, and at the very least given the opportunity to re-test free of charge with Form B. And if they aren't yet ready to accept that a problem exists, despite notification from me and several others now, then there seems to be a bigger problem.
Last but not least, I hope we WL teachers begin to have some conversations about how valid any computerized test of language proficiency can ever really be. We're evaluating learners' abilities to navigate interpersonal communications by having them "interact" with a pre-recorded face on a screen? Or how able they are to accurately comprehend and keep up with spoken language by clicking on the most correct box after hearing a short audio clip.
The computerized testing industry is powerful, and as for-profit companies they're not about to do anything that would affect the bottom line for their investors. That includes measures that would require more human involvement without raising the prices on our students. And we certainly can't ask families to put down any more money ($20 is already seems like too much) or we'll just see this process further perpetuate economic privilege.
There may be better ways to do this whole game folks, maybe even a way that doesn't involve a corporation. Consider this an ice-breaker for that important conversation. And be sure to email support@languagetesting.com if you think you got a bite of the rotten AAPPL too.
