Accuracy of expert fraud detection technology in remote test exams (proctoring)

The purpose(objective) of the empirical study is the measurement of the accuracy of expert-proctors in detecting cheating in online testing. 

Sample of the study. 35 test takers passed an online test of general knowledge on the basis of 30 multiple choice questions. Half of the subjects (18 persons) were “artificial cheaters” – they used cheat sheets with correct answers. 

Methods. The video recording of the testing process included a “screen capture” so that expert-proctors could observe all cursor movements, see a recording of the subject's facial expressions and a visual focus of attention in a separate window (recording from the front camera), and could listen to the subject pronouncing the task conditions and answers (“oral decision”). 14 experts took part in rating of video recordings, of which 8 experts showed satisfactory results in terms of the level of accuracy in detecting cheating (their accuracy that was measured using the Kappa coefficient was higher than 0.5). 

Conclusions. A high asymmetric validity of expert assessments is revealed. More accurate experts allowed a negligible (about 5 percent) number of errors of the “false alarm” type, but a relatively large number of errors of the “skip” type. Recommendations are made for the practical use of the expert assessment method in combination with automatic chronometric analysis of the degree of atypical protocols and subsequent control of face-to-face offline testing of all suspected subjects (examinees). 

References

Apton G. Analiz tablic soprjazhennosti. Moscow: Finansy i statistika, 1972. 143 s.

Chia-Yuan Chuang. Improving Proctoring by Using Non-Verbal Cues During Remotely Administrated Exams. A Dissertation Presented for the Degree Doctor of Philosophy. Arizona state university, 2015. 79 p.

Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 1960, 20 (1), 37–46.

Jadov V.A. Strategija sociologicheskogo issledovanija. Moscow: Akademkniga, Dobrosvet, 2003. 596 s.

Lurija A. R. Diagnostika sledov affekta. Soprjazhennaja motornaja metodika i ejo primenenie v issledovanii affektivnyh reakcij. – V kn: Problemy sovremennoj psihologii, t. 3. Moscow., 1928, s. 46.

Macmanus I.C., Lissauer T.J., Williams S. Detecting cheating in written medical examinations by statistical analysis of similarity of answers: Pilot study. British Medical Journal (online), 2005, 1064-6.

Orlov A.I. Jekspertnye ocenki. Uchebnoe posobie. Moscow: MZ-Press, 2002. 31 s.

Shmelev A.G. Mnogokriterial'naja ocenka pol'zovatel'skih interfejsov portal'nyh proektov. – V sbornike Internet-portaly: soderzhanie i tehnologii, Moscow: Prosveshhenie, 2002, tom 2, s. 346–361.

Shmelev A.G. Prakticheskaja testologija. Testirovanie v obrazovanii, prikladnoj psihologii i upravlenii personalo. Moscow: Maska, 2013. 688 s.

Shmelev A.G. Soglasovanie jekspertnyh ocenok s pomoshh'ju Internet-tehnologii shkalirovanija. – V sbornike “Jekspertiza v sovremennom mire: ot znanija k dejatel'nosti”. Moscow: Smysl, 2006, s. 175–185.

van der Linden W.J. A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 2006, 31 (2), 181–204.

 

Cross-validaion (2020). https://en.wikipedia.org/wiki/Cross-validation_(statistics)#k-fold_cross-validation

Inter-rater reliability (2020). https://en.wikipedia.org/wiki/Inter-rater_reliability

ProctorU. (2008). Pupilcity ProctorU. http://www.proctoru.com/

ProctorEDu(2015). Тесты в бизнесе и прокторинг. https://proctoredu.ru/interview

VProctor. (2014). VProctor. http://vproctor.com/

Chto takoe proktoring i chem on polezen na jekzamenah, testirovanii i v obuchenii (2020). https://finacademy.net/materials/article/proktoring