2008 BotPrize contest

The first BotPrize contest was held in Perth, on 17 December 2008, as part of the 2008 IEEE Symposium on Computational Intelligence and Games. The aim of the contest was to see if a computer game playing bot could convince a panel of expert judges that it was actually a human player.

The Judging Panel

  • David Fogel - President of IEEE Computational Intelligence Society
  • Robert (RJ) Spencer - COO of Interzone Entertainment
  • Penny Sweetser - senior game designer at 2K Australia and an AI expert
  • Cam Atkinson - a l33t game player
  • John Wiese - senior project manager in Special Operations Command Support Systems at Thales Australia

The Humans

  • Andrew Smith
  • Roderick Baker
  • Byron Pogson
  • Keith Johnson
  • Seb Davidson

The Teams

Team Affiliation Members
Amis Charles University in Prague
Michal Stolba
Jakub Gemrot
Juraj Simlovic
ICE-UT@RITS Ritsumeikan University, Japan
Daichi Hirono
Yuna Dei
Ruck Thawonmas
Intelligent Systems Centre Nanyang Technological University
Budhitama Subagdja
Ah Hwee Tan
Di Wang
Underdog University of Western Australia
Oren Nachman
University of Texas at Austin University of Texas at Austin
Jacob Schrum
Igor Karpov
Risto Miikulainen

* The team of Jon Wright, David Nichol and Russel Thom from Glasgow Caledonian University made the finals but was unable to attend.

Results

The prize winners were:

  • Best bot: AMIS (Michal Stolba, Jakub Gemrot, Juraj Simlovic)
  • Best human: Roderick Baker
  • Best Judge: John Wiese

Here is a summary of the results in the form of a table of bots and human confederates, sorted by human-ness, as rated by the judges.

Identity Mean rating #Judges convinced
Byron (human) 4 5
Andrew (human) 3.8 4
Roderick (human) 3.8 4
Keith (human) 3 2
Seb (human) 2.6 2
AMIS (bot) 2.4 2
ICE (bot) 2.2 1
ISC (bot) 2 2
UTexas (bot) 0.8 0
Underdog (bot) 0.4 0

All the humans were rated as more human than all the bots.

Complete round by round results can be found here.

The table below contains videos made from the games in the final, from the judge's viewpoint. Within each video, all players appear identical, but can be distinguished by name (e.g. Player123), which appears when the player is targeted by the judge. The table entries give the player names used in each game. You might like to try your hand at picking the bots! Can you do better than the judges?

Each video is between 30MB and 50MB. Warning: these videos depict simulated violence.

  Judge 1 Judge 2 Judge 3 Judge 4 Judge 5
Round 1
video
Player144=Judge
Player443=Human
Player932=Bot
video
Player502=Judge
Player212=Human
Player773=Bot
video
Player830=Judge
Player136=Human
Player415=Bot
video
Player875=Judge
Player433=Human
Player441=Bot
video
Player388=Judge
Player471=Human
Player639=Bot
Round 2
video
Player787=Judge
Player859=Human
Player470=Bot
video
Player752=Judge
Player739=Human
Player148=Bot
video
Player952=Judge
Player760=Human
Player884=Bot
video
Player173=Judge
Player408=Human
Player389=Bot
video
Player640=Judge
Player961=Human
Player178=Bot
Round 3
video
Player673=Judge
Player367=Human
Player833=Bot
video
Player711=Judge
Player431=Human
Player189=Bot
video
Player231=Judge
Player247=Human
Player703=Bot
video
Player845=Judge
Player525=Human
Player352=Bot
video
Player568=Judge
Player275=Human
Player794=Bot
Round 4
video
Player325=Judge
Player959=Human
Player228=Bot
video
Player165=Judge
Player707=Human
Player818=Bot
video
Player720=Judge
Player160=Human
Player822=Bot
video
Player475=Judge
Player859=Human
Player739=Bot
video
Player467=Judge
Player755=Human
Player886=Bot
Round 5
video
Player261=Judge
Player357=Human
Player217=Bot
video
Player237=Judge
Player980=Human
Player157=Bot
video
Player635=Judge
Player955=Human
Player494=Bot
video
Player373=Judge
Player642=Human
Player189=Bot
video
Player683=Judge
Player178=Human
Player578=Bot

The Turing Test

In 1950, artificial intelligence pioneer Alan Turing posed the question "Can Machines Think?" To answer this question, he proposed a test : if a machine can chat with you and convince you that it is human, then the machine is showing that it can think.

In 1990, philanthropist Hugh Loebner started the Loebner Prize , an annual competition that challenges programmers to create a program that can pass the Turing Test.

Although the prize has not yet been won, a number of "chatterbots" had their start in life at the competition.