Lessons from a restricted Turing test

Reviewer: Mark Halpern

The Loebner Prize competition, as I will assume the reader merely needs to be reminded, is an annual competition held since 1991, whose purpose is to encourage the development of a program that will successfully pass the Turing Test. The three closely related documents under review are a paper by Shieber assessing the scientific value of the Loebner Prize competition; a rebuttal by Loebner, donor of the prize; and a rejoinder by Shieber to Loebner. In his opening paper, Shieber makes it clear that he has the greatest misgivings about the validity of the Turing Test even when taken in its full rigor, but he deliberately waives his doubts on that score for purposes of considering the usefulness of the pared-down version of the Turing Test that Loebner has subsidized. In this paper, he accepts the Turing Test for purposes of argument, and limits himself almost entirely to showing that the severely limited version of the Turing Test on which the competition is based is irreparably flawed as a working model of the Turing Test, whatever the test's own flaws may be. In so narrowing his scope, he forgoes the chance to examine the deeper and (to me) more interesting questions about the Turing Test, but this sacrifice enables him to focus sharply on the Loebner Prize competition. Shieber's motive in inquiring into the competition is, he tells us, a desire to correct two abuses: first, the cost is not, as casual observers might easily think, all borne by Loebner; public money is involved too (specifically, National Science Foundation funds). Second, the nature of the competition, and the publicity given it, may have the effect of warping the direction taken by projects in computer science and AI, and<__?__Pub Caret> the public perception of such projects. Shieber, who took part as a referee in the first competition in 1991, and who is acquainted with many of the personalities involved, gives some interesting and colorful behind-the-scenes details of that event. The main value of his paper, though, lies in his analysis of the logic of the competition (and often, by implication if not explicitly, of the Turing Test). There is nothing startlingly original, it may be said at once, in that analysis; it is in the same vein as that by Eric Weiss and me [1], for example. Shieber says, in summary, that the structure of the competition virtually guarantees what we have seen in the three annual competitions held so far: the competing programs are of very low quality; they represent no advance whatever in AI, but rely (in particular, the program that has won all three so far relies) on shallow trickery whose success in fooling some people is better explained by Barnum than by Babbage. He concludes this paper by suggesting what he calls an improved and restructured Loebner Prize competition that might actually have some scientific value, but here he is merely being kind; what he is actually suggesting is not a patched-up but still recognizable Loebner Prize competition, but an utterly different sort of competition that would share with the present competition only its name. Loebner, in his response, opens with a trumpet blast—a slightly off-key blast. “Shieber,” he writes, “proposes to tell me how I should spend my money.” Shieber does no such thing; he does indeed regret what he sees as a waste of private funds, but he nowhere suggests that Loebner has no right to spend his own money as he wishes, waste or not. Then Loebner addresses the question, “Why a Loebner Prize__?__” which Shieber had used as a section title in his own paper, but somehow picks it up by the wrong end. He seems to think Shieber was asking, “What were the circumstances in which Loebner conceived the Loebner Prize competition__?__” when he was really asking, “What is the justification for the Loebner Prize competition__?__” A large part of what Loebner writes in this misdirected attempt at a reply is of little interest except to those who care about his private history. The rest deals with notions for improving the competition that are relatively trivial or marginal; he is dealing with a report by a structural engineer that says his house is unsound and must be demolished, and in response proposes to give that house a fresh coat of paint, discussing the advantages of one color over another. His response to Shieber culminates in the remarkable statement that the reader wanting a full discussion of the “lessons we have learned from our restricted Turing Test” should “see Shieber's companion article…”; remarkable because Shieber's conclusions are that the Loebner Prize competition has no lessons of value to teach us (unless we are connoisseurs of human gullibility), and ought to be changed so radically that what is really proposed is outright abolition. That Loebner imagines that the competition finds support in Shieber's piece suggests a more-than-ordinary failure of communication. In his brief reply to Loebner, Shieber tries to correct several of the misunderstandings I have just mentioned. His effort to set Loebner straight is understandable, but probably futile; he can take comfort, though, in knowing that any careful reader of the two earlier papers will have made the same observations. Incidentally, Shieber thanks a distinguished galaxy of scientists and scholars for reading early drafts of his main paper, but they have not served him as well as they might; none of them caught his statement that “humankind has dreamed of mimicking the power of human thought”; nor his calling the Turing Test a “litmus test” of human intelligence, when he has repeatedly emphasized that the Turing Test is a binary, yes-or-no test; nor such nonstandard English as “I am not ostentatious enough to provide examples….” But these are very minor blemishes; Shieber has performed a service in setting forth at length the real and serious flaws in the Loebner Prize competition, and has made it a little harder in the future for showmanship and publicity-seeking to subvert science.