There’s an interesting article in the Guardian today about whether AI could write the next great novel, and I’m beginning to suspect that it actually could. Whether it should is another matter, and whether it would insist on calling the novel something like The Whispering Algorithm is probably a separate ethical concern.
The article refers to a service called Pangram, which claims to be able to detect with near certainty whether text has been written by AI or by a human. This sounded interesting, so I thought I’d try it out with the full scientific rigour of someone sitting at a computer trying to prove a minor point. I used one paragraph that had been written by AI, and one paragraph from my blog written three years ago.
Pangram managed to get them both wrong. Not only that, it got them wrong while expressing high confidence, which is always the most enjoyable kind of wrong. There is a certain charm in uncertainty, but much less charm in a machine confidently informing you that your own pre-AI writing was produced by AI, while also letting AI wander past dressed as a human. At that point, the system is less a detector and more a very assertive guesser with a nice interface.
This seems worse to me than just reading the text and coming to your own conclusion. At least a human reader can be wrong with a pleasing degree of hesitation, perhaps accompanied by a thoughtful frown. Pangram is putting out false answers with authority. It does not claim to be 100% accurate, which is fair enough, but if it is not certain then it should not sound certain.
And, as another data point, AI wrote the above from lines that I fed it and I think that it’s noticeably AI written just visibly. And Pangram states:
I’d advise caution if using this apparently 99.98% accurate service…..