AI detection – Is it really that good?

As a result of a little experiment I’ve conducted, I feel almost compelled to write this post. It started out when I read a job ad and within it, the advertiser described that they would run all cover letters through an AI detection, since they wanted original work. My standard cover letter is very much written by me and haven’t been near ChatGPT, Jasper, QuillBot, or any other AI writers. Despite this fact, as I ran it through an AI detector, it got a score of 74% probability when it came to being written by an AI.

My first thought was, of course, if I am a robot and just unaware of that fact. As the curious soul I am, I thought that I needed to run it through at least one more, to confirm my robotness. It got a score of 9,8 percent. That’s more than a 64% difference, between only two different softwares. This, naturally, triggered my inner investigator. I fired up google and searched for ”free ai detection sites”.

I ran my Cover Letter through ten different engines and results varied from the earlier 74% and all the way down to 0,2% (this particular site highlighted the content it suspected was AI generated and to my surprise, it was my letters exit greeting)

Since I’m a writer, I have focused my work on this, around the detectors for written content, obviously. So, after trying out the free stuff I realized I needed to compare them with paid services and started searching for such platforms. Some of the free platforms also offered a paid service so, I of course had to try a couple of those out. I got the same varied results from the paid services, as from the free ones. A bit more consistency but overall a pretty spread out result. So this has me wondering how much trust we should put in our AI detectors.

1. Going from a 74% likelihood to 0,2%, with different platforms is just too much variable to trust, for me.
2. The fact that both free and paid services were almost equally bad at their task, which was trying to detect if my standard cover letter was written by an AI (again, it’s not)
3. Am I just an unaware robot?

Consistency and dependability are crucial when it comes to software performance. Users rely on accurate and reliable results, especially in critical situations where even a minor margin of error can carry huge consequences. With results spreading as wide as I’ve seen in this example, I think it’s smart to not put that much trust in the detectors…yet. And I say yet, because with AI developing faster than anything else, it’s just a question of time before we have software that is actually trustworthy, in this particular area.