Discussion about this post

User's avatar
Tolkien by the Numbers's avatar

I've had two personal collisions with AI slop as well, one ridiculous and one scary. I occasionally run a vanity search of my fandom name and the SWG, just to see where they are being discussed in the fandom. The last time I did this, the Google AI result at the top of the results informed me that Dawn Felagund is a character in a Tolkien novel. Okay, wrong but so ridiculously wrong that I could chuckle and move on and think most people doing that same search would as well.

Then while hanging out with a friend of mine, he posed the question to ChatGPT, "Who is Dawn Walls-Thumma?" (This was done with my permission.) This one was scarier because the answer it returned was just next to correct, i.e., not so blatantly wrong that most people would spot it as wrong immediately. It got most of me correct: I am the founder of the SWG, a humanities teacher, and an independent Tolkien scholar, and it got most of my research correct. It missed one of my biggest areas of research (historical bias), which was annoying but not disastrous. When it reached the Tolkien Fanfiction Survey, however, it erased Maria as my collaborator and stated that I'd done the survey in collaboration with the Organization for Transformative Works! Now I mostly support the OTW, but I've also spoken out against them (and AO3) on key issues in the past, and it's highly unlikely that I'd ever hitch my horse to their wagon (at least, not if I valued the horse).

This mistake was small enough that I doubt most people would have caught it. Fact-checking it would have taken enough effort to defeat the purpose of using AI in the first place. I imagined this information being used, say, in discussing my work in a blog post, thus giving it a degree of credibility to where it is replicated by other credible/ish (i.e., non-slop) sources and eventually becomes indistinguishable from fact. Maria and my criticisms of the OTW have been erased, and now this large institution is partly responsible for my little homespun survey!

My SWG comod Grundy compared it to making a photocopy of a photocopy. The first photocopy is just a slight degradation of the original but, copy it enough times, and it loses all resemblance to the original. With doing this kind of basic research being seen as one of the more acceptable uses of AI, at what point does this slop become so enmeshed with real content that it's impossible to trust any post-AI source anymore?

1 more comment...

No posts

Ready for more?