I've had two personal collisions with AI slop as well, one ridiculous and one scary. I occasionally run a vanity search of my fandom name and the SWG, just to see where they are being discussed in the fandom. The last time I did this, the Google AI result at the top of the results informed me that Dawn Felagund is a character in a Tolkien novel. Okay, wrong but so ridiculously wrong that I could chuckle and move on and think most people doing that same search would as well.
Then while hanging out with a friend of mine, he posed the question to ChatGPT, "Who is Dawn Walls-Thumma?" (This was done with my permission.) This one was scarier because the answer it returned was just next to correct, i.e., not so blatantly wrong that most people would spot it as wrong immediately. It got most of me correct: I am the founder of the SWG, a humanities teacher, and an independent Tolkien scholar, and it got most of my research correct. It missed one of my biggest areas of research (historical bias), which was annoying but not disastrous. When it reached the Tolkien Fanfiction Survey, however, it erased Maria as my collaborator and stated that I'd done the survey in collaboration with the Organization for Transformative Works! Now I mostly support the OTW, but I've also spoken out against them (and AO3) on key issues in the past, and it's highly unlikely that I'd ever hitch my horse to their wagon (at least, not if I valued the horse).
This mistake was small enough that I doubt most people would have caught it. Fact-checking it would have taken enough effort to defeat the purpose of using AI in the first place. I imagined this information being used, say, in discussing my work in a blog post, thus giving it a degree of credibility to where it is replicated by other credible/ish (i.e., non-slop) sources and eventually becomes indistinguishable from fact. Maria and my criticisms of the OTW have been erased, and now this large institution is partly responsible for my little homespun survey!
My SWG comod Grundy compared it to making a photocopy of a photocopy. The first photocopy is just a slight degradation of the original but, copy it enough times, and it loses all resemblance to the original. With doing this kind of basic research being seen as one of the more acceptable uses of AI, at what point does this slop become so enmeshed with real content that it's impossible to trust any post-AI source anymore?
That is scary--as you say, it's the idea of proliferating misinformation becoming more and more warped as it ripples out! And Grundy's metaphor is an excellent one.
I was horrified to stumble across a Wikpedia entry about myself--as far as I remember, it was just a bad pastiche of stuff from my old faculty page. I shall avoid ever asking an AI about myself (I think I read a post by John Scalzi that he asked one of them about himself and saw a similar pattern of errors).
I bet the OTW got thrown in because you have published with the journal.
One of my colleagues at my former university who was a full-fledged bibliographical scholar in the traditional sense (I describe my bibliographical stuff as out in the fuzzy boundaries) had one motto: "Trust no one."
I think that still applies, even more so these days.
I also remember getting hired to do some research as an undergraduate--it was just checking quotes my faculty mentor's book, but she was a Renaissance scholar so I spent a lot of hours in the library basement with the microfiche machine.
I was checking the quotes from primary sources that were in the published scholarship--so how accurately did the secondary scholarship convey the primary stuff--since my mentor was dealing with some of the same texts and issues.
I was horrified by how many errors I found: some were clearly typos, but some were 99.999% I was sure changed to support an argument.
So like many things today, I think that AI has accelerated (exceeding light speed and warp drive) what already existed in the world--misinformation (intentional or not), and what we can only call slop.
And I continue to find the news about data center proliferation and how environmental regulations and community votes are being ignored to be horrifying, especially given the impact of climate catastrophe.
I've had two personal collisions with AI slop as well, one ridiculous and one scary. I occasionally run a vanity search of my fandom name and the SWG, just to see where they are being discussed in the fandom. The last time I did this, the Google AI result at the top of the results informed me that Dawn Felagund is a character in a Tolkien novel. Okay, wrong but so ridiculously wrong that I could chuckle and move on and think most people doing that same search would as well.
Then while hanging out with a friend of mine, he posed the question to ChatGPT, "Who is Dawn Walls-Thumma?" (This was done with my permission.) This one was scarier because the answer it returned was just next to correct, i.e., not so blatantly wrong that most people would spot it as wrong immediately. It got most of me correct: I am the founder of the SWG, a humanities teacher, and an independent Tolkien scholar, and it got most of my research correct. It missed one of my biggest areas of research (historical bias), which was annoying but not disastrous. When it reached the Tolkien Fanfiction Survey, however, it erased Maria as my collaborator and stated that I'd done the survey in collaboration with the Organization for Transformative Works! Now I mostly support the OTW, but I've also spoken out against them (and AO3) on key issues in the past, and it's highly unlikely that I'd ever hitch my horse to their wagon (at least, not if I valued the horse).
This mistake was small enough that I doubt most people would have caught it. Fact-checking it would have taken enough effort to defeat the purpose of using AI in the first place. I imagined this information being used, say, in discussing my work in a blog post, thus giving it a degree of credibility to where it is replicated by other credible/ish (i.e., non-slop) sources and eventually becomes indistinguishable from fact. Maria and my criticisms of the OTW have been erased, and now this large institution is partly responsible for my little homespun survey!
My SWG comod Grundy compared it to making a photocopy of a photocopy. The first photocopy is just a slight degradation of the original but, copy it enough times, and it loses all resemblance to the original. With doing this kind of basic research being seen as one of the more acceptable uses of AI, at what point does this slop become so enmeshed with real content that it's impossible to trust any post-AI source anymore?
That is scary--as you say, it's the idea of proliferating misinformation becoming more and more warped as it ripples out! And Grundy's metaphor is an excellent one.
I was horrified to stumble across a Wikpedia entry about myself--as far as I remember, it was just a bad pastiche of stuff from my old faculty page. I shall avoid ever asking an AI about myself (I think I read a post by John Scalzi that he asked one of them about himself and saw a similar pattern of errors).
I bet the OTW got thrown in because you have published with the journal.
One of my colleagues at my former university who was a full-fledged bibliographical scholar in the traditional sense (I describe my bibliographical stuff as out in the fuzzy boundaries) had one motto: "Trust no one."
I think that still applies, even more so these days.
I also remember getting hired to do some research as an undergraduate--it was just checking quotes my faculty mentor's book, but she was a Renaissance scholar so I spent a lot of hours in the library basement with the microfiche machine.
I was checking the quotes from primary sources that were in the published scholarship--so how accurately did the secondary scholarship convey the primary stuff--since my mentor was dealing with some of the same texts and issues.
I was horrified by how many errors I found: some were clearly typos, but some were 99.999% I was sure changed to support an argument.
So like many things today, I think that AI has accelerated (exceeding light speed and warp drive) what already existed in the world--misinformation (intentional or not), and what we can only call slop.
And I continue to find the news about data center proliferation and how environmental regulations and community votes are being ignored to be horrifying, especially given the impact of climate catastrophe.