How Cheating Foreshadows Why AI Will Miss

"We're looking for data on people who cheat on their partners," the man said over the phone. "It could be a situation involving a spouse or another committed type of relationship." The man worked as an attorney and uncovered that a significant portion of their lawsuits involved infidelity. They didn't always involve married people either. Money poured into their legal firm and he wanted to scale the business. They wanted to become one of the leading firms involved in these situations.

I chuckled a bit because cheating is one of the earliest examples in my life where I learned why most people miss in research. I repeat over and over and over again two statements that people really don't notice, but pay attention:

Truth 1: No data is better than bad data [1]
Truth 2: People want to know what they want to know [1]

As you read both of these, you'll think "How is this relevant to cheating?" The issue is not about cheating even if that's an applied example; the issue is actually about research and why this will only become more important over time.

Before I start to show why, let's highlight why you should understand the importance of this now. The sooner you get this - and this won't be something you want to know - the better your future.

Problem: Applied AI solutions, like LLM, are amplifying the "search engine" problem. The search engine problem is where someone thinks that because they got an answer from a search engine, they know what they need to know. LLMs are ten times worse than this problem. As you're reading this, students are using LLMs to take shortcuts in their research and papers they write and some of this is then being fed back to the LLMs. Some teachers are catching this - and good for them. But never underestimate how many teachers would rather post dance videos on TikTok than be exceptional teachers. They let their students slide because it's easy money so they can go back to what they really love doing. The result is that critical thought becomes rare.

Like the internet and search engines compounded misinformation (ask anyone what money is and watch how no one can answer this simple question), AI will only make this worse.

Why Research Matters

Now, let's address why research matters. I'm intentionally using the example of a cheating spouse. Here's why: most of the people reading this have an immediate emotional reaction with this topic. The others who don't have what I call denial-syndrome. Denial syndrome is when you present as if you're detached from an issue because you don't believe that you'll participate in the issue. Our brains use denial-syndrome frequently because it saves us mental energy. As every statistician reading this will smile, "you really are the one exception!" That's denial-syndrome speaking.

Problem: emotions cloud critical thought.

But we haven't even entered the research problem with cheating. Cheating carries a stigma. People hate cheaters. What's funny here is that people who cheat also hate cheaters! Please make it make sense. Most novice or junior level researchers understand that when you're studying anything with a stigma, you'll get selective answers. There are so many scenarios of this and I don't want to overwhelm you because you've read this far and as a writer, I kind of like you as a reader who has the attention span to make it 300+ words into an article. Here's two examples of selective answers:

Respondent 1: "I've never cheated" [not said, "Because it was in a different zip code/area code/I was traveling in Las Vegas/some other exception]
Respondent 2: "I've never cheated" [not said, "Because the other person was a big, fat meany person thus it's not cheating when the other person is mean]

As the researcher, you got an answer - "I've never cheated" but that's actually a false answer. Remember my earlier point:

Truth 1: No data is better than bad data

Our research with these respondents just violated this principle. The worst part is that we don't know that they lied! Again, there are hundreds of these examples. Novice and junior researchers (and even some mid level researchers) will think of solutions. But their solutions end up creating even bigger problems.

One simple example here is the anonymous survey. Statistically, about half of the population thinks anonymous surveys are exactly that. The other half doesn't. Because you've read this far, you're probably a more skeptical person, as people who focus tend to be more cautious and skeptical. As I mentioned, researchers who propose this solution end up with the same problem - once again, they've invited bad data. As a fun experiment, think about how many times you've lied on an anonymous survey. Don't tell me, but you see the point because you know why you lied.

At this point, many researchers can cultivate their inner angsty teenager and declare, "We can never get to the truth!" While I appreciate the health of this sentiment rather than the know-it-all sentiment, it misses the mark too since giving up isn't a solution to the problem.

But There's A Bigger Problem

Finally, we get to the really big problem with research. "The way to solve the problem is not to ask people if they've cheated. Ask them if their friends have cheated. Now you got the answer." I heard this from a researcher a while ago and I chuckle at this "solution" she shared. She, like many people, has dynamic intuition [2]. This is how a person with dynamic intuition thinks. Dynamic intuition is often wrong even if popular. The assumption here is that people use stories of "friends" to really communicate things about themselves. This can be true, but it can be false.

And there's another big problem that she missed.

Even if this was 100% true - and it's not - by telling everyone to use this as a basis for truth, you've now affected how people will answer the question. Stated another way, answer the following question in your head:

Question: Has any of your close friends ever cheated? Don't tell me the answer; remember I like you.

Some of you already see where I'm going (this is why I like you). Now, consider that you know that I'm going to take your answer as a sign of you actually being the one who's cheated, so let me ask again:

Question: Has any of your close friends ever cheated?

And this is the problem. If you actually had the solution, you couldn't share it because people can change their behavior (and they will).

Unfortunately, she's missed the mark in her assumption, though people with dynamic intuition think this way. As a fun bonus for reading this far, I'll share with you a fun social experiment that you can use with people to spot if they have dynamic intuitions. Tell stories about your friends and watch how people respond to you in ways that (1) they think it's a friend or (2) they think you're talking about yourself. It's a fun way to spot people with dynamic intuition, and you can A-B test stories that are you stories and (or) your friend's stories. What you'll realize is that dynamic intuition misses the mark.

Since you're already running little social experiments, here's another one that will help you connect some of the dots in this article. Send this link to your friends to read. It's best if you do this in person and you'll see why. Later, ask them what they thought about the social experiments. You'll find that several of them will have "forgotten" but you'll sense something else is up. Statistically, only about 50% of people have read this far. Because I've just written this (along with the experiment), most people who've read this far will now read this entire article. If I had not written this, another percent would have dropped off after this. Focus is hard!

There Are Other Challenges Too

I only want to briefly highlight this point because I don't want to bore you with legality and ethics. In a nutshell, some research methods may be questionable in these areas. This is the commit a crime to find crime philosophy. Is that ethical? I can't decide for you, but it highlights a problem that some companies (or people) may dislike. I personally don't want to engage in behavior to uncover behavior. Not every researcher is like that and not every client will respect that either.

Luckily, there's a marketplace of competition. I can defer those clients to companies or researchers willing to break laws to catch those breaking laws.

Summary

I didn't want to overwhelm any of you with a wall of text, so I've highlighted an example of why research can be challenging. As I wrote earlier, no data is better than bad data. If you want any takeaway from this, then stick with no data. It's better, but it may limit your opportunities because that's why we seek information in the first place. Knowing the difference helps; I've shown clients some situations where they didn't need data at all. But sometimes they do. From there, it comes down to costs.

[1] Will AI Be the End of Employees and Workers
[2] I did not coin this term. I learned this term from Erik Markovik who wrote The Mystery Method under the pen name Mystery. While Erik claims that women have dynamic intuition, I have found that some members of both genders think or feel this way through social interactions.