Gameable Signals of Credibility

Gameable Signals of Credibility

A website can pass an appearance-based credibility test and still mislead. Professional design, polished prose, and scientific language are cheap signals: they can be purchased without earning trust.

What’s a cheap signal? Let’s start with an even more basic question: What do we even mean by a signal?

A signal is any observable element that we use to make judgments about something we can’t see. This sounds complicated, but we use signals all the time. You go to a job interview and you’re trying to get a sense of whether the company is an established entity or a fly-by-night operation. You might notice the building it’s housed in. Or where the interview takes place. You might recall where you initially saw the advertisement. An ad in a trade journal feels a bit more credible than one stapled to a telephone pole.

We shouldn’t dismiss a site because of a single signal. Instead, we should ask whether the signals we see align with the story the source is telling us. An ad for a dogwalker on a telephone pole makes sense. An ad promising a five-thousand-dollar monthly income stapled to a telephone pole feels a bit dodgy.

It works the same way with people. A date claiming to be a middle school social studies teacher shows up in a 2015 Prius? That tracks. On the other hand, a date who claims to be super generous but who leaves a server a 5 percent tip raises suspicion. These signals aren’t foolproof. But each signal allows us to ask whether the stories these people tell about themselves match the signals we see.

Many of the questions on the CRAAP test reflect an understanding that, like it or not, we rely on signals. Almost always. At least on the surface, the test’s questions make sense. We expect that a scholarly source would have an editing process that avoids spelling errors or that a major institute would be able to keep its website updated and free of linkrot.

The problem is that these signals are cheap and purchasable. We’re not saying you shouldn’t slow down and rethink an academic article that’s full of typos. Of course you should. But think about it: is good spelling and proper grammar that hard to pull off in an era of spellcheck and Grammarly? In the web’s early days, a professional website cost wads of money. But even back then, a well-designed site didn’t necessarily mean the organization was trustworthy. It did mean that the company behind it might at least be as big and influential as it claimed. Not anymore. Now, a slick-looking web template can be had for lunch money.

Taking at face value how an organization describes itself is like scrolling through someone’s Instagram feed and assuming they lead a charmed life. But it’s easy to look good when you control the camera angle and the filters are free. No one needs to explain that Instagram is all about image. Why would we expect anything different from an organization trying to cast itself as something it’s not? Would you expect the International Life Sciences Institute to come right out and give you the following warnings?

You should know that we receive major contributions from PepsiCo, General Foods, Hershey’s, Kellogg’s, and other giants in the food and beverage industry.

To draw you in, we “exploit the credibility of scientists and academics to bolster industry positions and promote industry-devised content.”

Former clients, like Mars Inc., makers of M&Ms and Skittles, have cut ties to us because they don’t “want to be involved in advocacy-led studies that so often, and mostly for the right reasons, have been criticized.”

The authors use the International Life Sciences Institute to show why appearance and scientific language need to be checked against information about funding, affiliations, and conduct.