Google Busts a Coffee Myth

Can You Actually Taste It?

If you buy whole bean coffee and grind the beans yourself, this video is for you.

If you buy it already ground, you’re probably someone who’ll drink anything. With enough milk and sugar you don’t notice anyway. Read this article only if you find it amusing.

How do you know how long ago those beans were roasted? James Hoffman, noted coffee expert, explains what “freshness” means:

Coffee is shipped as green beans. You can buy it that way and roast it yourself, but most people buy it already roasted.

When you buy whole beans from a supermarket, the bag usually won’t even tell you when it was roasted. It will say “best before <some date many months after roasting>”. So if you really care about the roast date, you should buy from a specialty coffee maker, which often will tell you. You might indeed care about it, since even in whole bean form, the beans are still exposed to oxygen, and thus degrading. The question is: how fast, and should you care?

Hoffman goes on to tell you that if you buy whole beans, they are probably still good for four to eight weeks after roasting, depending on the darkness of the roast.

There are other opinions, though. One answer on the coffee StackExchange forum says:

Coffee is at its best one week after roasting, and the shelf live [sic] is maximum 3-4 weeks after roasting.

and a comment to that says:

I am a home roaster, and sometimes need to purchase roasted coffee. My rule is to ONLY purchase coffee which shows the roasting date, and then only if it is within 7 days.

In another answer, a commenter says:

I generally freeze the roasted coffee beans at peak flavor, which is 3-4 days after roasting.

About six years ago, my friend Jerry Morrison and I did a test at Google on this. Google being Google, they had a manager whose only job was the coffee stations. In addition to running them, he ran an email list for coffee fanatics, and he was constantly taking grief about this topic. People would say, in effect:

Hey, you’re going to all this trouble to do everything the right way, and yet you’re using stale beans! They’re sitting around for weeks after being roasted! You should use freshly roasted beans.

Continuing with the “Google being Google” theme: we decided to get all scientific on it and test it empirically: can you really taste the difference? Google has world-

class statisticians and other researchers who know how to test things; but usually those “things” are “which ad is better for this query?” or something similar.

Lest you think this sort of coffee testing is always going on at Google: no, it’s not. This is the only such test I know of.

The Moment of Truth

You can do all the measurements you like on the volatile compounds, the size of the grains produced by the coffee grinder, the percentage of the coffee extracted by the brewing process, and on and on, but in the end, can you taste the difference? That person on StackExchange who said, “My rule is to ONLY purchase coffee which shows the roasting date, and then only if it is within 7 days” — could he tell the difference if we blindfolded him and gave him some coffee from fresh-roasted beans and some from 14-day old beans?

Maybe he could, actually. We’ll get to the difference between testing for yourself and testing for the general population later.

Blind Testing Shocks the World

Prior to 1976, it was almost universally believed that French wines were the best. The French thought so, of course. On May 24, 1976, a blind tasting was held in Paris. Nine of the most respected people in the wine world were given both French wines and California wines and not told which was which. Of course everyone assumed the French would win, so the news media mainly ignored it.

Shock and awe! The French wine experts preferred California wines. The wine world has never been the same, and now the world is full of wine-growing regions. The key thing here is blind tasting. What does it matter what the experts say, if you can’t taste it?

Triangle Tests

The Triangle Test is an international standard (also described here). Here’s a video that’s oriented towards production uses: it assumes you’re a manufacturer and you changed the formulation of a product somehow, and you want to know if users can tell the new product from one made in the old way:

In our test at Google, we give our testers three cups of coffee, two brewed with one set of beans, and one with the other. The sets are (1) 1.9 weeks since roast. and (2) 7.4 weeks and left open to the air. The testers didn’t know which was which, and their task was to pick out the one that was different. Jerry explains it to them at the start of this video:

The captions on the video explain how many testers would have to get it “right” to produce a statistically significant result, p < 0.05. As you’ll see, they didn’t even come close

So What Does It Mean?

Does this mean no one can tell the difference? Or that a representative sample of the world’s coffee drinkers couldn’t tell? Or that you couldn’t tell?

“No” to all of those.

Of course, though, this is data. It tilts the presumption, ever so slightly, in favor of “no one can tell.” More data might tilt it the other way.

No One Can Tell?

This was a sample of 14 people. All worked at Google (itself a major selection bias), all subscribed to the “coffee” mailing list (further selection bias), and all volunteered for the test (still more). It’s entirely possible you could give the test to 14 coffee-drinking people off the street and get different results.

How About a Representative Sample?

Political polls go to a lot of trouble to get proper representation of minorities, women, old, and young, and all important demographics. Even polling via telephone can risk over-representing people who have a phone and answer it.

Our sample was not representative at all, as we explain above.

Can You Tell?

This is where it gets interesting. Suppose you’re one of the testers who got it right, assuming several retests confirmed your perspicacity. In that case, it’s perfectly reasonable for you to insist on grinding only fresh beans, and no one’s entitled to ridicule you for it. You’ve got the data on your side!

Jerry and I did a lot of other tests with just the two of us, sometimes with one or two friends. A sample size of two, three, or four is nowhere near statistically meaningful. But here’s the thing: it’s meaningful to us.

Example: bamboo coffee filters vs. paper (maybe you didn’t even know these existed):

Both of us could taste the difference. I preferred the bamboo, because it lets through more flavor complexity, I thought. So we don’t have to care whether anyone else can taste it if we’re just buying for our own consumption. We know we can taste it.

Someone told me that Melitta claims to have “patented flavor-enhancing compounds” in their bamboo filters. I searched the patent literature and couldn’t find anything.

Try It Yourself

If you’re in the business of serving coffee to others, especially the general public, then your interests are different from someone just making it for himself & family. In the former case, your task is harder; you have to evaluate all the data.

It's fairly easy to enlist a friend or partner to give you a blind test. Can you tell pre-ground coffee from freshly ground? We’ll bet you can. Whether you care enough to grind beans every morning is another matter.

Other tests we’ve done for ourselves are:

• Should you freeze the beans? We couldn’t tell at 4 weeks; at longer durations it may matter.

• Does the grind setting on your grinder matter? It definitely does, especially if you use a drip machine; less if you use an AeroPress.

• If you use an AeroPress, does it matter if you push the plunger all the way down to the bottom, where you hear the hiss? We think not.

• Should you vacuum-seal the container the beans are in? We don’t think it matters.

All of these should not be taken as data. They’re anecdotes. Try the test yourself, if you care.