Walk the grounds of a barbecue competition like the Kingsford Invitational Barbecue Competition and you will repeatedly hear certain phrases about who made the best barbecue. From judges to spectators, snippets of conversations include the same questions: Did it blend "heat and sweet"? Did it make you want to smile? Was there a "command of heat"? Was there a perfect rib/bone connection? Was it sweet, spicy, and sexy?
The Daily Meal, fresh off its Ultimate BBQ Road Trip, headed to the Kingsford Invitation Barbecue Competition, which pits the best-of-the-best in the barbecue world, chosen from eight top-tier regional contests like Memphis in May and the Jack Daniel’s World Championships, against each other, to see how the pros judge barbecue contests. Their insight may surprise casual observers.
The first thing to realize about judging a barbecue contest is that pit creations and rules do not necessarily correlate to the barbecue found in a local restaurant; such concentrated flavors are too labor-intensive for the normal palate and too expensive to produce daily.
"Judging barbecue in a competition setting is a complete 180 degrees from the typical backyard experience," admits Scott Jones, a certified Kansas City Barbeque Society (KCBS) judge. "It’s a pretty intense proposition, especially when trying to stay objective. I suspect if judges are honest about the process, personal preferences always creep in."
As a result, most competitors are there to impress the judges with "orgasmic," in-the-moment creations, according to Brad Orrison, judge at the 2012 Kingsford Invitational Barbecue Competition and owner of The Shed BBQ & Blues Joint in Ocean Springs, Miss.
It’s about making the best on competition day, and how the judges’ palates taste it that day.
"I want to blow you, as a judge, away with one bite," said Tommy Houston, owner of Checkered Pig BBQ in Martinsville, Va., and one of the eight competitors at Kingsford Invitational Barbecue Competition.
Barbecue Rating Systems: A Primer
There are many types of rating systems, however, which vary from region to region, and some are pretty complex. In a typical Kansas City Barbeque Society competition, for example, there are three categories: appearance, taste, and tenderness.
Sounds simple, right?
But read further, because the rating method on the Kansas City Barbeque Society’s website looks like a complicated algorithm: "The scoring system is from nine to two… nine excellent, eight very good, seven above average, six average, five below average, four poor, three bad, and two inedible. The weighted factors for the point system are: appearance -0.5714, taste -2.2858, and tenderness -1.1428."
Confusing? When most people enjoy something, they say things like "Man, that tasted good," or "Wow, that was awesome." Not, "Tastes like 2.2858."
In Memphis, there is a different system called Memphis Barbecue Network, which focuses only on pork categories: whole hog, pork shoulder, and pork rib.
"In on-site judging, the teams are evaluated on their personal and site appearance, their presentation to the judges, the appearance of the entry, the tenderness and texture of their entry, and, finally, the flavor of their entry. In blind judging, only the appearance, tenderness, and flavor of each entry are evaluated. Each judge also assigns a score for Overall Impression to each entry, which takes into account the judge’s entire experience with that entry," according to the Memphis Barbecue Network’s judging guidelines.
Less perplexing, to be sure, but still a bit vague.
"I think that it is totally subjective to what the judge thinks is good barbecue, not what the competitor thinks. Too many times competitors think that they have the best stuff only to find out that what they made wasn’t good," said Richard Wachtel of the website Grilling with Rich.
"An interesting point to think about is that you can go to a barbecue competition in, let’s say, Virginia, win the competition and then put together the same exact flavor profiles and do the same exact thing in, let’s say North Carolina, which is only a couple of hours away, and come in dead last. Totally subjective," said Wachtel.
At the Kingsford Invitational, a new rating system was invented specifically for this contest, in which contestants competed for $50,000. The group of seven judges, six main and one tiebreaker, participated in double blind-box judging and scoring on a 14-point Hedonic Scale, co-created by one of the judges, Meathead Goldwyn, and based on a system Goldwyn had originally brought to the beverage world for taste comparisons.
"We needed a scale that works better and codifies repeatability while also bringing a simplicity to the process," said Goldwyn. "One number is all you need."
How does Kingsford judging differ fundamentally from other contests?
First, it allows judges to compare samples with one another. Second, the 14-point Hedonic Scale, with scores from 0 to 13 (from 12-13 for "like extremely" to 0-1 for "dislike extremely"), was developed by Goldwyn with Dr. Harry Lawless of Cornell University. In the Kingsford Invitational Scoring System, these particular ratings are "simple and easy. There is no arithmetic… There are no substitutions. Scales of 9 or 10 are easily ignored by judges who may choose to apply their own 10-point 'Miss America' scale."
"Is it an objective or subjective process? How does one define measurability when rating different types of barbecue from different regions with different styles? There is so much fragmentation across regions with variable sanctioning bodies," said Kingsford judge Tuffy Stone of Q Barbecue in Midlothian, Va., of the considerations made during judging.
At some point hairs will have to be split.