Sign up for the Opinionated About Dining™
mailing list to receive news and update alerts:



 

Seven Things Wrong With the 50Best

The release of the annual results of the 50Best List always causes a considerable amount of commotion in the restaurant world. For the record, even though the list has some obvious warts, I happen to like it and I find it useful. Face it, any publication that encourages people to dine out at the top restaurants in the world is good for anyone who fancies themselves as a dining aficionado. This is true even for myself as I have only been to 59 out of their Top 100, which gives me incentive to visit the restaurants I haven’t been to. More importantly, the list encourages people to look at dining out from a global perspective. Something that we need more of. Of course none of that precludes me from offering a critique of the list, and from what I understand, the methodology they use to compile the results.

Since the list was published, there have been a number of articles that attempt to explain how the results are derived. Steve Dolinsky’s article on the topic is best one I have come across: 50 Best Restaurants List Announced and Explained . And since Dolinsky’s article offers details about the methodology that is used (972 voters split into 27 different panels around the world each cast 7 votes, 4 for restaurants in their own region and 3 outside their region), it makes for a good starting point for a discussion. As such, may I point out the following which I have listed by bullet points. Hopefully, this will help you understand the list better, and offer some explanation as to why you might agree/disagree with their results.

1. An insufficient number of votes are cast - The list’s organizers collect 6804 votes each year (972 x 7 votes) which is an insufficient number of votes to create a worldwide ranking system that makes any real sense. And when you factor in various restrictions they place on how people can cast their vote (which I will get into in further detail below), things get much worse.

2. Selection bias – Rather than the voters being randomly selected which is the underpinning of any good survey, 34 of the 35 voters in each regional panel are selected by that region’s chair. While there is no way of knowing how this selection bias plays itself out (the 50Best lacks transparency), it is reasonable to assume that the chairs choose people that they believe are qualified to participate. So it is likely to assume that region’s chairs select people that are most like them when choosing the rest of the panel.

3. Dining experience differs across different regions of the world – If you can find me 35 voters who are based in Thailand who have eaten at a large percentage of the restaurants in the Top 100, I will happy to buy you a Pad Thai at the Thai restaurant of your choice. As I said above, I am one of the most experienced diners you will ever meet and I have only been to 59 out of their Top 100.

4. The wrongheaded assumption that the people with the most dining experience for a specific region, happen to live in that region – I have three friends who are expert on the restaurants of Japan. They respectively live in Hong Kong, Bangkok and Atlanta, Georgia. Yet the 50Best rules would prevent them from being on the Japanese voting panel.

5. Forced distribution of votes by region – Let’s assume my friend in Atlanta is on the voting panel for his region and he is forced to vote for 4 restaurants in his home region. But what if he feels that the top 4 restaurants in his home region do not deserve to be in the Top 100, let alone in his top 7? This is the reason that restaurants like Dinner by Heston or Test Kitchen in Capetown, South Africa over-perform. In regions that are not overflowing with great restaurants, the handful of good restaurants (or well-known ones) are named on every ballot. Meanwhile, because there are so many great restaurants in countries like France and Japan, the votes are split across so many restaurants that results for many great restaurants are diluted.

6. Constipation at the top of the list – It is reasonable to assume that not everyone who votes has visited the top three restaurants. And it is also reasonable to assume that that those are the restaurants that people are most likely to visit over the following year after the list is announced. Given this limitation, it will take the 972 voters a very long period of time to cycle through those three restaurants. This is why Cellar Can Roca, Noma and Osteria Francescana have been playing hopscotch at the top of the list for the past few years.

7. A lack of a unifying judging standard – Food journalists in different parts of the world see the dining experience differently. European food journalists (excluding the UK) are prone to seeing fine dining as an art form. Certain French journalists see dining as a matter of class and luxury. And many journalists in the US and UK write about dining from the perspective of consumerism and consumption. This is why the restaurants on the list from Spain veer towards the cutting edge, and why the U.S. restaurants are pretty much the same French restaurants that have been in business for 20 years. If you were to unpeel the onion as to why that is the case you will find that in general, the Spanish food media appreciates innovation in cuisine but the U.S. food media does not. Hence the lack of any cutting edge US restaurants on the list aside from Alinea.

Since my main complaint is with the list’s methodology and how it is organized, and since I am not a statistician, I turned to the world’s greatest statistical expert, Nate Silver of 538.com for his thoughts on the list and here is what he told me:

“The way they are asking people to vote within their regions seems strange. Their methodology is making it incredibly hard to compare restaurants “across" regions, which would seem to be the whole point of a world’s best list. How do New York diners feel about the food scene in Japan, and vice versa."

I guess that is a more succinct way of addressing a number of the issues I have mentioned above. I mean who here believes that the 35 people who make up the voting panel for the Eastern portion of the U.S. have a sufficient amount of experience dining in Japan? Most of the experienced diners I know would vote for places like Sushi Saito or Matsukawa as being the best restaurants in Japan. Furthermore, I am certain that they would find the idea that Ryguin and Narisawa are the best restaurants in the country rather amusing.

Which brings me to the word “best." While I understand that to many people the word best might mean the most artistic cuisine, and to another person it might mean the most profitable restaurant which also serves food above a certain standard, the idea that the world’s best restaurant can be devoid of any new artistry does not sit well with me. And to me, restaurants like Daniel, Le Bernardin, & the French Laundry haven’t created anything new in quite a long time. And I think anyone would be hard pressed to find an award ceremony where there is not a strong correlation between advancing the art or business of the subject in question, and who ends up on the list and where.

The reality is that the 50Best is really made up of two components. The first part is creating a list of the top 2-3 restaurant in each of their voting regions based on 4,050 votes which end up being cast for restaurants in the voter’s home region. Once that aspect of their list is complete, they then look at the 2754 votes that are cast for restaurants located outside of the voter’s home region in order to determine the order of the list. But as Nate Silver accurately points out, the second part of the exercise cannot fix the problems caused by the methodology they use in the first part which is using regional voting panels whose hands are tied in terms of who they can vote for by the rules. Of course, for marketing purposes it’s a great system as San Pellegrino can sell more water in South Africa and Moscow. Places where destination diners are not really traveling to eat despite the fact that they are appearing on the list. And thereby lies the rub.

Of course, this is how the media wants things to be when it comes to food. Their real interest is in attracting clicks and selling advertising, not reporting on what the best restaurants in the world might be. A case in point about this disease that they are afflicted with is Ryan Sutton’s analysis of the list in Eater titled: Five Charts That Demystify the 50Best which among other things, complains that places like India and China are “under-represented." As if it is automatic to assume that those countries actually have restaurants that deserve to be on the list. I don’t think I have seen a more blatant example of the inherent conflict between the food press and the hobby of eating out.

June 6 - Editors Note - Two different people (both of whom are Canadian which is a weird coincidence) wrote to tell me that the point I made in #5 is incorrect. It appears that 50Best voters
must cast at least 3 votes for restaurants outside of their regions. Which mean they cast all 7 votes for restaurants outside of their region if they choose. However, the existence of such a rule implies that regional voters are likely to be homers, and as such, are forced to case three votes elsewhere.



©2008-2010 SJP MEDIA, LLC • ALL RIGHTS RESERVED.
OPINIONATED ABOUT® IS A REGISTERED TRADEMARK OF SJP MEDIA, L.L.C.