The Secret Sauce of AMR's Top 25 Supply Chain Ranking

2011 marks the seventh year of AMR’s  annual Supply Chain Top 25 report. The goal of AMR’s research has always been to raise awareness of the supply chain discipline and how it impacts business. The most recent list shows Apple, Dell and P&G in the lead.

But how exactly are these lists developed?  Who says these are the best?  I interviewed Debra Hofman from AMR who shared with me some key insights into the list, and the research methodology used.

RH:  What is the history of the top 25?

DH:  The top 25 is a ranking that has been around since 2004.  My area when I first started at AMR in 2001 was in benchmarking.   I led a series of benchmarking initiatives – and this was straight quantitative benchmarks.  We did 70 of them across industries using 45 operational metrics that we defined – We then reported back to companies on how they fared against those numbers against same companies – and we got end to end benchmarks – not just the numbers – but look at interdependencies.  I believe that we don’t just look at a benchmark and use as a target, but needed to report back with a comprehensive discussion on where there are opportunities to optimize – instead of just taking a benchmark  I published the hierarchy of supply chain metrics.

I talk to companies every day about benchmarks and how to approach metrics and use them to set the targets and supply chain strategy and execute against them – and you have to segment different supply chains.

In 2007 I joined a colleague at AMR to do the top 25 ranking.  They are very connected in terms of defining by what we mean by excellence.  When I look at it from a micro perspective and determining who is best, I admit that we have adapted a high level look at that question – but how to define it?

There are no easy answers.  So for the top 25 the goal is to engender that debate – and it is a dynamic dicussion because it should be occurring and defined in a changing way.

The ranking is a combination of metrics.  One of the important things – the top 25 is not a detailed metric – it is about leadership, NOT excellence.  They have to do with each other however, and we believe that by identifying leaders we are also identifying excellence.  The top 25 are the ones that everybody points to – e.g. “I want to talk to P&G about what they are doing.”  Part of measuring leadership is about visibility.  They may be doing good things – but if nobody knows about them – then the company won’t get ranked.

RH:  So how is it exactly measured?

DH:  It is a combination of subjective and objective metrics.  We come up with a list of companies we for the aggregate list from Fortune 5000 and Forbes 500, and then also looked at companies above 10B.  We also don’t measure include non-physical supply chains.

We have 263 companies on that list – and we had 300 last year.

A composite score is made up of two major components – one is financial and one is voting and opinion.  We realized early on that we couldn’t visit every company and put out a ranking every year – and also realized that we would have to use publicly available audited financial statements.  We could get them – and we chose 3 metrics that are on their financial statements that had some relationship to supply chain activity.

The first measure is return on assets – this is a simple one that looks at high level view.  Inventory turns is the only supply chain financial statement term that is the second measure we use, and provides a view on what they are using, how they are using inventory as a buffer or not, etc..  Finally we also look at revenue growth – which is rated but not weighted as heavily.  We believed in combining operational excellence and financial excellence – and wanted to have a balance there.  We recognize that many things impact revenue growth including innovation.  So we chose those and moved to a 3 year weighted average for ROA and revenue growth – and went to a 4th quarter inventory average to avoid end of year shenanigans.  That helps to smooth things – although we don’t correct for acquisitions and divestitures.

Financial metrics thus make up 50 percent of the score in the rankings– ROA is  25, inventory 15, and revenue growth 10.  That is 50% of the score.

The other 50% is opinion-based, and this is divided equally between the peer opinion and the Gartner opinion.  Each of these are tallied tallied separately – the table in the report shows all these numbers and the opinion numbers are points NOT votes.  You get into a simple website – and put in your ranking an force rank them – and once you do that it is done – your number 1 vote gets 25 points, number 2 gets 24 points, 23 points, etc.  Then add up all the points in for all respondents, and we do the same for the Gartner peer vote.

The concept of voting is a modified Delphi method, which is based on the wisdom of crowds.  The aggregate answer you get is greater than anyone in the group’s individual opinion.  We sometimes get inquiries from people who say  “I don’t know about these companies!  But nobody knows and don’t have to know anything – you should vote based on what you know and the rest is in the noise.

When I look at the results – and when I sort by peer or Gartner voting – and it works very well – and this is what makes it interesting – there are things that you don’t agree with because it is not your list – and the transparency is very apparent.  We get people asking why didn’t I move up or down –  and we can explain very clearly with them on whether they are visible or not.

We made changes in the methodology.  It was easier to make changes when it started and harder now – to make sure not a big change.  We do welcome those comments.  We get some complaints that it is too US focused.  The answer to these questions is that these are global companies – we take the Fortune 500 not the Global 500 – as well as the Forbes Global 500.  Surprisingly the largest companies are in China not in the US!  India has only 5 large companies.  We are trying to address that over the years – and when you look at those 25 there are a lot of US companies.  It is a combination of the list and the voters and this year – I never looked at the list itself of 263 to see how many from each region.  I try not to bias the Asian companies and standardize the choice mechanism.  And we also increased the number of companies – and put a limit on number of companies in the rank.

We have struggled to expand the input from global opinion leaders, as 80% of voters are in the US.  I tried to do different things – and today 59% of voters are in the US.  Gartner had access to more people to drum up voters where they didn’t know us as well – and my goal is get the US portion down to 1/3 US 1/3 Europe and 1/3 in Asia.  Not as much footprint in Asia.  Last year there were 156 votes.

People complain you can’t get nominated unless you know somebody – which is simply NOT true.  People also call us and say that they may not want to be ranked – but we tell them you don’t have a choice…

We can’t compare the financials of non-physical inventory.  People can be smaller than 10B to vote – and would like to make the list even bigger – but not limited to voting.

How do you increase diversity of voters and companies?

We had 156 voters last year – and I have been playing with that number – and tried to get regional focus, not so much a high number, but the representation is more important to me.  We have a new system in place which allows us to do more cuts of the data.  For example, we could see how European and American and Asian voters vote.  And we published some things about that – and if you look at just European voters and how they ranked it – the first 5 companies were US companies – and BMW was 6…And if people from Europe complain that US voters focus is biasing the results, we can point to that fact.

These insights were extremely helpful in thinking about this list.  As pointed out, however, the Top 25 supply chain does not include non-physical supply chains (e.g. financial services, and insurance), and also doesn’t specifically look at process maturity associated with supply chain maturity.  This are two important factors to consider when viewing this list.