The Talent

October 10, 2011 4:00 PM

Moneyball for Law Firms

Posted by Ed Shanahan

By Steve Gibson, William Henderson, Caren Ulrich Stacy, and Chris Zorn

For more than a century, baseball has been been known as America's pastime. Change comes slowly to the game, and tradition is sacred. But as the recent film Moneyball details, one key part of the baseball edifice—the way that teams select and develop their players—has undergone a revolution, thanks to the data-driven approach famously employed by Billy Beane, the general manager of the Oakland A's.

Beane's core insight was that the engrained biases of scouts and coaches caused baseball teams to misidentify and misprice talent. With careful attention to data, even a small market team could clobber the competition. Over time, nearly all major league teams have borrowed from the Oakland A's playbook and won games, even championships, using the Moneyball philosophy. Is it possible for law firms—also tradition-bound institutions—to benefit from a Moneyball approach to talent? Corporations such as IBM, Xerox, and Proctor & Gamble have done so (well before professional sports), why not law firms?  As experts in quantitative analysis, legal labor markets, and law firm recruitment, we explored this exact question with multiple law firms. Our findings might surprise you.

Far and away, the most consistent finding is that there are huge gains to be made by focusing on traits or attributes that are actually correlated with performance. A Moneyball analysis explores the association between performance and several dozen success traits that can be observed on a lawyer's resume or transcript. These range from traditional success criteria such as grades, law review, clerkships, and law school rank to nontraditional criteria that many firms overlook or give less weight to—blue- or pink-collar work experience, advanced degrees, publications, participation in team sports, etc. Using this wide range of biographical data, Moneyball analyses reveal that law firms are often systematically overvaluing some attributes, ignoring others that really matter, and generally making bad tradeoffs in both entry level and lateral lawyer "drafts."

Consider grades, law firms' primary filtering device when making hiring decisions. It turns out that grades (like home runs in baseball) do matter for success, but that there are other Moneyball factors that typically matter more and, as suspected, some that matter less. When a firm focuses excessively on grades—which only explain a small fraction of associate performance—it often systematically overlooks better-performing candidates.

What are the factors beyond grades that are positively correlated to success at firms? The answer depends on the firm. In some Am Law 100 firms, lawyers with blue- and pink-collar work experience tend to be among the top performers. While at other Am Law 100 firms in the same profit bracket, associates with blue- and pink-collar backgrounds tend to underperform. Similarly, with educational credentials, JD/MBAs have thrived at one firm, but at another competing firm only a Masters or Ph.D. in a science or technical field is correlated with success. Performance of homegrown versus lateral associates varies by firm. There are also "nonfactors" at many firms that don’t rank as positive or negative predictors of success, including participation in moot court, mock trial competitions, and clinics.

The lawyer-success profiles vary significantly across firms. Although AmLaw 100/200 firms compete over market share, just like the Yankees and Red Sox compete over wins and losses, the law firms and lawyers seem to be playing different sports—one is rugby, another polo, a third lacrosse, etc. The data support the existence of distinctive firm personalities and cultures, which are likely the product of a variety of factors, including the dominance of particular practice areas and the principles that have guided the firm’s growth. Contrary to the conventional wisdom of law firm recruiting, many firms may be better off scouting for different success factors and, in turn, hiring different candidates than their so-called peer firms.

Firm culture, however, it only one piece of the puzzle. For a law firm to apply Moneyball principles to its lawyer hiring and development process, it has to confront a simple but difficult question at the very heart of its business model—how do we keep "score" of excellent performance? For example, does the firm measure success simply in terms of billable hours, or through lawyer competencies such as initiative, problem solving and interpersonal skills, which place a lawyer in higher demand both inside and outside the firm?

Some law firms (consciously or unconsciously) make billable hours the key to increased compensation or advancement. Yet, a firm does not win market share or get more challenging and price-sensitive work by amassing hours. The abundance of client work for most law firms is the result of proactive, client- and team-focused behaviors that consistently deliver value in excess of cost. When a firm fails to measure these behaviors and instead relies on billable hours and subjective evaluations that are not competency-based, the Moneyball factors tend to bake-in the biases most prevalent within the partnership.

Bias among brilliant equity partners? Yes, it happens. A good example is attitudes toward law school pedigree. The data suggests that, in several firms, a subset of partners who attended elite law schools often give higher performance ratings to associates who also attended elite law schools—even when nonelite associates are statistically identical on every other measure. In contrast, when looking at the same group of associates, partners who did not attend elite law schools observe no performance gap.

Depending upon the firm, indulging these biases can lead to problems. While some partners have a love affair with academic credentials, they are not always strong predictors of success. In fact, we have observed instances where undergraduate honors and law review (after controlling for law school grades) have been strong negative predictors of essential associate competencies or success traits. This may sound counterintuitive, but it is consistent with other research on the traits of high-performing lawyers.

For example, on workplace personality assessments, associates tend to score highly on analytical reasoning, learning orientation, and communication—all of which are rewarded in an academic setting.  High-performing associates and partners possess these factors in equal degree, but they also spike on factors such as business awareness, decision-making, problem solving, innovation, and customer focus.  These business-oriented and more pragmatic competencies are likely the key to winning over clients.  Yet, like the sluggers of the 1990s, lawyers who have enjoyed great academic success may find it difficult to not over swing the academic bat.

The most common question related to Moneyball is "what is the return on investment?" The short answer is enormous. To date, the Moneyball models employed at these various firms have successfully predicted approximately 50–60 percent of the low performers ("C-players") and 10–33 percent of the high performers ("A-players"). Our research at one Am Law 100 firm revealed that hiring ten midlevel A-players—instead of C-players—would contribute between $2 million and $5 million to the firm's bottom line annually. That doesn't even include the positive mental and emotional benefits to partners who have less work to redo and more capable associates to whom they can delegate critical work. Fewer C-players also frees up the huge amount of time partners and staff devote to course correction and outplacement.

The more nuanced answer to the ROI question is that Moneyball is dynamic, not static. Billy Beane may have invented Moneyball, but the Red Sox used it to win the World Series and the Tampa Bay Rays became the proof-of-concept for small market clubs through their World Series appearance. The "state-of-the-art" always evolves, with new analytics constantly being developed. But the statistics are only part of the equation. Arguably, Beane focused too much on the numbers and not enough on coaching, supporting, and inspiring his players. Law firms have the opportunity to start on a much higher plane.

Moneyball in law firms is no longer an academic idea. At least a handful of forward-thinking law firms are using it to enhance their lawyer selection and development practices right now, and more will come on line in short order.  As author Frederick B. Wilcox put it, "Progress always involves risks. You can't steal second base and keep your foot on first." Right now, stealing second is pretty easy. In a few years, some firms will be rounding home.

The authors are affiliated with Lawyer Metrics, LLC, which specializes in evidence-based systems for lawyer selection, development, and performance management. Henderson, a principal in the company, is a professor of law at Indiana University; Stacy is the company's president; Gibson, is the company's chief operating officer; and Zorn, a professor of political science at Penn State University, is also a principal. 

Make a comment

Comments (4)
Save & Share: Facebook | Del.ic.ious | | Email |

Reprints & Permissions


Report offensive comments to The Am Law Daily.

This story only convinces me that there is a lot of work to be done before alternative metrics will make a meaningful contribution to law firm hiring.
As to blue collar/pink collar success clusters, in a random world statisticians would expect to find some firms with concentrations of high achieving lawyers with any characteristics you care to look for (e.g., short necks) and other firms with concentrations of low achievers with the same attribute. No guidance there.
Money ball models that give you a 50-50 chance of indentifying below average performers equals random chance and a 10% success rate in identifying high performers (those in the top 10%?) is no better than random as well.

Part 2 -
Your analysis of high billable hours is shallow and doesn't demonstrate an understanding of the business. High billable hours are closely associated with high levels of experience in a variety of matters or deep involvement in particular matters. Either of those attributes is something that is highly sought out by clients. Sure high billable hours that come from performing document reviews in discovery proceedings are not the kind of work that commands high rates, but even document review experience is a positive attribute for clients with work that requires reviewing documents. High hours tend to show that a lawyer was trusted by a client or his/her fellow lawyers. Concluding that high hours has no relationship (on average) with the ability to attract business is just wrong. Will every high biller be good at client development? Of course not, but he has at least as good a shot as the next lawyer and probably correlates better than the 1-in-10 long-shot your alternative statistics offer. Moreover, on the compensation front, an associate with hours above a firm's target generates a windfall for the firm at a very real cost to the associate. Providing such associates with additional compensation is good policy even if the associate simply performed work provided to him/her by another.

Part 3
Historically, law firms could quite successfully use simplistic metrics and attributes to guide hiring because these produced all the talent needed. Given the pyramidal structure of law firms, it is not clear that exclusively hiring 99th percentile candidates for the first rung has great value. Hiring a more heterogeneous group may be a better approach because what makes a great associate is not what makes a great partner and law firms need both. While a law firm that needs 10 associates, would like to hire 10 lawyers that would make great associates, of those 10 no more than 3 or 4 need to also have the attributes that will make great partners (and even those should be a mix of rainmakers and service partners).
During the boom years, better selection tools would have been of real value as there was a great deal of competition for top quality candidates and traditional methods had difficultly identifying sufficient quantities to fill the available positions.
Now, however, class sizes have shrunk while law school production is up. So, law firms may find sticking with the traditional selection criteria identifies an adequate supply of quality lawyers available at reasonable compensation levels.
Hopefully, the use of more sophisticated metrics will advance beyond the state depicted in your article by the time demand once again outstrips supply.

Erik, the gains reported in the essay are those beyond the firms' current selection process. Further, the clustering we observe is always measured against what we would expect randomly--those are the gains reported above. As a matter of probability, they are far, far beyond random. Our time, reputation and effort all turn on getting the methodology right. These are careful, complex, methodical projects.

Post a comment

If you have a TypeKey or TypePad account, please Sign In

By: TwitterButtons.com

From the Newswire

Sign up to receive Legal Blog Watch by email
View a Sample