October 10, 2011 4:00 PM
Moneyball for Law Firms
Posted by Ed Shanahan
By Steve Gibson, William Henderson, Caren Ulrich Stacy, and Chris Zorn
For more than a century, baseball has been been known as America's pastime. Change comes slowly to the game, and tradition is sacred. But as the recent film Moneyball details, one key part of the baseball edifice—the way that teams select and develop their players—has undergone a revolution, thanks to the data-driven approach famously employed by Billy Beane, the general manager of the Oakland A's.
Beane's core insight was that the engrained biases of scouts and coaches caused baseball teams to misidentify and misprice talent. With careful attention to data, even a small market team could clobber the competition. Over time, nearly all major league teams have borrowed from the Oakland A's playbook and won games, even championships, using the Moneyball philosophy. Is it possible for law firms—also tradition-bound institutions—to benefit from a Moneyball approach to talent? Corporations such as IBM, Xerox, and Proctor & Gamble have done so (well before professional sports), why not law firms? As experts in quantitative analysis, legal labor markets, and law firm recruitment, we explored this exact question with multiple law firms. Our findings might surprise you.
Far and away, the most consistent finding is that there are huge gains to be made by focusing on traits or attributes that are actually correlated with performance. A Moneyball analysis explores the association between performance and several dozen success traits that can be observed on a lawyer's resume or transcript. These range from traditional success criteria such as grades, law review, clerkships, and law school rank to nontraditional criteria that many firms overlook or give less weight to—blue- or pink-collar work experience, advanced degrees, publications, participation in team sports, etc. Using this wide range of biographical data, Moneyball analyses reveal that law firms are often systematically overvaluing some attributes, ignoring others that really matter, and generally making bad tradeoffs in both entry level and lateral lawyer "drafts."
Consider grades, law firms' primary filtering device when making hiring decisions. It turns out that grades (like home runs in baseball) do matter for success, but that there are other Moneyball factors that typically matter more and, as suspected, some that matter less. When a firm focuses excessively on grades—which only explain a small fraction of associate performance—it often systematically overlooks better-performing candidates.
What are the factors beyond grades that are positively correlated to success at firms? The answer depends on the firm. In some Am Law 100 firms, lawyers with blue- and pink-collar work experience tend to be among the top performers. While at other Am Law 100 firms in the same profit bracket, associates with blue- and pink-collar backgrounds tend to underperform. Similarly, with educational credentials, JD/MBAs have thrived at one firm, but at another competing firm only a Masters or Ph.D. in a science or technical field is correlated with success. Performance of homegrown versus lateral associates varies by firm. There are also "nonfactors" at many firms that don’t rank as positive or negative predictors of success, including participation in moot court, mock trial competitions, and clinics.
The lawyer-success profiles vary significantly across firms. Although AmLaw 100/200 firms compete over market share, just like the Yankees and Red Sox compete over wins and losses, the law firms and lawyers seem to be playing different sports—one is rugby, another polo, a third lacrosse, etc. The data support the existence of distinctive firm personalities and cultures, which are likely the product of a variety of factors, including the dominance of particular practice areas and the principles that have guided the firm’s growth. Contrary to the conventional wisdom of law firm recruiting, many firms may be better off scouting for different success factors and, in turn, hiring different candidates than their so-called peer firms.
Firm culture, however, it only one piece of the puzzle. For a law firm to apply Moneyball principles to its lawyer hiring and development process, it has to confront a simple but difficult question at the very heart of its business model—how do we keep "score" of excellent performance? For example, does the firm measure success simply in terms of billable hours, or through lawyer competencies such as initiative, problem solving and interpersonal skills, which place a lawyer in higher demand both inside and outside the firm?
Some law firms (consciously or unconsciously) make billable hours the key to increased compensation or advancement. Yet, a firm does not win market share or get more challenging and price-sensitive work by amassing hours. The abundance of client work for most law firms is the result of proactive, client- and team-focused behaviors that consistently deliver value in excess of cost. When a firm fails to measure these behaviors and instead relies on billable hours and subjective evaluations that are not competency-based, the Moneyball factors tend to bake-in the biases most prevalent within the partnership.
Bias among brilliant equity partners? Yes, it happens. A good example is attitudes toward law school pedigree. The data suggests that, in several firms, a subset of partners who attended elite law schools often give higher performance ratings to associates who also attended elite law schools—even when nonelite associates are statistically identical on every other measure. In contrast, when looking at the same group of associates, partners who did not attend elite law schools observe no performance gap.
Depending upon the firm, indulging these biases can lead to problems. While some partners have a love affair with academic credentials, they are not always strong predictors of success. In fact, we have observed instances where undergraduate honors and law review (after controlling for law school grades) have been strong negative predictors of essential associate competencies or success traits. This may sound counterintuitive, but it is consistent with other research on the traits of high-performing lawyers.
For example, on workplace personality assessments, associates tend to score highly on analytical reasoning, learning orientation, and communication—all of which are rewarded in an academic setting. High-performing associates and partners possess these factors in equal degree, but they also spike on factors such as business awareness, decision-making, problem solving, innovation, and customer focus. These business-oriented and more pragmatic competencies are likely the key to winning over clients. Yet, like the sluggers of the 1990s, lawyers who have enjoyed great academic success may find it difficult to not over swing the academic bat.
The most common question related to Moneyball is "what is the return on investment?" The short answer is enormous. To date, the Moneyball models employed at these various firms have successfully predicted approximately 50–60 percent of the low performers ("C-players") and 10–33 percent of the high performers ("A-players"). Our research at one Am Law 100 firm revealed that hiring ten midlevel A-players—instead of C-players—would contribute between $2 million and $5 million to the firm's bottom line annually. That doesn't even include the positive mental and emotional benefits to partners who have less work to redo and more capable associates to whom they can delegate critical work. Fewer C-players also frees up the huge amount of time partners and staff devote to course correction and outplacement.
The more nuanced answer to the ROI question is that Moneyball is dynamic, not static. Billy Beane may have invented Moneyball, but the Red Sox used it to win the World Series and the Tampa Bay Rays became the proof-of-concept for small market clubs through their World Series appearance. The "state-of-the-art" always evolves, with new analytics constantly being developed. But the statistics are only part of the equation. Arguably, Beane focused too much on the numbers and not enough on coaching, supporting, and inspiring his players. Law firms have the opportunity to start on a much higher plane.
Moneyball in law firms is no longer an academic idea. At least a handful of forward-thinking law firms are using it to enhance their lawyer selection and development practices right now, and more will come on line in short order. As author Frederick B. Wilcox put it, "Progress always involves risks. You can't steal second base and keep your foot on first." Right now, stealing second is pretty easy. In a few years, some firms will be rounding home.
The authors are affiliated with Lawyer Metrics, LLC, which specializes in evidence-based systems for lawyer selection, development, and performance management. Henderson, a principal in the company, is a professor of law at Indiana University; Stacy is the company's president; Gibson, is the company's chief operating officer; and Zorn, a professor of political science at Penn State University, is also a principal.Make a comment