November 18, 2009 4:15 PM

Welcome to the Future: Rating the Ratings

Posted by Paul Lippe

Last week we tried to keep pace with the brouhaha over the Association of Corporate Counsel Value Index, culminating in the unauthorized publishing by Above The Law of the supposedly confidential data.

Launched at its October Annual Meeting, ACC is compiling in-house lawyers' assessment of value--grading firms' performances according to six criteria--and asking whether the responding ACC member would use the law firm again. (The ACC's Susan Hackett prefers to talk about "client scores," or assessment of performance, not rankings that imply a hierarchy.)

Peter Zeughauser, ex-GC, longtime industry guru, and former ACC chairman (and contributing editor to The American Lawyer), lambasted the Value Index, calling it a "Star Chamber." Among his objections: the ratings aren't available to firms; they assess firms and not partners; there is no prescriptive feedback for firms; and the information around substantive areas and offices is confusing. [Full Disclosure: Legal OnRamp has a complementary mission with the ACC, and we collaborate where we can, but there is no current commercial relationship.]

We can score this round: ACC, 1;  Zeughauser, 0.

Coming on the heels of many negative responses by law firms to AVVO's ratings initiative, it is legitimate to ask whether Zeughauser's critique is really intended to improve the Value Index or is it another manifestation of firms' (understandable, but unsustainable) desire to control reputation. 

The Value Index reflects the big changes roiling the universe (but until recently masked in law by the boom in demand). We now live in a legal world which is arguably the first professional services marketplace in which buyers (in-house lawyers) are as able to judge quality as sellers, and a Web world where information is ever more transparent. Even if the Value Index were hit by the proverbial neutron bomb, ACC members and other in-house lawyers would continue to candidly share assessments with each other, as they have done for decades.

Law firms are not universally opposed to ratings. This is a $300 million plus industry--Martindale-Hubbell, Chambers, SuperLawyers-- where lawyers manage their ratings through intermediaries.

In my old (software) company, we collected customer satisfaction data through a third-party service. The key question was, "would you recommend XYZCorp to your peers?" The data collection was anonymous (otherwise, obviously, the customers wouldn't have been as forthcoming), but we got very worried whenever we saw "would not recommend" go up and jumped into action. We also knew customers would judge us on their own experience rather than be unduly swayed by someone else's rating. If firms are worried about negative, but anonymous, assessments, they should be getting the feedback directly from clients before the ACC does.

I wrote a year ago that, "transparency is the strong force in the universe"--opposing transparency is almost certain to fail. In modern parlance, "you can't fight the web," or as Justice Brandeis wrote in Whitney v. California, "the remedy to be applied [to falsehood] is more speech, not enforced silence."

What matters is not the desire of those being rated to avoid critical ratings, which is natural and human, but the raters' need for better information. On Wednesday, I spoke at an Ark Conference in San Francisco on the theme of alternative fees. Surprisingly, many firm lawyers assumed malevolence by clients in the use of alternative fees, until forced to acknowledge that there was no evidence of such malevolence and no reason to think malevolence was in clients' interests. (Psych 101 would suggest a projection of motivation.) We need better ratings information to move away from the billable hour, both to assess performance in individual matters to determine the variable component of fees, and to help clients select firms for prospective matters.

Zeughauser has flagged some legitimate issues, although not a legitimate criticism of the ACC, which is not purporting to set up a comprehensive system. While Justice Brandeis was right in theory, in reality, negative assessments take on far more weight than favorable assessments, and damage to reputation is very hard to remedy, i.e., you "can't unring the bell."

An article in this week's New Yorker discusses Michelin's restaurant rating system: Michelin "operates on the principle that only reviews by anonymous, professionally trained experts can be trusted for accurate assessments."

A trusted rating system will contain the following elements (and other industries from Angie's List to Zagats have managed to move to effective ratings systems):

--Based on multiple criteria, not a single criterion, with the ability to refine criteria over time.

--Raters must be anonymous, but have some ability to share their identity with their trusted community, and some way of assuring they are both knowledgeable and not malevolent.

--Ratings should be provided by multiple raters with perhaps some minimum threshold before the rating could be published.

--Some ability to "appeal" or address negative ratings, although this has to be reconciled with the rater's desire for anonymity.

--End "pay to play."

--The rating organization must not have an incentive to suppress unfavorable ratings, or force ratees to pay for favorable ratings.

--Some normalization so you could see whether a particular rater was atypically harsh or favorable.

--Accessible in multiple places, so no one entity completely controls the ratings.

--Some ability to look at either the firm or partner level.

Any effective rating system must move us beyond Lake Wobegon--acknowledging that not every lawyer is above average--and provide clients and firms with information that helps them do their job better.

Make a comment

Comments (2)
Save & Share: Facebook | Del.ic.ious | | Email |

Reprints & Permissions

The comments to this entry are closed.