The career advice website Vault published another set of "best firms to work for" and "best firms for diversity" rankings. In total, there are twenty-four rankings, e.g. there is a ranking for hours, for training, for compensation, for culture, and so on. For each ranking, Vault asks lawyers to grade their own firm. It then uses these grades to rank the firms. For example, if Firm A receives an average grade of 10.0 from its attorneys, and Firm B receives an average grade of 8.5 from its attorneys, Firm A will be ranked higher than Firm B. Thus a firm can attain a high ranking by pressuring its attorneys to lie, and give it undeservedly high scores.
For example, take one of the rankings -- the "best firm for hours" ranking. O'Melveny is known for its "unpredictab[le]" and "long hours." Despite this, O'Melveny's attorneys rated it so highly on hours that it's ranked as the #1 firm for hours. Or look at the "best firm for diversity" ranking. O'Melveny is a white-privileged firm. Yet its attorneys rated it so highly on diversity that it's ranked as the #2 firm for diversity. In fact, O'Melveny's attorneys rated it so highly that it ranked #1, #2 or #3 in almost all twenty-four of these self-reported rankings. Due to this consistency, I wonder if any of its attorneys gave it the highest score on each question, without bothering to read it.
Why would they do that? I recall how O'Melveny's partners pushed them with voice-mails, emails and meetings -- similar to a tactic reportedly used to get Trump University a 98% approval rating. Maybe it was that. Maybe they're scared because they work for an organization that retaliates if they complain. Maybe they sincerely feel that O'Melveny is the #1 firm in all areas. Maybe it's something else. I don't know, but these rankings aren't useful, because Vault's method rewards dishonesty.
So let's reward honesty. By comparing what firms say about themselves to the truth, you can grade a firm's candor. You can then use those grades to rank firms according to their honesty.
For example, take Vault's "best firm for hours" ranking. You can get data on the number of hours worked at each firm. With that data, you can correctly rank firms according to hours. For example, if Firm X's associates work 2,042 hours per year on average, while Firm Y's associates work 1,830 hours per year, then Firm Y would rank higher, because it's a better firm for hours.
Now that you have the correct hours ranking, you can compare it to Vault's self-rated hours ranking. For example, say a firm rated itself so highly that it ranked #1 in Vault's ranking, but it only ranked #91 in the correct ranking (because its attorneys work more than those at 90 other firms). You can conclude that this firm is not as honest, as a firm that rated itself accurately. You can even give it an honesty grade, of negative 90 (its #1 position in Vault's ranking, minus its #91 position in the correct ranking). You can repeat this until every firm has an honesty grade and, finally, you can rank the firms according to that grade. This would give you an honesty ranking.
You can do the same thing with Vault's "best firm for compensation" ranking, because compensation can also be objectively measured and ranked. Unfortunately, I can't do these calculations because I don't have data on each firm's hours and discretionary compensation.
Looking at publicly available data . . . how about comparing American Lawyer's diversity ranking (which is based on something objective, the percentage of minority attorneys at each firm) with Vault's diversity ranking (which is based solely on a firm's rating of its own diversity). That's the chart below. Please note it only contains the twenty-five firms that rated themselves diverse enough to make Vault's diversity ranking. So, for example, it excludes seven of American Lawyer's ten most diverse firms.
Or consider Vault's "100 most prestigious firms" ranking. This is Vault's most widely read ranking, and there's something interesting about it. For this ranking, Vault doesn't let lawyers grade their own firm; a firm's grade comes from attorneys outside of the firm. So it's probably free of manipulation. A firm can pressure its employees to lie for it, but it can't control others. How about comparing this prestige ranking to Vault's "best firms to work for" ranking (where a firm's grade comes solely from its own attorneys)? Sure, these two rankings do not measure the same thing. The former ranks prestige, and the latter ranks the best firms to work for. But there may be some overlap between these two concepts. That's the chart below. Please note it only contains the twenty-five firms that rated themselves high enough to make the "best firms to work for" ranking. So, for example, it excludes the four most esteemed firms in the world. (Apparently, Vault advises lawyers to avoid those firms. Instead, they should work at a firm that manipulates a process to anoint itself. Thanks Vault.)
Or perhaps I just don't have the data to do this ranking. Regardless, a more incontrovertible point is that Vault's self-reported rankings are manipulable. If Vault wants to keep publishing them, there are a number of ways it can correct for manipulation. One would be a control question and an honesty score, which readers could use to adjust the results. As noted above, the questions on hours or compensation could be control questions. Or, instead of using self-reports, it could use objective data like American Lawyer does with its diversity ranking. Whatever the solution, Vault, please fix this to avoid misleading people into poor decisions.
Firsthand, First hand, Akin Gump, Baker Donelson, Cahill Gordon, Cleary Gottlieb, Clifford Chance, Crowell Moring, Debevoise Plimpton, Finnegan Henderson, Fried Frank, Gibson Dunn, Goodwin Procter, Jenner & Block, Jenner Block, Kirkland Ellis, Kramer Levin, Latham Watkins, Littler Mendelson, Milbank Tweed, Morrison Foerster, Orrick Herrington, Patterson Belknap, Paul Hastings, Perkins Coie, Pillsbury Winthrop, Proskauer Rose, Robins Kaplan, Ropes & Gray, Schiff Hardin, Shearman Sterling, Shook Hardy, Sidley Austin, Thompson Knight, White & Case, Williams Connolly, Vault most prestigious law firms, winter bonus, annual bonus, bonus news
Why would they do that? I recall how O'Melveny's partners pushed them with voice-mails, emails and meetings -- similar to a tactic reportedly used to get Trump University a 98% approval rating. Maybe it was that. Maybe they're scared because they work for an organization that retaliates if they complain. Maybe they sincerely feel that O'Melveny is the #1 firm in all areas. Maybe it's something else. I don't know, but these rankings aren't useful, because Vault's method rewards dishonesty.
So let's reward honesty. By comparing what firms say about themselves to the truth, you can grade a firm's candor. You can then use those grades to rank firms according to their honesty.
For example, take Vault's "best firm for hours" ranking. You can get data on the number of hours worked at each firm. With that data, you can correctly rank firms according to hours. For example, if Firm X's associates work 2,042 hours per year on average, while Firm Y's associates work 1,830 hours per year, then Firm Y would rank higher, because it's a better firm for hours.
Now that you have the correct hours ranking, you can compare it to Vault's self-rated hours ranking. For example, say a firm rated itself so highly that it ranked #1 in Vault's ranking, but it only ranked #91 in the correct ranking (because its attorneys work more than those at 90 other firms). You can conclude that this firm is not as honest, as a firm that rated itself accurately. You can even give it an honesty grade, of negative 90 (its #1 position in Vault's ranking, minus its #91 position in the correct ranking). You can repeat this until every firm has an honesty grade and, finally, you can rank the firms according to that grade. This would give you an honesty ranking.
You can do the same thing with Vault's "best firm for compensation" ranking, because compensation can also be objectively measured and ranked. Unfortunately, I can't do these calculations because I don't have data on each firm's hours and discretionary compensation.
Looking at publicly available data . . . how about comparing American Lawyer's diversity ranking (which is based on something objective, the percentage of minority attorneys at each firm) with Vault's diversity ranking (which is based solely on a firm's rating of its own diversity). That's the chart below. Please note it only contains the twenty-five firms that rated themselves diverse enough to make Vault's diversity ranking. So, for example, it excludes seven of American Lawyer's ten most diverse firms.
Or consider Vault's "100 most prestigious firms" ranking. This is Vault's most widely read ranking, and there's something interesting about it. For this ranking, Vault doesn't let lawyers grade their own firm; a firm's grade comes from attorneys outside of the firm. So it's probably free of manipulation. A firm can pressure its employees to lie for it, but it can't control others. How about comparing this prestige ranking to Vault's "best firms to work for" ranking (where a firm's grade comes solely from its own attorneys)? Sure, these two rankings do not measure the same thing. The former ranks prestige, and the latter ranks the best firms to work for. But there may be some overlap between these two concepts. That's the chart below. Please note it only contains the twenty-five firms that rated themselves high enough to make the "best firms to work for" ranking. So, for example, it excludes the four most esteemed firms in the world. (Apparently, Vault advises lawyers to avoid those firms. Instead, they should work at a firm that manipulates a process to anoint itself. Thanks Vault.)
Or perhaps I just don't have the data to do this ranking. Regardless, a more incontrovertible point is that Vault's self-reported rankings are manipulable. If Vault wants to keep publishing them, there are a number of ways it can correct for manipulation. One would be a control question and an honesty score, which readers could use to adjust the results. As noted above, the questions on hours or compensation could be control questions. Or, instead of using self-reports, it could use objective data like American Lawyer does with its diversity ranking. Whatever the solution, Vault, please fix this to avoid misleading people into poor decisions.