We recently wrote a piece in the American Prospect analyzing a recent report on arbitration by the U.S. Chamber Institute for Legal Reform—an affiliate of the U.S. Chamber of Commerce—that we found to be misleading and riddled with errors. In this blog post, written especially for those policy wonks who can’t get enough, we share more details about what’s wrong with the report.

The report—touting arbitration’s supposed benefits for workers—arrived just in time to be cited at a House hearing last month. That’s unlikely to be the last of it. The report will surely be presented by corporate lobbyists to a coterie of undecided legislators—legislators who care about access to justice and who are genuinely concerned about the impact of forced arbitration, but who also want to be responsive to the concerns raised by businesses that use it. The danger is that these legislators will believe the report’s spurious conclusion that arbitration is better for workers and vote accordingly.

Some initial skepticism is obviously in order. With its history of opposing reforms like paid family leave, a higher minimum wage, and strong overtime protections, the U.S. Chamber of Commerce is not an institution that is generally known as a champion of workers’ rights. And the report (entitled “Fairer, Faster, Better”) was written by a consulting firm that’s published previous reports like “Regulations: the more is not the merrier” and “The Regulatory Impact on Small Business: Complex. Cumbersome. Costly.”

But whatever the source, it’s important to engage with the substance, which is why we decided to take a deep dive. We discovered glaring errors, most of which we discussed in our American Prospect piece: the report’s omission of class actions and state court cases from its litigation data; its reliance on the mean rather than the median in its conclusions; its failure to account for cases that never come to light because of forced arbitration or to consider deterrence and injunctive relief; and its failure to engage with decades of prior academic studies on the topic.

However, there are even more in-the-weeds problems in addition to those in our op-ed. Here are a few:

Arbitration’s lack of transparency results in problems with the Chamber’s report. The available information on arbitration only reports the amount claimed by workers for about one-third of the cases, obstructing anyone’s ability to assess the real meaning of the results (since the amount awarded should be viewed in relation to the amount claimed). And in nearly one quarter of the 1030 arbitration cases in the study with a final decision, “information about the prevailing party was unknown or… there were awards to both plaintiffs and defendants.”

In part because of the limited sample resulting from the lack of transparency, the total number of arbitration decisions included in the report’s primary analysis was miniscule: 251 cases in which employees initiated and prevailed, over fifty states in a five-year period. This is out of 10,486 arbitration cases in a sample that’s already shrunk by the omission of class actions and state court cases. Is this sample sufficient to draw any broad conclusions? Probably not.

Also, the information about “awards to both plaintiffs and defendants,” while glossed over by the report’s authors, is in fact noteworthy. It’s pretty uncommon, if not highly unusual, for employer-defendants to get awards from courts in litigation cases initiated by ordinary employee-plaintiffs. But the arbitration results suggest that this may not be the case in arbitration. When comparing which of two options is better for workers (arbitration or litigation), if one seems to carry real risk of the worker owing money surely this factor merits discussion or at least further investigation.

The report emphasizes that most workers in the arbitration cases were not high earners. However, most employees in the arbitration sample were much higher earners than the general public. Of the arbitration sample’s employee plaintiffs, 21 percent made over $100,000, and 7 percent made over $250,000. In the general population of wage earners in 2014 (the first year of the period of the study) the shares earning over $100,000 and over $250,000 were 8 and 2 percent, respectively. So there were more than 2½ times as many high earners—and 3½ times as many very high earners—in the arbitration case sample compared with the general public. When higher-paid employees have employment disputes, their claims for lost wages, and therefore their recoveries, will logically be higher than lower-paid workers. This may make it look like workers “do better” in arbitration when actually they may just be higher earners in the first place. (Note, however, that the report does not include comparable information about plaintiffs in litigation, thus preventing any comparison of the two in this regard.)

As we explained in the American Prospect, the report’s methodology compares the results of employment arbitration with the results of litigation for the years 2014–2018, comparing cases that were initiated by employees and terminated with awards during that period. However, including only cases that were terminated with awards is misleading. Often in employment litigation, an employer files a pre-trial motion, like a motion to dismiss or motion for summary judgment. If the employer loses these motions (especially summary judgment motions), cases commonly settle shortly thereafter; such motions are often the immediate precursor to a settlement. By only comparing cases that terminated with an award from the adjudicator, the report fails to consider the considerable benefit of litigation in this regard.

The skewing of results has a particularly strong impact in relation to summary judgment motions. A summary judgment motion disposes of a case only if the employer wins, but not if the employer loses. Therefore, if there’s a summary judgment motion, and the employer wins, the report would include that case as an “employer win.” But if there’s a summary judgment motion and the employer loses, the report would not include that case as an “employee win” even though in practice, it more often than not functionally has that impact. This approach misleadingly makes the employee success rate look much worse in litigation than it really is.

The report never actually explains what it means by “employment litigation.” There’s not any one “Employment Law.” Are the cases included minimum wage and overtime cases under the Fair Labor Standards Act? Discrimination or harassment cases under Title VII? What about cases brought under the Family and Medical Leave Act, or ERISA, or other statutes? Were retaliation claims included? This lack of clarity about what laws we’re talking about makes it hard to analyze the data.

The report is like an “I Spy” book for hard-headed economists and lawyers—there are problems on so many pages.

Which is not surprising. In the context of massive worker pushback and organizing against forced arbitration, it’s a red flag from the get-go when a Chamber affiliate argues that workers don’t know what’s in their own best interest.

In this case, that red flag is warranted. The Chamber’s report has numbers and charts and percentage signs, but lacks solid evidence supporting its position. Workers aren’t stupid or gullible. They know what’s in their own best interest: not having to give up their right to seek justice as a precondition of getting a job.