The Brookings Institution issued a new report on Friday about the H-1B program—a temporary foreign worker program for “skilled” occupations, meaning those that require a college degree—and then issued corrections to it almost immediately afterwards.

The report claims to include a wage analysis on “new data” that, “suggests that the H-1B program helps to fill a shortage of workers in STEM [science, technology, engineering and math] occupations.”  

There are two critical problems with the report’s analysis of these data:

First, the data are proprietary, meaning the data are exclusively held by the authors, thus no one can critique or review the study’s presentation of the data or its findings. (The authors obtained the data from two other researchers, who first obtained it through a Freedom of Information Act request.) This kind of approach, where researchers use data that are not available publicly, means that the data and subsequent analysis can never be checked, leaving out a critical step in the scientific process.

This step is critical because mistakes happen. In fact, just hours before it was to be released the authors shared the study with me. They did not ask for a critical review, and it would have been impossible for me to conduct one given the short time frame (typically a reviewer is given two weeks to provide input), and more importantly, I did not (and still do not) have access to the proprietary data.

As I skimmed the report I found a few peculiar items in the study. Since I have spent more than a decade working on the H-1B program, and these data are presented at an aggregate level by U.S. Citizenship and Immigration Services (USCIS) on an annual basis, I’m quite familiar with them.

The Brookings researchers had only 105,000 H-1B workers in their data but USCIS reported (in its 2010 Characteristics of Specialty Occupations report) that almost 193,000 H-1B workers were approved that year. It turns out that the Brookings study was missing nearly 88,000 workers, or approximately 45% of the data, in their analysis. This is the exact same data, so there is no logical reason for the discrepancy.

I pointed out this serious discrepancy to the authors, and they dismissed my concerns. After I repeatedly pointed out why their data didn’t make any sense—after all, USCIS was reporting the same data with many more workers—the Brookings authors realized I was correct.

They had erroneously thrown out 45% of good data before even beginning their analysis. Subsequently, their study was changed on the fly to include the data. This was an honest albeit elementary mistake, since anyone who works on the H-1B program is familiar with the size of the program.

Mistakes get made in the course of research, which is why it is critical to have multiple pairs of eyes scrutinizing the data and the analysis. But when researchers use proprietary data, as the Brookings report does, it precludes these critical steps in the scientific process.

Without public access to the data there is no way for the Brookings report to be properly reviewed.

Second, notwithstanding the major data problems in the report, the study makes many questionable analytic choices. I will only point to one of them here.

The report attempts to evaluate whether H-1B workers are underpaid. The key issue is to design your analysis so that you are comparing apples to apples. But the Brookings study fails this basic standard in a number of ways.

Its key table on wage analysis (Table 1) tries to make a comparison of H-1B wage levels against comparable American workers.

However, it does so by using a very expansive definition of comparable American workers. In calculating “American Wages” it includes all computer occupations, including technician level workers like Computer Support Specialists. This category is a significant share of all computer occupations and has relatively very low wages. By including the low wage categories in its analysis, Brookings is systematically lowering the reported American wages, thereby making H-1B wages appear higher than they are.

The upshot is that the authors are making an apples to oranges comparison.

The Brookings report adds no value to the question of H-1B wages. USCIS actually provides more detailed and more useful tables of the same data in its annual report (for example, see Tables 10, 11, and 12 here).

The U.S. Government Accountability Office attempted a similar exercise with the same data set that Brookings is using (but from an earlier year, 2008). After describing the various weaknesses in the data (see pages 82-83) the report concluded: “In light of these limitations, caution should be used in interpreting differences found in comparing estimated 2008 median U.S. citizen worker salaries and the median salaries for H-1B worker petitions submitted in 2008.” Brookings does nothing to allay or account for, or even acknowledge these limitations.

And it is well known that the H-1B program is used for cheaper labor.

Industry veteran Neeraj Gupta testified before the Senate Judiciary Committee on April 22 that most H-1Bs are being used for cheaper labor. Just a few weeks ago, a Wall Street Journal article about proposed changes to the H-1B program reported that “Indian IT professionals working in the U.S. are typically paid about 25% less than their American counterparts.” And some H-1B employers themselves have told government auditors that they hire H-1Bs because they can be paid less (for example, see page 4 of this report).

I will post a more detailed critique of the Brookings data and report later this week. For another take on the weaknesses in the report’s methodology, see U.C. Davis Professor Norm Matloff’s analysis from his email newsletter.