On the 50th anniversary of the Freedom of Information Act, here are ProPublica reporters’ most frustrating public record failures.
This month marks the 50th anniversary of the Freedom of Information Act, which was designed to give the public the right to scrutinize the records of government agencies. Almost no one needs public records more than an organization like ProPublica, whose mission is producing work that “shines a light on exploitation of the weak by the strong and on the failures of those with power to vindicate the trust placed in them.”
Yet almost every reporter on our staff can recite aneurysm-inducing tales of protracted jousting with the public records offices of government agencies. Local, state and federal agencies alike routinely blow through deadlines laid out in law or bend them to ludicrous degrees, stretching out even the simplest requests for years. And they bank on the media’s depleted resources and ability to legally challenge most denials.
Many government agencies have gutted or understaffed the offices that respond to public records requests. Even when agencies aren’t trying to stymie requests, waits for records now routinely last longer than most journalists can wait — or so long that the information requested is no longer useful. This, in turn, allows public agencies to control scrutiny of their operations.
There’s little reason to hope things will improve. Last week, President Obama, who has repeatedly broken promises to deliver new levels of transparency, signed the FOIA Improvement Act of 2016. The act writes the presumption of disclosure clearly into law, pledges to strengthen the FOIA Ombudsman and creates a single FOIA portal for agencies to receive requests, among other user-friendly provisions. But the act explicitly provides no new resources for implementing these provisions.
To provide a sense of the difficulties encountered by ProPublica reporters trying to access public records, we are recounting some of our battles on the Freedom of Information front at all levels of government:
Cezary Podkul
Last year, a whistle blower reached out to ProPublica with an enticing tip: Shoddy oversight by New York city and state regulators was allowing real estate owners to improperly collect more than $100 million in tax breaks while depriving tenants of rent-regulated apartments as required by law.
The tip from Steve Werner, an analyst at the city’s Department of Housing Preservation and Development, eventually led to our “Rent Racket” series, which sparked outrage and prompted state and local lawmakers to introduce legislation to address the problems. But my experience with New York State’s FOI law along the way shows how public officials can easily, and without penalty, abuse it.
Werner told me he had years’ worth of memos and other documents detailing the problems he’d noticed and his repeated warnings to superiors. He couldn’t provide us with copies, however, because he was, and still is, employed at the agency. And we feared that asking him to do so could compromise his job or subject him to retaliation. So instead I filed a FOIL request for the information. Getting it via official channels, I thought, would protect Werner, give the agency a chance to weigh in, and still allow us to press ahead.
Knowing that documents exist makes such requests interesting tests of both the integrity and transparency of government agencies. In this case, I thought I would be able to describe the documents clearly enough that it would be hard for the agency to say they couldn’t find them.
So last October, I sent HPD a FOIL letter requesting emails between Werner and his superior that contained a few specific terms, such as “J-51” and “421-a,” the names of the two tax breaks at issue. I quickly got letters back denying this and similar requests because they did not “adequately describe the records you seek.”
I knew this wasn’t true because inside the agency two of my requests had been shared with Werner, who told me he was able to locate the exact records I sought. But, he said, as he was assembling the records for the agency’s then-FOIL officer, Donald Appel, he was told by his superior to stop working on my request.
So I filed a follow-up request asking Appel to provide me with “copies of any records Stephen Werner has assembled in response to my previous FOIL requests.” Appel gave this request a new number, 1527-15, and denied it a few days later on the grounds that HPD “does not possess or maintain any records responsive to your request.”
A few days after my second denial, Werner emailed HPD’s deputy general counsel, Mary-Lynne Rifenburgh, and cc’d me. He informed Rifenburgh that he’d assembled 340 pages of records in response to my FOIL requests and included the keywords in the titles and bookmarks of an attachment he’d emailed Appel.
I then responded to politely remind Rifenburgh that HPD did, in fact, have the records I was requesting and that I would refile a fresh request to receive them. That same day, Ifiled yet another request, this time for the records Werner referenced in his email to Rifenburgh – about as specific as one can get with any FOIL request.
Then I waited. And waited.
Finally, in April 2016, after five months had passed, I sent HPD an email demanding immediate action. I cc’d ProPublica’s president (our lawyer) and New York’s Committee on Open Government, which helps reporters navigate the FOI law. HPD then issued a vague summary denial citing sections 87(2)(a) and 87(2)(g) of the Public Officers Law. The two sections prohibit disclosure of records that are specifically exempt by state or federal law, and inter- or intra-agency communications that are not statistical or factual tabulations of data, among other things.
It is unclear why this denial was not sent in reply to my initial request or why the agency said it couldn’t locate the records when it clearly could. But it raises important questions: Does the agency routinely lie – without consequence – about the existence of public records? Do higher-ups interfere with the workings of the FOI law? Does the agency routinely drag out such requests to penalize and interfere with reporting efforts?
I asked HPD these questions. A spokeswoman said the agency handled my request “appropriately.”
Werner, 70, works as an analyst and programmer in the agency’s statistical division, so there is a very good chance that much of the work he’s produced is, in fact, statistical or factual in nature and not exempt from disclosure. At a City Council hearing in February, he described some of the work he’s done and how its disclosure could help better inform council members and policymakers about lax oversight over important housing tax breaks. That sounds like information that would have strengthened my reporting and helped document how long agency officials were aware they had a problem.
But short of suing the agency - an expensive proposition that could take years if the agency actively resists and appeals when it loses – I can’t find out. And neither will the public — unless lawmakers ask the agency for the records and ask the tough questions that I couldn’t get answered.
Justin Elliott
A simple, solitary FOIA request tells you everything you need to know about living under what President Obama has called The Most Transparent Administration Ever.
To wit: on March 16, I submitted a request for emails to the Department of Justice.
This was no fishing expedition: I wanted the emails of just two officials, the former U.S Attorney General and his chief of staff, and to make it even easier, I only wanted those from a few-months-long period and about a single lawsuit.
When I was given an estimated completion date of April 27 on the government’s FOIA website, I was satisfied: That would give me plenty of time to incorporate any documents into my story. So far, so good.
April 27 came and went and I received no word. A week after the deadline, I called and spoke to the FOIA officer handling my request.
“I’ve got to apologize because I had overlooked this one,” he told me. “And that means I need to work on it today.”
Ten days later, on May 13, I had still not received even an acknowledgement of my request. I called back.
“Unfortunately it doesn’t appear that they took any action in response to my email,” the FOIA officer told me, referring to another office in his department. This had happened, he explained, “because I sent them an email with a tagline from the system, which they tend to ignore. I need to send an email saying we need action on this immediately.”
Um, not reassuring.
Finally on May 16 — several weeks after I was supposed to have had the documents I requested in hand — I received a form letter officially acknowledging that my request had been received.
But now there was a new wrinkle. My request had been placed on the dreaded “complex” track.
I called my FOIA officer, James Davis, back. Why had my request, which I deliberately crafted to be as narrow as possible, been deemed complex?
“The complexity factor is based on the fact that we have to go to the Attorney General’s office and get the emails,” my FOIA officer told me. Huh?
The FOIA office, it turns out, is part of something called the Office of Information Policy. And, cue incredulous look, the only request to the Justice Department that would be deemed simple would be a request for materials from the Office of Information Policy itself. “Anything that goes outside our office is not a simple request,” he said.
So by asking the Justice Department for materials related to the man who headed it, my request automatically qualified as having “unusual circumstances” and was placed on the “complex” track.
Two weeks later, in June, I called for an update. My request, I was told, had been transferred to another FOIA officer.
By this time I should not have been surprised when the new officer told me it would be at least three more weeks before another part of the agency would even do a search for the emails I requested. At that point, they would need to be reviewed and redacted.
Then all redactions would be sent for review and approval by the Attorney General’s own lawyers. That process would take at least an additional two months.
If I’m lucky, I’ll receive any emails in the fall, months after my story will have been published.
Paul Kiel
In the summer of 2010, with the foreclosure crisis still grinding on, I filed a FOIA with the Treasury Department. By that time, I had already published plenty of stories about the government’s flat-footed response to the crisis, the centerpiece of which was a program called Home Affordable Modification Program or HAMP. To homeowners facing foreclosure, HAMP was a four-letter word, an acronym that served as a symbol for the anxiety and rage they felt at the sheer incompetence of the nation’s largest banks, which had been trusted with cleaning up the mess they’d made and were doing a really poor job.
My FOIA was pretty specific: I wanted the government’s audits of the big banks in the program. HAMP was premised on the idea that, with the proper financial incentives (billions of dollars), banks would willingly make the mortgages of struggling homeowners more affordable. But the program was long on carrots and short on sticks. I knew that auditors were supposed to be checking out these banks to see if they were following the government’s rules for performing modifications. For the Treasury Department, it should have been simple to find these reports and send them to me.
Instead, my experience turned out to mirror that of a homeowner caught in HAMP. Simply getting a response at all was the big challenge. Then, when the response came after 10 months of phone calls, it was a flat no. Why? Because releasing the reports would violate the banks’ trade secrets. I appealed. Among my points: How could breaking government rules constitute a trade secret? I also found that some of the banks had not even objected to having the documents released. So why was Treasury withholding them?
Months later, I was gratified to learn that I had won my appeal. But the Treasury Department released the documents in drips and drabs and with hefty redactions. As I read through them I knew why. In late 2012, two years after starting the process, I had enough material to publish my story. Headlined “Secret Documents Show Weak Oversight of Key Foreclosure Program,” the story revealed that the government had not completed a major audit of the two largest banks in the program, Bank of America and Wells Fargo, until over a year after the program launched. It also showed the friendly back and forth between the banks and auditors, even as the banks failed to comply with the program’s guidelines with no consequences.
It was an important story, but because of the Treasury Department’s unwarranted delay in releasing public records, it came late in the game. The crisis had passed its peak, and for millions of homeowners, it was too late. By dragging its feet, the Treasury Department had succeeded in keeping information from the public when it might have had the greatest effect.
Charles Ornstein
A few years back, after we’d compiled the first-ever public database of drug company payments to doctors, we thought it’d be smart to see if any doctors working for the federal government were taking money or meals from the companies, perhaps in violation of the government’s rules.
It should’ve been easy to find out. The U.S. Office of Personnel Management considers the names, titles and salaries of most federal employees to be a matter of public record and will release that information in response to a FOIA request. So, I filed one. (Here’s a sample of what an organization has done with the data.)
Some agencies — the CIA, the FBI, the Secret Service and the National Security Agency — won’t release the names of their employees for obvious reasons: They could be targeted. But doctors paid with tax dollars? No problem, I figured.
The Office of Personnel Management sent me a trove of records from most federal agencies, including the Department of Veterans Affairs. But a short time later, I received a letter from the Defense Department, which had been forwarded my request, saying it wouldn’t release the names of any of its employees, be they active-duty personnel or civilians. The reason: Since 9/11, we are at war.
I quickly appealed. The military health care system is vast and doctors who treat service members receive scant scrutiny.
“While I understand that some personnel are involved in sensitive operations, I cannot understand why a blanket exemption would apply to all DOD personnel,” I wrote in my appeal letter in March 2011. “For instance, what justification is there for denying information on civilian staff members, including clerical workers, political appointees and medical personnel? Even for enlisted men and women, not all are stationed abroad or involved in dangerous operations.”
A few months later, the Pentagon’s FOIA office sent along court rulings supporting its decision and said I would likely lose if I proceeded with my appeal. After a phone conversation, the DOD asked if I wanted to narrow my request.
That seemed fair, so I asked for information only on medical personnel. I noted that the Center for Public Integrity got some of this information by looking at DOD travel disclosure forms. “It seems that if the names are included in those public records, it’s hard to understand why they couldn’t be released in the context of my request,” I wrote.
Then, I waited some more. By now my seemingly simple request had taken three years (!), well past the time most journalists can afford to spend on a story.
In May 2014, I received a forehead-smacking email from a DOD FOIA appeals analyst: “I believe that due to the age of the request, it would be in your best interest to withdraw the appeal and submit this as a new FOIA request.”
No. I did not want go back to square one of the FOIA board game. “It’s frustrating,” I responded, “because I was asked at least once before whether I wanted to proceed with my appeal and every time, I have said yes. I don’t want to restart a process that could take many years to resolve. The military apparently does not believe it has to disclose this information — or respond timely to an appeal.”
Finally, in November 2014 — more than 3½ years after I filed my appeal — I received a final response: Denied. We are at war, I was told, and I am not entitled to the records.
It took 3 1/2 years for @DeptofDefense to respond to my #FOIA appeal. It was denied. pic.twitter.com/SI4i3sfyFy #FOIAFriday
— Charles Ornstein (@charlesornstein) December 5, 2014
There was no explanation of how doctors would be put at risk by releasing their names. No sense of when or if this “war” would ever end — or whether the DOD could use it as an excuse forever.
While I am frustrated that I never received the records I requested, I am more troubled that it took years to process my FOIA appeal. How could it have taken nearly as long putting a kid through college to formulate a response? It underscored just how broken the FOIA system is.
Jesse Eisinger
On a hot and sunny day in late July 2015, I filed a FOIA request with the Securities and Exchange Commission for all the investigative files from its closed probe of a certain bank.
For a while, save for a note acknowledging the receipt of my request, I heard nothing. Finally, on Nov. 15, I received a reply: The agency had determined there were 20 boxes of material containing such files. Since a pre-release review of the boxes would take longer than 32 hours, I would be placed in the “First-In, First-Out” track. Then came the kicker: “At present we anticipate that it may take thirty-six months or more before we can begin to process a request placed in our FIFO track.”
Despite being a finance reporter, I struggle with math. It took me a moment to take that in. The SEC was saying it couldn’t *start* working on my request for … three years. My editors are pretty patient at ProPublica, but not that patient.
The letter helpfully offered a possibility that things might go faster if I narrowed my request. So I got on the horn with a very nice FOIA officer at the SEC. He really seemed to want to help me. We walked through what materials might be in those 20 boxes in order to see if I could pare things down. Finally, we decided I’d only ask for the transcripts of interviews the SEC lawyers had conducted.
A few weeks later, he called me back. That had narrowed things down to 12 boxes. I was still FIFO’d.
I asked: Ok, what’s in those 12 boxes? I’d be fine starting with only a handful of executive interviews.
“We don’t know.”
“Come again?”
“They are not labeled,” he said. They were in a warehouse and all pertained to the investigation into the bank, but they couldn’t tell specifically what was in the boxes. I am not often surprised by government responses, but this was an exception.
I stifled my skepticism and asked if the SEC could look in the boxes and then tell me. But the FOIA officer explained that looking in the boxes would take longer than 32 hours – putting me back into FIFO.
Here I felt something like Jack Nicholson’s character in “Five Easy Pieces.” I just wanted some goddamn toast. Was that so hard? I made a suggestion: Let’s pretend I was an SEC attorney who was interested in a specific interview from the case. How would I go about getting it? Would I have to look through every box? He said he didn’t know because he wasn’t an investigating attorney. In fact, he wasn’t a lawyer at all. Nobody in the FOIA office was.
I was stuck. I whined to a big firm lawyer I knew, who came up with an ingenious solution: Ask for one box, any box.
I wrote a new request. I asked for one box, any box of their choosing. I offered to waive any media privileges and pay any fees the SEC incurred. I offered to come to their offices and go through the box myself. I tried to give them no out.
Seven months after my initial request, they came through. They gave me the materials from one of the boxes. I opened up the files eagerly. They all had all the names redacted. Even the SEC attorneys. Even names of deals. I couldn’t understand a thing. It was like looking at the scroll of code from the Matrix.
The lawyer thought we would have a great case to challenge the SEC’s response. She thought we’d win and get them to un-redact most of the names and details. But by that point, my story was done and I was moving on with a new project. The SEC had won.
Julia Angwin
In November 2013, a U.S. customs official refused to let Toronto resident Ellen Richardson board a flight to New York because she had been hospitalized for depression — and was therefore considered a mental health risk.
A Canadian privacy commissioner investigated and found that Ontario police were routinely uploading suicide calls into a database that was accessed by the FBI and U.S. Customs and Border Patrol. And it turned out that there was an obscure provision in the immigration law that allows border agents to deny admission to people who have mental disorders that could pose a threat to themselves or others.
Curious about how many other people might have been turned away at the border on mental health grounds, I filed a FOIA on May 19, 2014 to the U.S. Citizenship and Immigration Services seeking complaints by individuals denied admission on mental health grounds as well as training manuals governing such denials and other related documents.
I heard nothing. And then, two years later, I got an email from Yael Schacher, a professor at the University of Connecticut, who said she had received my FOIA response at her home address in West Hartford.
“I opened it, thinking that the name was wrong and it was for me because, as a Harvard PhD student studying immigration, I submitted several FOIAs to USCIS in 2013 and 2014,” that were still outstanding because they had been referred to other agencies for review, Schacher wrote to me. But inside the envelope was not her FOIA response, but mine.
Kindly, she mailed the response to my office. The contents were underwhelming to say the least. The two-year wait had produced a three-page printout of an e-mail thread, in which an immigration service employee outlined the criteria for when the agency must consult the U.S Department of Health and Human Services on mental health immigration denials.
The email answered a question I hadn’t asked, and was — to be honest — beside the point.
Meanwhile, Schacher is still waiting for her FOIA. “Who knows who has received my long awaited FOIA documents!?” she wrote to me. “If you happen to get any of my documents, please send them to my home address.”
Michael Grabell
Back in 2010, during the depths of the recession, the Obama administration awarded $7 billion in grants and loans to help bring high-speed internet to rural areas and inner cities. It was a key part of President Obama’s $800 billion stimulus package and an endeavor often compared to the New Deal program that brought electricity to farming communities.
As part of my reporting on the stimulus for ProPublica and a book I was writing, I spent some time driving the unpaved back roads of rural Vermont, talking to families who were desperate for faster, more reliable connections to the outside world.
To see when, and if, they would get it, I filed a FOIA request with the U.S. Department of Agriculture in October 2010, seeking the grant applications for six of the largest projects. My goal was to see what was being promised, so that when the projects were completed, I could check if the companies — and the Obama administration — followed through.
I’m no stranger to FOIA delays.
But despite the president’s pledge of unprecedented transparency with stimulus spending, I heard nothing for four-and-a-half years.
Meanwhile, construction on the broadband projects began. My book was published (you can buy it here!). And controversy grew over whether the government money was competing with private investment, whether the internet speed was fast enough, and why some communities weren’t getting connected.
Finally in April 2015, the USDA sent me a letter, saying they had located approximately 4,000 pages related to my request.
But they couldn’t give them to me yet. First, as is standard, they had to notify the companies behind the broadband projects to give them a chance to object to any information they might consider proprietary.
It wouldn’t take much longer, the agency assured me in the letter. The companies had to respond by the end of the month.
That was the last I ever heard of my FOIA request.
Since then, much of the money has been spent. While the administration promised to connect millions of rural households, in March 2014, it lowered that number to 729,000, according to congressional investigators at the Government Accountability Office. More than 40 projects were terminated before they even began. The agency was releasing so little information, the GAO said, that it was difficult to determine if the program had worked.
“We are left with a program that spent $3 billion,” a GAO investigator told Politico last summer, “and we really don’t know what became of it.”
Back in rural Vermont, a $116 million plan by VTel Wireless to bring broadband to all unserved households is officially considered complete. But the state’s congressional delegation recently sent a letter to the USDA asking why they’re hearing it’s “only available in a few areas.”
I decided to ask some local reporters how the grant application might inform their work, so I called John Lippman, who’s been covering the broadband project for the Valley News, which covers several communities in central Vermont and New Hampshire.
To my chagrin, Lippman said he already had the grant application. He received it under FOIA last year, he said, and it took only a matter of weeks. He published a story about it here.
“It was very valuable because it gave me some insight into a number of things,” he said. “In the application they said they were going to be doing x, y, z. So I was able to match what they said they were going to do with how far they were in the process.”
Vermont’s state auditor Doug Hoffer got a copy too — but heavily redacted.
“It’s a 949-page document, and 650 pages were redacted. It’s a complete joke,” he said. “I have to say, not only as the state auditor, but as a citizen, I’m outraged about what the U.S. Department of Agriculture agreed to in terms of redactions at the behest of the company. It’s appalling. We basically were not given any information.”
As I called people around Vermont, I heard back from the USDA’s chief FOIA officer, who was trying to track down my request. She said it was still in process and gave me the direct number for the analyst handling my request. So I called the number and got an automated message: The number was not in service.
After I notified her, she provided the right number and, suddenly, I started receiving the applications.
I’m still waiting for two more, including a project in Western Kentucky worth more than $100 million in federal funding. But the USDA says they’ll send them to me next week.
Derek Willis
Tillamook County, Oregon, is famous for its cheese. The Cheese Factory there has large bins at which visitors can freely sample its cheddar and other varieties before deciding what to take home.
Getting precinct-level election results from the county, however, is an exercise in frustration and expense. No free samples, or free anything.
In the middle of 2015 I began requesting precinct-level election results from counties in Oregon for OpenElections, a side project funded by the Knight Foundation. The Secretary of State maintains county-level results, typically in electronic PDFs, so to get down to precinct level we need to ask county clerks across the state for their data. Many of them post precinct results on their web sites, but some don’t, so we emailed a few to ask for results from 2000-2014. In doing so, we were prepared to pay reasonable fees for them, as the Oregon Revised Statutes permit.
Local officials were quick to get back to us in every case, and their responses were straightforward. Here’s an example, from Art Harvey, the Josephine County clerk and recorder:
“The reports you are interested in are available in PDF format. The cost would be $10.00 per election.”
Other counties charged fees ranging from $25 (Umatilla County) to $45 (Wasco County) to $86.50 (Linn County, which sent us paper print-outs of election results that we scanned). And then there’s Tillamook County, where Tassi O’Neil, the county clerk , set a price of $664 for PDF copies of precinct-level results for elections from 2000-2014.
I wondered how that price was calculated, so I asked. O’Neil responded:
“The fee for each election is $3.75 locate fee and then .25 cents per page. That is the fee if it is a paper copy or if we send it in a PDF. That is the charge that the Oregon Revised Statutes say that we can/or should charge.”
That is true, but there are two points here: One is that members of the public are being charged for pages of an *electronic* document. There are no paper copies involved in this process - the report itself is generated by a computer. The other is that the Oregon Revised Statutes also say this:
“The custodian of any public record may furnish copies without charge or at a substantially reduced fee if the custodian determines that the waiver or reduction of fees is in the public interest because making the record available primarily benefits the general public.”
There is no question in my mind that the public availability of election results is in the public interest. But O’Neil disagreed. When we asked for a waiver, she denied it. There is an appeals process in Oregon, but it involves the county’s district attorney, who is an elected official just like the county clerk. There is no independent board or state office that can review such appeals.
In the end, our volunteer project couldn’t afford $664 for Tillamook’s results, so we settled for paying more than $250 for elections from 2010-2014, which arrived promptly. As scanned PDFs.
As a final note, this year I checked to see if the Tillamook clerk had posted precinct results from the May presidential primary. She had not. As a desperate measure, I emailed a county commissioner and laid out the situation, making my case that election results should be freely available. In reply, the commissioner sent a copy of the precinct-level results PDF, but added no comments.
Knowing who to ask when the actual custodian of records is seeking to make money from your public records request is a skill, but it’s not one that most people should have to develop.
But my sojourn in Oregon underscored a few other points: The current public records laws are out of date for the digital age. Public officials like O’Neil can pretty much do whatever they want without fear of consequences, and too often treat public records as belonging to them, not the public.
When Sandra Fish, one of our volunteers, sought to take pictures of election results in Park County, Colorado, she was told that state law banned phones and computers from the clerk’s office (it doesn’t). She eventually got the results, but only after filling out a form, waiting several days and paying for them.
Given the cost and effort involved to get this basic record of our democracy, it would hardly be surprising if people gave up before getting what they requested. I doubt that’s what the authors of state and federal open records laws had in mind.
-Originally appeared at Propublica.org