A year after updating its open data policy, New York City is pushing ahead with plans to automatically refresh more of its data sets and tie public records requests to the open data process — moves hailed by open government advocates, even as some worry about the data's quality.
In a letter to Mayor Bill de Blasio and the city council earlier this month, a year after his "Open Data for All" initiative launched, Department of Information Technology and Telecommunications reported it entered 156 new data sets on the city's online portal over the last year, and noted that more than 200 of the roughly 1,500 data sets included on the site are now configured to update automatically.
Additionally, the letter highlighted how the IT department and the Mayor's Office of Data Analytics have been working to comply with the seven new laws de Blasio signed to improve the portal, including a new focus on reviewing data made available through Freedom of Information Law, or FOIL, requests for inclusion on the site.
"At its core, it's about making sure that the people who the data belongs to can get access to it," Amen Ra Mashariki, the city's chief analytics officer and director of the mayor's data office, told StateScoop.
Mashariki said some of the city's successes meeting the goals of the "Open Data for All" initiative, which de Blasio launched last July, can be chalked up to the fact that they're now managing the open data program more holistically. Rather than DoITT and the mayor's data office each working to monitor the portal, they've formed a single "open data team" to oversee their efforts in the area.
In particular, he thinks this increased emphasis on teamwork has helped the departments adapt to the new open data laws, and they "spent a lot of time diving into the laws and understanding their impact on the agencies." With the directive from council and the mayor to start examining FOIL requests to see if data released can be included on the city's portal, the open data team found themselves with plenty of complicated questions to answer.
But Albert Webber, DoITT's open data program manager, noted that examining open records requests can also help smooth the data release process. As they've started to look at the data sets people are frequently requesting, he thinks they can chart a course toward saving agencies time and money by releasing them proactively.
"Looking at the FOIL requests is sort of like crowdsourcing or letting the public help with the identifying of the data that should be public," Webber said.
John Kaehny, executive director of the New York government transparency group Reinvent Albany, is "really impressed" with those results as well. He points to the city's Taxi and Limousine Commission dataset detailing yellow cab trips as one that's "hugely, hugely useful" but only accessible via a cumbersome records request.
National observers also praised the city for the practice, noting that the notion of tying open data to open records requests is still a relatively rare one.
Stephen Larrick, open data program lead for the nonprofit Sunlight Foundation, said his research indicates that only 38 states and localities even mention public records in their open data policies and fewer still tie the release of data to information that people frequently request — in fact, just nine other localities have adopted that sort of policy.
"Open data is a continuation of that feedback loop that open records rely on," Larrick said.
But Kaehny is equally enthusiastic about the city's automation efforts, calling staffers' work to automate 100 data sets in the last year an "A-plus performance," making information more up to date and cutting down on human data entry errors.
Mashariki hopes those improvements get more people using the automated data sets, since they can be more confident in its accuracy. Meanwhile, Webber is thrilled by how much time each automation saves his team, particularly after they found a way to automatically update "geospatial data sets" like maps a few months back.
These advances aside, Kaehny said the "big criticism" he has left for the city's progress report revolves around what it doesn't include: any mention of how the department plans to address issues with the quality of some data sets.
From "inconsistent geospatial data" to "bad metadata" to erroneous data duplication, Kaehny sees issues that the city could address if it had a better process for accepting feedback from the public and reacting to those comments.
Kaehny suggests that something as simple as setting up "expert email lists," letting experienced users communicate directly with staffers about issues, has worked in places like Philadelphia and Chicago. Similarly, he thinks devoting more staff to this problem could help yield solutions as well.
Indeed, Kaehny recognizes that these are uncharted waters for the city to navigate, but he's anxious to see some results all the same.