Let’s Move Beyond Open Data Portals
Sorry...I just have to disagree with the OP. Several years ago, Socrata stopped by where I worked (a news organization) and told us of their idea to build a portal of city governments everywhere that would host datasets. They were new at the time and I just thought they were bonkers.<p>Now, I can&#x27;t believe what you can find on the various data portals. There&#x27;s a lot of shit data but that&#x27;s because lots of organizations collect shit data. But for the organizations that <i>do</i> have data, Socrata is such a huge step from what existed before.<p>I&#x27;ll ignore the many situations in which agencies just didn&#x27;t put out data at all. Dallas, Texas is one exception. It has been posting its crime data for years. Except it was on a FTP site with a convoluted structure. And it wasn&#x27;t all in one file. So you had to write a script that spidered the subdirectories, downloaded the files, unzipped them, and concatenated them (and I don&#x27;t think they were a straight-up concatenation).<p>Now, it&#x27;s just all on one page from which you can export the data as bulk CSV [1]. Because Socrata&#x27;s REST API is so straightforward, you can just script your data requests to hit the right end points. But not only is there the incident data, there&#x27;s the narrative data [2] (which had also been on the FTP site, but required its own spidering), and there&#x27;s tables I hadn&#x27;t seen before, such as [3] suspects and real-time active calls [4]<p>On top of it, the police department has even decided to put up the data for their <i>officer-involved-shootings</i>. Mind you, they were <i>already</i> ahead of the game, nationwide, last year when they created a parseable (via scraper) webpage with HTML tables and PDFs. They certainly didn&#x27;t have to make their data even easier to get, but they did [5]<p>Texas has always been generally good about public records because of their broad sunshine law. But it&#x27;s not that the law turned them all into free-data-hippies right away...the agencies just have a tradition of doing it -- it helps alot if you&#x27;re a Texas employee and you&#x27;ve seen how everyone else just agrees to potentially damaging records requests, and yet no one gets stressed out.<p>I have to think that Socrata, just by being <i>there</i> as an option, not just in Dallas, but everywhere, has made bureaucrats more aware of how data sharing can just be...<i>done</i>. Certainly, there are always officials who will push back, because they&#x27;re power-control-freaks or because they have something to hide. But plenty of bureaucrats don&#x27;t really care...they&#x27;ve just been told by their IT people that putting up data in an easy way would cost too much and be too much of a security compromise. Now that there&#x27;s an option of a general data portal, there&#x27;s fewer reasons to say no.<p>Just to give you an idea of how technically clueless many bureaucrats are (and I don&#x27;t really blame them, but their agencies for not prioritizing tech training)...it is still not unheard of to be denied access to machine-readable data -- e.g. they <i>print out a spreadsheet</i> and fax it to you, instead of just sending you the XLS -- because they think that if they give you the spreadsheet, you can &quot;alter the data&quot;.<p>Yes, it really is that dumb.<p>edit: to the author&#x27;s credit, he&#x27;s not saying that open data portals should be closed, just that governments should move beyond them. That&#x27;s a nice sentiment, but in reality, it&#x27;s an idea that takes away resources from <i>improving</i> data portals.<p>From TFA:<p>&gt; <i>Now we actually give that directly to Waze, so they can reroute people dynamically. Indeed, this is a good open data story — taking the data to where people are —but there’s something more interesting: it’s a two-way street. Not only does Waze now share pothole and road condition data it collects regularly through its app, they went one step further. They began to proactively collect and share data in the interest public safety.</i><p>But <i>that can already be done</i> with the existing LA data portal and its REST API. Why does the city of Los Angeles have to give Waze anything other than the GET endpoint, from which Waze engineers can download as they like. And not just Waze, but everyone else, in equal measure. So there&#x27;s nothing <i>wrong</i> with what the author wants, he just doesn&#x27;t appear to think that with APIs, developers can create far better and far more resources than what the city could do itself.<p>And no, the city (unless it has a magical source of revenue) can&#x27;t do both building out more &quot;human&quot; data applications while improving their open data pipelines. The latter has much, much further to go before the city can spend IT money on building out new apps.<p>[1] <a href="https:&#x2F;&#x2F;www.dallasopendata.com&#x2F;Police&#x2F;Dallas-Police-Public-Data-RMS-Incidents&#x2F;tbnj-w5hb" rel="nofollow">https:&#x2F;&#x2F;www.dallasopendata.com&#x2F;Police&#x2F;Dallas-Police-Public-D...</a><p>[2] <a href="https:&#x2F;&#x2F;www.dallasopendata.com&#x2F;Police&#x2F;Bulk-Police-Narrative&#x2F;inke-qqax" rel="nofollow">https:&#x2F;&#x2F;www.dallasopendata.com&#x2F;Police&#x2F;Bulk-Police-Narrative&#x2F;...</a><p>[3] <a href="https:&#x2F;&#x2F;www.dallasopendata.com&#x2F;Police&#x2F;Dallas-Police-Public-Data-Unknown-Suspects&#x2F;jitt-qwwh" rel="nofollow">https:&#x2F;&#x2F;www.dallasopendata.com&#x2F;Police&#x2F;Dallas-Police-Public-D...</a><p>[4] <a href="https:&#x2F;&#x2F;www.dallasopendata.com&#x2F;dataset&#x2F;Dallas-Police-Active-Calls&#x2F;9fxf-t2tr" rel="nofollow">https:&#x2F;&#x2F;www.dallasopendata.com&#x2F;dataset&#x2F;Dallas-Police-Active-...</a><p>[5] <a href="https:&#x2F;&#x2F;www.dallasopendata.com&#x2F;Police&#x2F;Dallas-Police-Public-Data-Officer-Involved-Shootin&#x2F;4gmt-jyx2" rel="nofollow">https:&#x2F;&#x2F;www.dallasopendata.com&#x2F;Police&#x2F;Dallas-Police-Public-D...</a>