FCC Releases All Net Neutrality Comments As Giant XML Files For Data Analysis

from the open-records dept

While it had trouble keeping its site up during times of intense commenting, the FCC’s IT team is now working to make all the submitted comments on its “open internet” net neutrality proposals available to download in a bunch of XML files:

Because of the sheer number of comments and the great public interest in what they say, Chairman Wheeler has asked the FCC IT team to make the comments available to the public today in a series of six XML files, totaling over 1.4 GB of data ? approximately two and half times the amount of plain-text data embodied in the Encyclopedia Britannica. The release of the comments as Open Data in this machine-readable format will allow researchers, journalists and others to analyze and create visualizations of the data so that the public and the FCC can discuss and learn from the comments we?ve received. Our hope is that these analyses will contribute to an even more informed and useful reply comment period, which ends on September 10. We will make available additional XML files covering reply comments after that date.

While the more cynical among you may see this as more of a statement on the rather weak capabilities of the current FCC’s system for handling searching through the submitted comments, it’s still nice to see at least a move towards openness and transparency in sharing this data for others to search through. As we’ve noted, we’ve been digging into some of the data on the comments, and hopefully this will make the process much easier.

Filed Under: , , , , , ,

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “FCC Releases All Net Neutrality Comments As Giant XML Files For Data Analysis”

Subscribe: RSS Leave a comment
Mike Masnick (profile) says:

Re: Re:

Since it’s government public data, it should be in public domain. So in theory no one should stop anyone from taking and redistributing these anyway they want.

That’s not accurate. The rules only apply to works produced by government employees. That is not the case here. Any copyrights remain with the creators of the content.

Arthur Moore (profile) says:

Re: Re: EULA?

Do you know what the terms and conditions were for submitted comments?

I’m guessing it was probably something like “Grant the FCC a perpetual license to copy and display this comment.” However, I don’t know for sure. I figure you all would have picked up on it if there was a copyright assignment clause, though.

I hope none of the people analyzing this data work for places with legal departments who don’t understand fair use. Especially since, that’s the only way that anyone reporting on this can directly quote any of the comments.

Another little aside is that while it might be legal to download these huge files to your personal PC, it’s almost certainly illegal to give a copy of them to anyone else. They have to go use up FCC bandwidth by obtaining the files from a “authorized source”.

MrTroy (profile) says:

A million monkeys typing on a million keyboards...

… though not particularly randomly.

I wonder why they compare the size of XML marked up data to raw text? I suspect the submission data would drop up to half of its size if displayed as raw text. Also, boo to offering XML files for download without compression! They serve their web pages gzip-encoded (compressed), why not their XML files? Or at least pre-compressed versions… the 5th file zips from 100M down to 5.6M, and 7zip takes it down to 3.5M! Anyway, I guess I wouldn’t expect the FCC to know anything about the internet by now.

Also, I like this from their release:

Finally, we hope that whatever visualizations are developed using this open data will comply with the standards that allow use and access by differently-abled individuals. The Chairman and the FCC CIO are committed to ensuring accessible web content in multiple forms for all.

While you’re helping us to do our work, would you mind conforming to the standards that we have to adhere to?

Joking aside, I’m glad they’re making this available to everyone. Even if we were to trust that they know exactly what they’re doing, if they were doing all the work themselves they’d only analyse the submissions in whatever ways they can both imagine and implement in time. Giving this to the internet lets them benefit from novel ways of parsing the data that they might not have thought about, and it puts the onus on everyone else to try to justify whatever they claim to find in the data.

If the FCC was to make all of the findings itself without making the data public, anyone who disagreed with the result would simply say that the FCC was selectively parsing the results. Now, whenever anyone tries to make any claims from the data, everyone else will be able to verify those claims… and if someone tries to make a claim without saying how they came to that conclusion, then that will be worth about as much as 1.5Gb of uncompressed XML text.

Violynne (profile) says:

I was disappointed to see the XML files don’t include IP address, though not surprising given the comment submission form didn’t exactly demand users verity themselves.

This means separation by city and state. It also means a possibility of gross inaccuracies regarding the data. Geez, this could get bad.

Oops, my cynicism is showing itself again. I suppose I could take the data on face value. Though, it’ll be difficult to determine a margin of error without the IP address.

Don’t take this as a fact, because I’ve not counted the responses in the actual files yet, but cursory scans seems to have a majority favoring classification as common carrier.

In addition to this, the commentary also seems to be sparse, as though people simply voted without leaving comment.

Once I do this, I’ll sit back and wait for others to post the results so I can determine who’s lying and who’s honest.

The FCC did good by releasing these files.

David says:

Downloaded it...

And just from a quick look there are lots of entries which the respondent copy and pasted the “Net neutrality is the First Amendment of the Internet, the principle that Internet
service providers (ISPs) treat all data equally. As an Internet user, net neutrality is vitally important to me. …” text. So someones campaign appeared to work!

TRL says:

I came across this article after searching for why my comments wouldn’t load on the FCC site.

I just came from trying to make a comment on the FCC comment site. Despite it being made for laywers and law firms and not regular people, the site was still unable to take my comment saying that it “could not add the text to the file” and, after turning it into a PDF and submitting it through their Expert submission “disk quota is full”. No matter how big this XML file is, it doesn’t represent half the comments people want to share about net neutrality.

Add Your Comment

Your email address will not be published. Required fields are marked *

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Techdirt Deals
Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...
Older Stuff
05:30 Survey Shows Majority Of GOP Voters Support Restoring Net Neutrality (31)
06:25 Big Telecom Finally Ends Quest To Stop States From Protecting Broadband Consumers (35)
05:56 Big Telecom's Quest To Ban States From Protecting Broadband Consumers Continues To Go... Poorly (13)
12:15 Courts (Again) Shoot Down Telecom Lobby's Attempt To Kill State-Level Net Neutrality Rules (5)
04:48 Dumb Telecom Take Of The Week: Because The Internet Didn't Explode, Killing Net Neutrality Must Not Have Mattered (23)
09:37 British Telecom Wants Netflix To Pay A Tax Simply Because Squid Game Is Popular (32)
04:55 Axios Parrots A Lot Of Dumb, Debunked Nonsense About Net Neutrality (54)
10:50 NY AG Proves Broadband Industry Funded Phony Public Support For Attack On Net Neutrality (10)
06:24 The GOP Is Using Veterans As Props To Demonize Net Neutrality (22)
06:03 Telecom Using Veterans As Props To Demonize California's New Net Neutrality Law (12)
09:32 AT&T Whines That California Net Neutrality Rules Are Forcing It To Behave (11)
06:23 The New York Times (Falsely) Informs Its 7 Million Readers Net Neutrality Is 'Pointless' (51)
15:34 Facebook's Australian News Ban Did Demonstrate The Evil Of Zero Rating (18)
04:58 'Net Neutrality Hurt Internet Infrastructure Investment' Is The Bad Faith Lie That Simply Won't Die (11)
05:48 Dumb New GOP Talking Point: If You Restore Net Neutrality, You HAVE To Kill Section 230. Just Because! (66)
06:31 DOJ Drops Ridiculous Trump-Era Lawsuit Against California For Passing Net Neutrality Rules (13)
06:27 The Wall Street Journal Kisses Big Telecom's Ass In Whiny Screed About 'Big Tech' (13)
10:45 New Interim FCC Boss Jessica Rosenworcel Will Likely Restore Net Neutrality, Just Not Yet (5)
15:30 Small Idaho ISP 'Punishes' Twitter And Facebook's 'Censorship' ... By Blocking Access To Them Entirely (81)
05:29 A Few Reminders Before The Tired Net Neutrality Debate Is Rekindled (13)
06:22 U.S. Broadband Speeds Jumped 90% in 2020. But No, It Had Nothing To Do With Killing Net Neutrality. (12)
12:10 FCC Ignores The Courts, Finalizes Facts-Optional Repeal Of Net Neutrality (19)
10:46 It's Opposite Day At The FCC: Rejects All Its Own Legal Arguments Against Net Neutrality To Claim It Can Be The Internet Speech Police (13)
12:05 Blatant Hypocrite Ajit Pai Decides To Move Forward With Bogus, Unconstitutional Rulemaking On Section 230 (178)
06:49 FCC's Pai Puts Final Bullet In Net Neutrality Ahead Of Potential Demotion (25)
06:31 The EU Makes It Clear That 'Zero Rating' Violates Net Neutrality (6)
06:22 DOJ Continues Its Quest To Kill Net Neutrality (And Consumer Protection In General) In California (11)
11:08 Hypocritical AT&T Makes A Mockery Of Itself; Says 230 Should Be Reformed For Real Net Neutrality (28)
06:20 Trump, Big Telecom Continue Quest To Ban States From Protecting Broadband Consumers (19)
06:11 Senators Wyden And Markey Make It Clear AT&T Is Violating Net Neutrality (13)
More arrow