Skip to main content

Big Data as Big Brother

Yesterday and today, it seems like one of the biggest news items is the Big Data of Big Brother. 

The argument goes like this:

The U.S. government is aggregating the same kind of social media and online user data that private companies use to understand their customers' sentiments (or potential customers) for the express purpose of counter terrorism, reporting on potential defense threats, and generally trying to figure out who the "bad guys" in the world are...

Using data from companies such as Apple, Google, Facebook, Microsoft, Skype, Yahoo, YouTube, and others, the National Security Agency is able to obtain all types of data (1). Now, when one considers the total size of this data it's clear to see that this exits the world of statistical analysis and enters the world of Data Science and Big Data. 

To give you an idea of the size of the data, "The amount of data in question is enormous. For example, the U.S. wireless-communications trade association C.T.I.A. estimates that as of December, 2012, there were over three hundred and twenty-six million wireless-subscriber connections, which use 2.3 trillion minutes of call time a year. Facebook has a billion users; eight hundred and thirty-three million of them are international. On average, over three hundred million photos are uploaded to Facebook’s servers per day. YouTube handles seventy-two hours of video uploads per minute." (1)

Indeed Big Brother is now in the business of Big Data.

The technical things the government is doing with Big Data involve:

- Machine Learning
- Visualization
- Sentiment Analysis
- Cluster Analysis

I've talked about Sentiment Analysis before with regards to using Twitter's API; however, what are these other technical things?

I'll leave that for next time.

References
(1) http://www.newyorker.com/online/blogs/elements/2013/06/nsa-prism-big-data-national-security.html

Comments

Popular posts from this blog

Downloading a Video from Facebook

Hey... here's a neat trick for downloading a video from Facebook:


First, notice that your "Saved videos" can usually be found in the upper right menu (where birthdays usually show up) or under the "Saved" button in the left menu.




Once you click on the saved video, you'll find the file... click on that bad boy to play it in the browser.




Once it's playing, notice that there's no way to download it (that's what we're gonna "hack")





In the URL of the playing video, replace www with m (for mobile)





You should be brought to the funky looking mobile site (even though you're on a regular browser)





Play the video and then right click, you should see the following menu:





Save it as an MP4, crack a beer, and enjoy your Video whenever you want without having Facebook open.


Data Science Course (Complaint # 2?)

Looks like I'm not the only one having problems with the first assignment for UW's Introduction to Data Science.... while Professor Howe has certainly whetted my appetite for wanting to learn more useful DS than I've learned before, the first assignment seems like a weed-out assignment. I remember from my economics days that this is actually a fairly common practice among econometricians (thanks Professor Levy). [Side note: I'm STILL working my way through Data Analysis and Regression: A Second Course in Statistics.]

I've submitted most of what was required in a Python file for A1P2; however, it's probably going to get a 0/10. I fully intend on sticking with this course after reading a lot of the complaints. Most of the people complaining seem to have the "must have now" mentality; fortunately my years of dealing with this kind of stuff is that it rarely comes easy... that said, my intent to blog along with this course is going to be less sporadic and …