Skip to main content

Big Data as Big Brother

Yesterday and today, it seems like one of the biggest news items is the Big Data of Big Brother. 

The argument goes like this:

The U.S. government is aggregating the same kind of social media and online user data that private companies use to understand their customers' sentiments (or potential customers) for the express purpose of counter terrorism, reporting on potential defense threats, and generally trying to figure out who the "bad guys" in the world are...

Using data from companies such as Apple, Google, Facebook, Microsoft, Skype, Yahoo, YouTube, and others, the National Security Agency is able to obtain all types of data (1). Now, when one considers the total size of this data it's clear to see that this exits the world of statistical analysis and enters the world of Data Science and Big Data. 

To give you an idea of the size of the data, "The amount of data in question is enormous. For example, the U.S. wireless-communications trade association C.T.I.A. estimates that as of December, 2012, there were over three hundred and twenty-six million wireless-subscriber connections, which use 2.3 trillion minutes of call time a year. Facebook has a billion users; eight hundred and thirty-three million of them are international. On average, over three hundred million photos are uploaded to Facebook’s servers per day. YouTube handles seventy-two hours of video uploads per minute." (1)

Indeed Big Brother is now in the business of Big Data.

The technical things the government is doing with Big Data involve:

- Machine Learning
- Visualization
- Sentiment Analysis
- Cluster Analysis

I've talked about Sentiment Analysis before with regards to using Twitter's API; however, what are these other technical things?

I'll leave that for next time.



Popular posts from this blog

Downloading a Video from Facebook

Hey... here's a neat trick for downloading a video from Facebook:

First, notice that your "Saved videos" can usually be found in the upper right menu (where birthdays usually show up) or under the "Saved" button in the left menu.

Once you click on the saved video, you'll find the file... click on that bad boy to play it in the browser.

Once it's playing, notice that there's no way to download it (that's what we're gonna "hack")

In the URL of the playing video, replace www with m (for mobile)

You should be brought to the funky looking mobile site (even though you're on a regular browser)

Play the video and then right click, you should see the following menu:

Save it as an MP4, crack a beer, and enjoy your Video whenever you want without having Facebook open.

Life Statistics

Today I used Excel to analyze total email volume and client specific email volume data (sent/received) weighted by an "effort" ranking to help highlight work flow at the office. In the interest of not having to check my email as much as I am currently, I hoped to shed light on prioritization of time spent with email vs projects. 

My hope is to only check my email twice a day... But, given the client service that my role entails, that's probably not gonna happen. One thing that came out of my analysis though is when getting the most email (day of week/time of day) is most probable. It turns out Tuesday mornings and Friday afternoons are the most likely times for stuff to hit the proverbial "fan."

In other life statistics news (and productivity measures), I'm tracking my time spent during the day in a manner similar to the "hyper tracking" concept I've learned from listening to The Better Guy Show podcasts (available on iTunes). In the next week I…