Skip to main content

Big Data as Big Brother

Yesterday and today, it seems like one of the biggest news items is the Big Data of Big Brother. 

The argument goes like this:

The U.S. government is aggregating the same kind of social media and online user data that private companies use to understand their customers' sentiments (or potential customers) for the express purpose of counter terrorism, reporting on potential defense threats, and generally trying to figure out who the "bad guys" in the world are...

Using data from companies such as Apple, Google, Facebook, Microsoft, Skype, Yahoo, YouTube, and others, the National Security Agency is able to obtain all types of data (1). Now, when one considers the total size of this data it's clear to see that this exits the world of statistical analysis and enters the world of Data Science and Big Data. 

To give you an idea of the size of the data, "The amount of data in question is enormous. For example, the U.S. wireless-communications trade association C.T.I.A. estimates that as of December, 2012, there were over three hundred and twenty-six million wireless-subscriber connections, which use 2.3 trillion minutes of call time a year. Facebook has a billion users; eight hundred and thirty-three million of them are international. On average, over three hundred million photos are uploaded to Facebook’s servers per day. YouTube handles seventy-two hours of video uploads per minute." (1)

Indeed Big Brother is now in the business of Big Data.

The technical things the government is doing with Big Data involve:

- Machine Learning
- Visualization
- Sentiment Analysis
- Cluster Analysis

I've talked about Sentiment Analysis before with regards to using Twitter's API; however, what are these other technical things?

I'll leave that for next time.



Popular posts from this blog

Downloading a Video from Facebook

Hey... here's a neat trick for downloading a video from Facebook:

First, notice that your "Saved videos" can usually be found in the upper right menu (where birthdays usually show up) or under the "Saved" button in the left menu.

Once you click on the saved video, you'll find the file... click on that bad boy to play it in the browser.

Once it's playing, notice that there's no way to download it (that's what we're gonna "hack")

In the URL of the playing video, replace www with m (for mobile)

You should be brought to the funky looking mobile site (even though you're on a regular browser)

Play the video and then right click, you should see the following menu:

Save it as an MP4, crack a beer, and enjoy your Video whenever you want without having Facebook open.

The life stats have waned... but there is this. . .

My last few weeks with a newborn have been trying to the point that tracking life stats hasn't been too feasible..But, for now, I want to talk about some articles I found while searching around for Data Science news:

First, from the Star Tribune, Minnesota Companies and Workers Cache in on Big Data, a quick word cloud shows words such as: Oracle, information, data, and software; however, it's the words like Hadoop, and Data Science that stand out to me. It's an exciting time for the field of Data Science. 

One encouraging quote from the article really makes me excited about the field:

'I would challenge you to describe to me an organization of any size in any industry or not-for-profit setting that will not be leveraging [Big Data],' said Isaac Cheifetz, a headhunter working to find the Mayo Clinic a head of information management and analytics. 'Name one. I can’t'"

What this tells me is that there is going to be rising demand for people with the skills t…