From Unstructured Data

June 25, 2012, 8:29 PM

Step 1: Create a monster by networking 16,000 ultrafast computer processors.

Step 2: Feed the monster 10 billion images chosen at random from YouTube videos.

Step 3: See what happens.

What Happened: The monster taught itself to recognize cats.

“We never told it during the training, ‘This is a cat.’ It basically invented the concept of a cat.” – Jeff Dean, speaking for the scientists at Google’s secretive X-Labs

The frightening part of this report is that modern computers appear to be capable of independent learning through extrapolation.

The comforting part of this report is that it takes 16,000 ultrafast processors working together to do something that’s completely effortless for a human toddler.

I drive 40 minutes to meet Jeffrey Eisenberg for lunch in a place that looks like it used to be a Denny’s. I hand him an advance copy of Pendulum. “Hold it up next to your face,” I said. He held it up and smiled. [click]

Jeff laid the book on the table and thumbed through it, “This really turned out nice.”

“So tell me what’s happening in Jeff-world.”

Jeff said he was developing applications of big data for some of America’s largest companies.

“What’s big data?” I asked.

“You’ve been teaching Practical Applications of Chaos Theory in the Magical Worlds course for about 12 years now, right?”

“Right.”

“Big data is just one more use of that idea.”

“How so?”

“Dump huge amounts of unstructured data into a computer, then wait to see the patterns it discovers. The bigger the dataset, the more obvious the patterns.”

Jeff went on to explain that ‘unstructured data’ included information from climate sensors, digital photos and videos, purchase transaction records, Tweets and other social media posts, GPS signals from cell phones, things like that.

YouTube, Facebook, Twitter, traffic and security cameras and the worldwide proliferation of portable digital devices add up to this: If every book, document, spreadsheet, register, newspaper and photograph created prior to July, 2010, were digitized, they would account for only about 10 percent of the world’s data.

Ninety percent of all the data in the world has been created in the last two years.

According to Ed Dumbill,

“Big data is data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the strictures of your database architectures. To gain value from this data, you must choose an alternative way to process it.”

Institutions can use big data to reduce fraud and errors. Hospitals can use it to improve patient care while reducing healthcare costs. According to IBM, one health care organization used big data this year to decrease patient mortality by 20 percent. A telecommunications company reduced processing time by 92 percent and a utility company improved the accuracy of power resource placement by 99 percent.

Huge organizations like these have been the first to embrace big data, but Jeff Eisenberg tells me that he and Bryan are working to make its power available to retailers and small businesses, as well.

When Jeff said big data was just another practical application of chaos theory, here’s what he meant: Chaos, in science, is not randomness, but precisely the opposite. Chaos is a pattern so vast that it won’t fit into the human mind.

Confronted with the product of a chaotic system, the pattern-recognition function of our brain’s right hemisphere senses a pattern that never seems to resolve, never seems to close, never seems to finish and begin again. As a result, we are drawn to a beauty that is too big for us. All the beauties of nature – mountains, canyons, trees, clouds, snowflakes and the movements of fire and water – are products of chaotic systems.

Likewise, works of art that pierce public consciousness to become mass-appeal, runaway successes – top-of-the-chart songs, bestselling books, Oscar-winning movies – always contain a third gravitating body* with a high degree of divergence and an explicit moment of convergence. Third gravitating bodies show up in effective advertising, as well.

I didn’t make up that term, “third gravitating body,” by the way. It’s a term that has been used by astrophysicists for more than 100 years to describe a mathematical function of the universe that relates to system evolution and gravity, the ability to attract.

Attract…

Attraction. That would be a good thing to master, don’t you think?

And now you know why it’s called The Magical Worlds Communications Workshop at Wizard Academy, the most interesting business school on earth.

Come. We’ll fill in the blanks.

It will all make perfect sense before you leave.

Roy H. Williams