predict foretell forecast reading room: Albert-László Barabási

* The difference between human dynamics and data-mining boils down to this: Data mining predicts our behaviors based on records of our patterns of activity; we don't even have to understand the origins of the patterns exploited by the algorithm. Students of human dynamics, on the other hand, seek to develop models and theories to explain why, when, and where we do the things we do with some regularity.

____________________________________

[p.31]

... In the same fashion, Dirk's super-diffusive law allowed him to predict how long before we lose track of the origin point of a pile of dollar bills spent in Queens. This is critically important discovery. Especially if you have a suitcase heavy with counterfeit money and don't want the FBI at your doorstep.

Dirk predicted that in sixty-eight days the coast would be clear. That is in about two months your counterfeit bills would be all over the United States, leaving the FBI no way to trace their origin.

There was only one problem with this prediction: It was blatantly wrong. Indeed, Dirk's own measurement showed that most bills released in New York one hundred days later were still in the city's vicinity. Despite their superdiffusive trajectory, an invisible hand slowed the bills down, forcing them to defy the laws of superdiffusion and stay put in their original neighborhood. ([ this shows a similar pattern to crime and criminal. Most criminals behave in a predictable pattern or fashion, liken to an M.O. (Latin. modus operanti, mode of operation). Unless you get a super-criminal, an outlier, someone who has unusual range and distance in pattern of criminal behavior. ]) ([ As every hunting students and trackers have been taught from generation to generation: Humans, like all animals, is a creature of habit. ])

____________________________________

[p.86]

... So, while the outcome of the next throw is always a mystery, true randomness has some magic uniformity to it.

Despite this apparent uniformity, there is nothing more random than a Poisson process, which is nothing but a sequence of truly random events. ([ very few events, and dare I say it, no events are random; there exist a cause and effect; the set of questions - who, what, when, where, why, how, what for, and so what - is, do we have some understanding of the event, or, we do not have any understanding of the events - for example, the Earth orbits around the Sun represents one (1) solar year and, the Earth rotates along its axis represents 24 hours period as signified by the repeated cycle of night (Sun set) and day (Sun rise), and when we do not have understand[ing] of the events, we fall back on, oh, it is a random event - for example, a 7.0 magnitude earthquake; or, oh, it is an act of God; or, oh, who could've predicted the 1st and 2nd world war; can you imagine the kind of human resource mobilization, material production, financing, energy consumption, etc ... to run [a] war longer than 30 days [or 90 days]; the kind of efforts, coordinations, communications, and mobilization that is required to prosecute any war of a length longer than 31 days [or 91 days]; so when you hear, see, or read the word, random, your antenna should go up. You should say to you self. Is this an expert. Does this person have some special knowledge about the man and the woman on the street question I am asking him or her. Is this person parroting a script - a mentally recorded message, a viral meme that has infected the mind. Randomness could simply be an excuse for I don't know and, I don't understand. However, rather than say, I don't know, or I have no explaination, they say, it is a random event. I am going to repeat it, again. We might not understand, we might not know, we might not have an explaination; we may never understand, we may never know, we may never have an explaination; but that does not mean it is random. ]) Therefore, deviations from Poisson's prediction are often taken as a signature of some hidden order. They offer evidence of a deeper law or pattern that remains to be discovered.

... As Richardson so eloquently pointed out, the atmosphere is governed by laws and equations, which today are successfully exploited by meteorologists everywhere. Many of these events--from solar eclipses to floods and droughts--were once considered the mysterious provenance of gods and spirits. Today they are predictable, however, telling us that deviations from randomness are often signatures of fundamental laws yet to be discovered.

____________________________________

[pp.171-172]

the patterns of human mobility

... About a year after the publication of my first book on networks I had grown used to e-mails and calls from readers seeking advice on inter-connected systems. This was one of the few times that someone had called not to ask but to give. He had my full attention.

The caller was a high-ranking executive at a mobile-phone consortium who'd recognized the value in having records of who is talking with whom. After reading 'Linked' he had become convinced that social networking was essential to improving services for his consumers. So he offered access to their anonymized data in exchange for any insights our research group might provide.

His intuition proved correct: My group and I soon found the mobile users' behavior patterns to be so deeply affected by the underlying social network that the executive ordered many of his company's business practices redesigned, from marketing to consumer retention. With that, he pioneered a trend that over the past few years has swept most mobile carriers, triggering an avalanche of research into mobile communications. Despite his crucial role in advancing network thinking in the mobile industry, his combination of modesty and caution prevented his ever wanting his name attached to any of it.

As my group and I immersed ourselves in the intricacies of mobile communications, we came to understand that mobile phones not only reveal who our friends are but also capture our whereabouts. Indeed, each time we make a call the carrier records the tower that communicates with our phone, effectively pinpointing our location. This information is not terribly accurate, as we could be anywhere within the tower's reception area, which can span tens of square miles. Furthermore, our location is usually recorded only when we use our phone, providing ... information about our whereabouts between calls. ([ Data can also be gather after the fact and-or in real-time of any devices that has GPS (Global Positioning System ]) Despite these contraints, the data offered an exceptional opportunity to explore the mobility of millions of individuals.

____________________________________

[pp.193-195]

... As a result we tend to romanticize college life, the cradle of youth culture, seeing students as perhaps the most spontaneous and thus least predictable segment of the population. Yet Sandy Pentland, an MIT professor who follows the chatter of hundreds of students every day, finds that concept preposterous.

In the early 1990s Pentland started a research program in wearable computing at the Media Lab at MIT, prompted by the the realization that, given the rate at which computers were shrinking, we soon would want to have them with us all the time. Sandy's vision of the future proved remarkably accurate, as today computers have become a part of our wardrobe, fashion accessories of a kind. In fact, for the most part we have stopped even calling them computers. We refer to them simply as smart phones.

In the fall of 2002 Nathan Eagle, a doctoral student in Sandy's lab, offered one hundred MIT students free Nokia smart phones, a desirable top-of-the-line gadget at the time. This was no handout, however; the catch was that the phones collected everything they could about their owners: whom they called and when, how long they chatted, where they were, and who was nearby. By the end of the year-long experiment, Nathan Eagle and Sandy Pentland had collected about 450,000 hours of data on the communication, whereabouts, and behavior of seventy-five Media Lab faculty and students and twenty-five freshmen from MIT's Sloan School of Management.

Trying to make sense of his data, Nathan arranged each student's whereabouts into three groups: home, work,and "elsewhere," the latter category assigned when they were neither at home nor at work but jogging along the Charles River or partying at a friend's house. Then he developed an algorithm to detect repetitive patterns, quickly discovering that on weekdays the students were mainly at home between the hours of ten P.M. and seven A.M. and at the university between ten A.M. and eight P.M. Their behavior changed slightly only during the weekends, when they showed an inclination to stay home at late as ten A.M.

None of these patterns would shock anybody familiar with graduate student life. But the level of predictability of their routines was still remarkable. Nathan found that if he knew a business-school student's morning location he could predict with 90 percent accuracy the student's afternoon whereabouts. And for Media Lab students, the algorithm did even better, predicting their whereabouts 96 percent of the time. ([ we are creatures of habits ])

It is tempting to see life as a crusade against randomness, a yearning for a safe, ordered existence. If so, the students excelled at it, ignoring the roll of the dice day after day. Indeed, Nathan's algorithm failed to predict their whereabouts only twice a week, during rare hours of rebellion when they finally lived up to our expectation that they be wild and spontaneous. Yet the timing of these unpredictable moments was by no means random--they were the typical party times, the Friday and Saturday nights. The rest of the week, twenty-two out of twenty-four hours a day, the students were neither the elusive Osama bin Laden nor the ubiquitously erratic Britney Spears but intead dutifully trod the deeply worn grooves of their lives. So maybe the Harlequins were onto something when they insisted on using an RNG(Random Number Generator). Had they studies at MIT, their whereabouts would have been no mystery--not to Nathan, nor to the Vast Machine.

But we may yet avert the dawn of an Orwellian world as described in 'The Traveler'. For me, this sense of hopefulness emerged in the summer of 2007 when I purchased a brick-sized wristwatch. It was a loud antifashion statement and doubled as a GPS device, which recorded my precise location every few seconds. After I had worn it for several months, Zehui Qu, a visiting computer-science student, applied Nathan Eagle and Sandy Pentland's predictive algorithm to the data collected by my GPS. Sure enough, after a few days of training, Qu was able predict my whereabouts with 80 precent accuracy.

While the algorithm's performance was impressive, the persistent gap between the 96 percent predictability Nathan found amoung the MIT students and my 80 percent raise a red flag. Neither I nor the MIT students were a fair representation of the population at large. Marta's study of the mobile-phone records had already explained why: When it comes to our travel patterns, we are hugely different. Some, like the MIT students and myself, are relatively home- and office-bound. Others are outliers, however, and travel a lot, tending to be less localized.

So does that mean there are people out there who are far less predictable than the MIT students and I? Truck drivers, perhaps, who travel the country for weeks at a time? Soccer moms, whose minivans shuttle between piano and fencing lessons? What about super-traveler Hasan Elahi, whose "suspicious movements" will undoubtedly land him in hot water again? How different are they from you and me? Are there Harlequins among us, individuals whose lives are driven by the roll of the dice to such a degree that their movements are impossible to foresee?

____________________________________

[pp.199-200],

Before we move on, let me clarify that there is a fundamental difference between WHAT we do and how PREDICTABLE we are. When it comes to things we do--like the distances we travel, the number of e-mails we send, or the number of calls we make--we encounter power laws, which means that some individuals are significantly more active than others. They send more messages; they travel farther. This also means that outliers are normal--we EXPECT to have a few individuals, like Hasan, who cover hundreds or even thousands of miles on a regular basis.

But when it comes to the predictability of our actions, to our surprise power laws are replaced by Gaussians. This means that whether you limit your life to a two-mile neighborhood or drive dozens of miles each day, take a fast train to work or even commute via airplane, you are just as predictable as everyone else. And once Gaussians dominate the problem, outliers are forbidden, just as bursts are never found in Poisson's dice-driven universe. Or two-mile-tall folks ambling down the street are unheard of. Despite the many differences between us, when it came to our whereabouts we are all equally predictable, and the unforgiving law of statistics forbids the existence of individuals who somehow buck this trend.

But let statistics forbid, halt, hinder, impede, refuse, and deny, there's still someone who won't be limited by it. Our friend Hasan Elahi.

[pp.201-202]

... When Zehui Qu ran the predictive algorithm on Hasan's data, though, it was an epic failure--only three times out of more than four thousand hourly attempts did he succeed in predicting Hasan's movements. .... For all practical purposes, Hasan was a Harlequin, fully unpredictable.

I confronted Hasan with our conclusions, telling him that, as far as we were concerned, he was completely random. His predictability was practically zero.

"Can't be zero, is it?" He laughed, then continued without missing a beat, "I mean, there are a few places I go now and then."

Sure, Hasan did visit the same spot in New Jersey 131 times, which, as we later learned, was his home at the time. .... By way of comparison, during the two-month period I dutifully toted my GPS device around, I was tracked at home on more than 880 occassions. The difference between Hasan and me boils down to this: While I was predictably at home every night, Hasan was just as likely to be on a train in Europe or asleep in an airport as spending the night in his own bed. He did go home occasionally, but there was no recognizable pattern to it.

For Hasan's perspective, his unpredictability wasn't all that surprising; and while he never explicitly said so, I think he found our whole analysis somewhat puzzling.* That he could readily explain each move he'd made convinced him that his behavior was absolutely normal.

"This is what I do," he said. "It's the transit points that's become my work."

Well, that didn't really cut it for me. Not because I doubted what he said. The real problem is that if power laws had governed our predictability, as they did the distances we cover, we'd expect to have a few outliers. Once power laws are absent, however, outliers are no longer normal. They are forbidden, making us all equally predictable. But no matter how we parsed the data, when it came to his predictability and lifestyle, Hasan was an outlier. Since outliers could not exist in this context, he was not normal any longer. Just as Homeland Security has suspected.

* "As of mid-April and on, I was on sabbatical leave from Rutgers," he told me, adding, "so I didn't exactly have a regular weekday schedule to go on. Even when I was at school, I would just basically fly in, teach my class, and then leave. So it does make sense." He then thought a moment, adding, "Because I literally was all over the place that year."

____________________________________

prediction, predictability, forecast

[p.204],

Despite the high predictability it represents, my low entropy is not a lock on my future--from it you can predict me only if you also know my history. Furthermore, if my entrophy is high, my past will reveal little of what lies ahead. If my entropy is low, my movement should be easy to foresee, but only if you have access to my past whereabouts. There is a simple lesson in this, ... : In order to predict the future, you first need to know the past.

[p.205]

... while low entropy does mean predict-ability, to predict your future location we need access ot your past whereabouts. And as penetrating as they are, phone records are insufficient to the purpose. To predict your future location, I need to know your hourly whereabouts for the past several months. <skip last sentence>

So at the end of the day, whether we are examining the events of today or the 16th century, the future is hard to foresee. And what if our past suddenly becomes transparent? Our future, both as individuals and as a society, may cease to be so mysterious. So, in order to reach into the future, we must first go back in time.

____________________________________

[pp.237-238]

depression

By 2020, depression will likely be second to only heart disease as our nation's deadliest killer. It is one of the greatest health scourges of our age, affecting approproximately 18 million Americans, and it is often lethal. Up to 60 percent of all those who commit suicide suffer from mood disorders. Depression is also one of the most mis-understood diseases, as more than half of the U.S. population views the condition as little more than personal weakness. Its sigma is rooted partly in its diagnosis--doctors rely mainly on self-reported symptoms, which are, by definition, subjective. And so if technologies were developed that could codify a depression diagnosis as clearly and stringently as we can diagnose cancer or heart disease, not only would millions be helped, but the confusion and ignorance surrounding the debilitating condition would be wiped out as well.

A common ... [...] Since by now we know what constitutes a normal activity pattern, we can ask, do depression patients do anything differently from the rest of us?

It was the Japanese research group from Tokyo University who first addressed this question. They provided 25 individuals sensitive wrist accelerometers, designed to pick up on even the slightest movement of the hand. The detectors confirmed that human activity is bursty, down to the slightest movements of our wrists. Indeed, the researchers determined that the length of periods of rest--when the subjects' hands did not move--follows a power law. Most rest periods lasted only seconds, or minutes at most. Yet these pauses co-existed with movement free intervals lasting hours, capturing sleep, rest, or meditation.

14 of the 25 subjects displayed a more intermittent rest pattern than did the others. They were clinically depressed. The differences in their movements were striking: The average rest period in the healthy subjects was about 7 minutes, compared to over 15 minutes for depression patients. Furthermore, the scaling exponent, a number that uniquely characterizes each power law, was larger for healthy individuals than for the depressed. And so it would seem that being "encased in cement" was more than a mere illusion, actually corresponding to detectable changes in a depressed patient's activity pattern.

[...]

____________________________________

[pp.257-258]

Scientists are poor prognosticators, often unable to see the consequences of their work. Luckily we have engineers and entrepreneurs to fill the gap between theorems and products. I am no exception to the rule--many companies have successfully monetarized my research on networks, none of which I foresaw. The same it true for human dynamics--I am too shortsighted to fully comprehend the full potential in forecasting your activity.

Lewis Richardson's first book was a remarkable failure. And yet today weather forecasting is routine, following the principles he outlined while driving ambulances on the battlefields of France. In his second book, The Statistics of Deadly Quarrels, he ventured farther, aiming to predict conflicts and wars in the hopes of helping us one day avoid them altogether. He failed once again. And so the real question is this: Have we matured enough to trust our predictive abilities?

____________________________________

[p.102]

Richardson was not the first to discover this pattern. Indeed, the 19th-century economist Vilfredo Pareto found that while the vast majority of people are poor, a few individuals amass outlandish wealth. The existence of the rich isn't surprising, as even if wealth is acquired randomly some will be richer than others. Pareto found, however, that the rich were far richer than a random distribution of wealth could ever explain. Richardson's and Pareto's work showed that wars and income follow what we call a power law. Specifically, many small events co-exist alongside a few extra-ordinary large ones.* It meant that for every world war and every Gates and Rockefeller there would be countless small conflicts and millions of poor.

____________________________________

predict foretell forecast reading room

Friday, December 10, 2021

Albert-László Barabási

No comments:

Post a Comment

Chin-tang sah

Blog Archive