The Signal and the Noise by Nate Silver

The Signal and the Noise by Nate Silver is a solid book on predictions and probability which at times is a bit weighty, but due more to the subject than anything Silver should have done differently in the writing.

Silver is the creator and still lead writer of the (now New York Times) blog FiveThirtyEight which began in advance of the 2008 Presidential election and has become a go-to place to get political predictions. Silver makes his calls based in probabilities and does so based on looking at a wide variety of poll numbers and factoring in weighting information about those polls. Out of the November 2012 elections, Silver correctly predicted the Presidential winner in each state as well as all but one Senate race (those unpredictable folk in Montana).

 The subtitle to Silver's book is Why So Many Predictions Fail — but Some Don't and he notes fairly in reference to his main title that "the signal is the truth. The noise is what distracts us from the truth." In terms of structure, he lays out in the intro that the first half of the book diagnoses problem with many predictions and the second looks at making predictions better, in large part through the application of something called Baye's theorem.

Some of the prediction areas that Silver notes as having room for improvement are below:

Many people felt the real estate plunge of a few years back couldn't occur because they made predictions based on the idea that home loan defaults were much more independent of each other than was the case. In actuality, what occurred is loan defaults turned out to be very much related as a direct result of home values being tied to one another.

Pundits making their calls about political winners based on feeling turn out to often be wrong. As evidence of this, Silver notes The McLaughlin Group and how predictions on the television show turned out to be roughly equal parts right, wrong, somewhat right and somewhat wrong.

Forecasting disasters such as terrorism, hurricanes, earthquakes, foods and influenza outbreaks.

As Silver gets into the idea of improving prediction, he brings up the aforementioned idea of Bayesian reasoning named after Thomas Bayes, which includes considering the likelihood of events in making predictions. Silver also notes that the concepts behind Baye's theorem fit heavily into principles that guide his FiveThirtyEight blog... (1) think in probabilities of things coming to pass rather than just making predictions, (2) make changes to forecasts as new info arises and (3) look for consensus in forecasts (which sounds to me like a Wisdom of the Crowds-related idea).

All in all, it's a fairly heavy read, but a worthwhile one (or at a minimum, a worthwhile skim of parts that resonate) for somebody interested in predictions, why many of them fall short and how they can be improved.

Penn State Crimes, Moral Failures & Journalism

While the typical path of this blog has been to link to and write about specific pieces or books that struck me as interesting, this post is about the Penn State scandal and it's mesmerizing (in mostly bad, but also some good) elements.

Details of the actual indictment and alleged crimes within are horrifying, but what's really captured attention has been the characters apart from the accused molester Jerry Sandusky, especially living legend and now ex Penn State coach Joe Paterno. In his case, you have someone who has done much good, but in this case (by all appearances) failed at the most important moment.

It's a fascinating subject, this concept of duality within an individual... his good deeds remain so, but one specific thing handled in entirely the wrong way impacted so much for the victims. Paterno is of course the most recognizable name involved that could have raised these allegations to police years ago, but there's also now ex Penn State President Graham Spanier and his head in the sand approach both years ago when Sandusky's crimes were raised and since the indictment came down over the weekend. If one can detach from the horrible nature of the crimes, it's a study in how people in power (now including the Penn State Board of Trustees that fired both Spanier and Paterno) react to events, create environments and set policy.

-----

So... the story itself has been riveting, but what's also held my attention over the past week has been the words from sports journalists reporting on the story. Some writers I respect a great deal have had fascinating observations to make and they've first appeared as real-time twitter musings and then as published columns.

There's been so many good pieces written already, but what many sports journalists and fans of sports journalism are waiting to eventually read is the announced earlier this year book from Joe Posnanski on Paterno and this season (which of course nobody could have envisioned turning out like it has). Posnanski thus far has written two different blog posts on the scandal, first Darkness and then Curiously Short Posts and I doubt he knows what his eventual book on Paterno will contain, but I have to imagine it's going to be an enthralling read. Just a guess of course, but it may well be heavy on the aforementioned duality of how a good person (which Paterno certainly seems to be) can do a bad thing (specifically, the limits to his actions taken when allegations were brought to him).

-----

Back to the subject of reporting and sports journalism... reading the columns and musings from good writers as this story has unfolded has gotten me thinking more about journalism and writing. To this end there's been three different pieces I've come across lately about the profession that all stand out as interesting.

First was an address given by Nate Silver to the Columbia School of Journalism. Silver founded the political blog FiveThirtyEight and in his speech imparts both his background and valuable career lessons for someone about the enter the field of journalism.

Second was a series of tweets from Tommy Tomlinson about an Ira Glass speaking event. I've also been at a live event by the This American Life creator and concur that Glass is a definite master storyteller.

Third was a tumblr site We Are Journalists I just came across today. Very interesting vignettes from people in the profession working to chase down and report well on stories... including things like the horrible crimes and subsequent inaction (and now action) out of Penn State.

-----

Updated on 11/14 with additional pieces that struck me on the child sex abuse scandal (not a "scandal, not a "sex scandal")...

- Good Riddance, Joe Paterno by Buzz Bissinger for The Daily Beast
- Omelas State University by author John Scalzi on his website
- The End of Paterno by Joe Posnanski on his Sports Illustrated blog

A horrific and amazing ongoing story.

Kids Mental Health / Ted Kaufman - from Time Magazine

Two different pieces from the Nov 1 issue of Time Magazine that struck me as interesting...



The first was "Keeping Young Minds Healthy" by Jeffrey Kluger about some of the psychological ailments that can manifest themselves in childhood. A really interesting story with important content. I also found that this piece was part of a larger section on the Time website titled "Health Checkup: Kids and Mental Health"... which featured additional content around kids and health.

The second thing which stood out was the Joe Klein In the Arena column "Ted Kaufman, the Temporary Senator". Interesting reading about a very solid sounding politician.... which reminded me of Nate Silver's excellent political forecasting site FiveThirtyEight (now part of the New York Times).

Sports Illustrated Article on FiveThirtyEight.com

The Oct 13, 2008 issue of Sports Illustrated has a brief piece titled "The Sports-politics Connection" on http://www.fivethirtyeight.com/.

The article describes the website as using a statistics-based approach to predicting the winners of political contests. The sports connection (and why this would appear in SI) is that the economics grad creator of the site also has a background in sabermetrics.

This statistical approach to predicting player performance in baseball was begun by Bill James and became more well known as a result of the Michael Lewis book "Moneyball" which was previously reviewed on this blog.

Back to FiveThirtyEight... it's a fascinating site and given that it's about politics and predicting winners, it's probably worthwhile to note that on the FAQ section of the site, the author, Nate Silver, describes himself as someone who votes Democratic the "majority of the time", but attempts to not have his own convictions reflected in any of the statistical predictions.

Really interesting stuff.