UPDATED: The Political Poll Bungee Cord: What a Difference Three Weeks Make

As American as apple pie, pie chart, that is.  

 

U.S. Political Party Affiliation. Image: Public Doman

On September 14, I published this poll chart below on the so-called national ballot for the House of Representatives from Pollster.com.  It appears obvious to any observer that the Democrats (indicated by the blue trend line) were losing ground as fast the Republicans were gaining it.  But looking inside the data, plus getting out the ol’ Excel spread sheet and doing some analysis of my own, I realized that the national poll was missing some key factors.

For one thing, the national poll aggregate is made up of individual state race polls and then computed using specific criteria applied by Pollster.com (the old adage that all races are local races is true).  I also knew that the aggregate contained data compiled from a wide range of methodologies, as well as polls that were directly tied to political parties, who, even with the best intentions, will often introduce biases into their questions that favor positive responses for their candidates.

Here is the national aggregate House poll chart from September 14:

 

2010 National Congressional Ballot. Image: Pollster.com

 

For the purposes of our discussion, ignore the earlier results.  Just look at the roller-coaster for both parties since January 2010 through the present.  May appears to be the moment of truth for the Republicans and they continued to increase in a nearly linear fashion from then on while the Democrats declined on a similar downward slope.

Not so Fast!

In my previous post on this topic, titled, Where the Wild Thing Are I made this comment:

If this chart was the only one you looked at you would conclude that the Republicans have made huge gains beginning about May 2010 and now hold a 47.1% to 40.6% lead over the Democrats. And you would be wrong. Something is missing. First of all what about the undecided voters? Where are they? How many of them are there? What is their trend? For that answer, click here.

In that post, I then led the reader through the process of using the Pollster.com User Tools to come up with a much different looking trend line because it eliminated all the polls that either were of questionable reliability or directly tied to a political party.

On September 26, I spent an evening working on generating some of my own statistics using the polling results from the Pollster.com website.  Here is my Excel chart of the aggregate data, all polling groups included, my results came out at 47% Republican and 44% for the Democrats:

 

National Congressional Poll Aug-Sep 2010. Data Courtesy Pollster.com

 

By carefully watching the movement of the poll results and tracking the changes in the gaps, I became more convinced that the trends were changing, that it was possible the Democratic candidates were gaining, although I could not estimate how much.  One factor likely appeared to be the ending of the primaries, and the results from those races, if one ignored the pundits and the party-motivated spokespeople, I wanted to see what the trend was emerging.  It was time to fire up the Excel and do a bunch of number crunching and running through the Chart Wizard.  Except the new Excel doesn’t really have a chart wizard, so I fortunately know how to build the charts, having done it several thousand times having used one form or another of Excel since 1992.

At this point if you want to read the wonkish discussion and statistical analysis you can go to that page by clicking here.

The trend in the chart above confirmed my gut.  There had been an upturn for the Democrats but also for the Republicans.  One limitation of every chart is to decide what it means.  A trend line, in this case a “moving average,” does give one a picture of change, but does not communicate what is pushing the change.  The meaning, in one sense, is secondary.  I was interested in the trend, because the dynamics pushing the trend begins with individuals.   And as I pointed out in my post, The Black Poll Wars, Part II, the concept of one person, one vote no longer accurately describes the inner process of the American voter.  Rather, a theory I dubbed “isovoting” is based on the assumption that,

The transformation of the vote into a compilation of isovotes [that is, the subpersonal meaning the person assigns to different issues that must be reasoned into a single vote on the ballot]  is the key to understanding the American Electorate…The Uncertainty Principle [as defined by Heisenberg] shows that the isovotes cannot fit the Classical Statistical models for voter behavior. Like quarks in atoms, isovotes behave in dynamic ways that cannot be predicted with certainty either before or after they are observed, and that the very behavior of the survey taker will have a direct affect on the nature of the isovotes, especially with regard to the person assigning meaning to them, creating a new future for that person’s set of isovotes that did not exist prior to being polled on his or her preferences.

In short the uncertainty naturally built into the isovote process each person goes through when voting is too complex to discern, and components within the isovotes can change, sometimes affecting the others and sometimes not.  Therefore, following the trending becomes the only reliable methodology to ascertain the what will possibly take place on November 2nd.

That trend is beginning to emerge, but with caveats discussed below:

 

House General Ballot Chart 3 October, 2010. Image: Pollster.com

 

This scatter plot with the trend line, covers the same length of time as the first chart in the post, so neither of them are as sensitive in representing the  change over the past two months as the second chart I built using Excel.  Unfortunately, the flash function of the Pollster.com chart cannot be copied onto this post.  However, you can look at the same time frame, with all polling organizations represented by clicking here.  The gap between the two parties has shrunk to 44.2% for the Republicans and 42.8% for the Democrats.

The results get even more interesting, though, when you eliminate the less statistically reliable polls (which I include as the internet polls and the robocall polls; the first being difficult to ensure true randomization, and the second on the basis it is easier to lie to a computer voice asking the questions than it is to a real interviewer).

Bungee Jumping With the Polls

The trend using this second set of criteria can be viewed by clicking here.  The trend lines now have crossed with the Democrats taking the lead by a 45.8% to 44.5%. But whether this set of percentages is really good news for the Democrats depends on three factors.   First, how many people are registered as democrats and will vote as a faithful member of the party.  Second how many of those individuals will vote in the election.  And third, the most difficult questions to answer is how many people who are not Democrats, who either identify themselves as Independents or are Republicans who plan to cross party lines with their vote, will vote Democratic.  These caveats are not difficult to ascertain, but reading the subtleties of the trending, since it is always in flux is much harder to determine.  Therefore, it is possible that despite a percentage majority showing in the polls, the party with the upper hand in terms of percentage may still end up losing more races than it wins.

 

Voting--The American Way. Photo Courtesy: http://www.etches-johnson.com

 

Concluding Remarks

Using the considerable resources of Pollster.com and the Gallup Polling organization, we can come up with some interesting speculation about the coming election.  For example, we know roughly how many people are going to vote, 46.8 million Democrats and 46.4 million Republicans, a total of  93.2 million voters.  The percentage difference is 50.2% (D) to 49.7% (R).  That’s a tiny difference of only 466,000 voters compared to the national scale.  But that analysis is actually not correct, because these numbers represent the categories of voters, D, R, and I that will vote either Republican or Democratic.  Tucked inside the party’s totals are  14.7 million Independents who will vote Democratic and 18.9 million who will likely vote Republican in this election.  That is much larger gap of 4.2 points in the favor of the GOP. Another factor we can look at is registered voters, who are more likely to vote, and numerous polls distinguish between registered and likely voters.    I analyzed the polls that interviewed registered voters and came up with 31 surveys.  Plotting out those surveys, I came up with the following chart:

 

Data Courtesy of Pollster.com

 

Note: for an explanation of the R² number, please click here

UPDATE: Since I wrote the post I came across this very illuminating article on the issue of choosing “likely voters” in contrast to “registered voters” as the survey sample on the Huffpost Pollster (The Huffington Post has just acquired Pollster.com and integrated its sites into Huffington’s), by Mark Blumenthal (who originally founded Pollster.com).  I recommend you read through his article.  He gives a nicely framed explanation of how pollsters choose who to survey and it is written for the general reader: “Likely Voters: How Pollsters Define and Choose Them.”

After reading Blumenthal’s article, I recalibrated the filters on the National Congressional Ballot on the Pollster site to only include those who surveyed registered voters.  To see the result, click here.  The results contradict the unfiltered chart that shows the Republicans up by over 7 points.  Instead by looking at the registered voters (which I hold are still in the highest percentage of all voters) the Republicans hold the thinnest of margin at 45.3% over the Democrat’s 45.0%  Statistically speaking this is a virtual tie.

Is the trend line good news for the Democrats?  Yes and no.  Any time one party gains ground and passes the other in the number of people who say they will vote for them, that is cause for at least cautious optimism.  But looking at the visual slope of the lines in the chart above does not indicate the Republicans have begun to dramatically slump.  A week from now, they could just as easily stopped the small downward slope and recovered  to  move above the Democrats again.  The positive factor for the Democrats is the R² of their trend is significantly stronger than the Republicans.  In other words, it may be evidence of more “oomph” behind the upward change in direction.

We are down to three weeks and counting.  The fun continues unabated.

Swimming through Boiled Okra: The American Political Stew, 2010

Swimming through Boiled Okra:

The American Political Stew, 2010

Some Background:

I lived in Texas for three years while attending seminary at Texas Christian University.  Being a native of the Northwest, I was exposed to a whole new cornucopia of foods: great Texas barbecue, Tex-Mex, and Southwest cuisine. I learned how to eat grits the right way (butter and salt and pepper, or cheese—none of this milk and sugar over Cream of Wheat stuff).  To this day I love grits.

My one culinary defeat was okra.  For those of you who haven’t experienced this little vegetable,

Okra--Secret Dispenser of Slime. Photo: FowlerNurseries.com

Okra--Secret Dispenser of Slime. Photo: FowlerNurseries.com

it certainly looks benign enough in its raw form.  It’s also as popular as broccoli in more northern climes and a staple vegetable in numerous countries around the world.  You can see from the photo, it can look quite appetizing.  Okra can be prepared in many ways.  It can be breaded and fried (of course, most anything in the South can be breaded and fried—and usually is).  It can be boiled with all sorts of other foods.  Many people love it pickled.  It’s a popular ingredient in a wide variety of soups and gumbos.  That’s all fine and good, except…

Boiled okra is slimy.  For those three years I tried my best to sample okra in its various gastronomic manifestations.  I discovered, despite all this experimentation, that the only way I could handle okra was fried.  Fried okra is not slimy.  It just doesn’t have much taste for my palette, other than an oily breaded mushy…well…veggie-something-or-other. Take a look at the crosscut round of okra.  It looks a lot like a crosscut jalapeño pepper.  Only the jalapeño has both flavor and zing.

Okra Seafood Gumbo. One of a thousand ways to create vegetative slime.

In fairness to the otherwise popular vegetable, I selected two of my friends (completely non-random, and no intent to be so), on separate occasions, and who don’t know each other, and asked if either liked okra?  Their responses were identical to mine: Okra in soup is slimy and I don’t really like it.  So, what’s the point?  There are at least two other males in the United States who, without prompting as to reason, agree that okra in soup is slimy.  And none of us like it because of that particular quality.  And that’s important because of what comes next.

A Test of Your Gag Reflex:

Back to slimy.  Not just slickish slimy, but stringy and slimy.  As for stringy: A hundred times

Ultimate Alien Slime, Aliens, 1979, Photo Courtesty MGM.

worse than my grandmother’s over-cooked rope-strength stringy asparagus.  And as for slimy: pure gaggy slimy: You start to chew the spoonful of boiled okra, which just seems to release this gelatinous goo, resembling the stuff that drips off of your typical movie monster’s tongue or, uh, snout. As you try to swallow the bite the goo elongates itself so that by the time the first half of your bite has traveled the length of your esophagus, reaches your cardiac sphincter, and dumps into your stomach the other half is still trying to get out of your mouth and down your throat.

Gag.  Retch.  Get me something to wash down this uck!  Now!

An Analogy of the Absurd—But the Bridge to the Topic at Hand:

Imagine stirring a pot of okra the size of an Olympic swimming pool with that potential for such slimy viscosity.  Imagine swimming through a pool filled with such.  Michael Phelps wouldn’t be pleased.  Michael, whatever his other imperfections, would have the good sense to avoid such an Olympic sized pool of mucousity.

Okra Stew--Imagine an Olympic Pool filled with this.

Now to the Main Dish:

On the other hand, that exact unpalatable characteristic makes for a good analogy of the current state of American politics.  The so-called melting pot of America has been emptied of its finest ingredients, civility, respect, loyalty, etc, and filled to the brim with okra soup.  The blogosphere is boiling over with okra slime, from both the right and the left, but all too often the temperature of rhetoric coming from the right is higher.  In this context, characters such as Limbaugh and Beck, Palin and the Miz Liz of Cheney are but spices in an otherwise noxious recipe of political okra gumbo.

Last summer, we were caught off guard by the sudden rise to the boiling point of the Town Hall Meetings held in August during the Congressional Summer Recess.  But this year, the pot is already bubbling resembling those gloppy mud pots in Yellowstone National Park, and it’s still over seven months until the November mid-term elections.  Itsa bubblin’, like they say.  I think.

This year there will be no surprises.  Members of Congress in both Houses and from both sides of the Aisle may find it best to attend their town hall meetings dressed in a heatproof full-body firefighter outfit.  The political okra soup pot likely will reach a rolling boil even before summer.  The slop-slinging will probably be fierce.

The question is can it be sustained?  My observation last year was that the Sturm und Drang of the Tea Party’s birth and its rapid rise to an ear-shattering keen burned out before the end of the month.  Part of that, in my opinion, was it was “newsed” into numbness.

Numb News—It’s the American Way:

The success of cable/satellite/internet news with its around the clock accessibility very quickly has habituated Americans (and most of the world, undoubtedly) to a very short news cycle.  Inside of two weeks last August, people at the Town Hall Meetings screaming the same rant over and over lost its punch if not its volume. Though the opponents of everything Obama were taking great delight in the attention and media coverage they were receiving, they missed an unintended consequence of that saturation.  They unwittingly became passé. Almost with the predictability of an autonomic response, the polls shifted subtly, but the shift was critical.  It was pro-reform.

Inside the shift was the data that kept Health Care Reform alive, passing the House in November and the Senate on December 24. Despite the loss of one senate seat in January messing up the Senate Democrats’ supermajority of 60 seats the rhetoric of the opposition from the elected officials, the right-wing pundits and the “angry” right-wing citizenry did not shift.  Their message, although loud, consistent, and vitriolic, became less and less influential as each day passed.

That message also offered nothing: the “Let’s start with a blank piece of paper gambit” failed, not because the Republicans were united but because they had nothing to counter with.  If at the great Health Care Summit, the Republicans had shown up with a bill that could have been plopped down beside the one that had already passed (since December 24th, remember) that was half the height of the Democratic bill, Americans’ attention would have been riveted to know what was in that piece of legislation.  Instead the Republicans brought a blank piece of paper and kvetched for seven hours about the size of the already passed bill.  The three best words for this colossal error are: stupid, stupid, and stupid.

The opponents did not recognize their strategic error. They thought they were being consistent and united.  The president and the Democratic leadership, on the other hand, correctly interpreted the message as dragging itself down: that the public sentiment in a shift of even one or two points in favor of reform was far more important than the actual percentage of support or opposition was being rolled out weekly by the multitude of pollsters.  TV, Internet, Radio, newspapers, magazines along with pundits of all stripes missed that nuanced reality, their focus locked on a depth of field most conducive to seeing their ratings and profits and not the meaning of the changes.

Health Care Reform passed, astonishing and infuriating the Republican leadership and all sorts of right-wing groups.

Elections 2010: Jump in, the Okra’s in Prime Slime!

What’s next?  More of the same: A big pot of slimy okra political soup we will be forced to swim around in between now and November.  The Republican formula of Stupid3 remains the strategy they are taking into the Fall under the illusion (or delusion) it will be different this time and they will snatch the majority status from the Democrats in a great uprising of voter rage, or a groundswell of secessionist sentiment threatening the fabric of the Republic, whichever comes first.

I almost hesitate to write this next section for fear of tipping off the Republicans and the Tea Party supporters to the actuality of the situation that the colossal error they made attempting to kill health care reform remains a colossal error as they plan for the mid-term elections.

Poster Advocating Revolution. But What Kind? Photo: Menifee Tax Day Tea Party

The emerging rhetoric calling the administration a “regime” accompanied by repeated less than veiled threats of revolution and civil war, marches with guns in plain sight are not the next phase they believe it to be; it is rather an escalation of the same message.  TV and the other media again will create the opposite intended consequence the Right Wing wants to convey.  Their agenda, antics, and demonstrations will be covered ad nauseum, giving them all the coverage they desire, but dulling the impact of their effort: Why?  Very simply, Americans want “new” news.  Every day.  That is what we have been habituated to expect from the media.  It’s not a matter of how it’s slanted or editorialized, or punditized.  It has to be new.

This is an essential lesson the Democrats must keep in the center of their political radar screen, locked on like the Space Shuttle launching toward the International Space Station.  They have to remain disciplined in their own rhetoric as the campaign heats up.  The key: New talking points must be rolled out every week or so of what Obama and the Dems have accomplished and what they plan to accomplish after the polls close.  They don’t even have to counter what the Republicans are saying.  The Republicans will continue to tighten the trap they set for themselves in the broken record scenario they initiated upon Obama’s election in 2008, and the contrast between a constantly renewing fresh message and the broken record message will quickly create a gulf between the two in the Democrats favor.

Guns and Bombs Belong in the Movies and TV, Not in Real Life:

Why?  People crave the status quo.  The vast majority of U.S. citizens do not want revolution or civil war or any other highly disruptive political or social action.  We’re not talking about changes to health care—we’re talking about being able to go shopping or out to eat or on vacation, Friday night football, going to worship, school plays, and a thousand other everyday things we routinely do.

Domestic tranquility is what Americans want.  The news video of the bloody revolt and possible regime-changing coup in Kyrgyzstan is exactly the opposite of what Americans believe political change should be.  The more violent that situation becomes and the more coverage it will get because of American military interests in the country, the more nervous the typical American will be about the language in the okra soup.  They see suicide bombers belonging in Baghdad and Kabul, not valiant freedom fighters, 21st Century Minutemen here in the U.S. claiming they are wresting their constitutional rights from an oppressive and socialist government.  Regardless of how dissatisfied they may be about their taxes, Americans do not want bombs and blood flowing down the streets of their community.  The reason is straightforward: Americans make political change through ballots, not through bombs and bullets.

Tea Party Protester Advocating the use of Guns. But to Shoot What or Whom? Photo: TalkingPointsMemo.com

Swimming in the Soup of the Statistically Illiterate:

We can be certain that the media will continue to misinterpret the polls because they do not understand how to correctly interpret them to begin with.  Most political groups will continue to misinterpret the polls because they are always looking for an advantage for their side and a disadvantage for their opponents.  “Spin” is a set of heuristic blinders. Here’s why: Raw percentages are a flawed indicator of support.

We quote percentages as if they mean something absolute.  They do not.  Percentages provide information, just not what we typically believe it is.  The fundamental error in interpreting poll percentages is equating a final percentage of a vote with a pre-vote percentage.  The two are not equal indicators of support.  The better pollsters understand this and couch their questions with very precise language and report their results with statistical caveats that, for the most part, the media and the public ignore, and then they are surprised when the results don’t match the polls.  They blame the polls (some justifiably) instead of having the insight to realize they didn’t correctly read the data in the polls. We will see a lot of this in the next seven months.

Central Limit Theorem: One reason calculating percentage trends is complex. "In probability theory, the central limit theorem (CLT) states conditions under which the mean of a sufficiently large number of independent random variables, each with finite mean and variance, will be approximately normally distributed (Rice 1995)." Source: Wikipedia

Should we not believe the polls?  No, but be cautious about reading into the number what you want it to mean, rather than looking at the gap and vector of the differences between the two percentages.  That is where the real information is.  And understand there are ways to use the percentages that sound valid but are really nothing more than concoction and spin.  Second, don’t make the mistake of treating a final vote result with a pre-vote survey.  Even if the final outcome numbers are identical, they are two quite different metrics, almost to the point of being two separate statistical species.

Understand the spin-meisters of all the political parties and their various PACs are not going to give you the slightest bit of help in understanding any of this.  They are paid to convince you the numbers are always favoring their party or candidate, even if the most accurate interpretation suggests disastrous defeat.  They want the political okra pot to be as slimy as possible to confound your ability to squeeze the slightest bit of truth from the numbers.  No, it’s not pretty, and never will be, apparently, until the pot is emptied and a new recipe of soup, sans okra, is placed on the stovetop.

Is it November yet?  I can’t stand okra.

Okra: Abelmoschus esculentus. Photo: digthedirt,com