Hospital Food for the Mind

Featured

Gallup didn’t call me for their USAToday poll on the Debt Ceiling bill.  The results indicated Democrats and progressives were more positive about the final product than Republicans and Tea Party members, a fact I find incredulous.  I would have been ready to give the pollster a detailed account of my thorough displeasure over the mess this bill just created.  And that’s my take being firmly on the progressive side of the opinion scale.

Let the Backfiring Begin!

One group is stumped over what happened.  Tea Party members report being dissatisfied by the bill by 80%. Despite the fact their faction acted as a curdling agent in the legislation, rendering it both unpalatible and inedible but still force-fed into law, they don’t like it.  Granted, the bill did not force the virtual dismantling of the federal government or wipe out their most despised social programs. Nevertheless, I have this sneaking suspicion that they really believed that once they had hijacked the bill, they could force their will onto the rest of the Congress Backfire #1.

Backfire #2 appears to be that the global stock markets were already weakened and skittish from the Great Recession. Near panic from the debt ceiling fight, they took one look at the junk attached to an otherwise one-page piece of legislation and that anxiety blossomed into a full-blown state of apoplexy and, among other things wiped out $ billions in the Tea Party adherents’ investments and pensions.  And, of course that crash pulled in the rest of us thanks to their gross inability to understand Macroeconomics 101.

Numerous backfires will continue to create havoc in our politics. The final one I’ll mention in this post is the credibility of conservative agenda. Magnified and distorted a thousand times by the Tea Party’s first, and we can only pray, last congressional disaster visited upon the Union, their believability has been reduced to next to nothing.  They won’t get it, of course. In fact I expect them to be noisier at least through the 2012 general election. But as the backfires continue to damage the country at home and abroad, their chance to be a sustained political voice will be muted more and more.

The tragedy for the rest of us is the consequnces we will be forced to endure. The Tea Party won’t get that either.

Hey NBC & WSJ–What’s With Your Poll Report?

Featured

Today NBC and the Wall Street Journal released a new survey conducted by the polling company Hart/McInturff.  MSNBC.com’s deputy political director, Mark Murray writes,

The GOP’s ‘likely’ advantage

In the survey, 50 percent of likely voters say they prefer a Republican-controlled Congress, versus 43 percent who want Democrats in charge.

Last month, Republicans held a 46 percent to 43 percent advantage among likely voters on this question.

The GOP’s current seven-point lead, McInturff observes, is on pace — historically — to result in a shift of power in Congress. “The Democrats, with two weeks left, are facing very, very difficult arithmetic.”

Yet among the wider universe of registered voters, Democrats hold a two-point edge, 46 to 44 percent, which is up from the 44 percent to 44 percent tie in September.

But Hart calls that lead “hollow,” because not all registered voters will participate, especially in a midterm election.

Indeed, among those expressing a high interest in voting this midterm season, Republicans hold a 13-point advantage on the generic ballot, 53 percent to 40 percent.

So, as I often do, I clicked on the link to the published survey results and read it through looking for the results described in the article.  First time through I thought I missed this 50% to 43% advantage of the Republicans over the Democrats.  So I read it again, now looking line by line where that percentage comparison came up.  I couldn’t find it.  Okay, so one more time, very deliberately reading through the survey.  Nada.  What I did discover were two very interesting questions that belie a different mood in the electorate.

Keep in mind this was a survey of 1000 registered voters, which, as I have explained in previous posts, I hold to the theory that registered voters provide a more reliable sample and predictor of which way the election is more likely to go.  In two weeks we’ll know.

Let’s look first at question Q11a:

Q11a:   What is your preference for the outcome of this year’s congressional elections — (ROTATE:) a Congress controlled by Republicans or a Congress controlled by Democrats?

The result favors the Democrats 46% to Republicans 44%. That is mentioned in the quote above.  It’s also within the published margin of error of +/- 3.10%.  One could say, therefore, it’s a wash, but there is some interesting info in the trends.  The median (the midpoint of the 11 surveys NBC/WSJ has conducted since January 2010 for the Republicans is 44% and the Democrats, 43%.  Now that is tight!  The slope of those eleven surveys for this question is also small, 0.21% for the GOP and 0.25% for the Dems.  Not what you might write home about.  But still, when you apply those numbers to tens of millions of voters, small changes can make the difference.  It also shows that the Democrats are perhaps not in quite as bad a shape as the pundits have been droning on about month after month.

But there is more.  Question Q12A reads:

Q12a: If you had the choice in your congressional district, would you be more likely to vote for a (ROTATE:) Republican, Democratic, Libertarian, or  Green Party candidate for Congress?

Since the smaller party candidates are at best wild cards, we can’t make a prediction if those who claim affiliation will actually vote for them.  But here, we find the responses again favor the Democrats over the Republicans, 44% to 41%.  Unfortunately, the survey does not provide the historical results of this question.

If you have looked at the published survey results, you might have noticed a number of questions are missing.  So, we can perhaps infer that NBC and the WSJ decided the most interesting information in the survey they wanted to keep to themselves, which is their prerogative since they paid for it.  And perhaps that 50% Republican advantage is among those survey items that were, shall we say, redacted.  But if the Democrats are in such dire shape going into the election and the survey shows that is very clear, why bother with cutting questions out of the published report?  Wouldn’t a reasonable person, or especially a partisan one, such as the Wall Street Journal’s clear editorial preference for Conservatives, want that information right there for everyone to see?  I can’t answer that question, but I do find it perplexing.

Nevertheless, there is enough information to ponder the strength of the GOP’s “surge” as reported.  My updated graph with the trend lines still shows the Democrats in a stronger growth curve:



Registered Voters Surveys 18 Oct 10. Data Courtesy HuffPost Pollster

I will be eagerly looking forward to the next batch of polls to be released.  Each data point provides a world of information about might or can happen on November 2nd.

UPDATED: The Political Poll Bungee Cord: What a Difference Three Weeks Make

As American as apple pie, pie chart, that is.  

 

U.S. Political Party Affiliation. Image: Public Doman

On September 14, I published this poll chart below on the so-called national ballot for the House of Representatives from Pollster.com.  It appears obvious to any observer that the Democrats (indicated by the blue trend line) were losing ground as fast the Republicans were gaining it.  But looking inside the data, plus getting out the ol’ Excel spread sheet and doing some analysis of my own, I realized that the national poll was missing some key factors.

For one thing, the national poll aggregate is made up of individual state race polls and then computed using specific criteria applied by Pollster.com (the old adage that all races are local races is true).  I also knew that the aggregate contained data compiled from a wide range of methodologies, as well as polls that were directly tied to political parties, who, even with the best intentions, will often introduce biases into their questions that favor positive responses for their candidates.

Here is the national aggregate House poll chart from September 14:

 

2010 National Congressional Ballot. Image: Pollster.com

 

For the purposes of our discussion, ignore the earlier results.  Just look at the roller-coaster for both parties since January 2010 through the present.  May appears to be the moment of truth for the Republicans and they continued to increase in a nearly linear fashion from then on while the Democrats declined on a similar downward slope.

Not so Fast!

In my previous post on this topic, titled, Where the Wild Thing Are I made this comment:

If this chart was the only one you looked at you would conclude that the Republicans have made huge gains beginning about May 2010 and now hold a 47.1% to 40.6% lead over the Democrats. And you would be wrong. Something is missing. First of all what about the undecided voters? Where are they? How many of them are there? What is their trend? For that answer, click here.

In that post, I then led the reader through the process of using the Pollster.com User Tools to come up with a much different looking trend line because it eliminated all the polls that either were of questionable reliability or directly tied to a political party.

On September 26, I spent an evening working on generating some of my own statistics using the polling results from the Pollster.com website.  Here is my Excel chart of the aggregate data, all polling groups included, my results came out at 47% Republican and 44% for the Democrats:

 

National Congressional Poll Aug-Sep 2010. Data Courtesy Pollster.com

 

By carefully watching the movement of the poll results and tracking the changes in the gaps, I became more convinced that the trends were changing, that it was possible the Democratic candidates were gaining, although I could not estimate how much.  One factor likely appeared to be the ending of the primaries, and the results from those races, if one ignored the pundits and the party-motivated spokespeople, I wanted to see what the trend was emerging.  It was time to fire up the Excel and do a bunch of number crunching and running through the Chart Wizard.  Except the new Excel doesn’t really have a chart wizard, so I fortunately know how to build the charts, having done it several thousand times having used one form or another of Excel since 1992.

At this point if you want to read the wonkish discussion and statistical analysis you can go to that page by clicking here.

The trend in the chart above confirmed my gut.  There had been an upturn for the Democrats but also for the Republicans.  One limitation of every chart is to decide what it means.  A trend line, in this case a “moving average,” does give one a picture of change, but does not communicate what is pushing the change.  The meaning, in one sense, is secondary.  I was interested in the trend, because the dynamics pushing the trend begins with individuals.   And as I pointed out in my post, The Black Poll Wars, Part II, the concept of one person, one vote no longer accurately describes the inner process of the American voter.  Rather, a theory I dubbed “isovoting” is based on the assumption that,

The transformation of the vote into a compilation of isovotes [that is, the subpersonal meaning the person assigns to different issues that must be reasoned into a single vote on the ballot]  is the key to understanding the American Electorate…The Uncertainty Principle [as defined by Heisenberg] shows that the isovotes cannot fit the Classical Statistical models for voter behavior. Like quarks in atoms, isovotes behave in dynamic ways that cannot be predicted with certainty either before or after they are observed, and that the very behavior of the survey taker will have a direct affect on the nature of the isovotes, especially with regard to the person assigning meaning to them, creating a new future for that person’s set of isovotes that did not exist prior to being polled on his or her preferences.

In short the uncertainty naturally built into the isovote process each person goes through when voting is too complex to discern, and components within the isovotes can change, sometimes affecting the others and sometimes not.  Therefore, following the trending becomes the only reliable methodology to ascertain the what will possibly take place on November 2nd.

That trend is beginning to emerge, but with caveats discussed below:

 

House General Ballot Chart 3 October, 2010. Image: Pollster.com

 

This scatter plot with the trend line, covers the same length of time as the first chart in the post, so neither of them are as sensitive in representing the  change over the past two months as the second chart I built using Excel.  Unfortunately, the flash function of the Pollster.com chart cannot be copied onto this post.  However, you can look at the same time frame, with all polling organizations represented by clicking here.  The gap between the two parties has shrunk to 44.2% for the Republicans and 42.8% for the Democrats.

The results get even more interesting, though, when you eliminate the less statistically reliable polls (which I include as the internet polls and the robocall polls; the first being difficult to ensure true randomization, and the second on the basis it is easier to lie to a computer voice asking the questions than it is to a real interviewer).

Bungee Jumping With the Polls

The trend using this second set of criteria can be viewed by clicking here.  The trend lines now have crossed with the Democrats taking the lead by a 45.8% to 44.5%. But whether this set of percentages is really good news for the Democrats depends on three factors.   First, how many people are registered as democrats and will vote as a faithful member of the party.  Second how many of those individuals will vote in the election.  And third, the most difficult questions to answer is how many people who are not Democrats, who either identify themselves as Independents or are Republicans who plan to cross party lines with their vote, will vote Democratic.  These caveats are not difficult to ascertain, but reading the subtleties of the trending, since it is always in flux is much harder to determine.  Therefore, it is possible that despite a percentage majority showing in the polls, the party with the upper hand in terms of percentage may still end up losing more races than it wins.

 

Voting--The American Way. Photo Courtesy: http://www.etches-johnson.com

 

Concluding Remarks

Using the considerable resources of Pollster.com and the Gallup Polling organization, we can come up with some interesting speculation about the coming election.  For example, we know roughly how many people are going to vote, 46.8 million Democrats and 46.4 million Republicans, a total of  93.2 million voters.  The percentage difference is 50.2% (D) to 49.7% (R).  That’s a tiny difference of only 466,000 voters compared to the national scale.  But that analysis is actually not correct, because these numbers represent the categories of voters, D, R, and I that will vote either Republican or Democratic.  Tucked inside the party’s totals are  14.7 million Independents who will vote Democratic and 18.9 million who will likely vote Republican in this election.  That is much larger gap of 4.2 points in the favor of the GOP. Another factor we can look at is registered voters, who are more likely to vote, and numerous polls distinguish between registered and likely voters.    I analyzed the polls that interviewed registered voters and came up with 31 surveys.  Plotting out those surveys, I came up with the following chart:

 

Data Courtesy of Pollster.com

 

Note: for an explanation of the R² number, please click here

UPDATE: Since I wrote the post I came across this very illuminating article on the issue of choosing “likely voters” in contrast to “registered voters” as the survey sample on the Huffpost Pollster (The Huffington Post has just acquired Pollster.com and integrated its sites into Huffington’s), by Mark Blumenthal (who originally founded Pollster.com).  I recommend you read through his article.  He gives a nicely framed explanation of how pollsters choose who to survey and it is written for the general reader: “Likely Voters: How Pollsters Define and Choose Them.”

After reading Blumenthal’s article, I recalibrated the filters on the National Congressional Ballot on the Pollster site to only include those who surveyed registered voters.  To see the result, click here.  The results contradict the unfiltered chart that shows the Republicans up by over 7 points.  Instead by looking at the registered voters (which I hold are still in the highest percentage of all voters) the Republicans hold the thinnest of margin at 45.3% over the Democrat’s 45.0%  Statistically speaking this is a virtual tie.

Is the trend line good news for the Democrats?  Yes and no.  Any time one party gains ground and passes the other in the number of people who say they will vote for them, that is cause for at least cautious optimism.  But looking at the visual slope of the lines in the chart above does not indicate the Republicans have begun to dramatically slump.  A week from now, they could just as easily stopped the small downward slope and recovered  to  move above the Democrats again.  The positive factor for the Democrats is the R² of their trend is significantly stronger than the Republicans.  In other words, it may be evidence of more “oomph” behind the upward change in direction.

We are down to three weeks and counting.  The fun continues unabated.

Krugman Verifies the Bungee Cord

I’m sitting in the SFO airport waiting for final leg home from a week in San Diego at the National Disciples Pastors Conference.  Getting caught up on Paul Krugman’s NYT blogs.  Just read this: http://krugman.blogs.nytimes.com/2010/09/22/not-over/.  He uses the same Pollster.com I used in my last post.

Just sayin’.  More when I get home and a good night’s sleep.  Typed this on my Droid, by the way.

Where the Wild Things Are–Reading the Polls

Tuesday night, September 14, 2010: The final day of the big primaries prior to the general election in November.  I’m sitting watching the results come in.  My network of choice tonight is MSNBC.  It’s just so much fun watching Rachel Maddow narrate the primary like it was a Super Bowl.

Statistics during the election season are thrown around like cheap bead necklaces at a Mardi Gras parade.  Polls are quoted like they mean something and are perfect predictors of the future.

Well, I’m going to tell you something the politicians and pollsters and pundits would prefer you don’t know:

Polls Results Can Be Stretched Like Bungee-Jumping.

Yep, its true.  And I can give some simple examples.

You can follow along.  But open a new window in your browser so you can click back and forth more easily.

First click here on Pollster.com.  This should take you to the 2010 National Congressional Ballot. This is for the House of Representatives only.  What we’re interested in is the polling chart.  It should look like this:

2010 National Congressional Ballot. Image: Pollster.com

So, how do you read this mess of dots and lines?  The dots are called a scatter plot and each one represents a poll taken on a certain day or period of days (usually 2-3).  The date of the poll is on the horizontal line and the percentage Republican (red) and Democratic (blue) is on the vertical line.  The squiggly lines in the middle are called a trend line and represents the mid-point of the dots for that day.  For those of you who may have taken statistics some time in the past, this is also called a linear regression. The colors represent the same political parties as the dots.  Correction: Dr. John Bogen, an Extreme Thinkover contributor, corrected my error in labeling the trend line as a “linear regression.”  Although the trend lines are based on regression formulas, I should have labeled it as Pollster.com calls it, a “trend estimate.”  For more info on Pollster’s statistical methods for trend estimates, click here.

Now take a moment to go to the Pollster.com site and look at their “live” chart.  Each dot will open a fly-by box telling you the pollster, date and the results.  Pretty nifty, huh.  There is also an expand box in the upper right hand corner if you want to open the chart to fill your screen.  The features still all work.

What, then, does this chart, as presented, convey?  Notice the date begins in November 2008, at the time of the Presidential election and covers the time since then.  Here the trend line is easier to read because you can see the ups and downs of the popularity of each of the two major parties over the past two years.

If this chart was the only one you looked at you would conclude that the Republicans have made huge gains beginning about May 2010 and now hold a 47.1% to 40.6% lead over the Democrats.  And you would be wrong.  Something is missing.  First of all what about the undecided voters?  Where are they?  How many of them are there? What is their trend?  For that answer, click here.

A new set of black dots with a trendline appears on the chart representing those voters who answered “undecided” on who they plan to vote for.  You can also see that as this year has progressed, the line has trended just slightly upward, and only since August have more of the people made up their minds.  As of today though, the undecideds are still 10.4% of the total, which is larger than the gap between the Republicans and the Democrats.  This is where it gets interesting.

I submit that in this gap is where the wild things are, to reframe the title of Maurice Sendak’s beloved children’s book.

Reading the gaps is where the information about the most dynamic trends in the electorate are.  Follow me on this.  Go back to the original chart.  You should already have the Red, Blue and Black trends open.  On the footer is a button titled “Tools.” Click on it and it will open another footer just above with six different choices on it.  Click on “Filter.”  This will open a small window with three check boxes: Live phone interviews, automated phone (i.e. robocall) interviews, and internet.  Place your cursor on each one and you will get a list of “filter options.”  Notice on the first option, Live phone interviews, there is an arrow in the top right hand corner.  This option has three pages and we’ll use them.

We want to narrow our polling data to the most relevant and the highest chance for honest answers.  To do that, based on my criteria (you are free to choose your own), I say let’s eliminate the internet surveys, first.  They are very hard to get a true random sample and very easy to lie on.  Next, let’s eliminate the robocalls, too.  Even though the calls go out to a random sample (supposedly) it is very easy to lie to a machine.  That leaves us with the live interviews.  These surveyors, you will notice are familiar big name pollsters, who have a reputation to uphold, and nearly all of them publish their survey questions and results online for free access for anyone interested in reading them (which would include geeks like me).   We want to cull some of these, still.  They are the pollsters for both political parties because there is a greater chance they will ask weighted questions that favor their side.

So, uncheck the internet, robocalls, and on the live call pages every pollster ID’d with either a R or a D.  Now we have a select set of pollsters who are as neutral as possible and use real people to talk to voters to decrease the chance for lying or misrepresentation.

One more thing.  We really only want to look at the results for the current primary season.  So again, click on tools and then on “Date Range.”  On the left date, click on the month and set it to “01”, the day, “01” and the year, “10” and then click on the blue “Set Range” button.

Look at your results on the chart now.  The polling results have changed.  The Republicans sit at 47.7%, the Democrats at 41.1% and the undecided at a whopping 17.1%!  This, I would suggest is a much clearer picture of the state of the electorate regarding the races in the House of Representatives.  By eliminating those polls that introduce bias into the big picture, either by the way they are administered, or by the way they are designed to benefit their candidates, we can see that the November election is far less certain than most pundits and politicians are leading us to believe.

The fact that apparently over 17% of the electorate is still vacillating about who they will vote for in the general election means the predicted gains by the Republicans has to be called into question, the predicted losses by the Democrats has to be called into question, and the outcomes across the country will very possibly be different than is now being predicted.  It may also mean that the gains or losses may be greater than predicted and one party or the other end up with a significant lop-sided outcome.

But one principle in polling must not be forgotten.  Each poll is a snap-shot in time and by itself can be either an accurate or inaccurate reflection of the voters’ will. It is also important to remember the truism that all politics is local and as Dr. Bogen also points out, the undecided percentage is likely to be smaller on the local scene.  He also rightly suggests this local phenomenon, all things being equal, favors the challenger.   This same principle applies to groups of polls as well because they are aggregates of local polls.  Political trending, although becoming more sophisticated all the time still cannot reliably predict the outcome on election day every time.  We have far to go to reach the algorithmic precision of Isaac Asimov’s Foundation “psychohistory.”  In the mean time we  have to search for the data where the wild things are.