A Condensed Guide to Political Polling

Accurate polling is difficult. To prove this point, Gallup decided to stop election prediction polling late last year after being unable to accurately predict the presidential election in 2012. While polling has always been a tricky business, there are several factors that make it increasingly difficult to get accurate results, including low participation rates among younger people, declining use of landline telephones, increasing annoyance at receiving phone calls from pollsters on cell phones, and the fact that people may report one thing and then end up doing another, which is particularly problematic when trying to predict an election. With over three decades of political polling experience in Utah and the Intermountain West, Dan Jones & Associates has gained a reputation as a respected polling firm throughout the area. This article, based on our experience navigating the rapidly-changing political landscape, addresses five key areas that are essential to consider when conducting a political poll in the 21^st century. These principles are applicable to polling in other states, but our specific examples will include some idiosyncrasies specific to Utah.

Sampling

Determining who to talk to may be the single most important factor in creating an accurate poll. On the surface, the question seems obvious: when polling to predict an election or measure opinions about political issues, it’s important to talk to voters. But in states where voter turnout is low, not everyone who is registered to vote will vote. It is, therefore, important to talk to the voters who are most likely to fill out a ballot and mail it in or show up to the voting booth on election day.

Typically, voters most likely to vote have a strong, consistent voting history, so the sampling criteria could require participation in one or two of the past elections. However, we may not want to apply these criteria to people under age 20 since they are too young to have voted in the last two elections. In this case, sample criteria could include a combination of voters who have voted in the past two elections OR who have first registered to vote within the last two years. If it is necessary to get more granular, which may be important when polling for a primary election rather than a general election or for a municipal rather than a presidential election, pollsters may want to require participation in a past election of the particular type you are measuring. Understanding typical voter turnout for the specific election being measured is also important to take into account. Voter turnout is much higher in presidential years.

Other methods of determining who to survey include asking respondents about their interest level in the upcoming election, their intention to vote, their voting history, and whether or not they know where their voting location is. It is also important to make sure the demographic make-up of the sample is consistent with the population being targeted. For a general statewide election, the proportion of voters who identify with a particular political party in the survey should closely align with the percentage of voters in the state who are registered for that party. Specifically in Utah, the proportion of survey respondents identifying as members of the LDS Church, the dominant faith in the state, should align with the percentage of the population that identifies as members of that religion. The same is true of other states in which a predominant religion may affect political opinions or results. The proportion of survey respondents of certain ages should correspond to the general age breakdown in the state (unless data confidently suggests that a certain age group is less likely to vote in the particular election being measured).

Location Targeting

Accurate location targeting requires understanding where the voters whose opinions you want to understand live. For example, when predicting the most recent Salt Lake City mayor’s race, we knew we needed to talk to voters who were registered to vote in Salt Lake City. However, if we only targeted Salt Lake City voters and paid no attention to where in Salt Lake City the voters lived, we could have easily oversampled east bench residents and left out voters from various other parts of the city. It’s possible that oversampling east bench residents wouldn’t make a difference if they had the same opinions as residents who live downtown. But we didn’t know if they had the same opinions, and we couldn’t make that assumption and unintentionally jeopardize the accuracy of our poll. One of the best ways to determine which locations to target and by how much is to look at historical turnout by location and use previous turnout rates as benchmarks. The more granular the location targeting, the more accurate the results will be. For a statewide poll in Utah, we might be tempted to assume that since the largest proportion of Utah voters lives along the Wasatch Front, we only need to survey voters who live along the Wasatch Front. However, opinions can largely differ in the towns and cities in other areas of the state, and without accurate location targeting, important pieces of information could be missed.

Methodology

What is the best way to contact survey respondents? Traditionally, polls have been conducted via landline telephone, but we live in a society where more and more people are ditching landline telephones in favor of cell phone-only households. According to the CDC’s National Health Interview Survey released in December 2015, 41 percent of households in the U.S. have both landlines and cell phones, and 47 percent of households only use cell phones. In addition to the difficulty of calling cell phones, those of us who have cell phones don’t always answer the phone if we receive a call from a number we don’t recognize. This shift in communication and response habits makes it difficult to conduct an accurate poll completely by phone because we know that there is a large portion of the population that is much more difficult to reach.

An additional complication for political polling is that voters in Utah and many other states are no longer required to include a phone number when they register to vote, so even when we dial both landline and cell phone numbers, there is a portion of voters whom we can’t contact. To get a truly random sample in which everyone has an equal opportunity of being selected to participate, other methods of contact are required. Each methodology has its own limitations, but when several methodologies are used together, the sample is more accurate.

Other methodologies that we can utilize for political polling include the following:

1. Mail paper survey
2. Mail postcard with survey URL
3. Email
4. Live intercepts

Other methodologies exist, such as social media targeting and banner ads, but for political polling, we have found the above methods to be most effective. Mail surveys can be used to reach additional respondents who may not be reachable by phone. Mail surveys can include a paper survey that can be returned and recorded or they can direct respondents to a URL where respondents can participate in the survey online. The primary limitation with mail surveys is the lack of control over who in the household fills out the survey. Additionally, if the survey directs respondents to a URL, it excludes participation from people who do not have access to internet. However, Utah has one of the highest rates of Internet connectivity in the nation, so this concern is not major. For political polls in particular, it is important that surveys are filled out voluntarily rather than incented. While as pollsters we want as high a response rate as possible, we want responses from those who are most interested and most likely to vote rather than those who participate to receive an incentive but who have no intentions of voting.

Email surveys are limited in their abilities to reach a representative sample. Since directories of email addresses do not exist, we are limited to obtaining access to email respondents through opt-in panels. Opt-in panels consist of a community of users who have agreed to take surveys and receive some kind of pre-determined incentive for each survey they complete. Opt-in panels allow us to reach people we may not have otherwise reached, and they can specifically target users who are registered to vote and live in a certain area. However, it is important to combine email sample with other forms of sample when attempting to get a representative, statewide sample. Live intercepts allow us to obtain opinions from people who may be out and about and won’t answer their phones but will stop and take a brief survey while waiting in line at a mall or coming out of a store. Location targeting is difficult with live intercepts, but screener questions that filter out respondents who don’t meet specific criteria can be used to mitigate that issue.

Ongoing Monitoring

Once the survey has been set up and the targeting has been defined, the hard part begins. Public opinion can change rapidly, particularly during high-profile elections. Name recognition increases over time, advertisements from candidates can effectively sway voters and inform and motivate voters who were previously disengaged in the political process. It is essential to recognize the difference between a shift in opinion and an incremental change in the data that could simply be due to sampling error. While never easy, one of the best ways to determine whether a difference in results is due to error or is a developing trend is to track results each day. Tracking requires more resources and the ability to adjust methodologies or targets at a moment’s notice to ensure you’re polling the right population using the right methods, but it provides real-time insights into the upcoming election that are hard to get by any other way.

When to End Data Collection

Knowing how much data is enough and when to stop data collection is one of the most nuanced aspects of political polling. The most accurate poll results come from exit polling, where interviewers randomly survey voters as they exit the voting booth and ask who they voted for. However, clients often need to know how an election is shaping up well in advance of election day, and the increasing use of vote-by-mail ballots means exit polling may soon become a thing of the past. If polling is ended too early, last-minute efforts by political candidates may have a significant impact that is not accounted for in the polling results. If polling continues right up until the election, the results are likely to be more accurate but at the expense of not getting results to clients in time for them to be useful.

Ultimately, deciding when to stop polling is largely determined by the timeline needs of clients. Regardless of the timeline, it is important to ensure that enough data has been collected to have a reasonable, predetermined sampling error. It is also important for both clients and pollsters to remember that poll results only reflect the attitudes and opinions of the population they represent for the time frame during which the poll was conducted.

There are many other aspects of polling that need to be taken into consideration to ensure accurate results, but these represent the most crucial pieces. Polling is complicated, but if a survey is set up correctly from the beginning and closely monitored and adjusted as necessary throughout the data collection process, it is more likely to contain accurate results.