Texas Hispanic Policy Foundation Internet Research Event Was Informative and Disappointing

 

Access to the Internet is an important topic these days, especially as it influences the education and general quality of life for families that have limited access to the Internet.  About 30 interested people, including myself, were eager to learn about the results of a recent study of Internet access among families in the Dallas Independent School District.  The research symposium, sponsored by the non-partisan Texas Hispanic Policy Foundation (TXHPF), was informative and revealed four key findings:

·        Hispanic families, especially the Spanish speaking segment, struggle with accessing broadband access to the Internet in their homes.

·        The primary reason for this lack of access is that broadband access was too expensive.

·        Also problematic is that families, especially Spanish speakers, lacked awareness of discounted packages to obtain broadband access in their homes.

·        Recommendations were made for broadband providers to launch Spanish-language campaigns to raise awareness of their discounted broadband plans among Hispanic families.

 

The research was conducted under the direction of Dr. Doug Jones, a recognized expert in Latin American studies who did an excellent job of presenting the study results.  A discussion of the study methodology was summarized by one slide at the end of the presentation.  Questions from the audience were then requested by TXHPF Director Jason Villalba.

Aside from interest in the research topic, research professionals such as myself are accustomed to asking questions about the study methodology. In the research courses that I teach, I usually advise students to resist embracing the results of any research study before understanding its methodology and the study limitations – especially in studies of multicultural audiences which often includes various sources of bias. Unfortunately, when it was my turn to ask some questions, Mr. Villalba abruptly interrupted me as I was beginning to introduce myself and praising their efforts by asking:  “Is there a question somewhere here?”  I was surprised by this interruption and offered to postpone my questions and allow other audience members to ask their questions.  Interestingly, Mr. Villalba did not interrupt the questions from other audience members.  I decided to raise my hand again to make a few observations and Mr. Villalba politely listened this time without interruption.

It was an especially disappointing experience to observe such a deliberate effort to mute criticism of such an important study. Indeed, I may have been the only recognized expert in the audience with decades of experience in conducting studies of Latino, Black and Asian audiences and continue to teach such related courses at area universities. Researchers ordinarily welcome feedback from research colleagues and often use the feedback to improve the design of research studies.  Following are some of the methodological concerns and observations that I was attempting to share about the study:

·        Reported levels of broadband access do not necessarily mean that families to not have access to the Internet.  Many obtain Internet access through their wireless devices alone.

·        Access to the Internet, whether through broadband or cell phones, is not the only or most important barrier.  Other research confirms that Hispanics experience navigational issues, anxieties and fears about sharing personal information on the Internet.  

·        It is not surprising that Hispanic response rates were strong to text-oriented efforts to complete a survey.  Pew Research has confirmed the value of cell phones in much of the research that they have conducted in past years.  Our research experience in Texas has shown a similar trend.

·        Telephone interviews, while involving more costs than other modes, is second only to personal interviews in being the most effective mode to obtain in-depth, reliable information from Hispanic and Black persons.  Responses to Census surveys confirms this trend.  Telephone interviews are not prohibitively expensive and lead to higher response rates when coupled with incentives and other modes.

·        Launching a Spanish-language campaign to increase broadband access among Hispanic families is premature and not supported by the research findings.  Hispanics who speak Spanish are not necessarily literate – that is, immigrants have limited education and often cannot read or write in Spanish despite having a speaking ability.  Consequently, different modes of advertisement may be needed.  Also, research did not address the issue of advertising effectiveness – that is, which aspects of the discounted programs offered by the broadband providers were considered undesirable by Hispanic families?

 

These are just a few of the observations that I made from the scant details provided about the study methodology at the event.  I would encourage Mr. Villalba, Director of the Texas Hispanic Policy Foundation, to be more transparent about the methodology of their sponsored research and to encourage, not discourage, questions about all aspects of these studies.  I would also encourage the public to not embrace the results of any research study until they understand the methodology that was used to collect the information --- especially when the target audience includes Blacks, Latinos or Asians.

The Texas Recipe for Muting the Hispanic Voice in Public Opinion Polls

If you are a tax-paying Texas resident, should your opinion matter in decisions related to publicly-funded programs or services in Texas?  Of course, you may say, the opinions of all Texas residents are important.  But one Texas state agency thinks that it is acceptable to exclude Spanish-speaking Hispanics from state-funded public opinion polls that are used to decide how tax dollars are spent.  I would like to share the details of an actual case study that vividly illustrates how one Texas agency is being allowed to silence the voice of Texas Hispanics in its public opinion polls.

The State of Texas has plans to spend billions of dollars to improve their transportation system, including the possibility of high speed rail. To ensure that the improved system meets the needs and expectations of Texas residents, the Texas Transportation Institute has the responsibility for conducting important surveys of Texans that reside in specific geographic areas, or corridors, that are likely to be impacted by these improvements.

A recently released report by the Texas Transportation Institute for the first of these two surveys, conducted during the Fall of 2012, provides concrete evidence that the voice of Texas Hispanics was muted by the survey planners.  Indeed, Hispanics represented only 20 percent of the survey respondents, despite their current representation of 38 percent in Texas (American Community Survey, 2011).  Even worse, only 19 percent of the few Hispanics included in the study were interviewed in Spanish – which compares poorly with other state surveys that have shown that 50 to 67 percent of Hispanics prefer a Spanish-language interview.

How could this occur, especially when the study design was reviewed by a “panel of experts” at the Institute?  A careful review of the study methodology reveals several missteps in the planning and execution of this survey:

• The 16,000 households selected as respondents received only an English-language version of the survey.

• The cover letter that was included with the English-language survey was provided only in English, and did not offer respondents any support to complete the survey in another language.

• A question that asked respondents to identify their race-ethnic background provided only one ethnic identifier for Spanish-speaking respondents -- “Hispanic” – which could partly explain the under-count of these respondents because other labels are often preferred over the “Hispanic” option.

• A call center was supposedly set up to receive incoming calls from survey respondents that had questions or needed Spanish-language support. But this call center probably received few calls from Spanish-speaking respondents since the cover letter did not provide the needed contact information.  Moreover, the report did not include a copy of the Spanish-language telephone survey that was supposedly used by the survey vendor's call center to capture incoming calls by the survey respondents.  

• The study design required that automated advanced calls (or “robo calls”) be made to the selected households prior to the survey mailing.  Automated calls are a recognized nuisance from telemarketers and political campaigns that often discourage response rates to legitimate public opinion polls.

• The report indicated that the survey “participation rate” was 34.6 percent – a rate that appears subjectively created and not recognizable by the American Association of Public Opinion Research (2011).  Instead, the overall survey response rate was more likely to be a much lower 9.7 percent (1,559 completions /16,000 invited participants) – not surprising given the recognized shortcomings in the methodology.

• While the vendor acknowledged that Hispanic respondents were significantly under-represented and non-Hispanic whites were over-represented, no explanation was provided about the potential causes or consequences of this imbalance.

• The fact that the survey planners ignored a previous warning about the potential flaws in the survey methodology suggests that the poor survey outcomes did not result from just simple carelessness.

         To make matters worse, the same survey vendor was awarded a second contract to conduct another public opinion poll of Texas residents using the same flawed methodology.  Why are state officials allowing such flawed practices to take place, especially at a time when the state’s population is being heavily impacted by the growing Hispanic population?

As one of the vendors that competed for both survey contracts, Rincón & Associates LLC monitored both competitions with some concern. In both competitions, state procurement staff decided to award the contract to the lowest bidder, which may not have been the brightest decision given the poor study outcomes.  As both studies required a mixed-mode survey methodology that few U.S. companies were capable of executing, more weight should have been given to proven experience using this specialized methodology with diverse communities in Texas. Procurement staff did not have to settle for the lowest bidder as other Texas vendors were ready, willing and able to conduct both studies.

Fortunately, an investigation was initiated on March 3, 2013 to find out why a state agency like the Texas Transportation Institute is allowed to deliberately design a study that minimizes the participation of Hispanic residents, especially Spanish speakers.  The outcome of this investigation is important because it could either (a) allow other state agencies to exclude Spanish-speaking Hispanics from state-funded studies, or (b) raise the standards of research for all state agencies to ensure that all state-funded studies provide adequate Spanish-language support.

The practical significance of this issue cannot be over-stated. Spanish-speaking residents are often the most likely to be overlooked in the delivery of public services, the most likely to receive the lowest quality services, and show distinctive attitudes or values that differ significantly from English-speakers.  Thus, excluding Spanish-speakers from opinion polls can lead to more positive satisfaction ratings than is actually the case, and result in erroneous public policy decisions.

It is time to require a higher standard for public opinion polling in communities that are linguistically and culturally diverse. Although professional research organizations have always defined quality and ethical standards for the research industry, it is apparent from the case just discussed that public agencies may not feel the need to follow these guidelines.  Following are a few ideas suggested from our experiences with public agencies like the Texas Transportation Institute:

• Research firms that compete for opinion polls in the public sector should be required to produce evidence that they have the staff, facilities and past experience to conduct polls in linguistically and culturally diverse communities. If a research firm does not produce a representative sample of such communities in a contracted study, they should not be rewarded with another contract that utilizes the same flawed methodology.

• The committee members convened by public agencies to evaluate research proposals may not have the expertise to judge these proposals in terms of their adequacy for diverse communities. The inclusion of experts with experience in conducting polls in diverse communities may have prevented the missteps in the Texas A & M studies.

• In the haste to award a contract to the lowest bidder, proposal evaluators do not regularly check the references provided by the different bidders, but oddly enough still find a way to rate the relevant experiences of the bidders without this information. Prior to contract award, an audit should be conducted to ensure that such references were verified for all of the vendors that submitted a bid in such competitions.

       It is unclear that the State of Texas got the “best value” by selecting the lowest bidder from outside of Texas. Indeed, what is the economic benefit to Texans when a contract is awarded to a non-Texas vendor whose payrolls, taxes and local spending for goods and services will only benefit another state? 

Legislators and advocacy organizations, especially those that represent the needs of Texas Latinos, should show their concern about public opinion polling practices that minimize or eliminate the voice of the constituents that they represent. Can we afford to remain silent on this issue? 

The more conservative members of the Texas community may believe that all public work should be conducted only in English, and that no special accommodations should be made to non-English speakers. Unless we are willing to also exempt non-English speakers from the payment of taxes as well, then I believe that they should be given the option of voicing their opinions on topics that impact their quality of life.  Although many Hispanics and Asian residents have proficiency in English and their native language, about 50 to 70 percent of these residents still prefer to express their opinions in their native language.  By providing the appropriate linguistic options, public opinion researchers are more likely to establish rapport, increase response rates, and obtain more valid responses to their questions from ethnic respondents – all desirable outcomes for high quality research.     

Texas public agencies, especially the Texas Transportation Institute, must be required to raise their standards when conducting opinion polls of Texas residents, and legislators must take a more assertive role to ensure this outcome. We cannot afford to bury our collective heads in the sand on this issue.