During fall 2021, D3 supported the research of Cadet Luke Stoner with the United States Military Academy at West Point by conducting a comparative assessment of face-to-face and Twitter-scraping studies in Kenya.
Nationally representative face-to-face studies represent the gold standard of public opinion research methods, but they require safe and stable ground conditions and can take several weeks to implement properly. These luxuries often do not reflect the reality in most environments where the U.S. Military is interested in conducting research for strategic, operational, and tactical planning. Consequently, Cadet Stoner set out to evaluate Twitter scraping—a process comprising the collection of tweets and analysis of underlying sentiments—as an alternative to face-to-face public opinion surveys in an environment where the U.S. Military operates: Kenya.
The study compared results from a nationally representative face-to-face poll of 1,061 Kenyan adults ages 18 and older with sentiments scraped and quantified from nearly six million tweets sent from Kenya. For the national sample, D3 and its Kenyan partner, Infinite Insight of Nairobi, designed a multi-stage probability-based study and implemented it using electronic data capture on Research Control Solutions (RCS) software. All respondents were randomly selected using a random number generator. The questionnaire included sections on technology and media use; general outlooks on the country and economy; foreign affairs; COVID-19; national security; and politics and elections.
Concurrently, D3 analysts and Cadet Stoner collected tweets from across Kenya using the RStudio package “rtweet”. Cadet Stoner employed Python’s Natural Language Toolkit (NLTK) to analyze the final sample of usable tweets using a Valence Aware Dictionary and sEntiment Reasoner (VADER) to calculate positive or negative sentiment scores for unique tweets on the same topics as those included in the face-to-face survey.
Cadet Stoner compared the face-to-face and Twitter scraping data and observed the following:
- Overall, tweets reflected positive sentiments.
- There were no statistically significant correlations between survey results and Twitter sentiments.
- There were statistically significant correlations between the volume of tweets and positive sentiments only for certain topics, namely great power competition and presidential elections.
Given that over half of Kenyan Twitter users are 18 to 29 years old, Cadet Stoner also tested for statistically significant correlations between face-to-face data from respondents in this age bracket with the full sample of Twitter scraping data. However, the results were largely the same, as there were no statistically significant correlations between sentiments expressed in the face-to-face survey for the stated age bracket and collected tweets.
The study indicates that Twitter scraping does not presently offer a reliable substitute for face-to-face research, primarily because:
- Compiling a usable sample of even 1,000 tweets is a lengthy process that requires scraping, cleaning, sorting, and analyzing millions of tweets.
- Quantifying sentiment in tweets is difficult, even with the aid of Natural Language Processing (NLP) programs.
- There do not appear to be statistically significant correlations between the sentiments in face-to-face and Twitter scraping data, suggesting the latter does not accurately represent the sentiments of a wider population.
Cadet Stoner presented his research at an invited session titled “Mathematics in Security and Defense” on March 19th at an American Mathematical Society Conference, with a summary provided below.
After graduation from West Point, Cadet Stoner will pursue a Master of Science in Data Science at Harvard University, further pursuing his interest in research for the U.S. Military. D3 congratulates Cadet Stoner on this impressive accomplishment, joining the alma mater of our own Founder & CEO, Dr. David Jodice. This collaborative research project yielded intriguing findings and has served as a basis for future collaboration between D3 and the United States Military Academy at West Point.
D3 is proud to announce that we will support the research of two West Point cadets in 2022, and we look forward to the exciting opportunities this partnership will manifest.