What 18,000 Declassified Documents (and a Computer) Reveal About the Credibility of Signals During Crises

By Eric Min of Stanford University

For the last couple decades, international relations scholars have fervently debated the credibility of signals during crises. Thomas Schelling’s work, followed by James Fearon’s audience cost theory and its offshoots, has popularized the belief that public signals are more credible because they implicate higher costs to take back. More specifically, costs from hand-tying through public words are greater than those from cost-sinking through public deeds. Private words are cheap. Some recent research has countered these claims by arguing that private signals can be just as costly in reputation as their public counterparts.

Two forms of path dependency have shaped and limited the contours of this discussion. First, likely due to Schelling’s enormous influence, most studies have focused on costs as the main and potentially only source of credibility. Second, without systematic data on diplomatic signals or how elites interpret them, scholars have used formal models and historical case studies to bolster their points. These methods have been valuable but also allow for different forms of cherry-picking.

At the 2017 MPSA conference, my co-author Azusa Katagiri and I presented a paper entitled “The Credibility of Public and Private Signals: A Document-Based Approach,” which offers two corresponding innovations to this debate. First, we present a novel rationale for why words exchanged behind closed doors may be more informative, and thus credible, than words exchanged in public. Second, we use text analysis and machine learning methods to create a comprehensive set of data on private statements, public statements, public actions, and White House evaluations of the Soviet Union during the Berlin Crisis of 1958 to 1963. These data permit the first quantitative study of diplomatic signaling and its effects.

The core of our theoretical argument is based on information processing/perception and relies on a more realistic understanding of the diplomatic environment, where policymakers are overloaded with tasks to perform and information to process. It is also intuitive. (Our argument’s framing has changed since we presented in 2017, so please check the revised paper.) So many statements are made in public that governmental actors—whether low-ranking desk officers or Cabinet members—cannot reliably keep up with or process them. Moreover, even though public statements are designed for and directed toward specific audiences, nobody can prevent other actors from noticing and (mis)interpreting those words. Public diplomatic statements are very noisy. In contrast, private diplomatic statements are infrequent, precise, and directed at one audience, all of which decreases their likelihood of being ignored or misperceived. This gives them important informational value and credibility.

We test these claims using over 18,000 de-classified documents from the Berlin Crisis of 1958 to 1963, which we photographed at the National Archives and the Eisenhower and Kennedy Presidential Libraries. This is a full collection of cables and memos during one of the most serious crises in American history. The documents come from three sources, which align with our concepts of interest: the Department of State (private statements from the Soviet Union), the Foreign Broadcast Information Service (public statements from the Soviet Union), and the White House (American elites’ assessments of Soviet intentions).

To convert these documents into quantitative data, we use a series of text analysis and supervised learning methods to make weekly-level measures of private statements of Soviet resolve (ostensibly costless), public statements of Soviet resolve (hand-tying), and White House perceptions of Soviet resolve. We also use The New York Times to manually create a measure of costly public actions by the Soviet Union (cost-sinking).

Statistical analysis points to three results:

• First, costly actions have far greater impacts on perceptions of resolve than either public or private statements.
• Second, private statements are indeed less frequent and more focused compared to public statements.
• Third, and consequently, private statements of resolve are more credible and affect White House perceptions of Soviet resolve than public statements.

These findings completely rearrange the hierarchy of signal credibility. Most research on costly signals would put them in this order of credibility:

Public statements > Public actions > Private statements

However, according to the information processing perspective, as well as our data, we get this:

Public actions > Private statements > Public statements

Our research has implications that are relevant to the current diplomatic situation with North Korea. President Trump’s bellicose tweets throughout 2017 and early 2018 were indeed unprecedented and increased the risk of grave unintended consequences, but fears about being committed to conflict were overblown. Not only have President Trump’s erratic and inconsistent statements already undermined his credibility, but his tweets were only one of multiple streams of public statements from his administration that were more measured or sought to engage in diplomacy. (For example, consider positions taken by Secretary Mattis or former Secretary of State Tillerson.) Conversely, claims that President Trump’s tweets recently intimidated North Korea into coming back to the table are also debatable. U.S. troop deployments and joint military exercises with South Korea have arguably been more important to American credibility.

Private diplomacy is a vital conduit through which to make clear and direct statements to one’s adversary, stripped away of incentives to perform for an outside audience. Unlike what President Trump has said about diplomacy with North Korea in the past, it is not “a waste of time.” Mike Pompeo’s secret trip to North Korea, while clearly unorthodox since he was then director of the CIA, underscores this point. But it is also for this reason that direct talks between Trump and Kim may be unwise. The higher up the ladder talks start (before lower-level officials already work out issues behind closed doors), the less likely they are to be truly private, and the more likely it becomes for both individuals to engage in public posturing instead of substantive discussions.

Our paper injects a new theoretical and empirical perspective to the study of signaling. Not only do we highlight the importance of private diplomatic activity, but we demonstrate how contemporary computational methods can be harnessed to explore the dynamics of crisis diplomacy. We look forward to extending this line of research in the future.

[youtube https://www.youtube.com/watch?v=0TGuhZF3YpU?rel=0&controls=0]

About the Authors: Azusa Katagiri is an Assistant Professor of Public Policy and Global Affairs at Nanyang Technological University in Singapore. His other research focuses on the micro-foundations of state behavior, and bureaucratic decision-making in conflict, and US-Japan relations. He can be reached atazusak@ntu.edu.sg.

Eric Min is a postdoctoral fellow at the Center for International Security and Cooperation at Stanford University and will be an Assistant Professor of Political Science at the University of California, Los Angeles starting in the fall of 2018. His broader research applies new data and methods to analyze the strategic interaction of fighting and negotiating during war. He can be reached at ericmin@stanford.edu and is also on Twitter.

Their research “The Effectiveness of Public and Private Signals: A Document-Based Approach“, presented at the 2017 MPSA conference, received the Robert H. Durr Award which recognizes the best paper applying quantitative methods to a substantive problem.