[null, {"document_url": "https://link.springer.com/article/10.1007/s10796-015-9599-5", "unformatted_src": "https://media.springernature.com/w306/springer-static/cover/journal/10796/19/2.jpg", "src": "https://media.springernature.com/w306/springer-static/cover/journal/10796/19/2.jpg"}, null]


Information Systems Frontiers 
April 2017, Volume 19, Issue 2, pp 377–396 | Cite as

Leader’s dilemma game: An experimental design for cyber insider threat research



One of the problems with insider threat research is the lack of a complete 360° view of an insider threat dataset due to inadequate experimental design. This has prevented us from modeling a computational system to protect against insider threat situations. This paper provides a contemporary methodological approach for using online games to simulate insider betrayal for predictive behavioral research. The Leader’s Dilemma Game simulates an insider betrayal scenario for analyzing organizational trust relationships, providing an opportunity to examine the trustworthiness of focal individuals, as measured by humans as sensors engaging in computer-mediated communication. This experimental design provides a window into trustworthiness attribution that can generate a rigorous and relevant behavioral dataset, and contributes to building a cyber laboratory that advances future insider threat study.


Insider threats Trusted human computer interactions Sociotechnical systems Online game simulation Experimental design



The first author wishes to thank National Science Foundation for the support of Secure and Trustworthy Cyberspace EAGER award #1347113 09/01/13-08/31/15, Florida Center for Cybersecurity award #2108-1072-00-O 03/01/15-02/28/16, and Conrad Metcalfe for his editing assistance.

Appendix A: Instrument items for participant observers


Research daily survey (misleading) questionnaire for the actor team-leader


Research daily survey questionnaire for the team players

These scales were adapted from Mayer and Davis (1999, 1995)

Appendix B: Demonstration of the efficacy of experimental design

We conducted a game simulation based on the research assumptions and considerations described in this manuscript to validate this experimental design of the “Leader’s Dilemma” Game. Discussion of our findings follows. Both Team-Leaders (from teams Crocodile and Dragon) who were presented with bait actually did betray their teams. The reliability of observers’ ratings to validate the experimental design is discussed in the B.4. Accuracy of the Measurement. The comparative content analysis was conducted for four different cases (2 control groups and 2 treatment groups) over 5-day archived data (including chats, blogs, and emails). We included some end-of-game qualitative survey results and face-to-face interviews.

Predictive results with qualitative attribution

According to the “Leader’s Dilemma” Game design principle, some information was made public and some was kept private. The Team-Leader in the game was empowered to be the sole person to contact the Game-Master, and was the only person allowed to submit the team’s answers at the end of each session. For example, according to the experimental design, Team-Leader Crocodile was treated and influenced with bait by the Game-Master. The bait was given in the form of 200 MerryBux (a micro-payment system). When the bait was presented to the Team-Leader Crocodile, Team-Leader Crocodile quietly decided to betray his team. This was reflected in his decision to delay the submission of the team answers. The conversation between the Team-Leader Crocodile and the Game-Master - which disclosed his or her intention to fail their team—is displayed in Table 4.


Crocodile team-leader and game-master private chats on day 5

Team-Leader Dragon was also treated with bait from the Game-Master. While the Team-Leader Dragon was the only one to submit answers on behalf of her team, her team players noticed and confronted the actor because the answers were not submitted properly (Table 5).


Dragon team private chat on day 4

In discussions between Team-Leader Dragon and the Game-Master, the Team-Leader seemed to pretend that he did not understand what the Game-Master was saying, and asked the Game-Master repeatedly about the rules around distribution of the 200 MerryBux. This Team-Leader also seemed to test the attitude of the Game-Master to determine the response if he were to distribute the bait to the team—just to be on the safe side (Table 6).


Dragon team-leader and game-master private chat on day 4

Predictive results with quantitative attribution

The design of the “Leader’s Dilemma Game” being situated within a small virtual group setting does not permit a large quantity of data to be collected about the actor’s perceived trustworthiness. Still, observers situated within a simulated insider threat scenario were able to evaluate the threat situation and assess the actor’s trustworthiness (Fig. 5) based on limited interaction with the actor, and with each other. In the games, Team-Leaders for Crocodile and Dragon both decided to take the bait and betray their team. The ensuing insider threat behaviors were apparent to the observers. Figure 5 denotes the trustworthiness of the four focal actors as attributed by their group members in four virtual teams, which was plotted on a 5-point scale. These team players neither had prior knowledge of how the game was designed, nor the dilemma that the Team-Leader faced.

Due to the nature of this experimental design, small virtual group interactions (rather than large-scale asynchronous game competitions) were tracked (Fig. 4). The resulting aggregate sample size does not allow for extensive statistical analyses, but patterns do emerge. The present experimental design allows for interactive behavioral observations across time in a longitudinal setting. This experiment method is in contrast with the method of collecting self-reported cognitions or indications of behavioral intention (e.g., survey research) (Warkentin et al. 2012). Furthermore, this experimental design allows researchers to capture and analyze two distinct behaviors; (1) the actual orthogonal trust violation itself as the behind-the-scenes truth when described by the perpetrator (e.g., Team-Leader) in the archived online dialogue, as well as (2) the third-party attribution of the trustworthiness of the perpetrator by team members. This methodological pluralism provides added rigor to the findings (Venkatesh et al. 2013). In addition, rich qualitative data (e.g., chat, blog, email, etc.) was obtained from this online game simulation, which created behavioral observation opportunities to obtain insights about the operations of virtual teams, and how focal actors make decisions during an ethical dilemma.

The survey results, illustrated in Fig. 5, indicate that the integrity in the dimension of justice value for the Team-Leader Buffalo (not influenced; group sensitivity reduced) exceeds – comparatively – that of any other team. The integrity of the Team-Leaders Crocodile and Dragon (who influenced, and betrayed their teams) in the dimension of justice was significantly lower. After removing the mole’s input, the rating for Team-Leader Dragon (influence; group sensitivity enhanced) was found to be lower than any other team. This infers that these observers were able to make their own independent judgments about the actor’s trustworthiness. After averaging the observations, the group believed that Team-Leader Dragon had low integrity. Data analysts interpreted that both Team-Leaders Alligator and Dragon did not communicate their values well to the members of Teams Alligator and Dragon. This finding has a significant implication in that the attribution of a person’s trustworthiness can be used as an indicator for whether this person is likely to betray his or her organization.

Privacy and general settings in the online game environment

Due to the sensitive nature of insider threats, and the nature of manipulation of human psychology, researchers need to be careful in the manipulation of human subjects. In our online game approach, team players found the experiment of the group dynamics to be very interesting, especially regarding how the players started to question the actor’s (i.e., Team-Leader’s) trustworthiness. Players also like the virtual way of communication that:

“People quickly gravitate toward their natural roles and the way the alliances are formed. Everyone wants a feeling of community, even in this very temporary virtual world. People want to fit in.”

“Privacy is a good aspect of this type of game,” as one player reported. “The room was well set up and no one knows who is who,” another player answered. Because this experiment is in temporary virtual setting, and because all players are given pseudonyms, no social responsibility is really ingrained outside of the 5-day games, thereby avoiding social desirability bias and ensuring that true personality behaviors are exhibited. As one player reported,

“I was also most interested in the fact that I formed an alliance at all. I had determined to be distant and anonymous for this game, but my natural talkativeness and willingness to try to answer questions took over. Immediately I noticed that [Ricky] and I answered similarly and that we had the most in common. I wasn’t planning to talk, but my wanting to take some kind of leadership role when there was a gap to be filled took over. I fell into my natural (my usual, however they were constructed) patterns of behaviors. Even though I planned to make the most out of having a fake ID, my personality took over and I was the same person I really am—I didn’t act like I was a different person; I started acting like myself. That was a very strong drive for me.”

Accuracy of the measurement

Measurement of the experiment includes two categories: (1) the accuracy of the participants’ views (accuracy) is illustrated in Fig. 6, and (2) judgments about the participants’ performance (outcome instrumentality) are illustrated in Fig. 7. Generally, each line in the graphs represents a group’s average view toward each category. Four lines represent four virtual teams. When lines are closer to outer circles, it means that the group’s views about the above four categories are higher (on 5-point Likert scale), and vice versa. If four lines in a graph are close to one another, it means that players’ views among four teams are close to one another.

Figure 6 illustrates that the participants’ judgment about the accuracy of the instrument was pretty close among each other for all four virtual teams. When the outlier data was removed (shown on the right hand side of the Fig. 6), the consistent pattern regarding the instrument’s accuracy, among team players in all four cases, can be found.

Figure 7 illustrates the Alligator team players’ own judgment about their performance was higher than all other three team players (outcome instrumentality). When the outlier data was removed (showed on the right hand side of Fig. 7), a consistent pattern regarding team players’ performance as outcome instrumentality, among team players in all four cases, was found.

An Interface design utilizing Google+ HangOut

Figure 8 illustrates an interface design of the “Leader’s Dilemma Game” utilizing Google+ Hangout. Participants’ role assignments are indicated in the lower right hand corner. It is important to note that participants’ privacy and data confidentiality are addressed in this instrument artifact by having their real identities replaced with pseudo identities. Participants’ group-oriented task assignments are given in the upper right hand corner. The human-to-human and human-to-computer interactions are captured in the chat boxes. Team Leaders’ private chat with the Game-Master is separated from the team’s group chat.







Ho, S.M. & Warkentin, M. Inf Syst Front (2017) 19: 377. https://doi.org/10.1007/s10796-015-9599-5



