extfieldsofstudy:
Computer Science
Physics
Mathematics

text:
Known by the company we keep: `Triadic influence' as a proxy for compatibility in social relationships

Networks of social interactions are the substrate upon which civilizations are built. Often, we create new bonds with people that we like or feel that our relationships are damaged through the intervention of third parties. Despite their importance and the huge impact that these processes have in our lives, quantitative scientific understanding of them is still in its infancy, mainly due to the difficulty of collecting large datasets of social networks including individual attributes. In this work, we present a thorough study of real social networks of 13 schools, with more than 3,000 students and 60,000 declared positive and negative relations, including tests for personal traits of all the students. We introduce a metric -- the `triadic influence' -- that measures the influence of nearest-neighbors in the relationships of their contacts. We use neural networks to predict the relationships and to extract the probability that two students are friends or enemies depending on their personal attributes or the triadic influence. We alternatively use a high-dimensional embedding of the network structure to also predict the relationships. Remarkably, the triadic influence (a simple one-dimensional metric) achieves the highest accuracy at predicting the relationship between two students. We postulate that the probabilities extracted from the neural networks -- functions of the triadic influence and the personalities of the students -- control the evolution of real social networks, opening a new avenue for the quantitative study of these systems.

Networks of social interactions are the substrate upon which civilizations are built. Often, we create new bonds with people that we like or feel that our relationships are damaged through the intervention of third parties. Despite their importance and the huge impact that these processes have in our lives, quantitative scientific understanding of them is still in its infancy, mainly due to the difficulty of collecting large datasets of social networks including individual attributes. In this work, we present a thorough study of real social networks of 13 schools, with more than 3,000 students and 60,000 declared positive and negative relations, including tests for personal traits of all the students. We introduce a metric-the 'triadic influence'-that measures the influence of nearest-neighbors in the relationships of their contacts. We use neural networks to predict the relationships and to extract the probability that two students are friends or enemies depending on their personal attributes or the triadic influence. We alternatively use a high-dimensional embedding of the network structure to also predict the relationships. Remarkably, the triadic influence (a simple one-dimensional metric) achieves the highest accuracy at predicting the relationship between two students. We postulate that the probabilities extracted from the neural networks-functions of the triadic influence and the personalities of the students-control the evolution of real social networks, opening a new avenue for the quantitative study of these systems.
Positive relationships help individuals thrive in society, whereas negative ones can jeopardize our chances of success and happiness. Social relationships arise from interactions between individuals and have been studied on different time scales and contexts [1,2]. As a result, social networks are formed, with individuals as nodes and interactions as links [3], and they can be studied and characterized using a complex network approach [4] in order to assess the many implications of social structure in our lives [5]. A great deal of research has been carried out on social networks by aggregating the interactions that occur over a certain period of time to define links, starting from the pioneering work of Moreno [6]. However, such an approach does not capture the dynamics of relationships, which is necessary to advance our understanding of the field [7]. Large efforts have been devoted to this question in recent years, mainly using empirical data with different degrees of time resolution, such as, e.g., letter exchanges [8], mobile phone communications [9,10], spatial mobility [11], or face-to-face interactions [12][13][14] (see also Ref. [15] for a review). All these analyses have led to many interesting insights on the evolution of relationships, but the issue of the mechanisms that explain how/why these relationships are created and evolve remains elusive.
Several models have been proposed to explain differ-ent aspects of the empirical observations. The first attempts were devoted to reproduce some of the structural properties observed in social networks, such as the small world phenomena [16] or the rich-get-richer effect [17,18]. Starnini et al. [19] proposed a simple model based on random walks and individual attractiveness to describe face-to-face interactions. For social networks, Jin et al. [20] studied networks with exponential decay of tie strengths to represent friendships. Other approaches have resorted to exponential random graph models [21] or stochastic actor-oriented models [22]. Finally, regression models that incorporate a selection of individual traits have also been considered for online social networks [23]. Still, none of these approaches sheds light on friendship formation in real life, taking into account the characteristics of the individuals and how some relationships can influence others.
In this paper, we contribute towards the understanding of friendship formation by adopting a different point of view, namely that of link prediction in networks [24]. The problem of link prediction, as originally formulated, is about temporal networks: given the graph of connections between certain entities or nodes during some interval, the task is to predict the set of links in a later interval. Notwithstanding this definition, the same idea applies to many different situations, such as recommen-dation systems [25], bioinformatics [26], scientific collaboration networks [27], criminal networks [28], or even estimating the reliability of network data [29], to name a few. In the case of online social networks, link prediction has been considered, for example, by Song et al. [30] or Hao [31] (see Ref. [32] for a review). Much less has been explored regarding real-world social networks, in particular friendship networks [33], due to the difficulty of collecting data on reasonably complete social networks that include personal attributes in real settings. For this reason, the discussion has been devoted in many cases to ego-networks (i.e., data on disconnected individuals who mentioned their friends) and to the meaning of friendship [34].
In this work we study social networks collected in 13 complete high schools in Spain, containing more than 3, 000 individuals and 60, 000 declared relationships between them. All students completed tests including information about their self-declared gender, cognitive results, and other variables that measured their selfishness/prosociality. Performing link prediction on this data, we are able to extract the probability that two students will be friends/enemies depending on their personalities. We also studied how this probability is affected by other relationships, defining a metric that we have termed triadic influence. Although we analyze static networks, our results suggest that the probabilities that we extract determine the mechanisms that control the initial formation of relationships and the evolution of the whole social network.

RESULTS
Data collection was carried out in 13 schools in different areas of Spain, with a total of 3, 395 students. They were asked to choose with whom they were related within their school by picking names from a school list. Then they had to rate the relationship as very bad, bad, good, or very good, which we codified as −2, −1, +1, and +2, respectively. We recovered 60, 566 declared relationship, see Supporting Information (SI) for more details. In addition, we also collected data on the students' gender (selfreported), cognitive skills (measured by the cognitive reflection test, CRT), and their prosociality (see Methods for details on these individual features). With this information, we build a directed weighted network, with each link representing a relationship that goes from the nominator to the nominee-two nodes can be connected by links in both directions-weighted by the reported rating. Additionally, each node represents one student and has his/her individual attributes (gender, CRT and prosociality). Figure 1 presents a sketch of the kind of social network that we will study. We have included several figures studying the structure of the social networks in the Supporting Information, see figures S1 to S3.
In this work, we study the correlations between the personal features of both students and the type of rela- FIG. 1. Diagram of a social network that includes personality traits and computation of the triadic influence. To predict the relationship from node 0 to node 1 we can use the individual features of both students (represented by the sliders within their body) and / or the triadic influence I01. The directions of these relationships are marked by arrows going from the nominator to the nominee, whereas the weight/intensity is represented with colors and edge labels (dark green: close friend, green: friend, yellow: dislike, orange: enemy). The thick arrows highlight the relationships that enter the calculation of I01. To compute I01 we select all directed paths of length 2 from node 0 to node 1 (0 → node → 1). In this example, they are 0-5-1 and 0-6-1. The path 0-3-1 is not a directed path (the direction of the edges is 0 → 3 ← 1) and therefore is not included in the calculation of I01. Thus, tionship between them, as well as the influence of other students on that relationship. We have used artificial neural networks to perform link prediction within our dataset from two complementary viewpoints: the first one focuses on the local structure, using the personality traits of both students and the influence of the nearestneighbors as described in the next section; the second one uses only the structural information of the network -the undirected and unweighted graph -to predict relationships. In what follows, we discuss these two approaches separately.
Predicting with the personality traits and the influence of the nearest-neighbors Figure 1 shows a sketch of the social network with all the information available to perform link prediction. It shows the students (nodes) with their traits (sliders) and relationships of different types between them. In this section, we use only local properties of the network to predict the relationship between two students, namely the individual features of both students (e.g. nodes 0 and 1 in Fig. 1) and the directed weighted paths of length 2 between them. Specifically, we define a variable that we term triadic influence as where w ik is the weight of the link that goes from node i to node k (see Fig. 1 for an example). The triadic influence condenses into one scalar the influence of third parties; e.g., if node i declares node k as a friend and k does the same with j, it adds a positive number to I ij (your friend's friends are likely to be your friends), whereas a path containing links of opposite sign will lead to a negative contribution (your enemy's friends or your friend's enemies are likely to be your enemies). I ij adds up the contribution from all directed paths of length 2 between i and j. For simplicity, we will train a neural network to correctly classify all relationships in the network into two classes: friends and enemies (see Methods for more details). We used different combinations of the triadic influence and the individual characteristics of the students as input for the deep neural network (NN) and trained it to output the correct value for each relationship in the training dataset (see Methods for a full description of the neural network and the training process). With our procedure, we obtain the probability that two students relate through a relationship belonging to one of the two classes (friends or enemies) as a function of the corresponding inputs. To avoid using a misleading metric of performance, since our classes are unbalanced-there are many more declared friends than enemies-we assess the performance of our method using the balanced accuracy on the test dataset. To compute it, after training the NN we feed it with all relations in the test dataset and assign the label "friend" or "enemy" to the class with the highest probability. The balanced accuracy is then computed as bAcc = 1 2 where N C α is the number of samples belonging to class α (+ friend, or − enemy) that were correctly classified from the total number of samples belonging to that class (N T α ). This is more informative than other performance metrics because if either the NN classified everything in the same class or guessed at random, we would obtain bAcc = 1/2 regardless of the number of samples in each class, whereas if all relations were correctly predicted, then bAcc = 1 (see Methods). Figure 2 collects the accuracies achieved using the NN to predict the relationships between students with different combinations of predictors. We first study relationships i → j with non-zero triadic influence (i.e., with at least one directed path of length 2 from i to j; see Fig. 1 and Methods for more details). The results are shown in the four upper bars of Fig. 2 (see the SI for the distribution of relationships per number of directed paths of length 2, Fig. S2). We train the classifier using four sets of predictors: (1) triadic influence and personal information (gender, CRT, and prosociality) of the pair of nodes, (2) triadic influence, (3) personal information, and (4) only students' prosociality. Just as a clarification, in case (1) we use as input for the NN the triadic influence Pairs of nodes not connected by directed paths of length 2.
(2) Triadic influence (1) triadic influence and personal information (gender, CRT, and prosociality), (2) triadic influence alone, (3) personal information alone, (4) just students' prosociality. In all four cases we trained 10 different NN with random initializations and show here the mean bAcc. Yellow bars correspond to the bAcc for relationships that have no directed paths of length 2. In this case, we use just two sets of predictors: (5) personal information and (6) students' prosociality. These cases use 10-fold cross-validation to estimate the performance of the prediction. Error bars represent the standard error of the mean in all cases.
(a scalar) and the individual traits of both students (a 6-dimensional array) to predict the correct label of that relation (friend or enemy). See Methods for a detailed explanation about how the value of the considered features: gender, CRT, and prosociality, are gathered and computed. The highest balanced accuracy, 86%, is achieved using the triadic influence as input, either in combination with personal information of both students (1) or alone (2). It is remarkable that such a high accuracy for the prediction of the nature of a relationship (friend/enemy) can be obtained with just a scalar (the triadic influence), and that a 6-dimensional array containing information about both students' characteristics does not improve on that. This suggests that the information on the individual features is already encoded in the network of relationships and captured by the triadic influence. The rationale for this is the following: if individuals i and j are both very prosocial, then i will be friends with j, as well as with many other intermediate individuals, with high probability; and these intermediate individuals will very likely be friends with j, since j is also very prosocial. Hence, I ij already encodes information about the prosociality of i and j. This suggests that I ij may act as a proxy for personal compatibility when individual traits are not available.
On the other hand, using only the personal traits of both students (3) yields bAcc = 60%. We studied the three attributes (gender, CRT, and prosociality) separately, and prosociality turned out to be the most predictive. In fact, using only students' prosociality to predict their relationship (4) already yields bAcc = 57%. Both figures are above the accuracy that a random guess would reach (50%). Note that prosociality is calculated with students' answers to three simple questions (see Methods). It is really remarkable that such a simple metric is already predictive for the nature of the social relationship between two individuals.
Finally, we study separately the relationships that do not have directed paths of length 2 connecting i to j (i.e. A 2 ij = 0, with A ij the adjacency matrix of the network); therefore, there is no triadic influence between i and j. These results are shown by the two bottom bars of Fig. 2. Since this dataset is much smaller (2% of all relationships, i.e. 1, 211 out of a total of 60, 566; see the SI Fig. S2 for more details), we assess the performance of the classifier using 10-fold cross-validation to ensure that our results are robust. We study two sets of predictors: (5) the complete personal information of the students (gender, CRT and prosociality) and (6) just the prosociality. The mean bAcc for the 10 realizations within 10-fold cross-validation is 57% for (5) and 55% for (6). Note that the mean bAcc seems to decrease compared to the case when A 2 ij > 0 (purple bars), although the significance of this difference is low given that error bars corresponding to cases (3) and (5), and (4) and (6) either overlap or are very close.

Interpreting the probabilities learned by the neural network
It is important to note that until now we have chosen to assess the performance of our prediction using bAcc for the sake of simplicity. However, the NN learns more than this; in particular, it learns to predict the probability that a relationship belongs to each of the classes in the dataset (see Methods for a detailed explanation on how this is achieved through the minimization of the cross-entropy loss function). The great advantage of using low-dimensional inputs is that we can interpret what the NN is learning. We can plot the probability that a sample belongs to a class (friend/enemy) as a function of the different predictors. In Fig. 3 (a) we plot this probability as a function of the triadic influence. We use the 10 different NNs trained for Fig. 2 (2) and plot the average probability of being friends and enemies for a pair of students with a given triadic influence. The colored area around both curves represents the standard deviation of the probabilities. The probability of being friends saturates to 1 when the triadic influence I ij 1, and drops to 0 if the triadic influence I ij 0 (the probability of being enemies is the complementary because both add up to 1). The probability curves of being friends and being enemies cross around I ij ≈ 5. Note that this is the only information used when computing the accu-racy bAcc, because we identify each relationship with the most probable one, as predicted by the NN. However, the probabilities learned by the neural network (which minimize the cross-entropy loss, see Methods) contain much more information and could be used to generate ensembles of social networks or to simulate their evolution using stochastic Markov chains. It is worth mentioning that, although the probability curves change abruptly around I ij ≈ 0, this change slows down as the triadic influence increases, thus displaying an asymmetric behavior on both sides of the crossing point I ij ≈ 5. Figures 3 (b) and (c) display the probability of being enemies and friends, respectively, as a function of the prosociality of both students (nominator/nominee), averaged over the 10 simulations used for case (4) of Fig. 2.
Similarly to the case of the triadic influence, even though bAcc is fully determined by the curve where the probability is 0.5, the profiles shown in these figures convey much more information. In particular, we can see that the probability that two students with 0 prosociality are enemies is 70%, which is in line with what one would expect: selfish people declare to have more enemies and are declared enemies more often than altruists (see the Supporting Information, where this can also be directly observed in the raw data, Fig. S4). Alternatively, two highly prosocial students are friends with a probability higher than 60%. Note also that both colormaps are approximately symmetric with respect to the diagonal. This implies reciprocity: the probability that i declares j as a friend is approximately the same as the probability that j does the same with i.

Predicting with the structural information of the social network alone
In the previous sections we use local informationindividual features and triadic influence-to predict relationships. Complementary to this, in this section we will attempt to make the same predictions using only the structure of the network-excluding weights, link directions, and individual features-hoping to shed light on the role played by the structure of the network for the creation of different relationships. We will merge labels {+1, +2} into a unique "friends" label, and labels {−1, −2} into a unique "enemies" label, so that predictions can be binary. In order to do that we will create node embeddings by assigning to each node a ddimensional array of features-which will replace the array of individual features used in the previous section. 128-dimensional embeddings are created with Node2Vec [35], an algorithm that explores the neighborhood of each node using biased random walks (see Methods for more details and figures S5 and S6 of the SI). The embeddings of all nodes are then used as inputs to train different models, in order to predict the relationships in the network. We show here the case where we train a neural network, although we have also used Random Forests (see the SI,  We create the embeddings for all nodes once and keep them throughout. We then train a neural network to predict the relationship between pairs of students (friends/enemies) using both their embeddings as input. This is akin to using the individual features of both students in the previous section, only this time embeddings encode information about the environment surrounding each node. We have trained and tested the neural network using two alternative treatments: in treatment I we have chosen at random 20% of the relationships from all high schools as the test dataset, and trained the neural network using the rest of the relationships; in treatment II we have created a test dataset with all the relationships inside one specific age level from one high school, and trained the model using all the other relationships. For treatment I, we trained the neural network 390 times, every time changing the train and test datasets as well as the initialization of the neural network (the embeddings do not change). For treatment II, there are 39 different age levels within the 13 high schools that we study, and we trained 10 different neural networks for each course-390 simulations in total.
The results corresponding to treatments I and II are summarized in Fig. 4. In this figure we show the accuracy as a histogram after carrying out treatments I and II for the 390 simulations-blue and orange bars, respectively. For treatment I, where we train and test on random relationships, the average accuracy is ∼ 75%, and the accuracy is always above 60% (purple bars). However, when we test on a complete age level that was excluded from the training dataset the performance degrades (treatment II), the mean accuracy is now ∼ 60% (orange bars), and there are many instances where the model is not doing better than a dummy model (bAcc ∼ 50%).
The fact that the model has predictive power using only structural information shows that there is a structural difference between the environments of friendly or adversarial relationships. Besides, since the predictive power of the model decreases when testing on an isolated age level, this suggests that the structure of most age levels contain specific information that is not present in the rest of the data. In other words, some of the information that is contained within an age level, and stored in the embedding of the students, seems to be exclusive to that level.

DISCUSSION
In this paper, we have applied techniques for link prediction to gain insight into the mechanisms behind the formation and evolution of social networks. This has been possible due to the large amount of data that we have collected, comprising individual features of more than 3, 000 students as well as their corresponding network of personal relationships-over 60, 000 connections.
The picture of the network dynamics that emerges from our work is as follows. Some initial relationships appear between pairs of students, promoted by their prosocial stance. As a matter of fact, we have shown that the prosociality of both students by themselves are capable of predicting isolated relationships significantly better than a pure random guess. This is actually a very strong claim, because many of those initial relationships are now hidden among many other relationships that emerged afterwards, and the isolated ones that we can find now are probably very sensitive to noise or trolling (e.g. students that label randomly other peers as friends/enemies). We hypothesize that isolated relationships continue to emerge until directed paths of length 2 dominate the dynamics of network formation. As discussed in previous sections, paths of length 2 are equivalent to intermediate students who can get two of their contacts in touch with each other. This mediation, quantified by the triadic influence, is an extremely good predictor of relationships, with accuracies as high as 86%. Interestingly, when we focus on relationships that are not isolated (there are directed paths of length 2 connecting both students), prosociality is still a good predictor of them. This suggests that some of these relationships might have originated as isolated relationships, and that prosociality is still important even when the relationship is not isolated. Complementary to this, we have observed that the accuracy achieved by the triadic influence does not improve if we also provide personal information about the students. This implies that the triadic influence somehow subsumes the information on the students' characteristics, rendering it irrelevant to predict relationships. It is still an open question whether information obtained from more elaborated personality tests could improve on the predictions achieved by the triadic influence alone.
On the other hand, we have used state-of-the-art algorithms to create an embedding for each student that contains information about their surrounding, considering only the undirected and unweighted network. We have shown that this structural information can be used to predict the type of relationship between two students. The embedding of each node is created using a random walk exploration of its surrounding, the depth of which is a parameter that we can vary (see Methods). Depending on the typical length of the exploring random walks this method can gather different structural information. The maximum length of the random walks used in this study is L = 4 (see the SI Fig. S6). Therefore, the Node2Vec algorithm is exploring the local structure of each student. This aligns with the results achieved using the triadic influence, suggesting that the closest contacts in the network-the local environment-are the ones that influence the creation/transformation of relationships the most. Still, it is remarkable that the triadic influence-a one-dimensional metric-is able to achieve a higher accuracy than the prediction based on the embeddings. This suggests that the triadic influence gathers more information than the metrics included in the embedding of the nodes-although note also that we are using different information in both cases because the embeddings are created irrespective of the weights and directions of the relationships.
Interestingly, Ref. [36] suggests that individuals with similar genotypes may not actively select into friendships. Instead, they may be placed into these contexts by institutional mechanisms outside of their control. Our conclusions could be interpreted in a similar way; the triadic influence may act as a social force that encourage students that are compatible (incompatible) to have positive (negative) relationships, akin to the popular knowledge "to be judged by the company you keep". In this case, prosociality would be still a good predictor of the relationship even though it was the social context-the triadic influence in our case-which promoted the relationship. This raises an important point that we want to stress: predictability does not imply causality. Another situation that highlights the difficulty of disentangling cause and effect is that at the time we collected the data many relationships that nucleated in isolation due to prosociality alone were now surrounded by multiple directed paths of length 2, and we have shown that the triadic influence is a very good predictor of the label of these relationships, even if their existence predated the paths entering the computation of the triadic influence. Therefore, while our results suggest a nucleation mechanism based on individual traits followed by a growth and evolution of the network dominated by the triadic influence, they do not prove that this is indeed the case. In order to assess to what extent this idea describes what is actually happening in real networks, a possibility would be to use the probabilities that we have learned through our link prediction techniques to simulate growing/evolving networks, and then compare these simulations with real data. In particular, it will be extremely interesting to collect data for the same network at different times to test the plausibility of different mechanisms of network evolution based on the probabilities learned here. If our proposal remains a good candidate to explain how networks form and evolve, then specific questions of interest arise, such as when the paths of length 2 begin to dominate over the primitive relationships existing in a network or how a local change in the sign of a relationship can lead to a cascade of changes with global effects on the social network.
Finally, it is worth mentioning that our results come from data from a large number of surveys but from a very specific population, namely, teenagers in secondary schools in Spain. Thus, the generality of our results should be validated by gathering similar data from other collectives and performing similar analyses.

Data collection
Surveys were conducted in 13 Spanish high schools (mandatory education, 11 to 15 years of age). The surveys were delivered through a computer interface and included direct questions about their relationships, as well as some others aimed at identifying personal attributes. To elicit relationships, students could choose from a list containing all the other students in their same year within their own schools. The number of classes participating in the study in each school depended on the availability of time and the decisions of the school direction. The data corresponding to one of the schools, also included in this work, was presented in full detail in Ref. [37]. For each student, we collected: • General data: School ID, course, class, and a student ID assigned by the software for the purpose of this study.
• List of relationships: All the relationships declared by the student (very good, good, bad and very bad) were collected with the student IDs of the nomenees and the corresponding labels (+2, +1, −1, −2).
-Prosociality, evaluated through the answer to the three following questions about sharing (q i ranks the level of selfishness of each answer): * What do you prefer? A) 10€ for you and 10€ for your partner (q 1 = 0) B) 10€ for you and 0€ for your partner (q 1 = 1). * What do you prefer? A) 10€ for you and 10€ for your partner. (q 2 = 1) B) 10€ for you and 20€ for your partner (q 2 = 0). * What do you prefer? A) 10€ for you and 10€ for your partner (q 3 = 0) B) 20€ for you and 0€ for your partner (q 3 = 1).

Predicting relationships using local information
Our social networks are directed graphs representing the relationships between all the students within each of the high schools of our study. We kept only the students that answered all the tests about their individual features (described above), a total of 3395 students and 60566 relationships. Relationships are gathered in the weighted adjacency matrix W , with elements w ij ∈ {−2, −1, 0, 1, 2} corresponding to the value of the relationship that student i declares to have with student j (w ij = 0 if there is no declared relationship). Note that w ii = 0 and that W is not symmetric (relations are not necesarily reciprocal). Additionally, the individual traits described above (self-declared gender, CRT, and prosociality) are stored in the nodes n i of the graph. A key quantity used in this work is the triadic influence I ij ≡ (W 2 ) ij = k w ik w kj . It quantifies the aggregated contribution of the directed paths of length 2 that go from i to j. Note that triadic influence considers only directed paths from i to j, and that I ij = I ji in general.
In order to use a neural network to predict the declared relationships between students, we would like to avoid having highly unbalanced classes, and therefore we define a task with only two classes: friends (we consider here only +2 relationships) or enemies (we merge here relationships −2 and −1). We have also considered a more unbalanced case, with the friend class corresponding to relationships with labels +1 and +2 and the results were qualitatively analogous. In any case, when we compute the triadic influence I ij , we keep all the labels in the network {−2, −1, 1, 2} (see Fig. 1 for an example). In this section, we use a deep neural network with one hidden layer, ReLu activation (see e.g. Ref. [40]), and 100 hidden units. The input dimension depends on the data we want to use to predict the relationship. Our neural network is a nonlinear function of the inputs and the internal parameters (numbers that change their value during training), which outputs a vector of dimension two. Let us call these outputs f (I, W) i , where I stands for the inputs corresponding to one specific relation (triadic influence, gender of both students . . . ), W are the internal parameters of the network and i = 0, 1 indicates one of the two classes in our dataset (friends/enemies). Then these outputs are put into a SoftMax function (see e.g. Ref. [40]) such that where q(I, W) i can be interpreted as the probability that a specific sample, characterized by inputs I, belongs to class i = 0, 1. Training the neural network amounts to minimizing a loss function such that q(I, W) i resembles the actual probability distribution p(I) i = δ i, (I) for each sample-(I) being the label of that input data and δ i,j = 1 if i = j and 0 otherwise. We use the cross-entropy loss function where the index k runs over all samples in the dataset. Note that if q(I k , W) (I k ) = 1 for all k, the network would predict with 100% certainty the correct label for all samples. In this situation L = 0, indicating that for the set of parameters W the function L reaches an absolute minimum. Hence, training the neural network amounts to minimizing L with respect to the parameters W. We have used stochastic gradient descent with an initial learning rate of 0.1 and a decaying factor of 0.99. We use a minibatch of size 20 and oversample the class with the smallest number of samples so that each minibatch has the same number of samples from each class. Unless otherwise stated, we minimize for 200 steps and compute the accuracy in the final step. In the case of the prediction of isolated relationships (two bottom bars of Fig. 2), the dataset is greatly reduced. To ensure that our results are robust we use a 10-fold cross-validation approach and report the mean value and an error bar representing the standard deviation from the mean. In this case, we train for 1000 minimization steps using a dynamical loss function with oscillations of amplitude 10 and a period of 5 minimization steps. A dynamical loss function weights the contribution of each class to the loss function with proportionality factors that oscillate during minimization. This process changes the topography of the loss function landscape [41], and helps the model find deeper and wider minima of the loss function (see Ref. [42] for further details).

Predicting relationships using global information
The steps followed in the process of creating the embeddings and predicting the class of a relationship are: • Passing the graph as an object to Node2Vec [35] yields a 128-dimensional vector for each node (an embedding). Node2Vec is defined by the two hyperparameters (p, q), which describe the space explored by the random walks. We use (p = 1, q = 4) after doing a hiperparameter optimization. The characterization of the typical random walk in this process can be found in the Supporting Information.
• We merge the embeddings for each pair of nodes to create the embedding of each edge (relationship), e.
• The structural representation for each edge, e, is the input that we use to predict the label, friends/enemies of the relationships in the training dataset. We oversample the training data (test data are left untouched) using the SMOTE technique [43]. This method produces new examples by interpolating close existing points in the 128dimensional space.
• We apply two different machine learning procedures: a random forest, and an artificial neural network.
The artificial neural network was implemented in the standard library Tensorflow [44] with four hidden layers (sizes 128, 64, 32, and 8) and the ReLu activation function. The final output included a sigmoid function. Our dataset compromises 13 schools with 3395 students and 60566 declared relationships, where we have already removed the students that did not answer all the questions about personality (around 3% of the students) and the relationships that included them. From the 13 schools considered, 3 of them are in the Region of Madrid and the rest are in Andalucía. In this Supplementary Materials we will name the schools in Andalucía as t11 1, . . . , t11 10, whereas the schools in Madrid will be t1, t2 and t6.
Students can declare to have a very good (+2), good (+1), bad (−1) or very bad (−2) relationship with any other student in their school. The distribution of the four types of relationships changes among schools although some features are common, see Fig. 5. For all schools, the most numerous relationships are good (+1) relationships, accounting between 40 and 50% in most schools. On the contrary, the least abundant relationships are the very bad ones (−2). In addition to this, we see some differences between the schools in Madrid and Andalucía but we leave a detailed study of these features for a future work.
According to the main text, the most important information to predict a relation between students i and j is the triadic influence. This quantity accounts for the influence that third people have on other relationships, and uses the directed paths of length 2 that connect i to j. For this reason, it is interesting to know how many relationships have a particular number of paths of length 2 connecting the starting node (i) and the final node (j). Figure 6 encodes this information. We have plotted each school separately (color lines) and all the schools together (black line). From the plot, we can see that 2% of all relationships do not have any path of length 2 connecting the starting and final node. The violin plot helps to visualize each school separately. Although there is variation between schools, the peak of the joint distribution (black line) is around 5 or 6 paths of length 2.
Prosociality is computed for each student. Students answer 3 questions that lead to a scalar that takes values 0, 0.33, 0.66 or 1 (see Methods). The proportion of students per prosociality value is similar between schools, see Fig. 7, and resembles the distribution of relationships (Fig. 5). The largest group of students in most schools is students with prosociality 0.66, while the smallest group corresponds to antisocial students (0). Again, there is an apparent difference between the schools in Andalucía and Madrid. The latter seem to have a larger proportion of very social students 1 compared to the schools in Andalucía. This seems to be in good agreement with the distribution of relationships where schools in Madrid showed more very good relationships (+2). We leave a detailed study of this phenomenon for future work. Figure 8 shows the average number of friends/enemies that are nominated by (or that nominate to) students of different prosociality values. Clear general trends indicate that students of high prosociality nominate and are nominated as friends more often, whereas students with low prosociality belong to negative relationships in a larger proportion than highly prosocial students. These results are in good agreement with the probabilities learnt by the neural network in the main text, where students with low prosociality had a larger probability of being enemies whereas highly prosocial students have a larger probability of being friends.
Appendix B: Predicting using only the structure of the network: additional information on the creation of the embedding and alternative results using a Random Forest We include here some additional information to clarify how the Node2vec algorithm works. Node2Vec builds the embedding for each node using random walks. The configuration of hyperparameters that we have chosen (p = 1 and q = 4) leads to local exploration. To create the embedding of one node we use 420 random walks, each of them composed of 30 movement attempts. In order to get some intuition, we show in Fig.9 the path followed by three random walks starting from different nodes, we show RWs that does not stay fixed in the initial node, although that is the most probable situation. The local exploration of the surrounding of the node by the random walks is confirmed by Fig.10, which shows the distribution of lengths for the random walks with the hyperparameters used in our work (see Methods).
Finally, to complement the results shown in the main text, we also include here the results for a second method using the embeddings as input. We have used a Random Forest (scikit-learn [45]). A Random Forest is an algorithm that divides the data into random ensembles of pre-dictors and data. From these randomly chosen pieces of data, the algorithm builds and trains decision trees. The final decision is then made on the most popular answer within this population of decision trees. We used a maximum depth of 7 levels for our decision trees. Fig. 11 depicts qualitatively analogous results to the ones achieved with the neural network and shown in the main text.   8. Statistics for the number of friends and enemies depending on the prosociality of the students. We merge here −2 and −1 relationships into an enemy category and +2 and +1 relationships as friends. Panel (a) shows the average number of people nominated as friends by a student of a specific prosociality, whereas panel (b) shows the average number of people that nominate someone of a specific prosociality as a friend. Panels (c) and (d) are analogous to (a) and (b) but for enemies. Students with high (low) prosociality tend to nominate and be nominated as friends (enemies) more frequentely. Error bars correspond to the error of the mean.
FIG. 9. Network representation of the students in one of the classes in the dataset, together with three examples of random walks used by Node2Vec to create the embedding of one node. The random walks are controlled by the hyperparameters (in our case p = 1 and q = 4), they measure the probability of exploring or staying in a certain node along the path. From the figure we can see that Node2Vec is exploring the local environment of the node.
FIG. 10. Distribution of path lengths in the random walks used by Node2Vec. This length is the shortest path between the origin of the random walk and the farthest node in that walk. A walk of length 0 means staying at the same node. We can therefore understand the locality of the algorithm. The shortest path length has an average and standard deviation of 0.80 ± 1.07, which shows again that embeddings are mainly built with local structure.
FIG. 11. Predictions achieved with a Random Forest. Distribution of balanced accuracy for the 13 high schools. Each histogram is composed of a sample of N = 390 points, that are different simulations for the same treatment, and then normalized such that the area of the histogram sums 1. The purple/dark histogram represents treatment I whereas the orange histogram represents treatment II, that is, using an specific age level from a high school as the test set.


DIALOG-22 RuATD Generated Text Detection

Text Generation Models (TGMs) succeed in creating text that matches human language style reasonably well. Detectors that can distinguish between TGM-generated text and human-written ones play an important role in preventing abuse of TGM. In this paper, we describe our pipeline for the two DIALOG-22 RuATD tasks: detecting generated text (binary task) and classification of which model was used to generate text (multiclass task). We achieved 1st place on the binary classification task with an accuracy score of 0.82995 on the private test set and 4th place on the multiclass classification task with an accuracy score of 0.62856 on the private test set. We proposed an ensemble method of different pre-trained models based on the attention mechanism.

Introduction
As the language neural nets got better at generating texts, it's getting harder and harder to distinguish the human-written text from generated one. So manual detection of these texts got impossible. For that reason, it's desirable to build a system that can automatically detect generated text.
The proposed system will use various features extracted from the text such as length, punctuation, word choice etc. To determine whether the text is human-generated or not. The accuracy of this system can be improved by using machine learning algorithms which will learn how humans generate texts and then use those features for detection purpose.
There are many ways to build such a system, but probably the most reliable one is based on machine learning algorithms. These algorithms can be trained on a large number of examples -both humangenerated and computer-generated texts. After being trained, they should be able to identify which texts are computer-generated with high accuracy.
This approach already works in other areas, such as spam detection. Some early experiments have shown promising results and indicate that this approach works well for the detection of the generated text.
The DIALOG-22 shared task of RuATD 2022 had 2 tracks for binary classification and multiclass classification. In this report, we will describe the data, how we handled it, the models we used, and the ensembling technique. For the binary classification task , we frame the generated text detection problem as follows: given a piece of text , label it as either human-written or machine-generated = { , }.

Datasets
Provided datasets offer the train and test splits. The part of the set was annotated automatically by different generative models. Various language models were fine-tuned on different tasks: machine translation, paraphrasing, summarization, simplification, and unconditional text generation -are used to generate texts. The texts written by a human were collected from open sources from different domains. (Shamardina et al., 2022b), (Shamardina et al., 2022c).

Related Works
In this section, we will discuss various methods for detecting machine-generated texts.
Over the past years, many approaches appeared for generated text detection. The latest works are usually based on using transformer-based models. Either fine-tuning the proposed task or using probabilities distribution to make decision-based on them (Ippolito et al., 2019). Here we list some examples of different methods: • First, we calculate the mean likelihood over all machine-generated sequences, then the mean of human-written ones. If the likelihood according to some language model is closer to the machinegenerated mean likelihood, then we consider it as generated text (Solaiman et al., 2019); • In GLTR (Gehrmann et al., 2019) described a method using a language model to compute the probability distribution of the next word given the text sequence. For each sequence position, we get the likelihood of the ground-truth next word within this list. Then these ranks are displayed on a histogram. Based on the distribution of bins, we can decide if this text is generated or not.
• Bert fine-tuning on the classification task. Having a label of text if it's machine-generated or not, we can fine-tune the language model to predict (Solaiman et al., 2019) • Also, there is possible to use human-machine collaboration. Real or Fake Tool provides a game-like interface for humans to decide at what point the text begins to be written by a computer (Dugan et al., 2020). More approaches are described in the survey (Jawahar et al., 2020).

Experiment Setup
In this section, we present the experiment configurations we use to solve binary and multiclass tasks.

Data preproccessing
We decided not to perform any preprocessing of the text itself. Regarding the data split: • Binary classification. We concatenated the train set with the validation set. This new concatenated dataset gave us an option to perform a 5-fold cross-validation; • Multiclass classification. We took the data as is without changing anything.

Models
All models we used for fine-tuning we got from transformers library (Wolf et al., 2020). Here we will describe these models: • sberbank-ai/sbert_large_nlu_ru (Antuhov, 2020), sberbank-ai/ruBert-large, DeepPavlov/rubert-base-cased are BERT models (Devlin et al., 2018). Sber-ai models were fine-tuned on a closed dataset collected by their research group (Sber, 2022). DeepPavlov's RuBERT was fine-tuned on the Russian part of Wikipedia and news data. • IlyaGusev/mbart_ru_sum_gazeta is a mBART model fine-tuned on the dataset for summarization of Russian news (Gusev, 2020). The original MBart model was pretrained on large-scale monolingual corpora in many languages using the BART objective. mBART is one of the first methods for pre-training a complete sequence-to-sequence model by denoising full texts in multiple languages, while previous approaches have focused only on the encoder, decoder, or reconstructing parts of the text (Liu et al., 2020). • MoritzLaurer/mDeBERTa-v3-base-mnli-xnli (He et al., 2021) This multilingual model is suitable for multilingual zero-shot classification. The original model was pre-trained by Microsoft on the CC100 multilingual dataset (Wenzek et al., 2020). This model was fine-tuned on the XNLI development dataset and the MNLI train dataset. The XNLI development set consists of translated texts for each of the 15 languages (Conneau et al., 2018). • DeepPavlov/xlm-roberta-large-en-ru-mnli is an XLM-RoBERTa model (Conneau et al., 2019) which was fine-tuned on the english-russian part of the MNLI (Williams et al., 2017) dataset.

Binary classification (with ensembling technique)
Five chosen models were used in the experiement: sberbank-ai/sbert_large_nlu_ru, sberbankai/ruBert-large, IlyaGusev/mbart_ru_sum_gazeta , MoritzLaurer/mDeBERTa-v3-base-mnli-xnli, DeepPavlov/xlm-roberta-large-en-ru-mnli Let's describe steps for training each of these models: 1. We split our training dataset into non-overlapping 5-folds and performed cross-validation ( Figure  1); 2. for each validation fold we predicted the target and as a result, we got out-of-fold predictions for training data; 3. for the test set, we predicted the target from every 5 models from cross-validation and averaged the result which is marked with red; As a meta-model, we choose Logistic Regression, which was trained on out-of-fold predictions. Then we predict the final results for the test set.

Multiclass classification
For the multiclass classification we chose these models: DeepPavlov/rubert-base-cased, DeepPavlov/xlm-roberta-large-en-ru, IlyaGusev/mbart_ru_sum_gazeta. We fine-tune these models without cross-validation only on provided train set. Table 3 shows each model accuracy score on binary classification task. The multiclass classification accuracy is shown in Table 4. On both tasks, the best performing single model was DeepPavlov/xlmroberta-large-en-ru-mnli. We managed to ensemble models in the binary task, so the ensemble of models showed the best accuracy.

Conclusion
In this paper, we described our pipeline for the DIALOG-22 RuATD challenge. Our solution achieved 1st place in binary classification using ensembling techniques and 4th place for the multiclass classification task using only a single model. However, the proposed solution requires a lot of computational power, so it cannot be used in real-time systems to detect generated texts. But it gives us an understanding that we still need to upgrade methods to distinguish generated texts from human-written ones.


Benchmarking Scientific Image Forgery Detectors

The scientific image integrity area presents a challenging research bottleneck, the lack of available datasets to design and evaluate forensic techniques. Its data sensitivity creates a legal hurdle that prevents one to rely on real tampered cases to build any sort of accessible forensic benchmark. To mitigate this bottleneck, we present an extendable open-source library that reproduces the most common image forgery operations reported by the research integrity community: duplication, retouching, and cleaning. Using this library and realistic scientific images, we create a large scientific forgery image benchmark (39,423 images) with an enriched ground-truth. In addition, concerned about the high number of retracted papers due to image duplication, this work evaluates the state-of-the-art copy-move detection methods in the proposed dataset, using a new metric that asserts consistent match detection between the source and the copied region. The dataset and source-code will be freely available upon acceptance of the paper.

I. INTRODUCTION
Integrity researchers have been reporting a threat to scientific image integrity for a long time [1], [2], [3], [4]. This improbity has achieved even areas like cancer [5], or more dramatically used on 'paper mills' services [6].
Although this serious problem has been an urge in the scientific community, to the best of our knowledge, few forensics works are dedicated to this topic. So far, no rich annotated forensic benchmark containing scientific tampering images was published. We believe that a large dataset would foster the forensic community to work more actively in this subject and assist state-of-the-art forensic techniques that might require large training datasets.
In this sense, we tried to collect known doctored scientific images, but we faced two main issues that make us avoid these methodology: legal and practical. To publish a dataset with real tampering cases, we would have to face copyrights and legal aspects of pointing third-party works that were retracted due to suspicious manipulation. Even if we decided to manage this legal aspect, we had to be guided by a retraction notice relative to the issued images. However, after reading several retraction notices, we realized that many of them are not precise enough to pinpoint the issued region's at a pixel level, which would not lead to an accurate ground-truth. The example of a real retraction notice due to an honest error, depicted in Fig. 1, shows this inaccuracy, in which the highlighted words 'some lanes' and 'not the appropriate ones' translate to an ambiguous region and cause.
On the other hand, we notice that a representative number of retracted images were due to duplication and basic image processing operations that could be automatically created. Therefore, this work presents the RECOD Scientific Image Integrity Library (RSIIL) that enables creating a synthetic scientific image tampering dataset with enriched pixel-wise ground-truth and without any associated legal issue. With this library, we created the RECOD Scientific Image Integrity Dataset (RSIID) with the most common image operations reported by scientific integrity researchers [2], [4]. To create this benchmark, we doctored 2,923 figures from creative common sources resulting in 39,423 tampered figures (26,496 for training and 12,927 for testing). In addition, we propose a new metric to evaluate copy-move forgery detection dedicated to scientific images using an enriched ground-truth map, that assert a consistent detection match between the cloned region and its source. Finally, using this new dataset and metric, we evaluate the performance of the state-of-the-art copy-move forgery detection [7], [8], [9], establishing a baseline and setting the ground for any future investigation. We organize the remaining of this paper into seven sections: Section II presents related work while Section III details the proposed library, RSIIL. Section IV presents the dataset RSIID while Section V brings out a new evaluation metric aimed at a more consistent copy-move detection evaluation. Section VI presents an analysis of state-of-the-art copy-move forgery detectors on the proposed dataset setting the ground for future research while Section VII presents the conclusions and future work directions.

II. RELATED WORK
To the best of our knowledge, few works try to design a tampering benchmark focused on scientific images. So far, we were only able to find two works that address scientific integrity image datasets. The first one is from Xiang and Acuna [10], which created a synthetic tampering dataset of scientific images from the web. They doctored microscopy and arXiv:2105.12872v1 [cs.CV] 26 May 2021 western blot images using three types of manipulations that they claim to be the common cause of problems in scientific papers: cleaning of an image region with a single color or noise (Cleaning); copying an alien content region into the image (Splicing); and applying visual adjustments in the image content (Retouching). Their dataset contains 747 manually manipulated scientific images, of which 616 are dedicated to Removal. As we were only able to find the pre-print version of [10], we could not find any released data. Due to this, the quality of their manipulations and the dataset license is still unclear. Despite the authors manually constructed the dataset to create a more realistic scenario, their dataset is still limited to a small size that might not represent the diversity of scientific images. Besides this, the dataset is highly concentrated on Cleaning, preventing one to properly evaluate the robustness of a forensic method among all modalities.
The second one is the work of Koker et al. [11], named as Bio-Image Near-Duplicate Examples Repository (BINDER), which have the pioneering idea of using legal issue-less scientific images for an integrity dataset. This dataset is limited to finding near-duplicate images, aiming to find image re-use across scientific publications. Their dataset has 10,179 nonoverlapping patches tiled in 256 × 256 or 128 × 128 pixels. To create their dataset, they gathered microscopy images from the following public repositories: NYU Mouse Embryo Tracking Database 1 (METD), the Broad Bioimage Benchmark Collection 2 (BBBC), the Adiposoft Image Dataset 3 (AID), and the Open Microscopy Image Data Resource 4 (IDR). Besides, they also applied some geometric, brightness/contrast, and compression transformations on some images. However, their dataset is still not as realistic as the figures presented in scientific publications. Despite scientific images often embed graphical elements and captions, hampering to detect re-use, the authors did not add these elements to the images. In addition, they did not apply any local tampering (region-level), which is also a typical manipulation in inappropriate image reuse [4].
In addition to these works, we also found two scientific integrity initiatives that collect real cases of retracted papers. The first is the Retraction Watch Database 5 maintained by the Retraction Watch 6 . This database has more than 20,000 metadata of retracted, corrected, or concerned papers. The metadata presents the paper's title, retraction reason, authors, and Digital Object Identifier (DOI), among other fields. Although this database is not dedicated to image integrity issues, it is possible to filter the retracted papers to this category. However, only the paper's metadata will be retrieved -due to legal aspects; it is not possible to retrieve the articles PDF, Figures, or Retraction notice -, which is a drawback of this database.
The second is the HEADT Centre Image Integrity Database 7 , an initiative focused on researchers and developers working on scientific image manipulation detection. Their database contains more than 500 images' metadata from retracted papers due to image manipulation. In addition to the basic information of a paper (Title, Authors, Publisher, Journal), they also added a text description of each manipulation, including the figure's panel in which the manipulation occurs and its category (e.g., copy-move). This text description is based on the retraction notice associated with the figure; therefore, some text also presents ambiguity as depicted in the Figure 1. Despite this text description, we could not find any manipulation map at pixel level in this dataset, which we believe is needed to evaluate a detection method properly.

III. RECOD SCIENTIFIC IMAGE INTEGRITY LIBRARY -RSIIL
Before working with synthetic data, we tried to gather real-world problematic scientific images. To avoid any bias from our side, we relied upon retracted papers due to image problems given that they have a retraction notice resultant of an integrity investigation. However, to publish an accessible benchmark for forensic research, we possibly would have to deal with some legal aspects (e.g., figure copyright and causing possible defamation to someone).
As reported by Adam Marcus [12], a retracted paper could make their authors feel their reputation harmed and make them sue journals for defamation. Azoulay et al. [13] also indicate that retraction due to misconduct --which are the most important papers to be included in a forensic benchmark-has a significant reputation penalty to their authors. Even co-authors that might not be involved in the image manipulation, who already suffered severe consequences [14], could be affected by such benchmark since it would promote their association with the retracted paper.
Besides this legal aspect, we also faced some practical issues regarding data annotation. When manually annotating the problematic figures' regions following their retraction notice, we experienced an absence of standard, including vague sentences (as illustrated in Figure 1), resulting on unreliable ground-truths.
Because of these issues, we decide to avoid using real-world scientific problematic image and create a photorealistic dataset using the library introduced in this section. Thus, this Section presents the types of forgeries implemented in the library (Sub-Section A), explains how the library mimics realistic figures as they usually are presented in scientific documents (Sub-Section B), and addresses the manipulation ground-truth (Sub-Section C). Finally, the section also discusses how the proposed library is amenable to extensions of new image manipulations types (Sub-Section D).

A. Library functionalities
The goal of the library is to implement the most common image manipulations reported in the scientific community. Although we are aware of the possibilities of more complexity tools for image manipulation, for example, the creation of scientific images using artificial intelligence (AI) algorithms [15], we suspect that these tools are not vastly used yet due to their complexities. Therefore, while designing the library, we adopt the forgery function based on the most common image processing operation accessible for a non-expert in AI or Computer Vision. We also design each block of the library to allow it to be extended to other more complex operations in the future. Following the research from Bik et al. [4] and Rossner and Yamada [2], we selected three main types of manipulations that can be recreated using common image processing software (e.g., Adobe Photoshop): 1) Retouching: The process of image beautification leading to an experiment misreading. This modality implements contrast, brightness, and blurring adjustments that highlight or obfuscate an image region. Figure 2 depicts an image that we applied retouching with our library. In Figure 2b, we used a Gaussian filter within the selected objects to obfuscate its content. Figure 2e illustrates an image with contrast and brightness adjustment, in which the method changes the selected object pixels intensity to cause an experimental misreading. 2) Cleaning: The result of obfuscating a foreground object using a background region. For this modality, we use inpainting and a brute-force routine. For the inpainting, we use the method of Criminisi et al. [16] implemented by Moura. 8 For the brute-force routine, we develop an in-house method to mimic the forgery procedure of a person seeking to cover an object using the background.
To implement this routine, we select a foreground object FO ; then, using brute force, we fit FO on a background region BR that has the most similar color histogram of this object; finally, we copy BR into FO and blur the border of FO, smoothing (feather edges) the difference from the copied BR and the neighborhood of FO.  of an image within the same or another image, using or not post-processing operations. Note that this definition includes both copy-move and splicing. We organized this category into three sub-categories: a) Copy-Move Forgery: Duplication of a region within the same image using geometric transformations (translation, rotation, flip, and scaling) and post-processing (e.g., retouching). All transformations can be combined with another. Due to the intrinsic result of scaling, we always combined it with another operation, otherwise it would cover the source object region. Besides these transformations, we also implemented a random object-tobackground copy-move (that we named Random). This routine copies a random object RO to a background region BR that has the same shape as RO. b) Overlap Forgery: Creation of two images with an overlap region from a single one. From a source image I, we select different regions of I that share an overlap area to create two images from these regions. Any of these new regions can suffer postprocessing to obfuscate its source. Figure 5 depicts the creation of an image with an overlap area. c) Splicing: Creation of an image composition that uses a donor figure's elements into a host one. Figure 6 depicts an Splicing forgery. Despite all cases are generated without any human interac-  tion, the result images may confuse even an attentive person.
To produce forgeries as realistic as possible, some functions from the library require as input an object map (segmentation map). The object map locates each object inside the image and assists a forgery function to execute the falsification more likely as a human would do.

B. Realistic Scientific Figures
As our key objective is to create scientific figures, we include two features (frequently present in such figures) in the library: captions/indicative letters and compound figures.
1) Caption/indicative letters: Scientific figures often present indicative letters or captions that overlay the image's content. As a result, this overlay is a splicing operation between a letter or a word within the experiment image that could raise a false alarm during forgery detection. Therefore, we add to the library the possibility to mimic this overlap behavior as it appears on scientific papers. We include three different levels of indicative verbosity. Level 1 includes only indicative letters around each panel of the figure. Level 2 includes the features of Level 1 and a random word around each panel. Level 3 includes all features from Level 2 and an indicative letter inside each panel. Figure 7 depicts all these levels of verbosity. 2) Compound figures: A Compound figure is a composition of multiple images that are organized in panels. These figures usually appear in articles to represent an overview of an experiment. To avoid creating unrealistic compound figures, we make use of figure templates based on real cases. These templates are image masks that can inform each panel's location to the method, as well as their type (e.g., graphs, photos). To create Compound figures, we implemented a routine that has as input a set of realistic compound figures templates T , a dataset of scientific images D (to be included in the   Figure 8 illustrates this routine. Thus, the method selects a template t from T with at least one panel whose aspect ratio is similar to the aspect ratio of S. Then, the routine applies the forgery f in S, creating S f (a forged version of S). Later, a figure with the same size of t is created, and S f is resized and placed in the panel of t with the most similar aspect ratio of S f . Finally, all other panels of t are filled with different images from D that have similar aspect ratio to those panels or with fake graphics.

C. Data Annotation
In spite of the importance of reliable ground-truth to evaluate a forensic method, to the best of our knowledge, there is no scientific forgery image dataset that presents an enriched ground-truth. Hence, all tampered operations implemented in the library provide detailed maps to indicate the manipulated regions. Each object involved in the tampering operation is indicated with a different ID in the ground-truth, which helps pinpoint the object's exact location before and after the forgery, as depicted in Figure 3g. The library also enables the creation of a JSON file containing metadata related to the forgery. This metadata includes the source images, the method and arguments used, and the location of each panel inside the Compound figure. As the metadata includes the source images and the forgery methods applied, one can evaluate provenance analysis [17] using these information as reference.

D. Library Extension
Given that scientific image tampering improves over time to convince even researchers [15], the benchmark of tampering detection also should include cutting-edge forgery techniques. In this sense, to facilitate the inclusion of new manipulation in RSIIL, we implemented a high-level routine that receives as one of its arguments a forgery function and applies it to an image. This routine is responsible for regulating the application of any new manipulation, asserting its guidelines to the ground-truth and the metadata associated with the forgery. Because of this, any new forgery function capable of returning the forged image along with its manipulation map -a pixel-wise map locating the forgery inside the image-can be easily added to the library to generate Simple or Compound tampering figures.

IV. RECOD SCIENTIFIC IMAGE INTEGRITY DATASET -RSIID
In addition to the library we introduced in the previous section, we created a dataset to serve as future benchmark for the area. For that, we selected the most frequent retracted types of images from the biomedical area. For this, we followed the orientation of Bucci [18] and Bik et al. [4] that report a high image manipulation rate on images from Western Blot techniques and Microscopy imagery. With this in mind, we downloaded real scientific images collected from diverse sources to apply the forgeries.
To avoid any legal aspect of creating manipulated images and aiming to publish the dataset with a common creative license, we only downloaded data available under public domain 9 (PD) or common creative attributed 10 (CC-BY) licenses. These license allow us to remix, transform, and reuse the images without asking for the author's authorization.
We use the following data source to gather the image collection: 1) Broad Bioimage Benchmark Collection 11 (BBBC), : A collection of freely downloadable microscopy image sets. From this source, we selected the datasets BBBC038, BBBC039, and BBBC019. The first two are dedicated to segmented nuclei images and have objectmask -that are needed for forgeries at object-level-. The last dataset (BBBC019) is dedicated to cell migration which we use for Overlap forgeries. 2) PubMed Central (PMC) 12 : PMC is a free article archive of biomedical and life sciences. To download each figure from this repository, we use an API 13   This dataset, announced in Naylor et al. [19], was designed aiming at nuclei segmentation of cells by deep neural networks. Therefore, this dataset has a highquality object map that we make use of to assists the forgery operations at object-level. Table I shows the number of source figures for each collection by type (Microscopy or Western blot) pointing if they have object-mask annotation, and the number of template images created based on the figures from PMC.

A. Dataset Construction
While designing the dataset, we project it to evaluate a forensic tool in different tasks with different complexities. Because of this, the dataset is organized so that a user can easily find the data and its annotation for each forgery modality. Thus, we divided the dataset into two types of figure complexities: Simple and Compound. To create this tampering type, we add a Simple forgery as one of the figure's panels. Forgeries that need more than one source image (e.g., splicing) or that generate more than one doctored figure (e.g., overlap) were not included in this modality.  b) Inter-Panel are forgered figures that have two or more panels involved in the manipulation process. This modality aims to evaluate duplications among two or more panels within the same figure. These duplications can be at object-level, region-level, or panel-level. At object-level, the objects from a donor panel are copied into a host, using splicing operation. At region-level, an overlap forgery operation creates two panels with overlapping areas. At panel-level, the entire panel is duplicated with or without post-processing (e.g., retouching, cleaning, or geometric transformations). For each Compound figure, we generated the three levels of indicative letters verbosity, as described in Section III-B. Figure 10 illustrates the organization of Compound figures in our dataset. To assist machine learning forensics techniques, we further divided the dataset into training/test sets. Tables II and III express the number of manipulated figures included in each modality. Note that, overlap forgery appears only in the test set, since this modality is similar to the copy-move, and this protocol will force the generalizability of a forensic tool among the methods.    Table IV.
As a drawback, these metrics cannot assert if both regions of a copy-move (the source and its copy) are in the ground-truth, since there is no consistency check. Because of this, some contradiction might occur during the evaluation. For instance, Figure 11 illustrates a detection map that has an inconsistent detection in which the copied objects with id = 1 and id = 2 are inconsistent with the ground-truth. For both objects, only the source or its copy overlaps with the ground-truth, which is inconsistent, since both (object and its copy) are expected to be included during detection. However, when evaluating this detection map with traditional true positive score, both regions would be considered as true positive hit.
To mitigate this drawback, this section introduces a new metric that takes advantage of the enriched pixel-wise groundtruth of the dataset. The proposed metric is a variation on how to consider a pixel as true positive in a detection map named as the Consistent True Positive score (CTP) and defined as: Given a ground-truth map GM with n ≥ 1 manipulated regions, a detection map DM with m ≥ 0 copy-pasted regions, each one of the n regions included in GM has c gm ≥ 2 connected components (the source object and all its copies), and each region in DM has c dm ≥ 1 connected components. Let R dm be a detected region from DM and R gm a tampered region indicated by the ground-truth. Also, let p be a pixel from DM , such that p ∈ R dm . Thus, p is a consistent true positive if exists R gm ∈ GM , such that, at least two connected components from R gm intersects R dm .
In other words, to consider a region R dm from the detection map as a consistent true positive, at least two components from R dm (the source and at least one of its copies) have to intersect the ground-truth.
As Figure 12 depicts, a region from the detection map can overlap with two or more region from the ground-truth. Given that the goal of CT P is consistency, we only consider as CT P the region of the ground-truth that has the maximum intersection area with the detected region. Hence, CT P ≤ T P . As a result, F N will have higher penalty on Precision and F1-score metrics, if calculated with CT P .
Thus, the equation of F1-score and Precision using CTP become: Precision CT P = CT P CT P + F P

VI. EVALUATING CMFD METHODS
Duplication of scientific images is one of the threats highly reported and studied in the literature [4], [2], [20] which includes copy-move, a well-studied forgery in the digital forensic field. Although this field presents multiples CMFD solutions for natural images, we could not find any study that evaluates their performance in the scientific image domain. In this sense, to assist any future forensic method with a baseline, we investigated the performance of popular CMFD solutions on natural images applied to the RSIID. In addition, we checked the difference of F1-score using the proposed consistent true positive metric and the regular true positive one. For this, we choose the following CMFD methods: 1) Efficient Dense-Field from Cozzolino et al. [7]. During the evaluation, we use the implementation of Ehret [21]. Ehret released two versions of [7] using Zernike  Fig. 11. Example of an inconsistent detection map. Although the detection map has two overlapping regions with the ground-truth, each object and its copy -indicated by the detection map-does not intersect simultaneously with the ground-truth. Example of CTP for a detected region that overlap two or more objects from the ground-truth. The ground-truth present three copied objects. The detection map present two connected components with the same ID that overlaps more than one object from the ground-truth. Only the object with the larger area is considered as true positive, the others would be considered inconsistent. and SIFT features. To distinguish this method from the others, we named them Zernike-PM and SIFT-PM, since these detectors use the PatchMatch algorithm [22] to match similar blocks contents. 2) CMFD library implemented by Christlein et al. [8]. We selected SIFT and SURF methods from this library since the others were not efficient enough to be explored on such a large dataset. To distinguish them from the previous CMFD detectors, we named them SIFT-NN and SURF-NN since they use a regular approximate nearestneighbor approach to match similar blocks. 3) Busternet from Wu et al. [9]. This method is a deep neural architecture for CMFD. During the evaluation, we use the pre-trained version of the model released by Wu et al. [9].
To evaluate SIFT-PM, Zernike-PM, SIFT-NN, and SURF-NN using CT P , we modified their implementation, including a routine that assigns each detected object and its copies a unique ID. For the sake of reproducibility, we released the methods source-code with this modification in the same repository of RSIIL. On the other hand, to evaluate Busternet, we normalized its output [0,255], then binarized all pixels greater than 100 to 1, otherwise 0. As Busternet is based on neural networks, we could not find an explainable methodology that would track the matching among different objects and their copies. Thus, to the CT P metric, all detected and ground-truth objects are set with the same ID = 1. Consequently, CT P would not be able to properly check inconsistencies on figures with more than one tampered object for Busternet's output; however, CT P is still valid and useful to check if Busternet's output overlaps with two or more connected components from the ground-truth. As a baseline approach, the evaluation protocol consists of running all methods without any training or fine-tuning and measuring their output with F1-score CT P . During the evaluation, we use all figures from the test set applicable for CMFD (i.e., images with duplicated areas within the same image). We group the baseline results into Simple and Compound scientific figures, which were divided by modalities. All copy-move modalities presented in Figures 13 and 14 can also include scaling. Since scaling cannot be applied alone, we did not indicate when this operation is combined with others. Figure 13 presents a radar graph visualization in which the forgery modalities are arranged in the radius axes. Each CMFD methods' result is represented with a different color in the radar char. In this visualization, we insert the score of each method along the modality axis (e.g., copy-move with flip) which start from the radar center (score zero) until its border (highest score); thus, as farther a method point (color point) is from the center as better is the method for the axis copy-move modality. After inserting all points of a method for each copy-move modality, we connected those points resulting in a polygon. The larger the polygon area, the better the method's robustness among different forgery modalities. Also, comparing each detector's robustness to the operations, this type of visualization helps identifying possible complementary behaviors among different methods. As an example, consider Figure 13a left panel. In this case, we have five modalities being compared (e.g., Copy-Move with Flip, Cleaning with Brute-Force, and Copy-Move with Translation). This char shows the results of five methods represented by each different polygon color (see legend on the right of the figure). The best method in this figure is Busternet (in orange) while the two worse methods (SURF-NN and Zernike-PM) are in superposition at the center (smaller areas). In the following, we discuss the forgery evaluation for each modality.

A. Simple Figure Forgery Baseline
In this modality, we tested the chosen methods on Simple figures, forged with Cleaning (Brute-Force) and Copy-Move. Although the chosen CMFD detectors have high efficacy on natural image benchmarks, their performance drastically decrease when applied to our scientific dataset. As  In this plot, each method fares differently for each modality. The polygons from SIFT-NN and SURF-NN have larger area than the others methods, indicating that they are robust to more operations than the others methods. The shrinking polygons area from F1-score CT P (V 1) to F1-score CT P (V 3) indicates the higher the caption Level, the lower is the method's effectiveness.  13a shows, the best CMFD method in the Simple Figure  Evaluation was Busternet [9], despite its modest scores.
For this modality, we also compare each methods' performance between F1-score using T P and CT P . We notice a difference in these scores for all methods, represented by the area reduction from their polygon chars, indicating the existence of copy-move inconsistencies on their detection maps, which is also depicted in Figure 14. The second row of this figure shows an example of an inconsistent detection map, in which Busternet activates just one of the connected components involved in the manipulation, resulting in F1-score CT P = 0. On the other hand, in the same row, SIFT-NN detects both regions (object and copy), resulting in F1-score CT P = 81.51.

B. Inter-Panel Figure Baseline
We evaluate the Inter-Panel tampered figures for all indicative verbosity levels in Copy-Move (at panel-level), Splicing, and Overlap forgeries. Figure 13b shows the result for the Inter-Panel forgery evaluation using F1-score CT P . In this modality, the radar visualization allowed us to notice some complementary performance among the chosen detectors. For instance, SURF-NN and Zernike-PM show a complementary behavior to copy-move with rotation and retouching. We believe that this complementary aspect indicates that a fusion/ensemble technique might enhance their individual robustness. For the Inter-Panel scenario, the flipped copymove, Splicing, and Overlap forgery showed to be the most challenging forgeries. In addition, the indicative letters are shown to have a perceptible impact in this scenario, reducing by up to seven points from level 1 to level 3 for some detectors. Althouth Busternet achieves the best performance in Simple Figure modality, when applied to compound figures, it leads to a higher false-positive rate, as depicted in Figure 14 by activating the entire image.
We also noticed that graphs and indicative letters are the most common causes of false alarms in the Compound Figures scenario, as illustrated by the third, fourth, and fifth rows of Figure 14, which SIFT-PM wrongly activates letters and graphs regions.
These findings help us to see where researchers should focus on when dealing with the scientific image forgery detection problem.

C. Intra-Panel Figure Baseline
We evaluate the Intra-Panel tampered figures for all levels of indicative verbosity in Cleaning (Brute-Force) and Copy-Move (at object-level).
As presented by Figure 13, this is the most challenging scenario, in which the detectors scored lower than four on F1-score CT P for all evaluated operations. A possible explanation for this is the lower percentage number of doctored pixels in these figures than in other modalities. The detectors' low performance does not allow us to measure the impact of verbosity levels in the figures properly. However, as Figure 14 shows, graphs and indicative letters would also be one cause of false alarms in this modality.

VII. CONCLUSIONS
In addition to the daunting scenario of fraud in sciencedue to the increase of image misconduct cases-, there is a legal issue related to copyrights and judicial aspects that prevents one from creating a large collection of fraudulent scientific images, even for an in-depth forensic study to benchmark and drive the development of appropriate detection methods.
Therefore, this work introduced a library and a dataset to assist the scientific integrity and forensic community to overcome this legal hurdle. We believe that by presenting a large dataset to the forensic community, we are fostering the development of more complex and robust detection tools (e.g., AI-based models).
The proposed library implements the most common image manipulation forgeries described by scientific integrity researchers. Also, it is extendable to more complex tampering operations. As a special feature, the library generates an enriched ground-truth addressing all regions affected before and after applying a tampering function, assigning a unique ID for the regions involved (when applicable). Using this library on creative common scientific images, we created a dataset with 39,423 manipulated figures freely available.
Leveraging the dataset's enriched ground-truth, we proposed a metric that avoids inconsistent detection during CMFD evaluation. Using this metric, we evaluate popular CMFD methods on our dataset. Although we choose high-cited and effective CMFD tools for natural images, all solutions presented a lower performance when transferred to the scientific image domain. This is not a fault of such algorithm as they were not designed for this specific setup. However, these findings show an important lack of methods and a tremendous research opportunity for new specialized detectors aiming at finding forgeries in scientific-related images. In addition, we notice that some of the chosen algorithms present complementary performance and might benefit from a fusion approach.
Notwithstanding the large size and diversity of the proposed dataset, we believe that science will report more sophisticated tampering operations in the near future, as warned by [15]. Thus, we are also concerned about more issue-less and freely available scientific integrity datasets with complex, enhanced, and realistic tampering modalities, aiding the design of more robust detectors.
Therefore, as future work, in addition to investigating robust forensic solutions using AI-based or fusion-based methods, we believe that studies on automated realistic scientific forgeries would also assist the forensic community in fighting scientific misconduct. Furthermore, we believe that the detailed pixelwise ground-truth of RSIID opens a research opportunity to explore eXplainable AI solutions that might assist analysts on sensitive cases, such as misconduct investigations.