text:
Robotic Inspection and Characterization of Subsurface Defects on Concrete Structures Using Impact Sounding

Impact-sounding (IS) and impact-echo (IE) are well-developed non-destructive evaluation (NDE) methods that are widely used for inspections of concrete structures to ensure the safety and sustainability. However, it is a tedious work to collect IS and IE data along grid lines covering a large target area for characterization of subsurface defects. On the other hand, data processing is very complicated that requires domain experts to interpret the results. To address the above problems, we present a novel robotic inspection system named as Impact-Rover to automate the data collection process and introduce data analytics software to visualize the inspection result allowing regular non-professional people to understand. The system consists of three modules: 1) a robotic platform with vertical mobility to collect IS and IE data in hard-to-reach locations, 2) vision-based positioning module that fuses the RGB-D camera, IMU and wheel encoder to estimate the 6-DOF pose of the robot, 3) a data analytics software module for processing the IS data to generate defect maps. The Impact-Rover hosts both IE and IS devices on a sliding mechanism and can perform move-stop-sample operations to collect multiple IS and IE data at adjustable spacing. The robot takes samples much faster than the manual data collection method because it automatically takes the multiple measurements along a straight line and records the locations. This paper focuses on reporting experimental results on IS. We calculate features and use unsupervised learning methods for analyzing the data. By combining the pose generated by our vision-based localization module and the position of the head of the sliding mechanism we can generate maps of possible defects. The results on concrete slabs demonstrate that our impact-sounding system can effectively reveal shallow defects.

INTRODUCTION
The civil infrastructure in the United States of America is reaching pass its life expectancy and the cost of repairs is going to tremendously increase over the next years. ASCE 2021 report card scores America's infrastructure with an average C-grade.
According to this report there are more than 617,000 bridges and 42% of them are over 50 years old, with 7.5% of them considered to be in a poor condition or structurally deficient. Condition of bridges is graded with C, a decrease from C+ from the ASCE 2017 report card. It is critically important to increase the inspection frequency so we can detect and repair the defects before any additional deterioration damages these structures beyond the repair point. The inspection of infrastructure is a time-consuming, expensive and potentially a dangerous process. Inspectors use NDT methods like impact-echo [1], acoustic inspection [2], ultrasonic [3], GPR [4], visual inspection [5] etc., to detect defects like cracks, delamination, voids, rebar corrosion etc. GPR [15] [16], is one of the most widely used methods in detecting defects like voids, delamination, buried objects and metallic rebars [6], corrosion [7] etc. Impact-echo method is used to detect delamination, cracks and voids in concrete structures [1]. However, impact-echo although successful still lacks any good result for shallow delamination [8].
The focus of this paper is detection of shallow defects using automated robotic impact-sounding and generation of high-resolution maps. First, an automated robotic solution with wall-climbing capabilities is developed. This automated solution drastically decreases the necessary time to collect data, improves the data consistency so all data points are collected under very similar condition, and finally frees the inspectors from the tedious task of collecting data manually. Second, we use FFT [9] to extract features out of the raw sound files. We show that by using other features other than just frequency peaks, we can use very noisy sound recordings recorded with an affordable microphone, without any physical filter required, and are able to detect voids and delamination in the concrete slab. This paper is potentially impactful, because we use K-Means [10] and Spectral Clustering [26] to analyze the data and we get very good results.

Robotic System and Software
We developed a robotic inspection system with vertical mobility which is called Impact-Rover (see Figure 1). The robot consists of two DC brushless motors for  differential drive locomotion, suction motor for climbing on concrete walls, a solenoid impactor on a sliding mechanism that is controlled by a stepper motor, specially designed embedded system control board, accelerometer with data acquisition module for IE, a microphone for IS, RealSense D435i RGBD camera for localization and Intel NUC on-board computer. Impact-Rover can collect data automatically in a pre-defined area.

Impact-Sounding and Impact-Echo Modules
The impact-sounding module consist of an Olson Solenoid for generating the impact. This solenoid can tap the surface and provide 0.011 to 0.019 Joules per hit. Sound data are recorded using 3.5 mm standard microphone. IS and IE devices both rest on the sliding mechanism to collect data at an adjustable spacing.
The IE module consist of three important parts. An accelerometer from PCB Inc. with sensitivity of 100 mV/g, frequency range (±3 dB) 0.5 to 20 kHz. Digital ICP Signal Conditioner Model 485B39 which connects through USB port and is used to transfer the data to the on-board computer. The servo motor is used to lower the IE transducer up and down to be in contact with the concrete surface for taking measurements.

Localization Module
In order to automate the data collection process, we need to determine the location of the robot with respect to the starting point and displacement of IE and IS sensors on the sliding mechanism. This way we can tag each measurement with position information. We use RealSense RGBD camera D435i to estimate the pose [11] [12][13] [14] and then fuse it with wheel's encoders to improve the localization accuracy of the robot.

IMPACT-SOUNDING BASED INSPECTION Data Processing and Feature Engineering
Due to the nature of the data and from our previous research experience [12][17] [18], we decided to use discrete Fourier Transform [19] or frequency domain analysis. This method decomposes the signal "($), where $ = 1, 2, … , + − 1 into spectral components and provides frequency information (see Equation 1). Thus, inputting a sound signal into Fourier Transform process, as output we get a function in the complex domain. This function represents the amount of each frequency present in the original signal. We perform Fast Fourier Transform (FFT) [20] on all impactsounding data we collected. For each sound signal we obtain the pair {. ! , / ! }, where / ! is the frequency vector and . ! corresponds to the amplitude vector, 2 = 1, 2, … , 3.
Feature extracting is the process of extracting non-redundant, abstract or meaningful information out of the data which better represents an underlying problem to the predictive models. Difference between sound signals collected over defective and nondefective concrete are not obvious in time-domain (see Figure 3 a, b and c). On the other hand, FFT signal shows a distinguishable pattern in energy (see Figure 3 d, e and f) i.e., magnitude of energy is lower over defects because they absorb impact energy. Our feature extraction method is inspired by [21], where we use FFT while they use Wavelet Transform [22]. Feature calculations is defined with the following expressions (see Equations 2 to 7 and Table I). These types of features capture different properties of the sound data. From mathematics we know that moments are quantitative measures related to the shape of the function. For these reasons we use spectral moments as features.

Unsupervised clustering and PCA
In this work we use unsupervised classification to classify non-defective and defective concrete. We use K-means [10], which is one of the most famous unsupervised clustering methods. Given the matrix of features 4 (see Equation 9), this algorithm partitions all observations (5 " , 5 # , … , 5 $ ) into 2 ≤ + sets Given an initial set of centroids 9 ":! " randomly initialized, algorithm [24] alternates between two steps. First, calculates the least squared Euclidean distance for each observation and assigns it to the closest centroid (see Equation 10). Second, algorithm recalculates the centroids for updated clusters (see Equation 11). Algorithm ends when 9 ":! ( ≈ 9 ":! ()" . We use Silhouette Coefficient [23] to determine the number of clusters.
Additionally, we use Principal Component Analysis (PCA) [25] for defect analysis and dimension reduction of feature matrix ;. PCA takes as input the matrix of data and calculates eigenvalues and eigenvectors, it only uses significant eigenvectors to recreate the data. We use this method in two ways. First, we use the filtering properties of this method for data processing that goes as input into K-means clustering. Second, we visualize some of the principal components and use image enhancement techniques [27] to improve the results.

EXPERIMENTS
We collected data in our concrete slab over the area that contains two types of defects, void and delamination. We collected 902 data points (.wav) with a 2cm spacing in x and y directions, over an area 20cm wide and 162cm long. Sounds are recorded using a sample rate of 44.1 kHz and a 16-bit AD. Each data point has an ID, which is tagged with the location information with respect to the starting point.

Feature Visualization
Features contain meaningful information; it helps to visualize the defects. From visualization (see Figure 5) we can clearly see the void box, while delamination circle is visible, but much less. We use enhancement on feature matrix to suppress the noise (Equation 12), leading to much clearer visualization results. For example, first and second moments are enhanced by squaring them, and power feature is enhanced by increasing it to the third power.

PCA Maps
We use PCA decomposition for the feature matrix ; (see Equation 9). For one component (C1) the amount of preserved information is 76.8%, for two components (C1+C2) 96.7% and for three (C1+C2+C3) 98.8%. For visualization we decided to use three components. We visualize all individual components and their linear combination (see Figure 6). They act as three combined predictors with equal weights. From results we can see that using PCA we are able to generate meaningful maps. Void is clearly visible, especially in the enhanced maps. Delamination is weakly detected.

Unsupervised Clustering Maps
Up to this point, we generated few types of data that can go into K-means and Spectral Clustering. First type is the feature matrix 4 (see Equation 8). Second type is enhanced feature matrix, where feature vectors are defined as 5 *+, 0 (see Equation 12). Third type is PCA. We use Silhouette coefficient to determine how good the clusters are separated, closer to 1 is better, -1 is worse. Table II shows the values for all cases when using two (defect/no defect) three clusters (void/delamination/no defect). According to the Table II, best clustering is achieved by Spectral Clustering using two clusters.
C2 C3 Figure 7. Clustering Results. a) Two clusters result matches with the ground truth and clearly finds the defects. b) Three clusters result shows that concrete around artificial defects is deteriorated. c) Scatter plots of PCA components with cluster information.
When we use ; and >@A(;, 3) as input for the K-means, the output is noisy and not very clear (see Figure 7a1, 7a2, 7b1 and 7b2). Feature enhancement improves the results (see Figure 7a1 and 7a3). Spectral Clustering uses the unmodified features ; and outputs very similar results with feature enhancement (see Figure 7a3 and 7a4). On the other hand, having three clusters differentiates the void and delamination (see Figure  7b). This figure shows parts of concrete close by artificial defects as weakened or deteriorated. Finally, 3D scatter plots are generated by using PCA decomposition with cluster information (see Figure 7c).

CONCLUSION
This paper introduces a robotic solution for data collection and an unsupervised learning method for impact-sounding data analysis. The robotic platform is equipped with visual, inertial and encoder fusion for high accuracy localization. It makes data collection much faster and improves data consistency. Data analysis algorithm starts with transforming the sound data into the spectral domain, then we calculate features. We use PCA to generate maps, then use K-means and Spectral Clustering for classification. Results from PCA maps and clustered maps can be used together to reveal defects. Experimental results show the effectiveness of our proposed method.

ACKNOLEDGEMENT
Financial support for this study was provided by the U.S. Department of Transportation under Grant No. 69A3551747126 through INSPIRE UTC. J. Xiao has significant financial interest in InnovBot LLC, a company involved in R&D and commercialization of the technology.
PCA Components Three Clusters

clusters 2 clusters
Two Clusters


Visual Abductive Reasoning

Abductive reasoning seeks the likeliest possible explanation for partial observations. Although abduction is frequently employed in human daily reasoning, it is rarely explored in computer vision literature. In this paper, we propose a new task and dataset, Visual Abductive Reasoning (VAR), for examining abductive reasoning ability of machine intelligence in everyday visual situations. Given an incomplete set of visual events, AI systems are required to not only describe what is observed, but also infer the hypothesis that can best explain the visual premise. Based on our large-scale VAR dataset, we devise a strong baseline model, Reasoner (causal-and-cascaded reasoning Transformer). First, to capture the causal structure of the observations, a contextualized directional position embedding strategy is adopted in the encoder, that yields discriminative representations for the premise and hypothesis. Then, multiple decoders are cascaded to generate and progressively refine the premise and hypothesis sentences. The prediction scores of the sentences are used to guide cross-sentence information flow in the cascaded reasoning procedure. Our VAR benchmarking results show that Reasoner surpasses many famous video-language models, while still being far behind human performance. This work is expected to foster future efforts in the reasoning-beyond-observation paradigm.

Introduction
Abduction · · · consists in studying facts and devising a theory to explain them.
- Charles Sanders Peirce (1839-1914 Abductive reasoning [51] was coined by Charles Sanders Peirce, the founder of American pragmatism, around 1865. It is inference to the most likely explanation or conclusion for an incomplete set of observations. Abductive reasoning is invariably employed in our everyday life; the generated hypothesis (H) is expected to best explain what happens before, after, or during the observation (O). Fig. 1 gives some examples. If you see O: "the road is wet", abduction * Part of this work was done when Chen Liang was an intern at Baidu. will lead you to the best explanation H: "it rained earlier" (i.e., H →O). One morning you find O: "sister leaves home hurriedly", then you conclude H: "she will be late for class" (i.e., O → H). You see O 1 : "a boy throws a frisbee out and his dog is running after it". One minute later you find O 2 : "frisbee is in the boy's hand". You can imagine H: "the dog just caught the frisbee back" (i.e., O 1 → H → O 2 ). Although abductive reasoning has long been considered as a core ability of everyday human cognition [40,57,59], it still remains an untouched domain in computer vision literature.
In this article, we propose Visual Abductive Reasoning (VAR), a novel task and dataset for investigating the abductive reasoning ability of AI systems in daily visual situations. In particular, inspired by the recent advance of causal reasoning in NLP community (i.e., abductive text generation [5] and counterfactual story revision [52]), we explore the use of natural language as the expression form to fully capture the complexity of real situations. This also better reflects the nature of human mind, which relies on linguistic thinking [38,39]. VAR requires the AI systems to describe the incomplete observation (i.e., visual premise) and write down the hypothesis that can best explain the premise. This allows to thoroughly evaluate the entire abduction procedure, as accurate understanding of the premise is the basis of abductive reasoning. Moreover, this can hasten the development of this new field, by comparing and embracing ideas for a relevant, well-established, yet different task -dense video captioning (DVC) [30]. In contrast to DVC that focuses only on describing the observation, VAR yields a new visual-linguistic reasoning paradigm -inference beyond observation. Three characteristics make VAR uniquely challenging: i) VAR needs imagination to find hypotheses outside the observation. ii) VAR seeks to discover the plausible causal structure among the observed events. iii) VAR is more related to the kind of human reasoning in daily situations where the information at hand is often incomplete [26] and absolutely certain conclusions cannot be reached [5,27].
Our dataset is collected to address the characteristics of the VAR task (cf. §3). It contains 9K examples from 3,718 videos. Each example consists of several chronologicallyordered events, most of which are logically related. For each event, abduction oriented description is written by people, and its role of premise or explanation is also annotated. When presenting each example to the AI system, the explanation event is masked and premise events are visible. The AI system is required to understand the partial, noisy observations (i.e., premise events) and construct the most plausible explanatory hypothesis -accurately describing both the observable premise events and the masked explanation event. The examples are on average 37.8s long, with 4.17 events, and harvested from diversely event-rich sources, i.e., YouTube Lifestyle videos, movies and TV shows.
To lay a solid foundation for future efforts, a new model, named REASONER (causal-and-cascaded reasoning Transformer), is proposed (cf. §4). Specifically, REASONER is building upon a Transformer encoder-decoder architecture. In the encoder of REASONER, a contextualized directional position embedding strategy is adopted to capture causal dependencies among the premise events. Hence the context of the premise events can be gathered in a causality-aware manner, enabling REASONER to learn discriminative representations for the premise and explanatory hypothesis. Then REASONER cascades a set of decoders for premise/hypothesis sentence production and refinement. For each generated sentence, the associated prediction score is viewed as the confidence and embedded into the next decoder as a signal for inspiring more information to flow from highconfident sentences to the low-confident ones. This leads to a confidence-guided multi-step reasoning strategy, boosting the reasoning power of REASONER eventually.
Extensive experimental results are provided in §5. First, to comprehensively probe deep neural models on this challenging task, we establish a group of baselines based on current top-leading DVC models. The benchmarking results on VAR dataset show that REASONER outperforms the best competitor by a large margin, e.g., 33.44 vs 28.71 in terms of BERT-S, but is still far behind human performance (42.96). This shows that VAR is especially challenging for current video-language models. Then a set of user studies and ablative experiments are conducted for a thorough evaluation. For completeness, we further test REASONER on the DVC task and confirm again its superiority. Concurrent to us, [17] studies image-based abductive reasoning: AI systems are required to identify, ground, or compare given inferences. Overall, we feel vision-based abductive reasoning is an intriguing problem worthy of exploring.

Related Work
Dense Video Captioning (DVC). Different from the classic video description task [29,46,67,68,76,80] that aims to describe a short video clip using a single sentence, DVC is to comprehensively describe all the events in an untrimmed video through a multi-sentence paragraph [30]. Typical DVC models [30,[43][44][45]60,74,81,82] follow a two-stage, bottomup paradigm: first parse a video into several temporal events and then decode a description from each detected event. As the problem of event detection is ill-defined [10], some alterative solutions either adopt a single-stage strategy to simultaneously predict events and descriptions [35,71], or turn to a top-down regime: first generate paragraphs, and then ground each description to a video segment [10,37]. A few other methods [23,32,50] focus purely on generating better paragraph captions from a provided list of events.
Both VAR and DVC are concerned with video-based text generation; a part of our dataset is sourced from Ac-tivityNet Captions [30], a famous DVC dataset. However, DVC is aware of general fact based plain narrative, while VAR addresses cause-effect chain based abductive reasoning. Rather than accurately understanding what is observed, VAR further requires invoking what might have happened or will happen. In our experiments, we involve several recent DVC models as baselines for our VAR task and also report the performance of our REASONER on the DVC task. Context-Aware Text Generation. Our work is also related to some context-aware text generation tasks in the NLP literature. For instance, text infilling [83], also known as the cloze task [64], is to generate a span of missing tokens in a text chunk, while sentence/story infilling [19,21] aims to generate missing sentences in long-form text. The generated tokens/sentences are expected to smoothly blend into and fit the context syntactically [83], semantically [19,21], and logically [24]. Counterfactual story revision [52] requires generating a new ending, given a story context altered by a counterfactual condition. Our work draws inspiration from abductive text generation [5], which investigates abductive reasoning via a natural language inference task: write an appropriate reason that could explain observations described by narrative text. Unlike these language tasks addressing inter-sentential relationship understanding only, our VAR task requires abduction and narrative for a sequence of partially observable visual events. Moreover, our VAR task setting is more general; it is not limited to the strict form of abductive reasoning in [5] Visual Future/State Prediction. Our work is, to some degree, relevant to future prediction -a popular research area in computer vision. In this area, a huge spectrum of topics/tasks are put forward, including forecasting future frames [42,70], future features [62,69], future actions [1,28,31,55,61], future human motions [14,22,41], future human trajectories [2], future goals [13], etc. Rather than studying the future generation at the semantic-category or color-pixel level, event-level prediction was recently addressed in [34] and [49]. However, [34] only requires choosing from two candidates for future event prediction, making the take less challenging. [49] targets to describe past, present, and future events for a single image, while our VAR task requires making full use of the information from a set of premise events. There are also some efforts made towards understanding the dynamics/transformations between states [18,48,75] or goal-conditioned procedure planning [7], while either relying on a pre-defined, limited action prediction space [7,18], or using simulated environments [7,18]. Our VAR task is essentially to discover and describe the causal relations in real visual environments. It is not constrained to a narrow view of predicting either future events or between-state changes, but tries to infer the missing parts in the cause-effect chains, even with some unrelated premise events. All of these together make VAR a unique and challenging visual reasoning task. Position Encoding in Transformers. Due to the permutation invariant nature of the attention operation, [58] learns and encodes position embeddings into Transformer tokens. Subsequent language-Transformers hence explore further variations, like incorporating sinusoid prior with more parameters [9], simplifying position embeddings as learnable scalars [53], disentangling special tokens ([CLS]) [25], etc. Some recent vision-Transformers [8,73] consider directional relative distance between 2D positions, and/or the interactions between visual tokens and position embeddings. REASONER encodes the relations of premise events in a directional and contextualized manner for causal relation modeling, and leverages the prediction scores of descriptions for confidence-guided multi-step reasoning.

VAR Task
Our visual abductive reasoning (VAR) task is designed to test the abductive reasoning ability of machine intelligence in everyday visual environments E. Formally, given a video example V ⊂ E that contains a set of N events, i.e., V = {O 1 , · · ·, O n−1 , H, O n , · · ·, O N −1 }, which are logically related and chronologically organized, we denote {O n } N −1 n=1 as premise events -partial observation of E, and H as explanation event that can best explain the premise events. Conditioning on the past and/or future visual con-  text {O n } N −1 n=1 only, the AI system is asked to describe these premise events, and, more importantly, infer and write down the most plausible explanatory hypothesis for the premise. Naturally, such a hypothesis is expected to be consistent with the content of the explanation event H. The abduction ability can thus be thoroughly examined by assessing the quality of both the premise-based descriptions and explanatory hypothesis sentences -adequent understanding of the premise is a necessary prerequisite for abductive reasoning.

VAR Dataset
Guided by the above task setup, we build a large-scale dataset for VAR. Fig. 2 depicts an illustrative example.

Dataset Collection
Data Source. VAR dataset is collected from three sources: • 23,457 Youtube lifestyle Vlog videos from ActivityNet Captions [30] and VLEP [34] datasets. These videos cover rich social scenarios and human activities. • 13,799 TV show and movie videos from TVC dataset [33] and a famous Youtube channel, Fandango MovieClips 1 . These videos are key scenes in popular TV shows and films containing wide-ranging genres. YouTube videos include diverse daily events, but last relatively short durations with short intervals (about minutes). While TV shows and movie videos usually have limited scenarios, they contain rich artificial cause-effect chains in their story-lines and last relatively long durations with long intervals (spanning even years). Thus gathering these videos together makes our dataset a good testbed for VAR. Data Cleaning. The collected videos are accompanied by event labels, and videos containing only one single event are first dropped. Then, for each of the rest videos, three human experts are invited to examine if there exists cause-effect relations between the video events. We only preserve qualified ones with more than two votes in the affirmative, finally resulting in 3,718 videos in total for further annotation.

Dataset Annotation
For each video in VAR, the annotation contains three steps: Step 1: Event Type Annotation. For an event E of video V, if E can well explain some other events in V; or in other words, if we can imagine that E could happen by only considering the other events V/E, event E will be labeled as  explanation and the other events V/E will be labeled as premise. Fig. 2 gives an example. For the video containing three events: E 1 "a man falls off a running horse", E 2 "the man lies on the ground and gets hurt", E 3 "the man is taken to the hospital", we can derive two legal examples for our VAR dataset: Step 2: Abductive Reasoning Aware Description Annotation. Although some videos are collected with event-level descriptions/plot summaries, we re-annotate all the events with abductive reasoning oriented descriptions. Specifically, instead of capturing all the visual details in video captioning, like {"a boy in a black jacket plays frisbee with his white dog in a park", "the dog catches the blue frisbee and runs back", "the boy smiles, takes the frisbee and pats the dog}, our descriptions are only aware of describing the visual content related to abductive reasoning, like {"a boy throws a frisbee out and his dog is running after it", "the dog catches the frisbee back", "the boy gets the frisbee"}.
Step 3: Validation. Finally, each annotated example is examined by three verifiers: the verifiers are shown with only the premise events and language-based explanation (i.e., description on the explanation event), and vote for: "Is the explanation sound?". If an example wins majority approval, it will be accepted; otherwise, it will be relabeled or dropped.

Dataset Features and Statistics
To offer deeper insights into our VAR dataset, we next discuss its distinctive properties and detailed statistics. Abductive Reasoning Orientated. VAR is the first dataset that underpins machine intelligence study of abductive reasoning in visual daily scenarios. It is designed to reason beyond visual premise for a plausible explanation, distinguishing it from existing video-language datasets/tasks. Diversity. To capture diverse cause-effect relations and abduction cases, our VAR dataset covers i) various daily events/activities, e.g., work, leisure, household; ii) rich scenarios, e.g., lifestyle recording, scripted drama; iii) different durations and intervals, ranging from minutes to years. Large-Scale. As shown in Table 1

Methodology
Problem Statement. Given a video V with N temporally ordered events, i.e., , and explanation event, i.e., H, are logically related. The AI system is only presented with a partially observable version of V, i.e.,Ṽ = {O 1 , · · ·, O n−1 ,H, O n , · · ·, O N −1 }, whereH is obtained by setting all the pixel values of H as zero. The AI system is required to not only describe the premise, but also reason about the most likely explanation for the premise, i.e., generate N sentences S={S O n } N −1 n=1 ∪S H that describe the content of the N events in V, while conditioning onṼ only: where w l is the l-th word in a generated sentence S ∈ S. It is worth mentioning that, when H = ∅, our VAR task is degraded into a classic DVC task [30] which focuses only on describing the content of observed events {O n } N −1 n=1 . Core Idea. Building upon a Transformer encoder-decoder architecture (Fig. 4), our REASONER is aware of two core challenges posed by the VAR task: i) inferring cause-effect relations, and ii) reasoning beyond the partial observation. To address i), a contextualized directional position embedding strategy is adopted to capture causal relations residing in the input videoṼ, leading to a Causality-aware encoder ( §4.1). To accommodate ii), a confidence-guided multistep reasoning strategy is developed, i.e., utilize the prediction scores of sentences to guide cross-sentence information flow, yielding a cascaded-reasoning decoder ( §4.2).

Causality-Aware Encoder
For notational simplicity, we redefine the partially observable videoṼ={O 1 , where E h refers to the masked explanation eventH, and {E n } =h indicates the visible, premise events {O n } N −1 n=1 . Let us denote the initial features of the N events as For each premise event E n =h , corresponding feature E n =h is obtained by aggregating the visual features of its frames. For the masked explanation event E h , we set E h = 0 d . The Causality-aware encoder is to leverage the context from the past and/or future observable events {E n } =h to reinforce their own representations as well as posit a meaningful representation for the most likely explanatory hypothesis, i.e., the masked explanation event E h .
The attention operation is the core of Transformer: where the output Y ∈ R N ×d is with the same length N and embedding dimension d as the input X ∈ R N ×d , and  Figure 4. Network architecture of REASONER. See §4 for more details. W q,k,v ∈ R d×d project the input into query, key, and value matrices, respectively. As the attention computation is invariant with respect to reordering of the inputs, explicit position encoding is widely adopted, in two typical ways: i) Absolute position encoding [65]: each position n is assigned an embedding, i.e., U n =F Abs (n) ∈ R 1×d , and the position embeddings are directly added to the input, i.e., X ← X+U . F Abs (·) can be a linear projection [12], a sinusoidal function [65], etc. ii) Relative position encoding [58]: the position embeddings are constructed considering the pairwise relationships between positions, i.e., U nm = F Rel (n, m) ∈ R. Contextualized Directional Position Embedding. Since the VAR task is essentially aware of the plausible chains of cause-effect, the relative ordering of the input events matters. We continue in the vein of relative position encoding [58,73] and adopt a contextualized directional position embedding strategy, i.e., U nm = F Rel (n, m, X n ) ∈ R: where R ∈ R (2N −1)×d is a learnable matrix, and (·, ·) is a directional indexing function, i.e., (n, m) = (m, n). The directional projection F Rel is conditioned on the visual context, i.e., X n , since the causal dependency between events is typically related to specific content, e.g., when we see people are laughing, we tend to look back only a short time into the past to figure out the reason; when we see a man falls off his horse, we worry about whether he gets hurt and the impact on his future life. Some more visual examples regarding our contextualized directional position embedding strategy can be found in Fig. 5. Then, U∈R N×N is injected by manipulating on the attention matrix A∈R N×N : We further set A nh = 0 to encourage leveraging the context from the observable events {E n } =h to infer the masked explanation event E h , rather than vice versa. The Causalityaware encoder in REASONER is therefore achieved by stacking several Transformer encoder blocks [65] with our contextualized directional position embedding strategy. We denote the output event representations as {Ṽ n ∈ R d } N n=1 .

Cascaded-Reasoning Decoder
With the discriminative representations {Ṽ n } N n=1 of the observable premise events {O n } N −1 n=1 as well as the explanatory hypothesisH, the cascaded-reasoning decoder first generates a descriptive sentence for each event/hypothesis individually, and then refines all the sentences in a comprehensive, confidence-guided, and step-by-step manner. Initial Description Generation. For each event representa-tionṼ n ∈ R d , a multi-modal, masked Transformer decoder is first adopted for initial description generation: where H n ∈R Ln×d is a set of L n words embeddings. During training, it is computed over the groundtruth description, i.e.,Ŝ En , and masked attention [65] is adopted to prevent the leakage of future words. During inference, it is recurrently generated. Learnable modal-type embeddings [11,32] are also added into the input yet omitted for brevity. By fusing visual and linguistic representations as the input, D 0 conducts cross-modal reasoning, and hence generates improved event representation, i.e.,Ṽ 0 n ∈ R d , and updated visual-linguistic state, i.e., H 0 n ∈ R Ln×d , for each event E n . Then a captioning head is adopted to map H 0 n into word distribution. The probability of l-th word is given as: where Ω ∈ R |Ω|×d is the embedding matrix of the word vocabulary Ω, and H 0 n (l)∈R d denotes l-th vector of H 0 n . As standard, the description S 0,En = {w En l } Ln l=1 for event E n is generated by greedy prediction, and we set the averaged prediction score as the confidence: c 0 n = 1 Ln i P (w En l ). Iterative Description Refinement. To better respond to the fundamental challenge of VAR task in reasoning beyond observation, we further cascade several Transformer decoder blocks over D 0 for iterative description refinement. This allows REASONER to make full use of both visual and linguistic context from the past and/or future observable events, and improves the explanatory hypothesis in a stepby-step manner, boosting the reasoning ability eventually.
Sentence from Sentence from time : He is using a brush to polish the wood surface.
: He is using a brush to cover each picket in a stain.
: He then wipes the table with a towel.
: The picket is stained and is waiting to dry. Figure 6. Sentences from the cascaded-reasoning decoder ( §4.2).
Specifically, our whole refinement procedure can be defined in a recursive, confidence-guided form: where D k refers to k-th refinement module and all the refinement modules are weight-sharing Transformer de- In this way, each D k can leverage inter-sentential relationship in previously generated descriptions {h k−1 n } N n=1 for refinement and better reason about the explanatory hypothesis. Moreover, we introduce the event confidence, i.e., {c k n } N n=1 , as a kind of bias into the refinement procedure: leverage the information from those more confident descriptions to help improve the predictions with relatively lower confidence. Without causing ambiguity, we denote X as the input of the decoder and omit the superscript k. For each input "token" X i , its confidence score c ni is the one of its sourced event E ni , and we normalize {c n } N n=1 over all the N events. Analogous to Eq. 4, the attention computation in D k is modified as: where the learnable vector r c ∈ R 2B−1 can be viewed as a bucket to store the relative confidence weight; and the directional indexing function ι(·, ·) is given as ι(c ni , c nj ) = c ni ·B − c nj ·B +B. With such confidence-guided decoding scheme, descriptions are refined by intelligently gathering context from more reliable sentences, while ignoring noisy cues from less confident ones. By stacking several such decoders {D k } k , outputs will be progressively improved (Fig.6). Related experiments can be found in §5.2.

Training Objective
Given the groundtruth sentences {Ŝ En } N n=1 corresponding to the N events {E n } N n=1 of videoṼ, REASONER is trained by minimizing the negative log-likelihood over the outputs of the cascaded-reasoning decoder {D k } K k=0 : Ln l=1 P (ŵ En l |ŵ En <l , H k n ), (9) whereŜ En ={ŵ En l } Ln l=1 . As the teacher forcing scheme [72] is used for training, H n in Eq. 5 and 7 is embedded over one-hot encoded groundturth words {ŵ En l } l . We further adopt a hypothesis reconstruction based optimization criterion, to provide the encoder with more explicit supervision signals for explanatory hypothesis reasoning: whereṼ h andV h are embeddings for the explanatory hypothesis obtained from the masked and original videos, i.e., V and V, respectively, and F Proj is a projection head, based on a small multi-layer perceptron. This auxiliary training objective forces REASONER to "imagine" an effective rep-resentationṼ h that better aligns with the original content of E h .V h is from the momentum version of the encoder.

Implementation Details
Details on implementing the algorithm are as follows: • Detailed network architecture: The encoder ( §4.1) of REASONER is implemented as two Transformer encoder blocks, and each decoder module ( §4.2), i.e., D k , is implemented as two Transformer masked decoder blocks. They have d = 768 hidden size and 12 attention heads. We use a bucket size B = 10 to quantize confidence scores (Eq. 8).
We stack a total of K = 3 decoders for cascaded reasoning.
• Data preprocessing: For each video event, action/appearance features are pre-extracted using ActivityNet [6] pretrained ResNet200 [15]/BN-Inception [20], as in [32,71,82]. We uniformly sample 50 frames per event and concatenate their features as the corresponding event representation which is denoted in a vector form in §4.1-4.3 for ease of notation. Sentences are padded or truncated into 20 words.
• Training/Inference: For the first decoder D 0 , we adopt scheduled sampling [4] to make the later decoders fully trained. The coefficient between the main and auxiliary training objectives is set as 0.2. During inference, the final descriptive sentences are generated from the last decoder D K , using deterministic decoding, i.e., greedy search. All the experiments are conducted on 2 NVIDIA GeForce RTX 2080 Ti GPUs with a 11GB memory per-card.

Experiments
We first provide benchmarking results on our VAR dataset ( §5.1). Then, to verify the efficacy of our core model designs, we conduct a set of diagnostic studies ( §5.2). Finally, for comprehensive evaluation, we test our REASONER on the classic, dense video captioning (DVC) task [30] ( §5.3).

Performance on VAR Task
Competitor. We benchmark five top-leading DVC models on VAR to reveal the abductive reasoning ability in existing techniques. They include three Transformer-based [9,32,82] and two RNN-based [71,74] models, which are trained on train set of our VAR dataset with pre-provided event segments using their original training protocols.  Table 3. User study of pairwise model preference (%). "Neutral" means A and B models are "equally good". Percentage of "equally bad" are omitted. See §5.1 for details.
Quantitative Result. Table 2 summarizes the benchmarking results on the test set of our VAR dataset. For detailed analysis, we report the performance over the observable premise events and invisible explanation events separately. Moreover, to probe the upper bound of model performance, we evaluate human performance by asking ten volunteers to perform VAR. Specifically, we randomly sample 500 examples from unique videos in VAR test. The volunteers are only provided with partially observable videos and requested to write down the corresponding descriptions and hypotheses. The human-written descriptions and hypotheses are evaluated by the automatic metrics, and evaluation scores are shown in the first row of Table 2. Several essential conclusions can be drawn from Table 2: i) Humans are good at VAR; although human-written hypotheses for explanation scored lower than the descriptions for the visual premise, they are still very plausible in absolute terms. ii) All traditional DVC models [9,32,71,74,82] struggle with VAR that humans excel at. Their generated hypotheses are usually untrusted, and far worse than their created premise narratives. This suggests that existing video-based language generation models are not good at reasoning beyond observation. iii) Our REASONER outperforms other AI models [9,32,71,74,82], in both explanatory hypothesis reasoning and premise description, demonstrating the effectiveness of our whole model design. Compared to other AI models, REASONER also yields a relatively smaller performance drop, from premise description to hypothesis reasoning. This suggests that REASONER can make better use of the context of observed events to infer the explanatory hy-pothesis. iv) Although our REASONER shows more promising results, there still remains a significant gap from human performance, that is waiting for more sophisticated abductive reasoning models to conquer.
User Study. For comprehensive performance assessment, we further carry out a subjective evaluation, based on pairwise model comparison. Specifically, we randomly sample 500 examples from unique videos in VAR test. Three volunteers are presented the outputs of a pair of systems (i.e., REASONER vs PDVC [71] or human) on the sampled examples, and requested to do a comparison about which one is better, or "equally good" or "equally bad". The human preference results are collected in Table 3, and again the statistics for premise events and explanation events are presented separately. The human subjective judgments are generally accordant with the trends reflected by Table 2. Specifically, the human pairwise comparison results confirm the superiority of REASONER over PDVC, the second-best model in Table 2: REASONER receives 34.2 and 29.9 percent preference votes on the premise description and explanatory hypothesis, respectively. However, human-written hypotheses and descriptions are much more favorable than our results, showing again VAR is a very challenging task.
Qualitative Analysis. A test video example in VAR dataset is shown in Fig. 13. It contains the explanatory hypotheses and premise descriptions from our REASONER and other competitors [32,71,82] as well as groundtruth sentences. We can find that our REASONER is able to discover and correctly describe the cause-effect chain, and hence generate a plausible hypothesis, i.e., making a small splash, that well explains the observed events, i.e., standing on the podium. In contrast, other competitors typically produce unsatisfactory results, especially for the explanatory hypothesis.

Diagnostic Experiment
A set of ablative studies is conducted on VAR test for indepth analyzing each component in our REASONER, using BLEU@4, CIDEr and BERT-S metrics, averaged over all the events. Key Component Analysis. We first study the efficacy of core model components. The first row in Table 8a gives the performance of a basic Transformer model, which simply uses ab- solute position embedding in the encoder and only adopts one single decoder, i.e., D 0 . The results in the first two rows reveal that contextualized directional position embedding ( §4.1) consistently improves the performance over the three metrics. Moreover, from the first and third rows we can observe that confidence-guided multi-step reasoning ( §4.2) indeed boosts the performance. By further considering the scores in the last row, we can safely conclude that combining the two model designs together leads to the best results. Contextualized Directional Position Embedding. Next, to thoroughly study the impact of our contextualized directional position embedding strategy ( §4.1), we report the performance of two alternatives in Table 8b. Specifically, "absolute" refers to the widely used, learnable absolute position embedding, while "directional" indicates learning relative position embedding without considering any input context. As seen, our contextualized directional position embedding is significantly better than the two alternatives. Cascaded Reasoning. Table 8c reports the performance with different steps of our cascaded reasoning ( §4.2), i.e., K = {0, 1, · · ·, 5}. When K = 0, only one decoder D 0 is adopted and the CIDEr score is just 32.72. However, after adding an extra refinement decoder, the score is greatly improved to 36.13. The increasing trend is gradually saturated until K > 3. We therefore use K = 3 as our default setting for balancing performance and inference efficiency. Confidence Embedding. We inject sentence scores into the cascaded reasoning for guiding information flow (Eq. 8). As shown in Table 8d, removing confidence embedding hinders the performance, e.g., 36.13→35.22 in terms of CIDEr. Training Objective. Finally we examine our training objective design ( §4.3). Table 8e demonstrates a beneficial impact of the hypothesis reconstruction loss L Aux (Eq.10).

Performance on DVC Task
For completeness, we report performance on DVC task. Dataset. As a gold-standard dataset for DVC, ActivityNet Captions [30] Table 5. Quantitative results ( §5.3) on the ae-val set of Activi-tyNet Captions [30]. The scores are mainly borrowed from [71].
(10,009/4,917/5,044 for train/val/test). Each video lasts 120s and is annotated with 3.65 temporally-localized sentences on average. Following [32,60,82], val set is further split into two subsets: ae-val with 2,460 videos and ae-test with 2,457 videos without overlapping. Evaluation Metric. As in [32,60,82], BLEU@4 [47], ME-TEOR [3], and CIDEr [66] metrics are used for evaluation. Quantitative Result. REASONER is trained on the train set and evaluated on ae-val set in paragraph-level. Since we focus only on descriptive quality, the sentences are generated from a provided list of events, like in [23,32,50]. As shown in Table 5, REASONER outperforms state-of-the-art DVC models over all the metrics, e.g., +2.81 performance gain in CIDEr. This proves the strong reasoning ability of REASONER and emphasizes the value of our VAR task in promoting innovations of powerful video-language models.

Conclusion
We introduce VAR (Visual Abductive Reasoning) -a novel task that investigates the abductive reasoning ability of machine intelligence in the visual world. We establish REASONER, a new Transformer based visual-language model, which captures the context from visual premise in a causality-aware manner, and generates premise descriptions and hypothesis sentences in a confidence-guided, step-bystep fashion. REASONER shows promising results on both VAR and dense video captioning tasks. We also observe a remaining large headroom for AI systems in VAR, which is expected to encourage exciting avenues in the future.
In this work, we build a headway of Visual Abductive Reasoning (VAR) as a new task and introduce a large-scale dataset for VAR that scaffolds the investigation of Abductive Reasoning ability in AI systems. Along with them, REA-SONER is further presented as a modest solution. In the supplemental material, we provide the following items that shed deeper insight on the aforementioned contributions:

A.1. Detailed Dataset Statistics
Our VAR dataset is curated from three main sources, i.e., YouTube Lifestyle video, movie and TV show, in Fig. 8b and Fig. 8c, we illustrate the detailed distribution of videos and examples by collected sources. As seen, most videos in VAR are from YouTube Lifestyle video, while movie videos tend to have more events, and thus leads to more complicated causal structures and more examples. We then study the distribution of frequently used words in VAR descriptions, which is illustrated as a word cloud in Fig. 8a. More frequent words are shown in larger font size. Finally, the distribution of premise events is shown in Fig. 8d.

A.2. Detailed Annotation Process
In the annotation process, each video is passed at least four times: (1) A first quick pass for filtering out videos without cause-effect relations; (2) The second pass for annotating the event type, i.e., the premise and explanation. In this phase, the entire video with initialized events and descriptions are all shown to the experts. We use the original event annotation provided in [30,33,34] to initialize event boundaries, while they might contain noise annotations. We thus request human experts to i) edit event boundaries when the initial separation can not well fit the description; ii) delete duplicate events when they are overlapped; iii) add additional events when they find a missing part in the cause-effect chain. And the annotation interface is shown in Fig. 9. After that, experts are requested to further annotate the event type based on the events he selected in the previous step, as shown in Fig. 11. (3) The third pass for abductive reasoning oriented description annotation. Human experts are only shown with premise visual events while the hypothesis event is hidden. Annotation interfaces for annotating the premise and hypothesis are shown in Fig. 10 and Fig. 12 respectively. Experts might vote to delete an example if they find that a plausible explanation can not be inferred from the premise. And the remaining examples are re-annotated with abductive reasoning oriented descriptions. Notably, a video might contain multiple examples (candidate cause-effect chains), and we manually control the example distribution to make sure the same video will not be shown to the same expert twice. (4) Forth, the final pass for validation. Both the annotated descriptions for premises and explanations for hypotheses are shown to another group of human experts. And they will vote for the validity.

B. Additional Details of Baselines
We benchmark five top-leading DVC models [9,32,71,74,82] on VAR. In this section, we detail the implementation and training protocol of these baseline methods.  MFT MFT [74] is an LSTM-based method that consists of a selection LSTM for relevant event filtering and a captioning LSTM for coherent sentence generation. We adapt it to VAR task by recurrently passing the given events into MFT and the selection LSTM is transformed into a visually coherent maintaining module. The unidirectionally casual structure can be captured that enables the inference on potential effect along the temporal order.
PDVC Similarly, PDVC [71] employs an LSTM-based captioning decoder, while it is conditioned on a deformable soft attention aggregated visual event. And the visual events are embedded with a Transformer-based encoder that could also capture bidirectional causal dependencies within it.  VTrans VTrans [82] is a fully attentional model that originates from the vanilla Transformer proposed in [65]. We follow the implementation in [32], which serves as a baseline that only considers a single event and independently generates a single sentence describing the given event. Thus causal structure can not be formulated in this method.
Trans-XL Transformer-XL (Trans-XL) [9] is originally proposed for modeling unlimited longer-term dependencies with a segment-level recurrent strategy. It can capture the intrinsic unidirectional causal structure within recurrent steps. Following the implementation in [32], gradients can flow through recurrent steps instead of being stopped. This enables stronger long-term modeling. MART MART [32] is also built on a fully Transformerbased encoder-decoder architecture, that maintains a summarized memory module to model dependencies among events. Similar to Trans-XL, the unidirectional causal structure is preserved in the memory. Whereas, experimental results show that it suffers more on our VAR potentially due to the content drift brought by masked visual hypotheses.
MFT and PDVC are benchmarked following the original training protocols. We adapt VTrans, Trans-XL and MART to the VAR task with the implementation provided by [32]. All of these baselines are trained with given events from our VAR train and evaluated on VAR test under the same setting of REASONER as reported in our main paper.

C. Additional Experimental Results
To shed light on the essence of both the Visual Abductive Reasoning task and our VAR dataset, we study two edge cases of the main setting: i) First, all events are made available to the model, so that no abductive reasoning is needed. And the VAR task is degraded to a basic Dense Video Captioning task. ii) Second, only ground-truth linguistic descriptions of premise events are supplied to the models. Therefore, models are expected to conduct abductive reasoning with and only with linguistic cues.
In Table 6, we summarize the quantitative results of these two settings. As seen, when there is no hidden event, i.e., the incomplete causal structure is directly provided, REA-SONER achieves even better performance. It reveals that fulfilling explanation events through abductive reasoning is indeed challenging and the causal structure understanding is also helpful to basic visual recognition. And for the next setting, when only premise texts are given, comparing to fully utilize both visual and linguistic cues, REASONER can not well-infer the hypothesis within linguistic modality only, which proves that the visual-based abductive reasoning is indispensable in the VAR task.
In our main paper, for the benchmarking results, we report the average scores of ten trained models with different random seeds. To prove the statistical significance of our results, here we further provide the corresponding standard deviations of REASONER and two representative methods [32,71] in Table 7.  Table 7. Average scores and their standard deviations of REA-SONER and two representative methods [32,71] ( §C).

D. Additional Qualitative Visualization
In Fig. 13-14, we show more qualitative examples from VAR test following Fig. 7 in the main paper. Generated sentences from competitors [32,71,82] along with our REA-SONER are presented in Fig. 13. In contrast to the competitors, both adequate descriptions on premises and plausible inferences for hypotheses are observed for our proposed method, REASONER, which demonstrates a superior abductive reasoning ability for capturing causal structures among visual events. Some failure cases and gold human-written explanations are shown in Fig. 14. Even though REASONER shows impressive performance on inferring with abduction, VAR task is still a mostly unsolved technical problem. And there remains a large headroom for future works to conquer.

E.1. Dataset Limitation
During annotation process, we observe a bias against women and minorities due to the highly biased nature of movie and web sourced video data [54,56,63]. VAR, derived from these data, inevitably runs into the same problem [16,77]. We thus suggest that models trained on VAR dataset should be cautiously examined before being deployed onto real-world applications. And we will devote further efforts to mitigating the issue in our later works.

E.2. Details of BERTScore Evaluation
BERTScore [79] leverages the pre-trained contextual embeddings from BERT-based models for similarity measurement. Thus the evaluated scores vary a lot with different model settings. In this paper, all reported BERTScores are evaluated under a hash code version: roberta-large L17 noidf version=0.3.0(hug trans=2.3.0)-rescaled. We encourage later works to follow the same setting for a fair comparison. A static version of BERTScore is released at: https://github.com/leonnnop/VAR.

F.2. Concerns on Personal Data Collection
VAR is annotated by human experts and we conduct user studies to evaluate the human-subjective generation quality. All human experts are noticed that the annotation and evaluation will be used for academic research and individual consents are reached with signed agreements. The annotation will not leak any personal information about the experts.

F.3. Potential Societal Impact
Endowing an AI system with human intelligence has long been dreamed by AI researchers, which could fundamentally change the experience of human-machine interaction. VAR takes an important step towards more human-like AI systems that are endowed with abductive reasoning ability, while it might provoke concerns about disinformation, e.g., fabricating deceptive facts. We encourage more technical researching efforts devoted to fake content detection, and at the same time, we will organize a gated release of our dataset and model to prevent potentially malicious abuses.


SkillSpan: Hard and Soft Skill Extraction from English Job Postings

Skill Extraction (SE) is an important and widely-studied task useful to gain insights into labor market dynamics. However, there is a lacuna of datasets and annotation guidelines; available datasets are few and contain crowd-sourced labels on the span-level or labels from a predefined skill inventory. To address this gap, we introduce SKILLSPAN, a novel SE dataset consisting of 14.5K sentences and over 12.5K annotated spans. We release its respective guidelines created over three different sources annotated for hard and soft skills by domain experts. We introduce a BERT baseline (Devlin et al., 2019). To improve upon this baseline, we experiment with language models that are optimized for long spans (Joshi et al., 2020; Beltagy et al., 2020), continuous pre-training on the job posting domain (Han and Eisenstein, 2019; Gururangan et al., 2020), and multi-task learning (Caruana, 1997). Our results show that the domain-adapted models significantly outperform their non-adapted counterparts, and single-task outperforms multi-task learning.

Introduction
Job markets are under constant developmentoften due to developments in technology, migration, and digitization-so are the skill sets required. Consequentially, job vacancy data is emerging on a variety of platforms in big quantities and can provide insights on labor market skill demands or aid job matching (Balog et al., 2012). SE is to extract the competences necessary from unstructured text.
Previous work in SE shows promising progress, but is halted by a lack of available datasets and annotation guidelines. Two out of 14 studies release their dataset, which limit themselves to crowd-sourced labels (Sayfullina et al., 2018) or annotations from a predefined list of skills on the document-level (Bhola et al., 2020). Additionally, * Equal contribution.
You will thrive working in a Dev/Sec Ops culture . none of the 14 previously mentioned studies release their annotation guidelines, which obscures the meaning of a competence. Job markets change, as do the skills in, e.g., the European Skills, Competences, Qualifications and Occupations (ESCO; le Vrang et al., 2014) taxonomy (Section 3). Hence, it is important to cover for possible emerging skills.
We propose SKILLSPAN, a novel SE dataset annotated at the span-level for skill and knowledge components (SKCs) in job postings (JPs). As illustrated in Figure 1, SKCs can be nested inside skills. SKILLSPAN allows for extracting possibly undiscovered competences and to diminish the lack of coverage of predefined skill inventories.
Our analysis ( Figure 2) shows that SKCs contain on average longer sequences than typical Named Entity Recognition (NER) tasks. Albeit we additionally study models optimized for long spans (Joshi et al., 2020;Beltagy et al., 2020), some underperform. Overall, we find specialized domain BERT models (Alsentzer et al., 2019;Lee et al., 2020;Gururangan et al., 2020;Nguyen et al., 2020) perform better than their non-adapted counterparts. We explore the benefits of domain-adaptive pretraining on the JP domain (Han and Eisenstein, 2019;Gururangan et al., 2020). Last, given the examples from Figure 1, we formulate the task as both as a sequence labeling and a multi-task learning (MTL) problem, i.e., training on both skill and knowledge components jointly (Caruana, 1997). Table 1: Contributions of Related Work. We list the recent works of Skill Extraction. Note that (*) indicates labels that are automatically inferred from some source (e.g., a predefined skill inventory) and not manually annotated. With respect to the annotation approach, "Manual" indicates uncertainty whether they used domain experts or not. Also note that many works do not release their dataset with annotations () nor guidelines (). The list is inspired by Khaouja et al. (2021).
Contributions In this paper: 1 We release SKILLSPAN, a novel skill extraction dataset, with annotation guidelines, and our open-source code. 1 2 We present strong baselines for the task including a new SpanBERT (Joshi et al., 2020) trained from scratch, and domain-adapted variants (Gururangan et al., 2020), which we will release on the HuggingFace platform (Wolf et al., 2020). To the best of our knowledge, we are the first to investigate the extraction of skills and knowledge from job postings with state-of-the-art language models. 3 We give an analysis on single-task versus multi-task learning in the context of skill extraction, and show that for this particular task single-task learning outperforms multi-task learning.

Related Work
There is a pool of prior work relating to SE. We summarize it in Table 1, depicting state-of-the-art approaches, level of annotations, what kind of competences are annotated, the modeling approaches, the size of the dataset (if available), type of skills annotated for, baseline models, and whether they release their annotations and guidelines.
As can be seen in Table 1, many works do not release their data (apart from Sayfullina et al., 2018 andBhola et al., 2020) and none release their annotation guidelines. In addition, none of the previous studies approach SE as a span-level extraction task with state-of-the-art language models, nor 1 https://github.com/kris927b/SkillSpan did they release a dataset of this magnitude with manually annotated (long) spans of competences by domain experts.
Although Sayfullina et al. (2018) annotated on the span-level (thus being useful for SE) and release their data, they instead explored several approaches to Skill Classification. To create the data, they extracted all text snippets containing one soft skill from a predetermined list. Crowdworkers then annotated the highlighted skill whether it was a soft skill referring to the candidate or not. They show that an LSTM (Hochreiter et al., 1997) performs best on classifying the skill in the sentence. In our work, we annotated a dataset three times their size (Table 2) for both hard and soft skills. In addition, we also extract the specific skills from the sentence. Tamburri et al. (2020) classifies sentences that contain skills in the JP. The authors manually labeled their dataset with domain experts. They annotated whether a sentence contains a skill or not. Once the sentence is identified as containing a skill, the skill cited within is extracted. In contrast, we directly annotate for the span within the sentence. Bhola et al. (2020) cast the task of skill extraction as a multi-label skill classification at the document-level. There is a predefined set of unique skills given the job descriptions and they predict multiple skills that are connected to a given job description using BERT (Devlin et al., 2019). In addition, they experiment with several additional layers for better prediction performance. We instead explore domain-adaptive pre-training for SE.
The work closest to ours is by Chernova (2020), who approach the task similarly with span-level annotations (including longer spans) but approach this for the Finnish language. It is unclear whether they annotated by domain experts. Also, neither the data nor the annotation guidelines are released. For a comprehensive overview with respect to SE, we refer to Khaouja et al. (2021).

Skill & Knowledge Definition
There is an abundance of competences and there have been large efforts to categorize them. For example, the The International Standard Classification of Occupations (ISCO; Elias, 1997) is one of the main international classifications of occupations and skills. It belongs to the international family of economic and social classifications. Another example, the European Skills, Competences, Qualifications and Occupations (ESCO;le Vrang et al., 2014) taxonomy is the European standard terminology linking skills and competences and qualifications to occupations and derived from ISCO. The ESCO taxonomy mentions three categories of competences: Skill, knowledge, and attitudes. ESCO defines knowledge as follows: "Knowledge means the outcome of the assimilation of information through learning. Knowledge is the body of facts, principles, theories and practices that is related to a field of work or study." 2 For example, a person can acquire the Python programming language through learning. This is denoted as a knowledge component and can be considered a hard skill. However, one also needs to be able to apply the knowledge component to a certain task. This is known as a skill component. ESCO formulates it as: "Skill means the ability to apply knowledge and use know-how to complete tasks and solve problems." 3 In ESCO, the soft skills are referred to as attitudes. ESCO considers attitudes as skill components: "The ability to use knowledge, skills and personal, social and/or methodological abilities, in work or study situations  To sum up, hard skills are usually referred to as knowledge components, and applying these hard skills to something is considered a skill component. Then, soft skills are referred to as attitudes, these are part of skill components. There has been no work, to the best of our knowledge, in annotating skill and knowledge components in JPs.

SKILLSPAN Dataset
Data 5 We continuously collected JPs via web data extraction between June 2020-September 2021. Our JPs come from the three sources: 3. TECH: The StackOverflow JP platform that consisted mostly of technical jobs (e.g., developer positions).
We release the anonymized raw data and annotations of the parts with permissible licenses, i.e., HOUSE (from a govermental agency which is our collaborator) and TECH. 6 For anonymization, we perform it via manual annotation of job-related sensitive and personal data regarding Organization, Location, Contact, and Name following the work by Jensen et al. (2021). Table 2 shows the statistics of SKILLSPAN, with 391 annotated JPs from the three sources containing 14.5K sentences and 232.2K tokens. The unlabeled JPs (only to be released as pre-trained model) consist of 126.8K posts, 3.2M sentences, and 460.5M tokens. What stands out is that there are 2-5 times as many annotated knowledge components in TECH in contrast to the other sources, despite a similar amount of JPs. We expect this to be due the numerous KCs depicted in this domain (e.g., programming languages), while we observe considerably fewer soft skills (e.g., "work flexibly"). The amount of skills is more balanced across the three sources. Furthermore, overlapping spans follow a consistent trend among splits, with the train split containing the most.

Data Annotation
We annotate competences related to SKCs in two levels as illustrated in Figure 1. We started the process in March 2021, with initial annotation rounds to construct and refine the annotation guidelines (as outlined further below). 6 Links to our data can be found at https://github. com/kris927b/SkillSpan. The annotation process spanned eight months in total. Our final annotation guidelines can be found in Appendix B. The guidelines were developed by largely following example spans given in the ESCO taxonomy. However, at this stage, we focus on span identification, and we do not take the fine-grained taxonomy codes from ESCO for labeling the spans, leaving the mapping to ESCO and taxonomy enrichment as future work.
Further Details on the Annotation Process The development of the annotation guidelines and our annotation process is depicted as follows: 1 We wrote base guidelines derived from a small number of JPs. 2 We had three pre-rounds consisting of three JPs each. After each round, we modified, improved and finalized the guidelines. 3 Then, we had three longer-lasting annotation rounds consisting of 30 JPs each. We re-annotated the previous 11 JPs in 1 and 2 . 4 After these rounds, one of the annotators (the hired linguist) annotated JPs in batches of 50. The data in 1 , 2 , and 3 was annotated by three annotators (101 JPs).
We used an open source text annotation tool named DOCCANO (Nakayama et al., 2018). There are around 57.5K tokens (approximately 4.6K sentences, in 101 job posts) that we calculated agreement on. The annotations were compared using Cohen's  (Fleiss and Cohen, 1973) between pairs of annotators, and Fleiss'  (Fleiss, 1971), which generalises Cohen's  to more than two concurrent annotations. We consider two levels of  calculations: TOKEN is calculated on the token level, comparing the agreement of annotators on each token (including non-entities) in the annotated dataset. SPAN refers to the agreement between annotators on the exact span match over the surface string, regardless of the type of SKC, i.e., we only check the position of tag without regarding the type of the entity. The observed agreements scores over the three annotators from step 3 are between 0.70-0.75 Fleiss'  for both levels of calculation which is considered a substantial agreement (Landis and Koch, 1977) and a  value greater than 0.81 indicates almost perfect agreement. Given the difficulty of this task, we consider the aforementioned  score to be strong. Particularly, we observed a large improvement in annotation agreement from the earlier rounds (step 1 and 2 ), where our Fleiss'  was 0.59 on tokenlevel and 0.62 for the span-level.
Overall, we observe higher annotator agreement for knowledge components (3-5% higher) com-  Table 3: Most Frequent Skills in the Development Data. Top-5 skill components in our data in terms of frequency on different sources. A larger example can be found in Table 8 and Table 9 (Appendix C).
pared to skills which tend to be longer. The TECH domain is the most consistent for agreement while BIG shows more variation over rounds, likely due to the broader nature of the domains of JPs.
Annotation Span Statistics A challenge of annotating spans is the length (i.e., boundary), SKCs being in different domains (e.g., business versus technical components), and frequently written differently, e.g., "being able to work together" v.s. "teamwork"). Figure 2 shows the statistics of our annotations in violin plots. For the training set, the median length (white dot) of skills is around 4 for BIG and HOUSE, for TECH this is a median of 5.
In the development set, the median stays at length 4 across all sources. Another notable statistic is the upper and lower percentile of the length of skills and knowledge, indicated with the thick bars. Here, we highlight the fact that skill components could consist of many tokens, for example, up to length 7 in the HOUSE source split (see blue-colored violins). For knowledge components, the spans are usually shorter, where it is consistently below 5 tokens (see orange-colored violins). All statistics follow a similar distribution across train, development, and sources in terms of length and distribution. This gives a further strong indication that consistent annotation length has been conducted across splits and sources.
Qualitative Analysis of Annotations Qualitative differences in SKCs over the three sources are shown (lowercased) in Table 3. With respect to skill components, all sources follow a similar usage of skills. The annotated skills mostly relate to the attitude of a person and hence mostly consist of soft skills. With respect to knowledge components, we observe differences between the three sources. First, on the source-level, the knowledge components vastly differ between BIG and TECH. BIG postings seem to cover more business related components, whereas TECH has more engineering components. HOUSE seems to be a mix of the other two sources. Lastly, note that both the skill and knowledge components between the splits diverge in terms of the type of annotated spans, which indicates a variation in the annotated components. We show the top-10 skills annotated in the train, development, and test splits for SKCs in Appendix C. From a syntactic perspective, skills frequently consist of noun phrases, verb phrases, or adjectives (for soft skills). Knowledge components usually consists of nouns or proper nouns, such as "python", "java", and so forth.

Experimental Setup
The task of SE is formulated as a sequence labeling problem. Formally, we consider a set of JPs D, where d 2 D is a set of sequences (i.e., entire JPs) with the i th input sequence .., y T } (e.g., "B-SKILL", "I-KNOWLEDGE", "O"). The goal is to use D to train a sequence labeling algorithm h : X 7 ! Y to accurately predict entity spans by assigning an output label y t to each token x t .
As baseline we consider BERT and we investigate more recent variants, and we also train models from scratch. Models are chosen due to their stateof-the-art performance, or in particular, for their strong performance on longer spans.
SpanBERT (Joshi et al., 2020) A BERT-style model that focuses on span representations as opposed to single token representations. SpanBERT is trained by masking contiguous spans of tokens and optimizing two objectives: (1) masked language modeling, which predicts each masked token from its own vector representation. (2) The span boundary objective, which predicts each masked token from the representations of the unmasked tokens at the start and end of the masked span.
We train a SpanBERT base model from scratch on the BooksCorpus (Zhu et al., 2015) and English Wikipedia using cased Wordpiece tokens (Wu et al., 2016). We use AdamW (Kingma and Ba, 2015) for Figure 3: Performance of Models. We test the models on SKILLS, KNOWLEDGE, and COMBINED. We report the span-F1 and standard deviation (error bars) of runs on five random seeds. Note that the y-axis starts from 50 span-F1. STL indicates single-task learning and MTL indicates the multi-task model. Differences can be seen on the test set: JobSpanBERT performs best on SKILLS, JobBERT is best on KNOWLEDGE, and JobBERT achieves best in COMBINED. Exact numbers of the plots are in Table 5 (Appendix E).
2.4M training steps with batches of 256 sequences of length 512. The learning rate is warmed up for 10K steps to a maximum value of 1e-4, after which it has a decoupled weight decay (Loshchilov and Hutter, 2019) of 0.1. We add a dropout rate of 0.1 across all layers. Pretraining was done on a v3-8 TPU on the GCP and took 14 days to complete. We take the official TensorFlow implementation of SpanBERT by Ram et al. (2021).
JobBERT 7 We apply domain-adaptive pre-training (Gururangan et al., 2020) to a BERT base model using the 3.2M unlabeled JP sentences (Table 2). Domain-adaptive pre-training relates to the continued self-supervised pre-training of a large language model on domain-specific text. This approach improves the modeling of text for downstream tasks within the domain. We continue training the BERT model for three epochs (default in HuggingFace) with a batch size of 16.
Experiments We have 391 annotated JPs (Table 2) that we divide across three splits: Train, dev. and test set. We use 101 JPs that all three annotators annotated as the gold standard test set with aggregated annotations via majority voting. The 101 postings are divided between the sources as: 36 BIG, 33 HOUSE, and 32 TECH. The remaining 290 JPs were annotated by one annotator. We use 90 JPs (30 from each source, namely BIG, HOUSE, and TECH) as the dev. set. The remaining 200 JPs are used as the train set. The sources in the train set are divided into 60 BIG, 60 HOUSE, and 80 TECH.
Setup The data is structured as CoNLL format (Tjong Kim Sang, 2002). For the nested annotations, the skill tags are appearing only in the first column and the knowledge tags are only appearing in the second column of the file and they are allowed to overlap with each other. We perform experiments with single-task learning (STL) on either the skill or knowledge components, MTL for predicting both skill and knowledge tags at the same time, while evaluating the MTL models also on either skills or knowledge components. We used a single joint MTL model with hard-parameter sharing (Caruana, 1997). All models are with a final Conditional Random Field (CRF; Lafferty et al.,  2021) show that BERT models with a CRF-layer improve or perform similarly to its simpler variants when comparing the overall F1 and make no tagging errors (e.g., B-tag follows I-tag). In the case of MTL we use one for each tag type (skill and knowledge). In the STL experiments we use one CRF for the given tag type.
We use the MACHAMP toolkit (van der Goot et al., 2021) for our experiments. For each setup we do five runs (i.e., five random seeds). 9 For evaluation we use span-level precision, recall, and F1, where the F1 for the MTL setting is calculated as described in Benikova et al. (2014).

Results
The results of the experiments are given in Figure 3. We show the average performance of each model in F1 and respective standard deviation over the development and test split. Exact scores on each source split and other metric details are provided in Appendix E. As mentioned before, we experiment with the following settings: SKILL, we train and predict only on skills. KNOWLEDGE, train and only predict for knowledge. COMBINED, we merge the STL predictions of both skills and knowledge. We also train the models in an MTL setting, predicting both skills and knowledge simultaneously. We evaluate the MTL model on both SKILL and KNOWLEDGE separately, and also compare it against the aggregated STL predictions.
Performance on Development Set In Figure 3, we show the results on the development set in the upper plot. We observe similar performance between the domain-adapted STL models-JobBERT and JobSpanBERT-have similar span-F1 for 9 For reproducibility, we refer to Appendix D. SKILL: 60.05±0.70 vs. 60.07±0.70. In contrast, for KNOWLEDGE, BERT base and JobBERT are closest in predictive performance: 60.44±0.58 vs. 60.66±0.43. In the COMBINED setting, JobBERT performs highest with a span-F1 of 60.32±0.39. On average, JobBERT performs best over all three settings. Surprisingly, the models for both SKILL and KNOWLEDGE perform similarly (around 60 span-F1), despite the sources' differences in properties and length Figure 2. In addition, we find that MTL is not performing better than STL across sources. For exact numbers and source-level (i.e., BIG, HOUSE, TECH), we refer to Appendix E.

Performance on Test Set
We select the best performing models in the development set evaluation and apply it to the test set. Results are in Figure 3 in the bottom plot. Since JobBERT and JobSpan-BERT are performing similarly, we apply both to the test set and BERT base . We observe a deviation from the development set to the test set: JobSpan-BERT 60.07±0.30!56.64±0.83 on SKILL, Job-BERT 60.66±0.43!63.88±0.28 on KNOWLEDGE. For COMBINED, JobBERT performs slightly worse: 60.32±0.39!59.73±0.38. Similar to the development set, we find that on all three methods of evaluation (i.e., SKILL, KNOWLEDGE, and COM-BINED), STL still outperforms MTL. For SKILL and KNOWLEDGE, STL is almost stochastically dominant over MTL (i.e., significant), and for COMBINED there is stochastic dominance of STL over MTL, indicated in the next paragraph.
Significance We compare all pairs of models based on five random seeds each using Almost Stochastic Order (ASO; Dror et al., 2019) tests with a confidence level of ↵ = 0.05. The ASO scores of the test set are indicated in Figure 4. We show that MTL-JobSpanBERT for SKILL shows almost stochastic dominance (✏ min < 0.5) over all other models. For KNOWLEDGE and COMBINED, We show that STL-JobBERT is stochastically dominant (✏ min = 0.0) over all the other models. For more details, we refer to Appendix F for ASO scores on the development set.

Discussion
What Did Not Work Additionally, we experiment whether representing the entire JP for extracting tokens yields better results than the experiments so far, which were sentence-by-sentence processing setups. To handle entire JPs and hence much longer sequences we use a pre-trained Longformer base (Beltagy et al., 2020)  . This drop in performance is difficult to attribute to a concrete reason: e.g., the Longformer is trained on more varied sources than BERT, but not specifically for JPs, which may have contributed to this gap. Since the vanilla Longformer already performs worse than BERT base overall, we did not opt to apply domainadaptive pre-training. Overall, we show that representing the full JP is not beneficial for SE, at least not in the Longformer setup tested here.
Continuous Pretraining helps SE As previously mentioned, due to the domain specialization of the domain-adapted pre-trained BERT models, they predict more skills and frequently perform better in terms of precision, recall, and F1 as compared to their non-adaptive counterparts. This is especially encouraging as we confirm findings that continuous pre-training helps to adapt models to a specific domain (Alsentzer et al., 2019;Lee et al., 2020;Gururangan et al., 2020;Nguyen et al., 2020). However, there are exceptions. Particularly in Table 5 on TEST for KNOWLEDGE, BERT base comes closer in predictive performance to JobBERT (difference of 1.5 F1) than on SKILLS. Our intuition is that knowledge components are often already in the pre-training data (e.g., Wikipedia pages of certain competences like Python, Java etc.) and therefore adaptive pre-training does not substantially boost performance.

Difference in Length of Predictions
The main motivation of selecting models optimized for long spans was the length of the annotations (Figure 2). We investigate the average length of predictions of each model ( Figure 5) to find out whether the models that are adapted to handle longer sequences truly predict longer spans. Interestingly, the average length of predicted skills are longer than the annotations over all three sources. There is a consistent trend among SKILL: BIG and TECH have similar length over predictions (>4), while HOUSE is usually lower than length 3. For both BIG and TECH, JobSpanBERT predicts the longest skill spans (4.51 and 4.48 respectively). We suspect due to the domain-adaptive pre-training on JPs, it improved the span prediction performance. In contrast, the Longformer predicts shorter spans. Note that the Longformer is not domain-adapted to JPs.
Regarding KNOWLEDGE, there is also a consistent trend: BIG has the overall longest prediction length while TECH has the lowest. The Longformer predicts the longest spans on average for BIG and TECH. Knowledge components are representative of a normal-length NER task and might not need a specialized model for long sequences. We show the exact numbers in Table 7 (Appendix E) and the number of predicted SKILL and KNOWLEDGE: JobBERT and JobSpanBERT have higher recall than the other models.
Performance per Span Length SKILLS are generally longer than KNOWLEDGE components in our dataset (Figure 2). The previous overall results on the test set ( Figure 3) show that performance on Figure 6: Average Span-F1 per Span Length. We bucket the performance of JobBERT according to the length of the spans until 10 tokens and show the performance on each length, averaged over five random seeds. Indicated per bar is the support. The model performs best on medium-length skill spans (i.e., spans with token length of [4][5]. For knowledge spans, on average, it performs best on short-length spans (i.e., spans with token length of 1-2).
SKILL is substantially lower than KNOWLEDGE. We therefore investigate whether this performance difference is attributed to the longer spans in SKILL. In Figure 6, we show the average performance of the best performing model (JobBERT) on the three sources (test set) based on the gold span length, until a length of 10.
In SKILL components (upper plot), we see much support for spans with length 1 and 2, which then lowers once the spans become longer. Spans with length of 1 shows low performance on BIG and TECH (around 40 span-F1), which influences the total span-F1. Short skills are usually soft skills, such as "passionate", which can be used as a skill or not. This might confuse the model. In contrast, performance effectively stays similar (around 60 span-F1) for span length of 2 till 7 for all sources. Afterwards, it drops in performance. Thus, the weak performance on SKILL seem to be due to lower performance on the short spans.
For the KNOWLEDGE components (lower plot), they are generally shorter. We see that there is a gap in support between the sources, TECH has a larger number of gold labels compared to BIG and HOUSE. Unlike soft skills, KCs usually consist of proper nouns such as "Python", "Java", and so forth, which connects to the high performance on TECH (around 76 span-F1). Furthermore, support for spans longer than 2 drops considerably. In this case, if the model predicts a couple of instances correctly, it would substantially increase span-F1. Contrary to SKILL, high performance of KNOWL-EDGE can be attributed to its strong performance on short spans.

Conclusion
We present a novel dataset for skill extraction on English job postings-SKILLSPAN-and domainadapted BERT models-JobBERT and JobSpan-BERT. We outline the dataset and annotation guidelines, created for hard and soft skills annotation on the span-level. Our analysis shows that domainadaptive pre-training helps to improve performance on the task for both skills and knowledge components. Our domain-adapted JobSpanBERT performs best on skills and JobBERT on knowledge. Both models achieve almost stochastic dominance over all other models for skills and knowledge extraction, whereas JobBERT in the STL setting achieves stochastic dominance over other models.
With the rapid emergence of new competences, our new approach of skill extraction has future potential, e.g., to enrich knowledge bases such as ESCO with unseen skills or knowledge components, and in general, contribute to providing insights into labor market dynamics. We hope our dataset encourages research into this emerging area of computational job market analysis. 11. If there's a knowledge/skill component in the position, we exclude it as well.
12. Only annotate the skills that are related to the position.
12.1. This includes skills that are specific for the position as well (e.g., skills of a ruminants professor versus math professor).
12.2 Also skills that the person for the position is expected to do in the future.
12.3 This does not include skills, knowledge or attitudes describing only the company, the group you will join in the department, and so on. Only annotate if it is specified or implied that the employee should possess the skill as well.
13. We annotate industries and fields (that the employee will be working in) as knowledge components.

C Type of Skills Annotated
In both Table 8 and Table 9, we show the top-10 skill and knowledge components that have been annotated. We split the top-10 among the data splits (i.e., train, development, and test set), and also between source splits (i.e., BIG, HOUSE, TECH).  We use the default hyperparameters in MACHAMP (van der Goot et al., 2021) as shown in Table 4. For more details we refer to their paper. For the five random seeds we use 3477689, 4213916, 6828303, 8749520, and 9364029. All experiments with MACHAMP were ran on an NVIDIA ® TITAN X (Pascal) 12 GB GPU and an Intel ® Xeon ® Silver 4214 CPU.

E Exact Number of Performance
In Table 5, we show the exact numbers of the plot indicated in Figure 3. In addition, we also show the results of each respective split. For the STL models, we observe differences in performances over the sources which is particularly pronounced for knowledge components: The TECH source is the easiest to process (and has most SKCs), while SKCs identification performance is the lowest for BIG. This might be due to the broad nature of this source.
In the exact results table (Table 5) we add a ( †) next to the highest span-F1 if the model is truly stochastically dominant (✏ min = 0.0) over all the other models. (*) denotes that the best model achieved almost stochastic dominance (✏ min < 0.5) over-at minimum-one other model (e.g., in TEST rows w.r.t COMBINED: MTL-JobBERT ⌫ MTL-JobSpanBERT with ✏ min = 0.06) and stochastically dominant over the rest.
In Table 6, we report the precision and recall of the models, SKILL and KNOWLEDGE show the precision and recall of the STL models. MULTI shows the precision and recall of the MTL models.
Last, in Table 7, we show the exact numbers of the length of predictions Figure 5. We also add the number of predicted SKILL and KNOWLEDGE Overall, JobBERT and JobSpanBERT predict more skills in general than the other models. This is also the case for knowledge components. We hypothesize that this might be due to the BERT models now being more specialized towards the JP domain and recognizing more SKCs.

F Significance Testing
Recently, the ASO test (Dror et al., 2019) 10 has been proposed to test statistical significance for deep neural networks over multiple runs. Generally, the ASO test determines whether a stochastic order (Reimers and Gurevych, 2018) exists between two models or algorithms based on their respective sets of evaluation scores. Given the single model scores over multiple random seeds of two algorithms A and B, the method computes a test-specific value (✏ min ) that indicates how far algorithm A is from being significantly better than algorithm B. When distance ✏ min = 0.0, one can claim that A stochastically dominant over B with a predefined significance level. When ✏ min < 0.5 one can say A ⌫ B. On the contrary, when we have ✏ min = 1.0, this means B ⌫ A. For ✏ min = 0.5, no order can be determined. We took 0.05 for the predefined significance level. In Figure 7, we show the ASO scores on the development set.     Table 8: Most Frequent Skills in the Data. Top-10 skill components in our data in terms of frequency.