Abstract Visual storytelling (VIST) is a typical vision and language task that has seen extensive development in the natural language generation research domain. However, it remains unclear whether conventional automatic evaluation metrics for text generation are applicable on VIST. In this paper, we present the VHED (VIST Human Evaluation Data) dataset, which first re-purposes human evaluation results for automatic evaluation; hence we develop Vrank (VIST ranker), a novel reference-free VIST metric for story evaluation. 1 We first show that the results from commonly adopted automatic metrics for text generation have little correlation with those obtained from human evaluation, which motivates us to directly utilize human evaluation results to learn the automatic evaluation model. In the experiments, we evaluate the generated texts to predict story ranks using our model as well as other reference-based and reference-free metrics. Results show that Vrank prediction is significantly more aligned to human evaluation than other metrics with almost 30% higher accuracy when ranking story pairs. Moreover, we demonstrate that only Vrank shows human-like behavior in its strong ability to find better stories when the quality gap between two stories is high. Finally, we show the superiority of Vrank by its generalizability to pure textual stories, and conclude that this reuse of human evaluation results puts Vrank in a strong position for continued future advances. In visual storytelling (VIST) (Huang et al., 2016), a generation model tells a short story to describe the given five images. Automatic generation of visual stories is challenging because it has the complexity of cross-modal understanding with the diversity  * denotes equal contribution 1 Dataset VHED and metric Vrank can be found on GitHub: https://github.com/AcademiaSinicaNLPLab/VHED.git the city was very busy. there were many different kinds of bikes. some were very unique. they were all very fast. i had a great time. i went to the park station. it was a train trip to the museum. the train was very long. we had to go on our way out of the trains. this dog is so happy to see us. Model 1 (BLEU-1: 0.605, Human Rankers: ) Model 2 (BLEU-1: 0.354, Human Rankers: ) Figure 1: Ranking of two stories generated by Model 1 and 2, by human rankers versus BLEU-1 score. Reference : i decided my dog would like a train ride. off to the train station we go. this is the train we will be taking our short trip on. my friend is the conductor. he is getting ready to attach the cars. here is the train all together. as you can see, my dog had a fantastic time. and sophistication of creative writing (Zhu et al., 2020). Extensive efforts in model developments have decreased the distance between machine-generated and human-written stories, but research on VIST evaluation remains stagnant. Automatic metrics and human evaluation are widely used to examine natural language generation. Traditional n-gram-based or reference-based autometrics such as BLEU (Papineni et al., 2002), CIDEr (Vedantam et al., 2015), and METEOR (Banerjee and Lavie, 2005) are common for VIST evaluation. However, preliminary findings have shown that these metrics have many drawbacks and hence are incompatible with VIST (Wang et al., 2018b). In particular, they assume that human-written stories are always better than machine-generated stories, limiting the advance of models yet not conforming to our observation on human judgment. Rethinking this postulation in evaluation, we believe the dependence on references should be minimized and human evaluation results should be fully utilized instead, because human judgements contain more 6365 meaningful signals. Recent hybrid and reference-free metrics such as BLEURT (Sellam et al., 2020) and UNION (Guan and Huang, 2020) have not yet been implemented or studied in VIST. Nevertheless, BLEURT utilizes few human results in fine-tuning, and UNION still regards human references as gold labels, which results in poor correlation to human judgment. On the other hand, human evaluations are relatively reliable for performance reports, and recent studies often include them to provide more convincing experimental results (Hsu et al., 2020, 2021a,b). However, human evaluations are expensive, time-consuming, and difficult to reproduce. Therefore, results should be recycled to benefit future evaluations. Accordingly, we re-collected the human evaluation results from multiple published papers and organized the data into story pairs (Wei and Jia, 2021) as the VHED (VIST Human Evaluation Data) dataset. We then re-purposed VHED to create a better metric Vrank for VIST to rank visual stories. Vrank is a reference-free SIMCSE (Gao et al., 2021) based metric trained on VHED to learn to rank visual stories. We believe a storytelling metric should be independent of the references because stories are highly diverse by nature (Zhu et al., 2020), and it is reasonable for them to be dissimilar to the references (Guan and Huang, 2020) As shown in Fig. 1, the story generated by Model 1 is assigned a higher BLEU score because larger portions of text overlap with the reference. However, human rankers recognize description in isolation and object detection error in Model 1, and instead rank Model 2 better. We conduct experiments to show that Vrank is superior to existing metrics, many of which lack properties essential to evaluating stories in a human-like fashion. Therefore, we utilize VHED to understand and analyze human judgment in evaluating visual stories, and to provide additional metric assessments to reveal the shortcomings of existing metrics. The metric assessment experiments are conducted as the story-pair ranking task in which two stories are ranked based on their story quality. We observe three characteristics and design corresponding assessments to demonstrate Vrank's merits. First, larger rank differences in story quality are easier for people to differentiate. We measure the performance of metrics in story pairs with large gaps versus small gaps to determine whether all metrics have this property. Our assessment indicates this property is exclusively hold by Vrank. Second, human-written stories are not always better than machine-written stories. Indeed, 38% of machine-generated stories are better than the references, which suggests that the afore-mentioned assumption may need to be revisited (Clark et al., 2021). We examine the ability of metrics to rank such human-machine pairs, which Vrank performs relatively well. Finally, most generated stories still contain many errors, which serve as signals for human rankers (Modi and Parde, 2019). Hence we evaluate the ability of metrics to detect errors and show that Vrank is a better indicator of errors. Also, we show that Vrank is able to generalize to other datasets without bias to VHED. In conclusion, Vrank excels in the above assessments and able to follow human behaviors in ranking. Moreover, Vrank can rank machine and human stories decently and is better at detecting story errors. Specifically, we make three major contributions:  We re-collect and organize human evaluation results from recent VIST papers to form a new dataset: VHED. We propose a novel valid metric Vrank for visual storytelling which appropriately evaluates VIST model performance. We propose three assessments for metrics according to human properties and a generalization test to better illustrate the shortcomings of existing VIST metrics. Visual Storytelling (VIST) Visual storytelling was introduced by Huang et al. (2016) as the task of generating a coherent story given five images. They provided a dataset, Sequential Images Narrative Dataset (SIND), containing images and references in which references are human-written short stories describing images. For every image prompt (one sequence of photos), there are 2 to 5 references. VIST requires deeper understanding of the photo events to prevent descriptions in isolation (i.e., image captions). Researchers have proposed various methods for this task. Knowledge graphs are often integrated in models to encourage diversity of terms and plots in the stories (Hsu et al., 2020, 2021a; Chen et al., 2021). Some studies use reinforcement learning to reward models that generate stories that contain fewer errors and are more topically-focused (Huang et al., 2019; Hu et al., 2020a). However, existing evaluation methods are 6366 unable to capture the true quality of the generated stories. Thus we examine automatic metrics to devise a better way for machines to evaluate stories. VIST-Human Evaluation Several VIST generation models use human evaluation to evaluate model performance. Recent studies apply aspect-based rating evaluation. Hu et al. (2020b) and Wang et al. (2020b) ask workers to rate stories based on pre-defined aspects. 2 However, it is difficult to normalize these aspects as the definition of aspect varies from paper to paper. Also, these aspects are not mutually independent, making it difficult to analyze results based on these ratings. Therefore, we consider the ranking method as it is commonly used (Hsu et al., 2020; Wang et al., 2020b; Hsu et al., 2021a) among authors. Hsu et al. (2020) asks human annotators to rank five stories from different models based on overall quality. Hu et al. (2020b) and Wang et al. (2020b) conduct pairwise human evaluations to rank stories according to different story aspects, where the latter is judged to be closer to human-level. These human evaluation results are valuable resources for observing human judgments in visual storytelling. Hence, in our work we collect this information for analysis and model training. Automatic Metrics Automatic evaluation metrics are widely used in language generation tasks. Most reference-based metrics (e.g., BLEU (Pap-ineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and ROUGE (Lin, 2004)) evaluate the n-gram similarity between a generated text and the reference. However, referenced metrics correlate poorly with human judgment (Wang et al., 2018b; Hsu et al., 2019; Modi and Parde, 2019) in dialog generation and story generation tasks: the generated text is given unreasonable scores due to incongruity with the reference. To account for this, several reference-free metrics (Sinha et al., 2020; Guan and Huang, 2020) have been designed to measure generated texts without any reference. BERT-Score (Zhang et al., 2020), for instance, uses contextual embedding to calculate the similarity between candidates and references, and BLEURT (Sellam et al., 2020) uses referenced automatic metrics as supervision signals for pretraining and is fine-tuned on a human judgment evaluation dataset. UNION (Guan and Huang, 2 Hu et al. define relevance, coherence, and expressiveness, and Wang et al. define focus, coherence, detail, share, grounded, and human. 2020) uses pre-defined negative samples to train a model in an attempt to provide a metric that specializes in story generation. In our analysis, current metrics remain unable to mimic human judgment to discern quality differences in story pairs. The VHED dataset is a collection of human evaluation results from three VIST studies: KGStory (Hsu et al., 2020), PR-VIST (Hsu et al., 2021a), and Stretch-VST (Hsu et al., 2021b). All papers followed Hsu et al. (2020)'s human evaluation method using Amazon Mechanical Turk. For each task, the workers were to rank the story by overall quality, from the best story to the worst story. Specifically, each task displayed N stories, and each worker ranked each story from 1 to N . Details about each paper are listed in Table 1. The construction of VHED is shown in Figure 2. Collected from the aforementioned papers, we obtained 4,500 task results. Further, we grouped N stories into story pairs, where the number of story pairs per task is CN 2 . The resulting story pairs ( x 1 , x 2 ) are either two machine-generated stories from two different models or one reference and one machine-generated story. For each story pair, there are five attributes:  Stories: A story pair consists of a better-ranked story and worse-ranked story. The story pair is either a reference with a machine-generated story, or two machine-generated stories. Image Sequence IDs: A list of IDs for each of the five images from the SIND dataset (Huang et al., 2016). Average Rank: The average of the five workers' story rankings is divided by N for normalization. N varies from paper to paper (Table 1). Ranking Gap: The ranking gap is calculated as the average ranking of x 1 minus the average 6367 Paper HumanEvaluation Sampling Tasks N KGStory 2 500 1,000 5 PRVIST 6 250500 1,000 34 Stretch-VST 7 250500 2,500 24 Table 1: Statistics of human evaluation results of KGStory (Hsu et al., 2020), PRVIST (Hsu et al., 2021a), and Stretch-VST (Hsu et al., 2021b) ranking of x 2 . The ranking gap distribution is shown in the appendix (Table 6). Human Agreement: Human agreement is when k workers agree that the better-ranked stories are better than the worse-ranked stories. Note that human agreement = 2 is equivalent to human agreement = 3, because 1 person agreeing that story A is better than B is equivalent to 4 people agreeing that story B is better than A. Therefore, we kept human agreements = 3,4,5 for simple notation. For quality control, we remove story pairs with zero ranking gap. This yields 13,875 story pairs in total. The train-test-validation sets were split at a ratio of 8:1:1 to 11,208, 1,351, and 1,316 story pairs. The descriptions of VIST models' generated stories are included in the appendix. As we acquired data about human preferences in story pairs, we conducted analyses to understand the potential patterns for workers when assigning story ranks, the quality gap between machine-generated and human-written stories, and the errors in the stories. The results of this observation are crucial for assessing the performance of a metric. Worker Ranking Analysis Story pairs are grouped by the same human agreement. k denotes a sub-dataset containing story pairs with human agreement = k . In Table 2, we calculate the number of story pairs as well as the averaged ranking gap of each sub-dataset. For story pairs, we note that story pairs with k = 3 account for 53% of the dataset, meaning that half of the tasks have inconsistent annotations. Regardless, this paper evaluates the story pairs with k  4 to filter out inconsistent human annotations. We also note that the ranking gap increases as human agreement increases. The ranking gap indicates the quality difference between a better-ranked and a worse-ranked story. That is, the difference between a ranked 1 story and a ranked 5 story should be larger Subset Story pairs Ranking gap Machine better  3 6,494 (53%) 0.123 918 (45%)  4 3,677 (30%) 0.247 523 (35%)  5 2,110 (17%) 0.416 110 (22%) Table 2: The number and percentage of story pairs, average ranking gap of each sub-dataset. than that between a ranked 2 story and a ranked 3 story. From Table 2, we find that story-pairs with lower agreement are closer in ranking. In other words, a story pair with a marginal quality difference easily leads to inconsistent worker annotations, because it is harder to rank two similar stories. Essentially, we expect the metrics to exhibit similar behavior: the larger the ranking gap, the easier it is to rank . Who Wins? Machine vs. Human Stories Next we revisit the assertion that references are always superior. We select story pairs with a reference and a machine-generated story. We analyze the number and percentage of references that are ranked better than the generated stories on three human agreements. From Table 2, we observe that when more humans agree on the ranking results, the percentage of the reference being better also increases. In addition, further analysis shows that, on average, 38% of the machine-generated stories are in fact better than the references, showing that references are not always better than machine-generated stories. Error Analysis To understand the difference between betterand worse-ranked stories, deeper analysis into the story content is necessary. We randomly sampled 200 stories from VHED (67 human and 134 machine generated) and manually labeled the stories according to the following error aspects:  Grammatical error (Gram): Erroneous usage of past/current tense and mistakes in misplaced modifiers. Repetition (Rep): Repetitive sentences or phrases at sentenceand story-level. Description in isolation (Desc): Sentences that lack consistency, resulting in isolated captions instead of a fluent story. Absurdity (Abs): Ambiguous sentences or nonsensical phrases that are incomprehensible to humans. Event mismatch (Event): Stories that are off-topic, which present events that are not relevant to the image stream. Object mismatch (Obj): Irrelevant nouns that do not appear in the images and are not semantically related. We first labeled stories based on all 11 error aspects defined in (Modi and Parde, 2019) and we select the most occurring errors, which are grammar, repetition, description in isolation, and absurdity. These four error aspects focus primarily on story coherence and within-story consistency. However, visual storytelling requires generated stories to fit the given story images. Rohrbach et al. (2019) show that humans are aware of the correctness of image descriptions. Also, Wang et al. (2020a) show that mismatched events in stories can lead to poor story quality. Therefore, we added event and object mismatch into our analysis. The error examples and correlation between the error are illustrated in the appendix (Table 9 and Figure 5). From our observation, 79.8% of the sampled machine generated stories contained at least one of the errors in the categories, meaning most VIST models are unable to generate perfect stories. In Table 3, the high percentage of object and event mismatch errors also show that current VIST models do not capture visual groundings accurately. This can lead to humans assigning higher scores to human-written stories since they are most likely to be relevant to the given images. Grammatical errors and absurdities are also common in generated text, which can lead to ambiguous stories that humans are unable to comprehend. The prevalence of errors makes it essential for evaluation metrics to automatically detect these errors. We propose Vrank, a reference-free automatic metric that inputs story pairs to predict human preferences between the two stories. We utilize SIMCSE (Gao et al., 2021) to leverage better sentence representations. SIMCSE uses contrastive learning with dropout as augmentation, then trained on natural langauge inference datasets to obtain better sentence embeddings from BERT (Devlin et al., 2019). First, we pre-trained the SIMCSE model using SIND reference stories with the Masked Language Model objective. Then, we input two stories with a [SEP] token in between through the pre-trained model. We use the acquired sentence embeddings and feed it through a regression layer to predict a ranking gap. We used mean squared error to calculate the loss between the predicted ranking gap and true ranking gap. After obtaining the ranking gap, we predict which story is better according to the sign of the predicted ranking gap. Although Vrank is a simple model fine-tuned solely on human judgment, it still outperforms current existing metrics in our assessments. This suggests further potential for use with VHED; more studies can be conducted to replace Vrank with stronger neural network models. During model training, since the number of positives and negatives were not balanced in the original dataset, we augmented the data to create a symmetric dataset of VHED to minimize dataset bias. 3 The ranking gap in the resulting dataset was close to normally distributed. We hypothesize that utilizing this feature makes it possible to extract more information, making it easier for the model to learn human judgment. However, due to the small amount of data available, high variance is likely (Mosbach et al., 2020) to occur during inference. Hence, we used all data from VHED, including human agreement = 3 to increase the stability of our model following Mosbach et al. (2020). In this section, we describe a series of assessments conducted on existing metrics on VHED, in which the assessment methods are based on the analyses in VHED. The objective is to examine whether Vrank is superior to other metrics based on our analysis of VHED. Story-Pair Ranking A recent study (Wei and Jia, 2021) illustrates that pairwise accuracy reflects metric performance better than using correlation with human evaluation. Hence, we propose simple story-pair ranking to evaluate automatic evaluation metrics for visual storytelling. The task is to determine the correct ranking order of the stories in 3 Other configurations, including utilizing visual features and changing the task objective to classifying betterand worse-ranked stories did not perform better. a story pair based on the story quality scores predicted by the automatic evaluation metrics being assessed. Given the story pair ( x 1 , x 2 ) , the automatic metric being assessed predicts the corresponding story quality scores ( s 1 , s 2 ) which we compare to the averaged ranks y 1 and y 2 of x 1 and x 1 from human evaluation. The performance of the evaluation metric on the i -th story pair is formulated as ranking _ acc i =   1 , if s 1 > s 2 and y 1 < y 2 1 , if s 1 < s 2 and y 1 > y 2 0 , otherwise, (1) where ranking _ acc i = 1 indicates correct (incor-rect) prediction. Note that low scores indicate high rank. The overall metric performance over M story pairs is defined as: avg _ ranking _ acc = 1 MM (cid:88) i =1 ranking _ acc i . Datasets In addition to VHED, we also collected VIST-Edit 4 (Hsu et al., 2019) for story-pair ranking. VIST-Edit includes 2,981 visual stories generated by AREL (Wang et al., 2018a) and GLAC (Kim et al., 2018), and 14,905 human-edited visual stories, that is, AREL and GLAC-generated stories edited by workers. Their paper shows that the crowd workers' edits systematically increased the lexical diversity of the stories. Since the purpose of the editing was to improve the machine-4 VIST-Edit: https://github.com/tingyaohsu/ VIST-Edit generated stories, we paired up human-edited stories and machine-generated stories as better-ranked and worse-ranked samples (labeled as 1 and 2), resulting in 14,905 story pairs. Comparing VHED to VIST-Edit, VHED contains reference and multiple models' generated stories, but VIST-Edit has only human-machine story pairs. Additionally, VIST-Edit is not in Vrank's training data. VIST-Edit is utilized only for metric performance reports, serving as an unseen dataset for Vrank. Baseline Automatic Metrics We first consider traditional n-gram-based reference-based metrics, BLEU, ROUGE-L, METEOR, and SacreBLEU (Keenan, 2017). We also implement the more recent BERT-Score, BLEURT, and UNION as baseline metrics. In addition to the above automatic metrics, we also include a random baseline, denoted as Random in Table 4, to provide a random score for each story as the lower bound. A common practice for reference-based metrics: a candidate story is scored against each reference r j in a gold reference set R = { r i } ni =1 ; the highest score was used. However, applying this method on a reference-machine story pair would always result in reference having a full score, because of the exact match between reference and the gold reference set. To ensure a fair evaluation and avoid meaningless matching, we first check that the gold references do not include the reference. To this end, we propose the Reference Absent Algorithm for evaluating story pairs containing the reference story (or stories) as in Eq. 3, which removes the r j from R when any of the candidate stories in a story 6370 pair ( x = { x 1 , x 2 } ) is identical to r j . where metric(  ) can be any reference-based metric and s j is the story quality scores for the j -th story in a story pair. This algorithm only applies when evaluating story pairs containing references, i.e., reference-machine pairs in this paper. We believe such behavior A v e r a g e R a n k i n g A cc u r ac y 0.3 > r >= 0.2 0.2 > r >= 0.1 0.1 > r Ranking Gaps (r) r >= 0.3 A v e r a g e R a n k i n g A cc u r ac y Figure 3: Average ranking accuracy for each metric on four sub-datasets with different ranking gaps r . Pairwise Story Evaluation Accuracy: Metric's ability to determine the correct ranking order in story pairs. The average ranking accuracy of each automatic metric on VHED and VIST-Edit are presented in Table 4 (left). Around 50% corresponds to random guessing, as shown as Random in the table. Vrank shows superior performance in VHED and VIST-Edit, which VIST-Edit is the unknown dataset to Vrank. High performance on VIST-Edit and VHED indicates Vrank has the ability to distinguish diverse story pairs. In contrast, we observe unexpectedly low performance for most baseline metrics, as they perform no better than the Random baseline. BLEU-4 especially struggles to rank the stories in both datasets. Further analysis suggests that BLEU-4 marked  80% of the stories as 0, and Equation 1 coincidentally treated them as incorrect prediction because it discourages ties. BLEURT, in turn, also performed poorly because it relies on reference-based metrics as signals for training. Reference-free metrics, especially UNION, perform well on VIST-Edit. However, its design is not generalizeable to VHED. Worker Ranking Behavior on Metrics: The larger the ranking gap, the easier is it to rank. The ranking gap is the difference between a better-ranked and worse-ranked sample's average ranks. VHED is categorized into four sub-datasets with different ranking gaps. This assessment tests each metrics' ability to mimic worker ranking behavior observed in the analysis. Story pairs with larger gaps suggest stronger linguistic differences and are likely easier to rank, whereas those with smaller gaps are likely more difficult. In Fig. 3, all baseline automatic metrics, including metrics not reported in the figure, show randomly distributed scores, most of which remain around 50%, thus failing to exhibit such behavior. On the contrary, Vrank yields an ideal decrease. Starting with ranking gaps over 0.3, the accuracy reaches  0.85 and a grad-ual decrease afterward. Machine and Human on Metrics: Machines are sometimes better than humans. Two aspects are studied in this section. First, we evaluate the ability of Vrank and reference-based metrics to rank reference-machine (R&M) pairs. Although some machine texts have progressed to human-level, to our knowledge, there has been little investigation of metrics' ability to evaluate references and machines. We apply reference-based metrics with Eq. 3. This results in poor performance for reference-based metrics as shown in R&M in Table 4 5 . An explanation is that since the reference is removed from the reference set by Eq. 3, the reference needs to match with the remaining references in the reference set. Although most references are on topic, the stories are highly diverse (Zhu et al., 2020). These metrics are unable to calculate the similarity to semantic levels; thus, they result in poor performance. On the contrary, Vrank is a deep learning model, trained on VHED and thus learned to rate based on story quality rather than similarity. We also find that Vrank ranks correctly when machine is better than reference, showing that Vrank yields 26.5% recall when the other metrics have 0 recall without Eq. 3 and  18% with Eq. 3. Second, we observe the performance of metrics on M&M (machine-machine pairs). M&M ranking gaps are smaller than those of R&M pairs (0.18 v.s. 0.21), making them harder to rank because their story qualities are closer. However, Vrank still shows promising performance when ranking 5 A complete table without Eq. 3 can be found in the appendix (Table 7) 6371 Error Types Human Vrank UNION-ROC UNION-WP BLEURT BERT-Score ROUGE-L METEOR Gram -0.107 -0.021 -0.099 -0.087 -0.228 -0.124 0.024 -0.167 Desc -0.212 -0.154 -0.149 0.154 -0.081 0.080 0.114 -0.018 Rep -0.130 -0.042 -0.120 -0.411 0.168 0.134 0.079 -0.034 Abs -0.309 -0.308 0.003 0.120 -0.113 0.105 0.092 -0.025 Obj -0.067 -0.157 -0.089 0.158 -0.302 -0.111 -0.048 -0.098 Event -0.191 -0.093 0.008 -0.001 -0.131 0.043 0.138 -0.099 Table 5: This table shows the correlation of human rankings, automatic metric scores with the corresponding error categories. Errors in Metrics: Metric's ability to detect errors. Current generated stories often contain errors which prompt human evaluators to assign lower scores. It is crucial for automatic metrics to also recognize such errors to judge generated text. To do this, we adapted the point-biserial correlation coefficient to analyze the correlation between binary annotated errors and metric scores. The correlation between metrics and errors is presented in Table 5: existing metrics are not able to detect errors as the correlation coefficients are low. From the correlation coefficients between the human ranking score and each error aspect, we observe that human evaluation for stories may be influenced by error aspects, especially absurdity and description in isolation. In general, Vrank performs best in detecting absurdity and description in isolation. UNION-WP performs best in correlation with repetition, which is reasonable since UNION is trained to discriminate erroneous stories that are repetitive in structure. In summary, current metrics remain unable to detect errors to evaluate coherency efficiently. Metrics ability to detect errors may give clearer indications of the quality of generated texts. In addition to VIST, we expect Vrank to reasonably evaluate the quality of text as well. To determine whether Vrank generalizes to textual stories, we selected MANS dataset (Guan et al., 2021), an image-free storytelling dataset in which the stories are derived from the ROCStories corpus (Mostafazadeh et al., 2016). MANS includes 200 story prompts, where each prompt includes five model-generated stories and a reference. However, it does not contain human story rankings. Thus, for each story prompt, we asked five workers from Amazon Mechanical Turk to rank the five stories to obtain ranking scores. Following the VHED construction procedure, the ranked stories were converted into story pairs, making for 1,112 story pairs for which 3 workers agreed on the ranking, 605 story pairs for which 4 workers agreed, and 132 story pairs for which 5 workers agreed. Likewise, we evaluate story pairs with k  4 . The results of Vrank and the baseline automatic metrics when ranking MANS are shown in Table 6. We find that Vrank outperforms baseline metrics in story pairs with k  4 , whereas the latter still show limited abilities to rank the MANS dataset. In general, the accuracy of automatic evaluation on MANS is lower than that on VHED. This may be due to the comparably unconstrained writing styles of pure textual stories. An example of the evaluation on stories is given in the appendix (Table 8). We present VHED and Vrank, the first dataset of human evaluation results and evaluation metric for VIST. We show that Vrank performs significantly better in three assessment tasks and generalizes to other datasets. Also, recent automatic 6372 metrics are ill-suited to evaluating visual stories, especially human-level written stories. We welcome researchers to share their human evaluation results to the community to broaden the data domain to obtain more knowledge about human judgment and improve the performance of Vrank. As the gap between machines and humans continues to decrease, stronger metrics will be needed to evaluate machine and human stories. Improving Vrank performance to replace reference-based metrics is our future goal. This research is partially supported by Ministry of Science and Technology, Taiwan under the project contract 108-2221-E-001-012-MY3 and 110-2634-F-002-051-. We also thank the crowd workers for participating in this project.
abstain abstain abstain abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective abstain abstain abstain abstain abstain result method abstain result abstain result result abstain objective abstain method abstain abstain abstain method abstain result result abstain abstain objective objective objective other other other other other other other other method other other other other other method other other other method other other other other other other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain other abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain method abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain other abstain abstain abstain abstain abstain result abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain objective result abstain result abstain objective other other