Abstract Story visualization is an underexplored task that falls at the intersection of many important research directions in both computer vision and natural language processing. In this task, given a series of natural language captions which compose a story, an agent must generate a sequence of images that correspond to the captions. Prior work has introduced recurrent generative models which outperform text-to-image synthesis models on this task. However, there is room for improvement of generated images in terms of visual quality, coherence and relevance. We present a number of improvements to prior modeling approaches, including (1) the addition of a dual learning framework that utilizes video captioning to reinforce the semantic alignment between the story and generated images, (2) a copy-transform mechanism for sequentially-consistent story visualization, and (3) MART-based transformers to model complex interactions between frames. We present ablation studies to demonstrate the effect of each of these techniques on the generative power of the model for both individual images as well as the entire narrative. Furthermore, due to the complexity and generative nature of the task, standard evaluation metrics do not accurately reflect performance. Therefore, we also provide an exploration of evaluation metrics for the model, focused on aspects of the generated frames such as the presence/quality of generated characters, the relevance to captions, and the diversity of the generated images. We also present correlation experiments of our proposed automated metrics with human evaluations. 1 1 Introduction While generative adversarial networks (GANs) have achieved impressive results on a variety of 1 Code and data: https://github.com/ adymaharana/StoryViz . image generation tasks (Zhu et al., 2019; Qiao et al., 2019), the task of story visualization (Li et al., 2019b) is a variation of image generation that is more challenging and underexplored. In this setting, there is a story which consists of a sequence of images along with captions describing the content of the images, e.g., a web comic. The goal of the task is to reproduce the images given the captions (Figure 1). The benefits of investigating this task are far reaching. It combines two interesting and challenging sub-areas: text-to-image synthesis and narrative understanding, providing an excellent test bed for exploring and developing multimodal modeling techniques. From an application perspective, such a system could be used to enhance existing textual narratives with visual scenes. This tool would be especially useful to comic artists, who are infamously overworked, allowing them to automatically generate initial drawings speeding up their workflow. Additionally, such a system would have many applications in an educational setting, allowing educators to cater to a more diverse set of learning styles by automatically generating visualizations for a given topic, such as the water cycle in a science lesson. Furthermore, the data in this domain is cartoon-style, meaning the generated images avoid many of the ethical issues associated with real-world data. For a more detailed discussion, see Section 9. The challenge of this task extends beyond tasks such as text-to-image or text-to-video synthesis. Namely, there is an explicit, narrative component to the data, which must first be accurately extracted from the text, and then consistently reproduced throughout the images. If the setting or a description of a character is provided in the first caption, this must be carried throughout the scene unless modified by a subsequent caption. Furthermore, the scenes in a single story can change drastically as the story progresses, requiring models to produce a greater variety of images than in a text-to-video task, which typically consists of short videos displaying a single action. To address these issues, we consider the task as proposed in Li et al. (2019b), which provides a baseline architecture, StoryGAN, along with datasets for the task. We introduce techniques that build on existing work and are focused on improving consistency across frames, resulting in images of higher visual quality. First, we augment the model with Dual Learning via video redescription. The output images are fed through a video captioning model, which is trained to reproduce the ground truth story captions. This provides an additional learning signal to the model, forcing it to semantically align with the given narrative. Next, we add a Copy-Transform module that can take generated images from previous timesteps and copy the most relevant features of those images into the next generated frame, thus making the images more consistent in appearance. Finally, we propose the use of Memory-Augmented Recurrent Transformer (MART) (Lei et al., 2020) to model the correlation between word phrases in the input text and corresponding regions in the generated image. The recurrent nature of MART allows for the learning of sophisticated interactions between the image frames, yielding images that are more consistent in terms of character appearances and background imagery. We call the model architecture with the aforementioned additions DU ( AL )-C O ( PY )-S TORYGAN or DUCO-STORYGAN. Next, we focus on exploring alternative evaluation methods for story visualization models. While modeling improvements are crucial for progressing in this domain, evaluating these models is a challenge in itself. Like many other generative tasks, it is nontrivial to evaluate a story visualization model. Human evaluation is the most reliable option, but its monetary and time costs make this ill-suited to be the only evaluation method. Most prior work relies upon standard GAN evaluation metrics, which may provide some insight into how well the images were reproduced, yet miss out on other aspects of the story visualization task, such as the visual consistency of the setting across frames and global semantic alignment. Therefore, we make evaluation another focal point of the paper, exploring a variety of automatic evaluation metrics, which capture various aspects of the task, e.g., evaluating the quality of the images, the relevance to the story, the diversity of the generated frames, and the model's ability to accurately represent the characters. We present results from our model and baseline models on all metrics along with qualitative results, demonstrating the improvements from our proposed techniques. Using these metrics, we also provide ablation analyses of our model. 1. For the story visualization task, we improve the semantic alignment of the generated images with the input story by introducing dual learning via video redescription. 2. We enable sequentially-consistent story visualization with the introduction of a copy-transform mechanism in the GAN framework. 3. We enhance prior modeling techniques in story visualization with the addition of Memory Augmented Recurrent Transformer, allowing the model to learn more sophisticated interactions between image frames. 4. We present a diverse set of automatic evaluation metrics that capture important aspects of the task and will provide insights for future work in this domain. We also conduct correlation experiments for these metrics with human evaluation. Li et al. (2019b) introduced the task of story visualization and the StoryGAN architecture for sequential text-to-image generation. There have been a few other works that have attempted to improve upon the architectures presented in this paper. Poro-roGAN (Zeng et al., 2019) aims to improve the semantic relevance and overall quality of the images via a variety of textual alignment modules and a patch-based image discriminator. Li et al. (2020) also improve upon the StoryGAN architecture by upgrading the story encoder, GRU network, and discriminators and adding Weighted Activation Degree (Wen et al., 2019). Song et al. (2020) is a more recent work which makes improvements to the StoryGAN architecture; the primary contribution is adding a figure-ground generator and discriminator, which segments the figures and the background of the image. Our model improvements of MART, dual learning, and copy-transform build upon more recent techniques and we support them with a detailed series of ablations. Text-to-Image and Text-to-Video Generation. While story visualization is an underexplored task, there has been plenty of prior work in text-to-image synthesis. Most papers in this area can be traced back to StackGAN (Zhang et al., 2017). Subsequent work then made various modifications to this architecture, adding attention mechanisms, memory networks, and more (Xu et al., 2018; Zhu et al., 2019; Li et al., 2019a; Yi et al., 2017; Gao et al., 2019). Huang et al. (2018) and Qiao et al. (2019) are direct precursors of our work. Both of these works subject the generated output as an image captioning task which attempts to reproduce the original text. Our proposed dual learning approach is an expansion of this module, where we use a state-of-the-art video captioning model based upon the MART (Lei et al., 2020) architecture to provide an additional learning signal to the model and increase the semantic consistency across images. In the domain of text-to-video synthesis, Li et al. (2018), Pan et al. (2017), Gupta et al. (2018) and Balaji et al. (2019) generate videos from single sentences. In contrast to videos, story visualization does not have the requirement that the frames flow continuously together. Therefore, it allows for more interesting interactions and story-level dynamics to be captured that would only be present in longer videos. Interactive Image Editing. Another task related to story visualization is interactive image editing. In this setting, rather than going from purely text to image, the model is given an input image along with textual instructions/directions, and must produce an output image that modifies the input image according to the text. This can take the form of high level semantic changes to the image, such as color and shape, as in Liu et al. (2020), Nam et al. (2018), and Chen et al. (2018), or this might take the form of Photoshop-style edits, as in Laput et al. (2013), Shi et al. (2020), and Manuvinakurike et al. (2018a). Alternatively, Cheng et al. (2020), Manuvinakurike et al. (2018b), and El-Nouby et al. (2019) are slightly closer to our task due to their sequential nature, where an image is modified repeatedly according to the textual feedback provided via a dialogue. However, unlike story visualization, these tasks do not have a narrative component. Furthermore, they involve repeatedly editing a single object at each timestep instead of generating diverse scenes with dynamic characters. Formally, the task consists of a sequence of sentences S = [ s 1 , s 2 , ..., s T ] and a sequence of images X = [ x 1 , x 2 , ..., x T ] , where the sentence s k describes the contents of the image x k . The model receives S as input and produces a sequence of images  X = [ x 1 ,  x 2 , ...,  x T ] , attempting to accurately reproduce X . As detailed in Li et al. (2019b), there are two aspects of this task. The first is local consistency, which is concerned with the quality of individual pairs in the sequence; an example is locally consistent if image  x k accurately represents the contents of sentence s k . The second aspect is global consistency, which is concerned with the quality of the entire sequence. Namely, whether the sequence of images  X accurately captures the content of the sequence of sentences S . The general approach to this task as followed by StoryGAN (Li et al., 2019b) is as follows: The story encoder creates the initial representation h 0 of the story S . This is then passed to the context encoder, which is a recurrent model that takes a sentence s k as input and forms a representation o k . Each of these representations o k are then fed to the image generator, which outputs an image  x k . The generated images are passed to two discriminators, the image discriminator and story discriminator, which each evaluate the generated images  x k in different ways and produce a learning signal that can be used to adjust the parameters of the network. The framework of our model is based on the StoryGAN architecture. We improve upon the context encoder and expand the network with dual learning and copy-transform mechanisms. The image and GRUI m age G ene r a t o r MART Transformer MemoryModule Story Encoder F u ll y C onne c t ed R epa r a m e t e r i z a t i on C ap t i on s P oo li ng Copy Transform Video Captioning Encoder Decoder C ap t i on s Story Discriminantor 3D Convolution Fully Connected Image Discriminantor 2D Convolution Fully Connected  X emb Real/FakePrediction Real/FakePrediction F il t e r i ng S   2 h 0 h 0 m t  1 m t c k g k o k Figure 2: Illustration of DUCO-STORYGAN architecture. story discriminators, and the story encoder from the original model are retained in DUCO-STORYGAN; each contributes to a separate loss term i.e. L img , L story and LKL respectively. See Appendix for details on the loss terms. An overview of our model architecture can be seen in Figure 2. MART Context Encoder. One of the primary challenges of story visualization is maintaining consistent background imagery and character appearances throughout the story. This is addressed with a recurrent context encoder which has access to the global narrative while encoding the caption in each time-step. We use the Memory Augmented Recurrent Transformer (MART) (Lei et al., 2020), where the memory is initialized with the conditioning vector h 0 from the story encoder. It takes word embeddings W k = [ w k 1 , w k 2 , ....w kL ] where w ij  R 1  d w , corresponding to the frame caption at each timestep and produces contextualized embeddings which are then pooled to a single weighted representation c k using attention. This allows the context encoder to capture sophisticated interactions among the words which the image generator can then capitalize on: [ m k 1 , ....m kL ] , h k = MART ([ w k 1 , ....w kL ] , h k  1 ) c k = L (cid:88) i =1  ki m ki ;  ki = exp ( m Tki u ) (cid:80) exp ( m Tki u ) where u is a query vector learned during training. The Transformer encoder is followed by a layer of GRU cells that take the contextualized embedding as input along with isometric Gaussian noise, (cid:15) k , and produce an output vector g k . The outputs c k and g k are concatenated and transformed into filters, and subjected to convolution with a projection of the sentence embedding s k , resulting in output vector o k . See Appendix for more details. Image Generator. The image generator follows prior text-to-image generation approaches (Qiao et al., 2019; Xu et al., 2018; Zhang et al., 2017) and uses a two-stage approach. The first stage uses outputs o k ; the resulting image is fed through a second stage, which aligns the contextualized word encodings m k from MART with image subregions generated in first-stage and reuses weighted encodings for image refinement. Dual Learning via Video Redescription. Dual learning provides the model with an additional learning signal by taking advantage of the duality of certain tasks, i.e., if X can be used to produce Y, then Y can be used to produce X. Here, our primary task is story visualization, and we consider the secondary task of video captioning. We refer to this process as video redescription. To execute the idea of learning via video redescription, we employ a video captioning network which takes the sequence of generated images and produces a corresponding sequence of captions. The video captioning network is based on a recurrent encoder-decoder framework ( V enc ( . ) , V dec ( . ) ) and is trained using a cross-entropy loss on the predicted probability distribution ( p ) over its vocabulary. Specifi-cally, L dual = (cid:80) Tk =1 (cid:80) Li =1 log p ki ( w ki ) . The hidden state in recurrent model helps the captioning network to identify narrative elements in the sequence of images and penalize the generative model for a lack of consistency in addition to semantic misalignment. We pretrain the video captioning network using ground truth data and freeze its parameters while training the generative model. We also include a multiplier,  dual , which allows us to scale the loss. The implementation of the encoder-decoder framework can vary. For our primary model, we adapt the MART video captioning network (Lei et al., 2020) to accept a 2D matrix of features at each time step where each column corresponds to an image sub-region (see Sec. 5). Sequentially-Consistent Story Visualization. While certain components, such as character positions, will change from frame to frame, there are other components like background and appearances which usually carry over to adjacent frames. To take advantage of this continuity, we augment the model with a copy-transform mechanism. This mechanism can take into consideration the generated image from previous timesteps, and reuse aspects of those prior images during the current timestep. The copy-transform module F copy ( . ) performs attention-based semantic alignment (Xu et al., 2018) between word features m k  RD w  L in the current timestep and image features i k  1  RD i  N from previous step. Each column of i k  1 is a feature vector of a sub-region of the image. The word features are first projected into the same semantic space as image features i.e. m (cid:48) k = Um k , where U  RD (cid:48)  D . For the j th image sub-region, the word-context vector is calculated as: c jk = L (cid:88) i =0  ji m (cid:48) ik ;  jik = exp ( h Tj m (cid:48) ik ) (cid:80) Li =0 exp ( h Tj m (cid:48) ik )  jik indicates the weight assigned by the model to the i th word when generating the j th sub-region of the image. The weighted word-context matrix is then concatenated with the generative image features from the current timestep and sent for upsam-pling to the image generator. Objective. Bringing it all together, the final objective function of the generative model is: min  G max  I , SLKL + L img + L story +  dual L dual where  G ,  I and  S denote the parameters of the entire generator, and image and story discriminator respectively. See Appendix for more details. Dataset. We utilize the Pororo-SV dataset from the original StoryGAN paper which has been adapted from a video QA dataset based on animated series (Li et al., 2019b) 2 . Each sample in Pororo-SV contains 5 consecutive pairs of frames and captions. The original splits of Pororo-SV from Li et al. (2019b) contain only training and test splits with nearly 80% overlap in individual frames. For a more challenging evaluation, we use the test split proposed in (Li et al., 2019b) as validation split (2,334 samples) and carve out an "unseen" test split from the training examples. The resulting dataset contains 10191, 2334 and 2208 samples in training, validation and test splits respectively. In this version, there is 58% frame overlap between the validation and train splits and 517 samples in the validation split contain at least one frame which is not present in the training set. Conversely, the test split has zero overlap with the training split. Experimental Settings. Our model is developed using PyTorch, building off of the original StoryGAN codebase. All models are trained on the proposed training split and evaluated on validation and test sets. We select the best checkpoints and tune hyperparameters by using the character classification F-Score on validation set (see Appendix). As with any task, evaluation is a critical component of story visualization; however, due to the complexity of the task and its generative nature, evaluation is nontrivial. For instance, characters are the focal point of any narrative and similarly should be the focus of a model when producing images for the story. Hence, Li et al. (2019b) measure the character classification accuracy within frames of generated visual stories in order to compare models. However, it is also important that the characters and background are consistent in appearance, and together form a cohesive story rather than an inde-pendent set of frames. Inspired by insights such as this, we explore an additional set of evaluation metrics that capture diverse aspects of a model's performance on visual story generation. Character Classification. We finetune the pretrained Inception-v3 (Szegedy et al., 2016) with a multi-label classification loss to identify characters in the generated image. Most earlier work in story visualization report the image-level exact-match (EM) character classification accuracy. However, 2 We opt to not use the CLEVR-SV dataset as we believe that this dataset lacks a narrative structure and is not suitable for story visualization. we contend that the exact match accuracy is not sufficient to gauge the performance of generative models, and the micro-averaged F-score of character classification should also be reported. For example, if Model A generates one of two characters in a frame with better quality than Model B (which generates none), it results in the same EM accuracy as Model B but an improvement in the recall/F-Score of the model, making the latter more reliable as a metric for quality. Our conclusion is based on the observation of consistent improvement in character classification scores with increasing training epochs and manual evaluation of image quality (see Fig 4). Video Captioning Accuracy. In order to measure global semantic alignment between captions and generated visualizations, we propose to use video captioning models which have been pretrained on ground truth data to identify narrative elements in a sequence of frames. We use the Memory-Augmented Recurrent Model proposed in Lei et al. (2020) and add a CNN encoder (Sharma et al., 2018) on top of the Transformer encoder to extract image embeddings. The final convolutional layer ( Mixed_7c ) in finetuned Inception-v3 is used to extract a local feature matrix f  R 64  2048 (reshaped from 2048  8  8 ) for each image in the story. We then use this trained video captioning model to caption the generated frames. The generated captions are compared to the ground truth captions via BLEU evaluation 3 , and this functions as our proposed metric for measuring global semantic-alignment between the captions and generated story. This pretrained model is also used as the video captioning dual network during training of DUCO-STORYGAN. Discriminative Evaluation. Generative metrics such as BLEU are known to be noisy and unreliable. Hence, we also develop a discriminative evaluation setup. In order to compute similarity between generated image and ground truth, we compare the feature representations from either images in this discriminative setup. The training dataset for story visualization may contain one or more frames with the exact set of characters that are referenced in captions in the evaluation data. When we are checking for the presence of these characters in a generated image, we do not want to reward the model for copying the exact same frame from the training set 3 We use the nlg-eval package (Sharma et al., 2017) for BLEU evaluation. instead of generating a frame suited to the input caption. In order to evaluate this consistency, we propose discriminative evaluation of the story visualization model. Using the character annotations for the final frame of each sample in the test splits, we extract a set of 4 negative frames which are taken from elsewhere in the video but contain those specific characters (see Fig. 7 in Appendix). The human evaluation accuracy on this dataset is 89% (  =0.86) and is used as an upper bound when interpreting model accuracy performance. The cosine similarity between Inception-v3 features of final generated frame and candidate frames is computed and the frame with most similarity is selected as predicted frame. We report Top-1/2 accuracies. R-Precision. Several prior works on text-to-image generation report the retrieval-based metric R-Precision (Xu et al., 2018) for quantifying the semantic alignment between the input text and generated image. If there are R relevant documents for a query, the top R ranked retrieval results of a system are examined; if r are relevant, the R-precision is r/R . In our task 4 , R = 1 . The encodings from a pretrained Deep Attention-based Multimodal Similarity Model (DAMSM) are used to compute cosine similarity and rank results. Since this model only evaluates a single text-image pair for similarity, it is not suitable for evaluating story visualization. Therefore, we train a new version of DAMSM to extract global representations for the story and sequence of images, referred to as Hierarchical DAMSM (H-DAMSM) (see Appendix). The models used in the aforementioned evaluation metrics are trained independently of DUCOSTORYGAN on the proposed Pororo-SV splits and the pretrained weights are used for evaluation. See Appendix for other upper bounds. The results for Pororo-SV validation set can be seen in Table 1. The first row contains the results using the original StoryGAN model (Li et al., 2019b) 5 . The second row functions as another 4 The R-precision score is obtained from 10 runs with 99 randomly picked mismatched story candidates in each run. 5 We use a reduced training dataset as compared to the original StoryGAN paper (see Sec 4). However, we evaluate our StoryGAN code base on their exact splits and get 26.1% exact-match accuracy, which is approximately equivalent to the 27% reported in the original paper where they demonstrate that StoryGAN outperforms previous baselines such as ImageGAN, SVC, and SVFN. baseline, where we replace the GRU-based context encoder in StoryGAN with a Bidirectional Transformer (Devlin et al., 2019). The conditioning augmentation vector is not used to initialize the context encoder in this model since a non-recurrent Transformer lacks a hidden state. We see 1-2% improvements in character classification and retrieval with this model over StoryGAN. The third row contains results from the more recent CP-CSV model (Song et al., 2020) which uses figure-ground segmentation as an auxiliary task for preserving character features. Consequently, it results in 2.68% improvement in character classification over StoryGAN and smaller improvements for other metrics. The final row contains results with DUCOSTORYGAN, which significantly outperforms previous models (including CP-CSV) across all metrics. The character classification F-Score improves by 7.16% suggesting that the characters generated in our images are of higher visual quality. Similarly, we see consistent improvements in BLEU as well as R-Precision with our model. As demonstrated in Sec 7.1, the improvement in BLEU can be attributed to the addition of dual learning, which directly optimizes the dual task of video captioning. The R-Precision indicates that our model learns better global semantic alignment between the captions and images. Lastly, the Top-1/2 accuracy scores show that our model is learning to generate diverse images, rather than copying scenes that feature the same characters from the training data. DUCO-STORYGAN performs dramatically better than other models on the unseen test split (see Table 2). As can be seen in Fig 3, StoryGAN performs rather poorly on unseen samples compared to DUCO-STORYGAN. While the former produces images that are blurry and character shapes that are faint, the latter generates frames with sharp character features. This is reflected in the wide improvement margins on character classification scores in Table 2. Similar improvements are also observed for BLEU and R-Precision metrics, indicating that our model generates images which are more relevant to the input caption. When generating stories for the Pororo-SV test split, models tend to copy background elements from the samples seen in the training set, since the captions lack sufficient information about the setting. Hence, we observe little improvement over random chance in the discriminative accuracy scores for different models on test split. For instance, instead of generating the tinted background in ground truth in Fig. 3, the models produce a clear blue sky which is closer to samples seen in the training set. However, discriminative evaluation will be valuable for future work in this domain when inputs contain detailed information about the visual elements. We also provide per character results for the Character F-Score. With DUCO-STORYGAN, we see up to 20% improvement for less frequent char-Win % Mean Rating Attribute Ours StGAN Ours StGAN Visual Quality 82% 3% 2.06 1.22 Consistency 78% 3% 2.94 1.78 Relevance 26% 2% 1.28 1.04 Table 3: Human evaluation on Likert Scale 1-5. We conduct human evaluation on the generated images from DUCO-STORYGAN and StoryGAN, using the three evaluation criteria listed in Li et al. (2019b): visual quality, consistence, and relevance. Two annotators are presented with a caption and the generated sequence of images from both models, and asked to rate each sequence on a scale of 1-5. Results are presented in Table 3. With respect to pairwise evaluation, predictions from our model is nearly always preferred over those from StoryGAN (see Win% columns). Similarly, we see large improvements in mean rating of stories generated by DUCO-STORYGAN. However, we also see higher Tie% and low mean rating for the attribute Relevance, suggesting that much work remains to be done to improve understanding of captions. Correlation Experiments: We also examine the correlation between our proposed metrics and human evaluation of generated images. We compute the Pearson's correlation coefficient between human ratings of 50 samples on three different attributes using the 1-5 Likert scale and their corresponding automated metric evaluation scores. Significant correlation (  = 0 . 586 ) was observed between our proposed Character F-Score metric and Visual Quality, lending strength to its use an automated metric for story visualization. Table 4 contains plus-one ablations for DUCOSTORYGAN. The first row is the StoryGAN baseline and the second row is the StoryGAN + Transformer model, as discussed in Section 6. We then iteratively add each of our contributions and observe the change in metrics 6 . First, we upgrade the 6 Statistical significance is computed with 100K samples using bootstrap (Noreen, 1989; Tibshirani and Efron, 1993). Transformer encoder to MART, which brings about the largest improvements across all metrics. The use of word embeddings with access to global conditioning vector and attention-based semantic alignment proves important to the task of story generation. Next, we use the MART context encoder with our proposed dual learning and copy-transform improvements. With the addition of video captioning as a learning signal, we see 0.20% (p=0.071) improvement in character F-score and 1.12% improvement in R-Precision (p=0.032) over MART. The highest improvements are observed for BLEU score, since the model is optimized on video captioning. Next, we evaluate the addition of the copy-transform mechanism where features from generated images in previous timesteps are copied to the image in current timestep. We observe 1.04% improvements for character classification and a slight drop in performance on video captioning. Similarly, there is 1.14% improvement in Top-1 accuracy for the discriminative dataset. As discussed in Section 3, we explore a variety of implementations for the dual learning component of our model. While MART-based video captioning works the best, we provide a discussion of other approaches in the Appendix. Figure 5 contains two generated examples from the Pororo-SV dataset. The top row in each example contains the ground truth images, the middle row the images generated by StoryGAN, and the final row the images generated by our model. In All our improvements in DUCO-STORYGAN are statistically significant, except for discriminative evaluation and frame accuracy scores for the dual learning module. split of Pororo-SV dataset. example A, we demonstrate the superior visual quality and consistency of the frames generated by DUCO-STORYGAN, as compared to StoryGAN. The MART encoder allows our model to comprehend long captions as well as attend to each word while generating images. The retention of native character features throughout the story during regeneration can be attributed to the copy-transform mechanism in our model. In contrast, we see that both models fail at generating defined characters in example B. This may be due to the fact that Poby is an infrequent character in the dataset and hence, both models fail to learn its features. We perform visual analysis of the captions and predictions from DUCO-STORYGAN and observe two major recurring themes. First, the frequency of characters in the training data is a significant deciding factor for generated image quality. We looked at the samples that contained at least Pororo (most frequent character) and found that generated stories are better when there is only a single character in the frame's narrative as compared to multiple characters. This points to the inability of current story visualization models to align captions with multiple subjects/objects to the corresponding images. Second, generated images are poor for scenes containing infrequently occurring objects such as book/plate/boat/plane etc. in the caption. This behavior is expected since the model is unaware of real-world objects that do not already appear in the training set with sufficient frequency. Moreover, since the Pororo-SV dataset has been adapted from the annotations of a video QA dataset, the captions often contain information that can only span over multiple frames (Pororo wakes up and then says what to Crong. Pororo stares at Pororo and throws a ball), or cannot be visualized through images (Poby gets the box and Eddy asks not to open it.). Hence, our results with metrics like BLEU and R-Precision which are supposed to capture the relevance between images and caption stay relatively low (see Tables 1 and 2). In this paper, we investigate the underexplored task of story visualization. We improve upon prior modeling approaches and demonstrate the effectiveness of these new approaches by performing a robust set of ablation experiments. We also present a detailed set of novel evaluation methods, which we validate by demonstrating improvements across various baselines. Evaluation for story visualization is a challenging open research question in itself, and we hope that these methods will encourage more work in this domain. From an ethics standpoint, we provide a brief overview of the data that the model is trained on in Section 4 and a more detailed discussion in the Appendix. We provide some analyses of the data and refer the reader to the original StoryGAN paper, where the dataset was created, for further details. All of the language data consists of simple English sentences. Our experimental results are specific to the story visualization task. Pororo-SV is the most challenging story visualization task available; therefore, our results would likely generalize to other story visualization datasets. While story visualization is an exciting task with many potential future applications, the generated images still contain many obvious visual artifacts and therefore models trained on this task are still far from being deployed in any real world settings. Story visualization minimizes many of the ethical issues associated with image and video generation. DeepFakes, which are algorithmically generated fake images, have become increasingly problematic (Nguyen et al., 2019). Oftentimes, these images are indistinguishable from real images, raising privacy concerns and providing a source of misinformation. The images that we generate here are not subject to this same issue, due to the fact that they are Cartoons, and are therefore unable to be confused with real images. The focus of the task is not on the realism of the images, but rather on the multimodal narrative. Therefore, cartoons are actually better suited for the task as real-world images only add additional visual complexity that is not relevant to the narrative. We thank Peter Hase, Jaemin Cho, Hyounghun Kim, and the reviewers for their useful feedback. This work was supported by DARPA MCS Grant #N66001-19-2-4031, DARPA KAIROS Grant #FA8750-19-2-1004, ARO-YIP Award W911NF-18-1-0336, and a Google Focused Research Award. The views are those of the authors and not of the funding agency.
abstain abstain abstain abstain method abstain abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method other abstain other other other abstain other other other other abstain other abstain other other other other other other other method other other other other other other other other other other other other method method other abstain abstain method abstain abstain abstain method abstain other abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain method method abstain method abstain abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain result result abstain abstain result abstain abstain abstain abstain result abstain result abstain abstain result result method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain result abstain method result abstain abstain method abstain method result abstain abstain method abstain abstain result abstain result abstain abstain abstain abstain result objective objective objective method method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain other other other