In this paper, we propose the Associative Conversation Model that generates visual information from textual information and uses it for generating sentences in order to utilize visual information in a dialogue system without image input. In research on Neural Machine Translation, there are studies that generate translated sentences using both images and sentences, and these studies show that visual information improves translation performance. However, it is not possible to use sentence generation algorithms using images for the dialogue systems since many text-based dialogue systems only accept text input. Our approach generates (associates) visual information from input text and generates response text using context vector fusing associative visual information and sentence textual information. A comparative experiment between our proposed model and a model without association showed that our proposed model is generating useful sentences by associating visual information related to sentences. Furthermore, analysis experiment of visual association showed that our proposed model generates (associates) visual information effective for sentence generation.
The paper proposes to augment (traditional) text-based sentence generation/dialogue approaches by incorporating visual information.  The idea is that associating visual information with input text, and using that associated visual information as additional input will produce better output text than using only the original input text.
The basic idea is to collect a bunch of data consisting of both text and associated images or video.  Here, this was done using Japanese news programs.  The text+image/video is used to train a model that requires both as input and that encodes both as context vectors, which are then combined and decoded into output text.  Next, the image inputs are eliminated, with the encoded image context vector being instead associatively predicted directly from the encoded text context vector (why not also use the input text to help predict the visual context?), which is still obtained from the text input, as before.  The result is a model that can make use of the text-visual associations without needing visual stimuli.  This is a nice idea.
Actually, based on the brief discussion in Section 2.2.2, it occurs to me that the model  might not really be learning visual context vectors associatively, or, that this doesn't really have meaning in some sense.  Does it make sense to say that what it is really doing is just learning to associate other concepts/words with the input text, and that it is using the augmenting visual information in the training data to provide those associations?  Is this worth talking about?
Unfortunately, while the idea has merit, and I'd like to see it pursued, the paper suffers from a fatal lack of validation/evaluation, which is very curious, given the amount of data that was collected, the fact that the authors have both a training and a test set, and that there are several natural ways such an evaluation might be performed.  The two examples of Fig 3 and the additional four examples in the appendix are nice for demonstrating some specific successes or weaknesses of the model, but they are in no way sufficient for evaluation of the system, to demonstrate its accuracy or value in general.
Perhaps the most obvious thing that should be done is to report the model's accuracy for reproducing the news dialogue, that is, how accurately is the next sentence predicted by the baseline and ACM models over the training instances and over the test data?  How does this compare with other state-of-the-art models for dialogue generation trained on this data (perhaps trained only on the textual part of the data in some cases)?
Second, some measure of accuracy for recall of the associative image context vector should be reported; for example, on average, how close (cosine similarity or some other appropriate measure) is the associatively recalled image context vector to the target image context vector?  On average?  Best case?  Worst case?  How often is this associative vector closer to a confounding image vector than an appropriate one?
A third natural kind of validation would be some form of study employing human subjects to test it's quality as a generator of dialogue.
One thing to note, the example of learning to associate the snowy image with the text about university entrance exams demonstrates that the model is memorizing rather than generalizing.  In general, this is a false association (that is, in general, there is no reason that snow should be associated with exams on the 14th and 15th—the month is not mentioned, which might justify such an association.)
Another thought: did you try not retraining the decoder and attention mechanisms for step 3?  In theory, if step 2 is successful, the retraining should not be necessary.  To the extent that it is necessary, step 2 has failed to accurately predict visual context from text.  This seems like an interesting avenue to explore (and is obviously related to the second type of validation suggested above).  Also, in addition to the baseline model, it seems like it would be good to compare a model that uses actual visual input and the model of step 1 against the model of step 3 (possibly bot retrained and not retrained) to see the effect on the outputs generated—how well do each of these do at predicting the next sentence on both training and test sets?
Other concerns:
1. The paper is too long by almost a page in main content.
2. The paper exhibits significant English grammar and usage issues and should be carefully proofed by a native speaker.
3. There are lots of undefined variables in the Eqs. (s, W_s, W_c, b_s, e_t,i, etc.)  Given the context and associated discussion, it is almost possible to sort out what all of them mean, but brief careful definitions should be given for clarity. 
4. Using news broadcasts as a substitute for true dialogue data seems kind of problematic, though I see why it was done.