Information Theoretic Co-Training
This paper introduces an information theoretic co-training objective for unsupervised learning. We consider the problem of predicting the future. Rather than predict future sensations (image pixels or sound waves) we predict ``hypotheses** to be confirmed by future sensations. More formally, we assume a population distribution on pairs where we can think of as a past sensation and as a future sensation. We train both a predictor model and a confirmation model where we view as hypotheses (when predicted) or facts (when confirmed). For a population distribution on pairs we focus on the problem of measuring the mutual information between and . By the data processing inequality this mutual information is at least as large as the mutual information between and under the distribution on triples defined by the confirmation model . The information theoretic training objective for and can be viewed as a form of co-training where we want the prediction from to match the confirmation from . We give experiments on applications to learning phonetics on the TIMIT dataset.
 This paper presents a novel algorithm for unsupervised learning based on information theoretic co-training. The objective is to find parameters that maximize the mutual information between x and y through hidden variable z without computing the probabilities of x and y directly. This means that this technique allows to find a structure from data in an unsupervised way without modeling x and y explicitly. The proposed method is applied to unsupervised learning of phonemes on TIMIT corpus. The experimental results show that the learnt representations correspond to the labeled phonemes 35.1% on all frames. The idea looks original and the formulation is presented precisely. Comprehensive comparisons to related models are given. The experimental results are not very strong, but the authors claim that they are initial experiments and can be improved with just a little more experimentation. The experiments could be improved to show the effectiveness of the proposed approach better. The experimental results show that it can learn something but it is difficult to know how good the number is. Probably, it is useful to compare the method with some simple baseline even if it is not the objective of the paper to achieve high accuracy. Also, I am not sure if the phoneme discovery is the best task for the experiments. If the purpose is to discover phoneme as accurate as possible, probably, one might take another approach. Probably, it is better to emphasize that this is just to show the ability of the approach. It will be useful to show a list of possible applications. If more experiments can be added with such applications, it will be great. I would recommend to have a figure to explain the procedure of the algorithm, i.e., how to update phi and psi.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Information Theoretic Co-Training
This paper introduces an information theoretic co-training objective for unsupervised learning. We consider the problem of predicting the future. Rather than predict future sensations (image pixels or sound waves) we predict ``hypotheses** to be confirmed by future sensations. More formally, we assume a population distribution on pairs where we can think of as a past sensation and as a future sensation. We train both a predictor model and a confirmation model where we view as hypotheses (when predicted) or facts (when confirmed). For a population distribution on pairs we focus on the problem of measuring the mutual information between and . By the data processing inequality this mutual information is at least as large as the mutual information between and under the distribution on triples defined by the confirmation model . The information theoretic training objective for and can be viewed as a form of co-training where we want the prediction from to match the confirmation from . We give experiments on applications to learning phonetics on the TIMIT dataset.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper presents a novel algorithm for unsupervised learning based on information theoretic co-training. The objective is to find parameters that maximize the mutual information between x and y through hidden variable z without computing the probabilities of x and y directly. This means that this technique allows to find a structure from data in an unsupervised way without modeling x and y explicitly. The proposed method is applied to unsupervised learning of phonemes on TIMIT corpus. The experimental results show that the learnt representations correspond to the labeled phonemes 35.1% on all frames. The idea looks original and the formulation is presented precisely. Comprehensive comparisons to related models are given. The experimental results are not very strong, but the authors claim that they are initial experiments and can be improved with just a little more experimentation. The experiments could be improved to show the effectiveness of the proposed approach better. The experimental results show that it can learn something but it is difficult to know how good the number is. Probably, it is useful to compare the method with some simple baseline even if it is not the objective of the paper to achieve high accuracy. Also, I am not sure if the phoneme discovery is the best task for the experiments. If the purpose is to discover phoneme as accurate as possible, probably, one might take another approach. Probably, it is better to emphasize that this is just to show the ability of the approach. It will be useful to show a list of possible applications. If more experiments can be added with such applications, it will be great. I would recommend to have a figure to explain the procedure of the algorithm, i.e., how to update phi and psi.
assistant
 This paper presents a novel algorithm for unsupervised learning based on information theoretic co-training. The objective is to find parameters that maximize the mutual information between x and y through hidden variable z without computing the probabilities of x and y directly. This means that this technique allows to find a structure from data in an unsupervised way without modeling x and y explicitly. The proposed method is applied to unsupervised learning of phonemes on TIMIT corpus. The experimental results show that the learnt representations correspond to the labeled phonemes 35.1% on all frames. The idea looks original and the formulation is presented precisely. Comprehensive comparisons to related models are given. The experimental results are not very strong, but the authors claim that they are initial experiments and can be improved with just a little more experimentation. The experiments could be improved to show the effectiveness of the proposed approach better. The experimental results show that it can learn something but it is difficult to know how good the number is. Probably, it is useful to compare the method with some simple baseline even if it is not the objective of the paper to achieve high accuracy. Also, I am not sure if the phoneme discovery is the best task for the experiments. If the purpose is to discover phoneme as accurate as possible, probably, one might take another approach. Probably, it is better to emphasize that this is just to show the ability of the approach. It will be useful to show a list of possible applications. If more experiments can be added with such applications, it will be great. I would recommend to have a figure to explain the procedure of the algorithm, i.e., how to update phi and psi.