Answers.Body: <p><em>I tend to think this question is border-line and may get close. A few comments for now, though.</em></p>
  
  <hr>
  
  <p>wrongx
  There are (at least) two issues with reproducing the work of a company like DeepMind:</p>
  
  <ul>
  <li>Technicalities missing from publications.</li>
  <li>Access to the same level of data.</li>
  </ul>
  
  <p>Technicalities should be workable. Some people have reproduced some of the <a href="https://github.com/kristjankorjus/Replicating-DeepMind" rel="nofollow noreferrer">Atari</a> gaming stunts. AlphaGo is seemingly more complex and will require more work, yet that should be feasible at some point in the future (individuals may lack computing resources today).</p>
  
  <p>Data can be more tricky. Several companies open their data sets, but data is also the nerve of the competition...</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-04T08:02:33.260Z
Answers.Text: I'm actually trying to find those borders... would you say it is off-topic? Too broad? Or what?
Answers.UserDisplayName: rcpinto
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-04T08:09:37.123Z
Answers.Text: I am not decided yet. I wonder what matters w.r.t. AI if we can or cannot reproduce some company's claims. I can see people questioning themselves about it and come here to get some answers, but we are not really talking about AI. Your question is still young. Let's see the community's decision. I find it "border-line acceptable".
Answers.UserDisplayName: Eric Platon
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-04T07:58:33.453Z
Answers.LastActivityDate: 2018-04-03T19:47:18.407Z
Answers.LastEditDate: 2018-04-03T19:47:18.407Z
Answers.UserDisplayName: Eric Platon
Answers.Body: <p>On the suggestion of the O.P. rcpinto I converted a comment about seeing "around a half-dozen papers that follow up on Graves et al.'s work which have produced results of the  caliber" and will provide a few links. Keep in mind that this only answers the part of the question pertaining to NTMs, not Google DeepMind itself, plus I'm still learning the ropes in machine learning, so some of the material in these papers is over my head; I did manage to grasp much of the material in <a href="https://arxiv.org/pdf/1410.5401.pdf">Graves et al.'s original paper</a>{1] though and am close to having homegrown NTM code to test. I also at least skimmed the following papers over the last few months; they do not replicate the NTM study in a strict scientific manner, but many of their experimental results do tend to support the original at least tangentially:</p>
  
  <p>• In <a href="https://arxiv.org/pdf/1607.00036.pdf">this paper</a> on a variant version of NTM addressing, Gulcehere, et al. do not try to precisely replicate Graves et al.'s tests, but like the DeepMind team, it does demonstrate markedly better results for the original NTM and several variants over an ordinary recurrent LSTM. They use 10,000 training samples of a Facebook Q&amp;A dataset, rather than the N-grams Graves et al. operated on in their paper, so it's not replication in the strictest sense. They did however manage to get a version of the original NTM and several variants up and running, plus recorded the same magnitude of performance improvement.<a href="https://arxiv.org/pdf/1607.00036.pdf">2</a></p>
  
  <p>• Unlike the original NTM, <a href="https://arxiv.org/abs/1505.00521">this study</a> tested a version of reinforcement learning which was not differentiable; that may be why they were unable to solve several of the programming-like tasts, like Repeat-Copy, unless the controller wasn't confined to moving forwards. Their results were nevertheless good enough to lend support to the idea of NTMs. A more recent revision of their paper is apparently available, which I have yet to read, so perhaps some of their variant's problems have been solved.<a href="https://arxiv.org/abs/1505.00521">3</a></p>
  
  <p>• Instead of testing the original flavor of NTM against ordinary neural nets like LSTMs, <a href="https://arxiv.org/pdf/1510.03931v3.pdf">this paper</a> pitted it against several more advanced NTM memory structures. They got good results on the same type of programming-like tasks that Graves et al. tested, but I don't think they were using the same dataset (it's hard to tell from the way their study is written just what datasets they were operating on).<a href="https://arxiv.org/pdf/1510.03931v3.pdf">4</a> </p>
  
  <p>• On p. 8 of <a href="https://arxiv.org/pdf/1605.06065.pdf">this study</a>, an NTM clearly outperforms several LSTM, feed-forward and nearest-neighbor based schemes on an Omniglot character recognition dataset. An alternative approach to external memory cooked up by the authors clearly beats it, but it still obviously performs well. The authors seem to belong to a rival team at Google, so that might be an issue when assessing replicability.<a href="https://arxiv.org/pdf/1605.06065.pdf">5</a></p>
  
  <p>• On p. 2 <a href="http://www.thespermwhale.com/jaseweston/ram/papers/paper_6.pdf">these authors</a> reported getting better generalization on "very large sequences" in a test of copy tasks, using a much smaller NTM network they evolved with the genetic NEAT algorithm, which dynamically grows topologies.<a href="http://www.thespermwhale.com/jaseweston/ram/papers/paper_6.pdf">6</a>  </p>
  
  <p>NTMs are fairly new so there hasn't been much time to stringently replicate the original research yet, I suppose. The handful of papers I skimmed over the summer, however, seem to lend support to their experimental results; I have yet to see any that report anything but excellent performance. Of course I have an availability bias, since I only read the pdfs I could easily find in a careless Internet search. From that small sample it seems that most of the follow-up research has been focused on extending the concept, not replication, which would explain the lack of replicability data. I hope that helps.</p>
  
  <p><a href="https://arxiv.org/pdf/1410.5401.pdf">1</a> Graves, Alex; Wayne, Greg and Danihelka, Ivo, 2014, "Neural Turing Machines," published Dec. 10, 2014. </p>
  
  <p><a href="https://arxiv.org/pdf/1607.00036.pdf">2</a> Gulcehre, Caglar; Chandar, Sarath; Choy, Kyunghyun and Bengio, Yoshua, 2016, "Dynamic Neural Turing machine with Soft and Hard Addressing Schemes," published June 30, 2016. </p>
  
  <p><a href="https://arxiv.org/abs/1505.00521">3</a> Zaremba, Wojciech and Sutskever, Ilya, 2015, "Reinforcement Learning Neural Turing Machines," published May 4, 2015. </p>
  
  <p><a href="https://arxiv.org/pdf/1510.03931v3.pdf">4</a> Zhang; Wei; Yu, Yang and Zhou, Bowen, 2015, "Structured Memory for Neural Turing Machines," published Oct. 25, 2015.    </p>
  
  <p><a href="https://arxiv.org/pdf/1605.06065.pdf">5</a> Santoro, Adam; Bartunov, Sergey; Botvinick, Matthew; Wierstra, Daan and Lillicrap, Timothy, 2016, "One-Shot Learning with Memory-Augmented Neural Networks," published May 19, 2016.    </p>
  
  <p><a href="http://www.thespermwhale.com/jaseweston/ram/papers/paper_6.pdf">6</a> Boll Greve, Rasmus; Jacobsen, Emil Juul and Sebastian Risi, date unknown, "Evolving Neural Turing Machines." No publisher listed</p>
  
  <p>All except (perhaps) Boll Greve et al. were published at the Cornell Univeristy Library arXiv.org Repository: Ithaca, New York.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-10-28T09:00:13.520Z
Answers.LastActivityDate: 2016-10-28T09:00:13.520Z
Answers.UserDisplayName: SQLServerSteve
Body: <p>DeepMind has published a lot of works on deep learning in the last years, most of them are state-of-the-art on their respective tasks. But how much of this work has actually been reproduced by the AI community? For instance, the Neural Turing Machine paper seems to be very hard to reproduce, according to other researchers.</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-10-27T20:19:20.190Z
Comments.Text: I'm not sure about reproducing the original paper's results, but I've run into around a half-dozen papers that follow up on Graves et al.'s work which have produced results of the caliber. Most are on variants of the NTM theme. I can post some links if that would help.
Comments.UserDisplayName: SQLServerSteve
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-10-28T01:01:14.450Z
Comments.Text: This comment + links would be a good actual answer.
Comments.UserDisplayName: rcpinto
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-10-28T08:08:27.243Z
Comments.Text: I'll convert it into an answer shortly, as soon as I can hunt up the web addresses again.
Comments.UserDisplayName: SQLServerSteve
Tags: <neural-networks><deep-learning><research><deepmind><neural-turing-machine>
Title: How much of Deep Mind's work is actually reproducible?
UserDisplayName: rcpinto
ThreadText: # How much of Deep Mind's work is actually reproducible?
  
  (Asked by: username_0 on 2016-08-04)
  
  DeepMind has published a lot of works on deep learning in the last years, most of them are state-of-the-art on their respective tasks. But how much of this work has actually been reproduced by the AI community? For instance, the Neural Turing Machine paper seems to be very hard to reproduce, according to other researchers.
  
  **Tags:** neural-networks, deep-learning, research, deepmind, neural-turing-machine
  > **Comments:**
  > **username_1**: I'm not sure about reproducing the original paper's results, but I've run into around a half-dozen papers that follow up on Graves et al.'s work which have produced results of the caliber. Most are on variants of the NTM theme. I can post some links if that would help.  
  > **username_0**: This comment + links would be a good actual answer.  
  > **username_1**: I'll convert it into an answer shortly, as soon as I can hunt up the web addresses again.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2016-10-28. Score: 10 | Accepted answer
  
  On the suggestion of the O.P. rcpinto I converted a comment about seeing "around a half-dozen papers that follow up on Graves et al.'s work which have produced results of the caliber" and will provide a few links. Keep in mind that this only answers the part of the question pertaining to NTMs, not Google DeepMind itself, plus I'm still learning the ropes in machine learning, so some of the material in these papers is over my head; I did manage to grasp much of the material in [Graves et al.'s original paper](https://arxiv.org/pdf/1410.5401.pdf){1] though and am close to having homegrown NTM code to test. I also at least skimmed the following papers over the last few months; they do not replicate the NTM study in a strict scientific manner, but many of their experimental results do tend to support the original at least tangentially:
  
  • In [this paper](https://arxiv.org/pdf/1607.00036.pdf) on a variant version of NTM addressing, Gulcehere, et al. do not try to precisely replicate Graves et al.'s tests, but like the DeepMind team, it does demonstrate markedly better results for the original NTM and several variants over an ordinary recurrent LSTM. They use 10,000 training samples of a Facebook Q&A dataset, rather than the N-grams Graves et al. operated on in their paper, so it's not replication in the strictest sense. They did however manage to get a version of the original NTM and several variants up and running, plus recorded the same magnitude of performance improvement.[2](https://arxiv.org/pdf/1607.00036.pdf)
  
  • Unlike the original NTM, [this study](https://arxiv.org/abs/1505.00521) tested a version of reinforcement learning which was not differentiable; that may be why they were unable to solve several of the programming-like tasts, like Repeat-Copy, unless the controller wasn't confined to moving forwards. Their results were nevertheless good enough to lend support to the idea of NTMs. A more recent revision of their paper is apparently available, which I have yet to read, so perhaps some of their variant's problems have been solved.[3](https://arxiv.org/abs/1505.00521)
  
  • Instead of testing the original flavor of NTM against ordinary neural nets like LSTMs, [this paper](https://arxiv.org/pdf/1510.03931v3.pdf) pitted it against several more advanced NTM memory structures. They got good results on the same type of programming-like tasks that Graves et al. tested, but I don't think they were using the same dataset (it's hard to tell from the way their study is written just what datasets they were operating on).[4](https://arxiv.org/pdf/1510.03931v3.pdf)
  
  • On p. 8 of [this study](https://arxiv.org/pdf/1605.06065.pdf), an NTM clearly outperforms several LSTM, feed-forward and nearest-neighbor based schemes on an Omniglot character recognition dataset. An alternative approach to external memory cooked up by the authors clearly beats it, but it still obviously performs well. The authors seem to belong to a rival team at Google, so that might be an issue when assessing replicability.[5](https://arxiv.org/pdf/1605.06065.pdf)
  
  • On p. 2 [these authors](http://www.thespermwhale.com/jaseweston/ram/papers/paper_6.pdf) reported getting better generalization on "very large sequences" in a test of copy tasks, using a much smaller NTM network they evolved with the genetic NEAT algorithm, which dynamically grows topologies.[6](http://www.thespermwhale.com/jaseweston/ram/papers/paper_6.pdf)
  
  NTMs are fairly new so there hasn't been much time to stringently replicate the original research yet, I suppose. The handful of papers I skimmed over the summer, however, seem to lend support to their experimental results; I have yet to see any that report anything but excellent performance. Of course I have an availability bias, since I only read the pdfs I could easily find in a careless Internet search. From that small sample it seems that most of the follow-up research has been focused on extending the concept, not replication, which would explain the lack of replicability data. I hope that helps.
  
  [1](https://arxiv.org/pdf/1410.5401.pdf) Graves, Alex; Wayne, Greg and Danihelka, Ivo, 2014, "Neural Turing Machines," published Dec. 10, 2014.
  
  [2](https://arxiv.org/pdf/1607.00036.pdf) Gulcehre, Caglar; Chandar, Sarath; Choy, Kyunghyun and Bengio, Yoshua, 2016, "Dynamic Neural Turing machine with Soft and Hard Addressing Schemes," published June 30, 2016.
  
  [3](https://arxiv.org/abs/1505.00521) Zaremba, Wojciech and Sutskever, Ilya, 2015, "Reinforcement Learning Neural Turing Machines," published May 4, 2015.
  
  [4](https://arxiv.org/pdf/1510.03931v3.pdf) Zhang; Wei; Yu, Yang and Zhou, Bowen, 2015, "Structured Memory for Neural Turing Machines," published Oct. 25, 2015.
  
  [5](https://arxiv.org/pdf/1605.06065.pdf) Santoro, Adam; Bartunov, Sergey; Botvinick, Matthew; Wierstra, Daan and Lillicrap, Timothy, 2016, "One-Shot Learning with Memory-Augmented Neural Networks," published May 19, 2016.
  
  [6](http://www.thespermwhale.com/jaseweston/ram/papers/paper_6.pdf) Boll Greve, Rasmus; Jacobsen, Emil Juul and Sebastian Risi, date unknown, "Evolving Neural Turing Machines." No publisher listed
  
  All except (perhaps) Boll Greve et al. were published at the Cornell Univeristy Library arXiv.org Repository: Ithaca, New York.
  
  
  ---
  
  ### Answer by username_2 on 2016-08-04. Score: 4
  
  *I tend to think this question is border-line and may get close. A few comments for now, though.*
  
  ---
  
  wrongx
  There are (at least) two issues with reproducing the work of a company like DeepMind:
  
  * Technicalities missing from publications.
  * Access to the same level of data.
  
  Technicalities should be workable. Some people have reproduced some of the [Atari](https://github.com/kristjankorjus/Replicating-DeepMind) gaming stunts. AlphaGo is seemingly more complex and will require more work, yet that should be feasible at some point in the future (individuals may lack computing resources today).
  
  Data can be more tricky. Several companies open their data sets, but data is also the nerve of the competition...
  
  > **Comments:**
  > **username_0**: I'm actually trying to find those borders... would you say it is off-topic? Too broad? Or what?  
  > **username_2**: I am not decided yet. I wonder what matters w.r.t. AI if we can or cannot reproduce some company's claims. I can see people questioning themselves about it and come here to get some answers, but we are not really talking about AI. Your question is still young. Let's see the community's decision. I find it "border-line acceptable".  
  
  ---
  
  