en en en pt es en en en en cv az en en en en en en en en en en en en en ca es en es ne en en en en en en en en en fr en en en en en en en de en en en en en en fr en en en en en en en en en en en en en en en en en en en en en en en en zh nl en en en en en en en en en en en en en en it en en en en en en en pt en en en en en en en en en
A month ago, I submitted an updated version of my basic block versioning paper to an academic conference (the previous version can be viewed here). We have an improved system, more working benchmarks and better results across all metrics. I was quite proud of this paper, and thought our odds of getting into the said conference were quite high. Unfortunately, the improved paper was again rejected. Comments from the reviewers were mixed. Some quite positive, others extremely negative and harsh. Notably, the paper was shot down by a famous industry researcher who has done extensive work on trace compilation, a competing approach.
Below are some excerpts of the comments from four different reviewers:
I fail to see the difference to trace compilation (and the predecessor of trace compilation, dynamic binary translation) […] Constant propagation and conditional elimination in a trace compiler lead to the same type check elimination that you present.
I have had a peek at the code and it very elegant. The paper is well written too. The paper has some weaknesses but  I don’t believe that they are show stoppers. […] Overall this is a nice idea well worth looking at in more detail.
The one big problem I have with the paper is that it does not motivate and put into context lazy block versioning properly. The paper needs to do a better job at explaining which precise problems of current Javascript JIT approaches that are used in production are solved by lazy basic block versioning.
In this improved paper, we’ve managed to show that basic block versioning reduced the number of type tests much more effectively than a static analysis could, and that our approach was able to yield significant speedups. We’ve also addressed one of the most important criticisms of such an approach by showing that it does not result in a code size explosion in practice, but only a moderate code size increase. This, however, wasn’t sufficient for some reviewers. The main criticism they raised is that basic block versioning has similarities with tracing, and that as such, we should have compared its performance against that of a tracing JIT.
I have no issue with writing a detailed comparison of both approaches on paper, but it seems the only way to compare the performance of basic block versioning against that of tracing in a fair manner would be for me to implement a tracing JIT compiler in Higgs. Not only that, I’d probably be expected to implement a tracing JIT with all the latest developments in tracing technology. This could well require an entire year of work. I feel like there’s an unfortunate disregard for the amount of infrastructure work required to do compiler research. It’s becoming increasingly difficult to do academic research on compiler architecture. There’s a high amount of pressure to publish often, but at the same time, conferences expect us to pour several man-years into each publication and to immediately provide an absolute proof that ideas brought forward are better than everything else out there. Curious exploration of new ideas is systematically discouraged.
I was understandably feeling quite frustrated and discouraged after this second rejection. This blog post by Daniel Lemire lifted my spirits a little. It’s clear in my mind that basic block versioning has potential. We have something clearly novel, and it works! No matter what, I can at least take some comfort in the knowledge that basic block versioning has already been presented at several industry conferences, such as DConf, mloc.js and Strange Loop.
I may not be doing very well at the publication game thus far, but thousands of people have watched my talks online. The idea has already been spread around, and I’ve gotten positive responses. I haven’t put this much work into Higgs and my thesis to just give up. I’ve already found two conferences where I can submit further updated versions of the paper, and the deadlines are luckily coming up soon. In addition to this, I’ve already started work towards implementing typed shapes in Higgs, which should lead to another publication next year.
From → Grad studies, Higgs
I read the early version of the paper you linked and in my mind there is no need for a comparison to tracing as your approach is focused on profile-driven generation of type specialized variants within a traditional SSA-oriented compiler. I am not terribly familiar with the latest in the tracing world, but it doesn’t seem like the problem you’re solving is the same problem they’re solving. As the reviewer said, the type specialization comes “for free” with tracing, but then you’re stuck with the downsides of tracing you pointed out such as needing to reconstruct the control flow graph from the traces.
In some ways I think you’re both trying to achieve the same goal, but through opposite approaches – your approach starts with inlining a traditional SSA representation to create blocks that can be usefully specialized; the tracing approach starts from traces that must be outlined in order to reconstruct a flow graph (and avoid trace explosion). Your approach further intrigues me because of the possibility of using it in a profile-directed optimization mode to generate meaningful source-level annotations that could be inserted in the code to bypass generation of uncalled block versions in future runs, something I think would be more difficult in tracing.
The level of fighting between academics who favor different approaches or ideas is one of the main reasons I left academia. A colleague of mine who has been finishing his PhD has been suffering this as well, so it sounds like the problem hasn’t gotten any better. It seems like the main “solution” is to find a friendly venue for the publication to bypass the reviewers who want to diss anything that isn’t their religion, but this just leads to even more of a silo problem.
Many years after I left, I read Olin Shivers’ advice on writing dissertations and it occurred to me that many of the issues I had with conference reviewers could have been avoided had I followed his advice. Perhaps there’s a way to focus the abstract and introduction on a clear thesis that can’t be rejected or disputed by a comparison to tracing. But I haven’t read the newest version of your paper, so please don’t take this as a critique of something I haven’t read :-)
I haven’t put the most up to date version online because we’re resubmitting it very soon. I thought it was a very nice improvement over the first version, however. The results are much, much better, we show improvements in execution time, and we worked on improved examples/illustrations to better explain our approach.
I’d be curious to read Olin Shivers’ advice. Do you have a link to a blog post?
Olin’s advice is here: http://www.ccs.neu.edu/home/shivers/diss-advice.html
Perhaps you have already accomplished this in the new version of your paper – which I look forward to reading. It was merely my own reflection that the advice to focus the thesis and repeat it as necessary to demonstrate the point would have helped with some of my reviewer issues – and later was very helpful in a corporate research context.
As someone who got a PhD and left Academia immediately after I would say that this blog and the code you are producing are far more important than some additional publications. Counting publications is a very stupid way to count the points and once outside it because so perfectly clear that the real world does not really care about academic papers :D
So keep posting and coding and do not lose too much time doing useless thing to satisfy some old, unfair reviewer.
Fellow Grad Studnet permalink
I’m currently finishing up a PhD. I’m dealing with into the same thing. What a stupid system. What a waste of time, effort, money, and sanity. I don’t know of any solution other than to leave (hopefully with the PhD though). Good luck, I hope it works out for you!
I am getting a little frustrated at how many barriers are in the way of research in the academic world. It’s pretty clear to me that I don’t want to continue on the academic path after my PhD. I like to try and innovate without having to constantly worry about what is or isn’t publishable and what’s the smallest publishable unit we can try and quickly get out the door.
Academics are often pretty conservative and are generally encouraged to have strong opinions and group into cliques. This makes innovative research extremely hard to publish, especially when some other people already worked on the problem you are trying to address with a different technique. Another problem is the quality of reviews, a recent paper of mine (in PL) got rejected from a prestigious conference with very short reviews filled with typos, misunderstandings, and plain wrong statements. Your area is harsh, because, as you phrase it, the infrastructure necessary for experiments is huge, I think your work is interesting (if only because it is alternative) and deserves publication. You should really be proud that industry people receive your work positively, they are often interested by actual results; as opposed to some bad researchers that spend most of their energy making their ideas the state of the art by simply denying to consider other’s.
Basic Block Versioning – Published At Last! | Pointers Gone Wild
Why I Chose Industry Over a Postdoc | Pointers Gone Wild
Leave a Reply Cancel reply
Enter your comment here...
Fill in your details below or click an icon to log in:
Email (required) (Address never made public)
You are commenting using your WordPress.com account. ( Log Out /  Change )
You are commenting using your Twitter account. ( Log Out /  Change )
You are commenting using your Facebook account. ( Log Out /  Change )
Notify me of new comments via email.
Notify me of new posts via email.
« Dogfooding Higgs with the Turing Turtle
The Constness Problem »
Typed vs Untyped Virtual Machines
Minimalism in Programming Language Design
Democracy, Dictatorships and Access to Information
Code that Doesn’t Rot
The Internet Changed My Life
What’s Could You Use Tesla Bot For?
NoiseCraft: a Browser-Based Visual Programming Language for Sound & Music
Art Artificial Intelligence Artificial Life Assembly Biology Compilers Creativity D Language Electronics Future Game Development Genetic Algorithms Grad studies Graphics Higgs JavaScript Microcontrollers Microprocessors Music Programming Programming Languages Psychology Rant Robotics Tachyon Technology Uncategorized work x86 Zeta
Archives Select Month June 2022  (1) May 2022  (1) March 2022  (1) February 2022  (2) January 2022  (2) December 2021  (1) June 2021  (1) February 2021  (2) December 2020  (1) September 2020  (1) January 2020  (1) November 2019  (2) October 2019  (2) June 2019  (1) January 2019  (1) December 2018  (1) May 2018  (2) February 2018  (1) January 2018  (1) December 2017  (1) October 2017  (1) June 2017  (1) April 2017  (1) January 2017  (1) December 2016  (1) November 2016  (2) September 2016  (2) May 2016  (1) March 2016  (1) February 2016  (2) January 2016  (1) November 2015  (4) October 2015  (1) September 2015  (3) August 2015  (2) July 2015  (3) June 2015  (1) May 2015  (3) April 2015  (2) March 2015  (1) February 2015  (1) December 2014  (1) November 2014  (2) October 2014  (2) September 2014  (3) July 2014  (3) May 2014  (1) April 2014  (1) March 2014  (1) February 2014  (1) January 2014  (3) December 2013  (3) October 2013  (2) September 2013  (1) August 2013  (2) July 2013  (3) June 2013  (2) May 2013  (3) March 2013  (2) February 2013  (1) January 2013  (3) December 2012  (4) November 2012  (1) October 2012  (1) September 2012  (3) July 2012  (4) June 2012  (2) May 2012  (2) April 2012  (4) March 2012  (2) February 2012  (3) January 2012  (6) December 2011  (2) November 2011  (3) October 2011  (8) September 2011  (4) August 2011  (3)