We propose an energy-based model (EBM) of protein conformations that operates at atomic scale. The model is trained solely on crystallized protein data. By contrast, existing approaches for scoring conformations use energy functions that incorporate knowledge of physical principles and features that are the complex product of several decades of research and tuning. To evaluate the model, we benchmark on the rotamer recovery task, the problem of predicting the conformation of a side chain from its context within a protein structure, which has been used to evaluate energy functions for protein design. The model achieves performance close to that of the Rosetta energy function, a state-of-the-art method widely used in protein structure prediction and design. An investigation of the model's outputs and hidden representations finds that it captures physicochemical properties relevant to protein energy.
The paper proposes an Energy-Based-Model (EBM) for scoring the possible configurations of amino acid side chain conformations in protein structures with known amino acid backbone structure. The energy of the side-chain conformation (the chi-angle) for a given amino acid in the structure is calculated as a function of a local neighbourhood of atoms (A), where each atom is embedded into a 256d vector using its cartesian coordinates, atom identity, atom side-chain position and amino acid identity. The model is trained using approximate likelihood where the model samples are generated using precalculated table (from literature) of possible Chi angles conformations conditioned on the back-bone amino acid identity and back-bone angles. The results seem comprehensive comparing the transformer based energy function parameterization with two sensible baselines as well as the Rosetta energy function  which is the de facto standard tool for these types of calculations. Using rotamer recovery accuracy as the benchmark measure the empirical results are close to performance as the Rosetta energy model however always slightly worse. Further visualizations of the energy levels for different Chi angles seems to support that the learned energy function captures well known characteristics of the rotamer configuration energy landscape.
Score:
Overall I think that the paper is solid and tackles an interesting problem of learning energy functions for physical systems from data. The experimental results are comprehensive and mostly supports the claims made in the paper. My main (slightly minor) concern about the paper is the obsession with discarding years of learned knowledge about handcrafted energy functions with fully learned functions. It seems to me that combining domain knowledge with a learned model should close the last small gap (and presumably surpass) the performance of e.g Rosetta. Combining I think the paper should be accepted at the ICLR.
Comment/Questions:
Motivation
Q1.1) Paper motivation: The paper several times seems to suggest that energy functions derived from physical knowledge are problematic (e.g. abstract). For many physical systems I don’t think this is true and would like the author's comments on why learned energy functions are preferable - arguably they can only capture properties present in the data and not any prior knowledge?  
Q1.2) Given that the paper is motivated by fully learning an energy function from data without injecting any prior knowledge i would like the authors comment on the construction of the q(x|c) distribution. This is based on essentially a contingency table between Phi/Psi/Chi angles and to me seems like injecting prior knowledge about physically possible rotamer configurations into the learned procedure? 
Method / Experimental Results:
Q2.1) With respect to the primary results in table 1) and table 2). The authors claim comparable results to the Rosetta Energy function (page 5. Sec 4.3, page 9, sec 6). However the experimental results are always slightly worse than the Rosetta energy function and I (strongly) suggest that the authors rephrase those statements to reflect that.
Q2.2 With Respect to Table 3) Firstly, Why are the only results for 16 amino acids in the table ? Secondly, just naively counting the Atom Transformer are better than Rosetta in 11 of 16 amino acids - this seems slightly at odds with the main result where the Atom Transformer is performing slightly worse than Rosetta?
Clarity
Q3): The notation in section 3 and 4 is slightly confusing. Especially I think q(x|c) is slightly misleading since it, (to my understanding) is a conditional distribution over Chi, conditioned on Psi/Phi/AminoAcid and not rotamer (x) and surrounding molecular context (c).