text:
Q: PHP loop with null check and multiple data query and echo (+ smaller if possible) I need to get the most recent member input from a form.
There are multiple fields in this form.
This is for a Joomla!
Here is what I have so far...
<?php
$user = JFactory::getUser();
$usr_id = $user->get('id');
$db = JFactory::getDbo();
$query = $db->getQuery(true);
$query->select($db->quoteName("ABC1", "ABC2", "ABC3", "ABC4", "ABC5", "ABC6", "ABC7"));
// this goes on up to 150 but I'm testing it out with the first 7.

$query->from($db->quoteName('data_from_the_form_in_db'));
$query->where($db->quoteName('created_by') . ' LIKE '. $usr_id);
//Trying to get the data for specific user.  (Bonus) Not sure yet how to only get the last form input.

$db->setQuery($query);
$res = $db->loadRow();

for ($a='F0'; $a++;){
    echo $a. "\n";
    echo "
";

    if (isset($res [$a])) if ($res [$a] != "") {
        echo "Err.."; exit;
    }

    echo (isset($res [$a]));
    echo "
";

    if  ($a > "F150"); {
       break;
    }

}
echo "the end";
?>

I am playing around with the code and I get into multiple problems with this.
Sometimes I get undefined index, sometimes I get undefined offset, sometimes I get only the first cell echoed.  Also (Bonus), I was wondering what would be the smallest code for all this.  But all in all, I just want something that works. :)
Many thanks in advance for any help.

A: Well I got it to work like this:
<?php
defined('_JEXEC') or die;

$user = JFactory::getUser();
$usr_id = $user->get('id');

   $db = JFactory::getDbo();
   $query="SELECT * FROM #__table WHERE created_by = $usr_id ORDER BY id DESC LIMIT 1";
   $db->setQuery($query);
   $rows=$db->loadObjectList();

   for($a=0; $a<count($rows);$a++){
      $row=$rows[$a];
      $id=$row->id;
   }

   echo '
';
   print_r($rows);

<?


text:
Feature, Issue #18
nonsite.org - Feature - Issue #18
The Dialectics of Damage: Art, Form, Formlessness
A Reply to Jennifer Ashton
By Theodore Martin
Is damaged art still art? There are two ways to approach the question. The first is ontological; it is a question of how much a work of art can be changed, damaged, or altered (the water-logged painting, the shattered sculpture, the abridged novel) and still be thought of as the same work. The other way to approach it is political: as a question of what it means for art to represent or reflect the damage—the compromise, concession, and instrumentalization—that is an inescapable consequence of its place in a market economy and a capitalist world. To put the question this way is to inquire into the conditions of an artwork that doesn't pre-exist its damage, one that arrives already in damaged form; "totaled in advance," to use Jennifer Ashton's excellent phrase. But the artwork that is totaled in advance raises in turn a further question, which is whether it is possible to differentiate intentional damage from unintentional damage, the preemptive from the inevitable. Does art that takes up the unavoidable demands of the market as its subject thereby somehow escape those demands? Or is the very point of art under such conditions simply to demonstrate its own impossibility in the face of the limitless reach of capital itself?
Ben Lerner's 10:04 is either perfectly exemplary of the damage capitalism does to art or paradoxically immune to that damage. It is either a novel that overcomes literature's enmeshment in the market, or it is a novel that was itself written only because the market solicited it. The difficulty of deciding which it is is what this short essay aims to think through. In "Totaling the Damage: Revolutionary Ambition in Recent American Poetry," Ashton argues that "10:04 imagines, in short, that instead of the work of art being subsumed within the inevitable damages of capital, the damages of capital are subsumed within it. …[W]e are looking at a work that has been totaled in advance. Its damage—the fact of it being totaled—is no longer understood as something that has happened to it but as something that is—in the form of the novel—subsumed by it."1
In Ashton's reading, 10:04 is a novel that has intentionally damaged itself—exposed its market concessions, revealed its economic compromises—because it knew that it would eventually end up damaged anyway. In that case, though, what can 10:04 really be showing us other than the inevitability of damage, the unavoidable horizon of capitalist instrumentality? While Ashton reads Lerner's totaled-in-advance novel as one that overcomes its concessions to the market by making them an intentional part of its form, I want to ask whether her theory of damaged form has more complex, less salutary consequences for how we understand the fate of art in a time of capitalist crisis. What is there that prevents the work that is "totaled in advance" from being nothing more than a record of the impossibility of avoiding damage, avoiding the market, and avoiding capital? What allows us to think that the role of art in a damaged world can be anything other than to remind us that there is no longer such a thing as undamaged art?
For Ashton, these questions are resolved by attending to the difference between being damaged and subsuming the damage; the difference, if you like, between sabotage and self-sabotage. The intentional self-consciousness of the latter, Ashton suggests, offers one solution to the structural inevitability of the former. In both cases, however, the result is exactly the same: a damaged novel. So how can we tell what kind of damage we're looking at? This question is raised most explicitly in 10:04 with respect to the New Yorker short story that Lerner published in 2012, and that became the basis not just for 10:04 but for the "'strong six-figure' advance" he was given in order to write it.2 The story was damaged from the start. The New Yorker editors "wanted a major cut…the section I considered the story's core." The narrator of 10:04—also named Ben—initially balks: "I wasn't going to make a cut whose primary motivation was, on some level, the story's marketability" (56). But then he does exactly that: "The next day my agent helped me word my mea culpa…. The magazine was gracious and decided to run the revised story quickly" (57). Of course, we are reading about all of this in a novel, as the experience of writing the story and being forced to cut it is then folded back into the narrative of 10:04, which includes the verbatim text of the published story. This is the "subsumption" that Ashton is talking about: the primal scene of Lerner's capitulation to "marketability" reappears in 10:04 as a confession of that capitulation that doubles, Ashton suggests, as its negation ("insofar as it's part of the meaning of the work, part of what the artist intends for it as a whole, it also becomes integral to the form of the work." Yet the re-publication of the compromised story within 10:04 may only be further evidence of the novel's compromise. 10:04 makes much of its refusal to conform to the expectations of popular literary fiction. "'Develop a clear, geometrical plot; describe faces … ; make sure the protagonists undergoes a dramatic transformation'" (156), the narrator's agent tells him, all advice that 10:04 scrupulously ignores—except for one thing. "'If they like it, fine,'" says his agent. "'But you need to keep the New Yorker story in there, I think.'" And so Lerner does. What choice does he have, really? If the publishers "rejected it," he reminds us, "I'd have to give the money back" (155).
What is the difference between the decision to allow the initial cuts to the story and the decision to include the butchered story in the novel? How can we be expected to differentiate between the novel that is compromised by the market and the novel that preemptively makes market compromise part of its form? To put it simply, the problem here is that these two different literary forms—which carry two very different political connotations—look exactly the same. The problem can also be rendered visually, like so:
These two pictures—or, two copies of the same picture (a photograph of the poet Claude Roy taken by Henri Cartier-Bresson)—appear on page 135 of 10:04. The picture on the left has the caption: "Our world"; the one on the right: "The world to come." The question they pose together is: what's the difference? The problem of difference posed by the utopian desire for a different future, a different world, is first raised in the novel's epigraph: "The Hassidim tell a story about the world to come that says everything there will be just as it is here. …Everything will be as it is now, just a little different." 10:04 is obsessed with the imperceptible differences that might constitute a redeemed future, an obsession communicated in part through the repetition of that key phrase from the epigraph, always just a tiny bit different: "Everything will be as it is now—the room, the baby, the clothes, the minutes—just a little different" (54); "everything is the same but a little different" (109); "it was the same, only totally different" (133); "What if everything at the end of the book is the same, only a little different?" (156).3 Yet the minor differences among these articulations of utopian difference distract us from the more complicated and possibly unsettling assertion of sameness, an assertion made most forcefully by the duplicate Cartier-Bresson photographs. The world to come may be a little bit different; but it will also be exactly the same. This raises what is essentially a hermeneutic dilemma: if the utopian future will look exactly the same as our damaged capitalist present, how will we recognize that future when we see it? The answer suggested by the two indistinguishable photographs—just as it suggested by a novel in which the editorially damaged story and the formally redeemed story are, in every way, the same story—is that it's possible that we won't.
But if the problem of both the two photographs and the two stories is that they are aesthetically indistinguishable, the problem with the narrator's actual political beliefs, as they are articulated in 10:04, is that they are frequently contradictory or irreconcilable. The content of the novel is, indeed, nothing but contradiction: between the political and the personal; between radicalism and liberalism; between the unrealized desire for collectivity and the incurable pathology of self-obsession. The narrator knows, for instance, that he ought to resist "the conflation of self-care and political radicalism" (46), then immediately conflates them: "I wanted a child, wanted one badly" (47). Then he berates himself for the conflation in a way that either overcomes the contradiction or doesn't overcome it at all:
Your gesture of briefly placing a tiny part of the domestic—your bathroom—into the commons leads you to redescribe the possibility of collective politics as the private drama of the family. …What you need to do is harness the self-love you are hypostasizing as offspring, as the next generation of you, and let it branch out horizontally into the possibility of a transpersonal revolutionary subject in the present and co-construct a world in which moments can be something other than the elements of profit. (47)
Collective politics, transpersonal revolution, a world beyond capitalist profit: sounds good! And yet something about the jargon here ("transpersonal," "co-construct") seems suspiciously overdone; the passage reads more like a parody of "transpersonal revolutionary" politics—a phrase that no one who held this politics would actually use—than an affirmation of such politics. Then, too, there is the neurotic, oscillating rhythm of epiphany that brought us to this point: the rejection of self-care as politics, the unconscious desire for a child, the realization of the ideological poverty of wanting a child in such circumstances. Where does it end? Actually, it ends with the child. The final pages of the novel reveal that the narrator and his friend Alex have successfully conceived. The Ultrasound that confirms Alex's pregnancy becomes yet another example of a world that is "the same, just a little different" (the narrator is reminded of his own earlier experience with an Ultrasound, used to diagnose the enlargement of his aorta), while the collective damage of Hurricane Sandy, bearing down in these last pages, devolves, through the flourish of metaphor, back into the "private drama of the family": "On the flat-screen hung high on the wall, we see the image of the coming storm, its limbs moving in real time, the brain visible in its translucent skull" (233).
The narrator wants a child but knows he shouldn't want it; wants political community but ends up with a child; imagines the ecological and economic catastrophe of the storm to be a potential site for collectivity; then imagines that the real "coming storm" is actually his unborn child. The narrator knows very well what politics he is supposed to have…but still, he can't quite bring himself to have them. Collectivity in 10:04 is thus there but not there, always a little bit—if not totally—different from itself. Every moment of political epiphany in the novel follows this formula, devolving into contradiction and self-negation. Consider the co-op that the narrator belongs to, toward which he has, over the course of three pages, at least three distinct attitudes. First, there's the need to demonstrate that "you weren't foolish enough to believe that belonging to the co-op made you meaningfully less of a node in a capitalist network, that you understood the co-op's population was largely made up of gentrifiers" (95). "And yet," continues the narrator, "I didn't think the co-op was morally trivial. I liked having the money I spent on food and household goods go to an institution that made labor shared and visible and that you could usually trust to carry products that weren't the issue of openly evil conglomerates" (96). And yet, it is precisely the existence of the co-op—what with all those products that are, thank god, not "openly evil" and all that "produce [that] was largely free of poison" (96)—that has produced "a new biopolitical vocabulary for expressing racial and class anxiety" (97). If the co-op is not "morally trivial," in other words, that is because it reveals the triviality of morals. The co-op does not simply make "labor shared and visible"; it also makes the mere "visibility" of labor insufficient grounds for the cooperative class politics that the larger economic structures routed through the co-op work actively to disable.
10:04 is, in this way, a thoroughly contradictory novel: a novel about contradiction that also makes contradiction the structuring secret of its narrative form. The damage on display in the novel's market concessions (its inclusion of the New Yorker story at the agent's behest) is reiterated at the level of the narrator's beliefs, which only ever rise to the level of political compromise. There is thus clearly a parallel between the damaged form of the novel and the "bad" or damaged "forms of collectivity" (108) that the narrator must constantly settle for—forms that inevitably collapse into their opposite: individual self-obsession, reproductive futurity, coupled domesticity. The novel that both is and isn't damaged (that subsumes its own damage) spirals out into the co-op that both is and isn't "morally trivial"; the cuts to the New Yorker story that both are and aren't unacceptable; the "bad forms of collectivity" that both are and aren't collective. This series of both/and's shows how the novel is structured by contradiction: the contradiction between the commitments Lerner's narrator wants to have (to the collective) and the commitments he does have (to himself); between "art before or after capital" (134) and art during capital; between "the world to come" and the world we're stuck with.
The literary form of contradiction is even visible in the novel's ideas about form. Form is the central term in Ashton's reading of 10:04: the "totaled work" becomes a model for the "total work"—the totalizing work—insofar as Lerner, according to Ashton, views aesthetic form as total, whole, unified. Such a view is born out most prominently in the scene that follows the narrator's first visit to the Institute for Totaled Art, having been emotionally moved by "those objects in the archive that both were and weren't different" (more contradiction), as he walks home across the Manhattan Bridge: "everything my eye alighted on seemed totaled in the best sense: complete in extent or degree; absolute; unqualified, whole" (134). The wholeness or totality of the work of art—its unqualified form—is, for Ashton, what protects it from damage; or, more accurately, what transforms the work's outwardly caused damage into its own intentional formal choice. And yet the idea that the totality of the novel's form can fully contain and preempt its damage is significantly complicated by 10:04's own more ambivalent commitment to form; a commitment that looks, as often as not, like a commitment to formlessness. If the Institute for Totaled Art is the place where Lerner raises the possibility of art as an allegory for totality, it is the narrator's visit to Marfa, Texas to view the minimalist sculptures of Donald Judd that calls that possibility almost totally into question. "I had never had a strong response to Judd's work," the narrator admits;
I believed in the things he wanted to get rid of—the internal compositional relations of a painting, nuances of form. His interest in modularity and industrial fabrication and his desire to overcome the distinction between art and life, an insistence on literal objects in real space—I felt I could get all those things by walking through a Costco or a Home Depot or IKEA; I'd never cared more for Judd's "specific objects" than any of the other objects I encountered in the world, objects that were merely real. (178)
Lerner's narrator "believe[s]" in the "nuances of form," while Judd's sculptures are utterly formless, refusing not only certain "compositional relations" but also the very "distinction between art and life"; a refusal that the narrator rightly notes does not so much elevate everyday objects into art as make erstwhile artworks indistinguishable from "any…other objects." What structures the narrator's distaste for Judd is his antipathy toward Judd's aesthetic project of refusing to distinguish artwork from object, art from life.
Yet this may strike the reader of 10:04 as a somewhat surprising objection for the narrator to voice—considering that it would seem to be precisely the "distinction between art and life" that 10:04 itself is constantly trying to undo. "Part of what I loved about poetry," says the narrator, whose life is more or less exactly the life of Ben Lerner, "was how the distinction between fiction and nonfiction didn't obtain" (171). This is the answer to the question of why Lerner the poet has chosen to write a novel at all: because he is trying to write a novel as if it were a poem, which means writing a novel in which—as in poetry—the difference between fiction and nonfiction no longer matters. That's exactly the type of novel 10:04 is: a book that continually reminds us of the fact that we are "reading [it] now" (194); a book whose narrator frequently comments on the process of writing "my book—not the one I was contracted to write about fraudulence, but the one I've written in its place for you, to you, on the very edge of fiction" (237). 10:04 is, in short, the novel being written by the narrator of 10:04, while the book that readers of 10:04 are reading turns out to be a book that is more than anything about our own experience of reading it.
But why would a writer who believes in the "nuances of form" write a book so perilously close to the "edge of fiction"—a precipice where the framing power of form, its singular capacity to separate art from life, no longer obtains? The answer, in this case at least, is: because that writer has changed his mind about Donald Judd. The chapter set in Marfa climaxes with the narrator's transformative experience of the sculptures at Chinati: "All those windows opening onto open land, the reflective surfaces, the differently articulated interiors, some of which seemed to contain a blurry image of the landscape within them—all combined to collapse my sense of inside and outside, a power the work had never had for me in the white-cube galleries of New York. At one point I detected a moving blur on the surface of a box and I turned to the windows to see two pronghorn antelope rushing across the desert plain" (179). The "power" of Judd's work is its capacity to "collapse my sense of inside and outside," to dissolve the distinction between the artwork, where it is set, and what it reflects (and just so we don't miss the fact that what art reflects is the living, breathing world, Lerner gives us the antelope). The newly discovered power of Judd's minimalism, coupled with the narrator's obsessive reading of Walt Whitman's "bizarre memoir," Specimen Days—in which Whitman similarly strives to blur the difference between his "empirical person" and a "pronoun in which the readers of the future could participate" (168)—ultimately transforms the narrator's, and Lerner's, sense of what kind of book 10:04 should be:
I decided to replace the book I'd proposed with the book you're reading now, a work that, like a poem, is neither fiction nor nonfiction, but a flickering between them; I resolved to dilate my story not into a novel about literary fraudulence, about fabricating the past, but into an actual present alive with multiple futures (194).
This is the point at which "the nuances of form" fully disappear from 10:04, replaced by the formlessness of "an actual present alive with multiple futures": a formlessness that is at once an indeterminate or as-yet-undetermined shape ("multiple futures") and a refusal of aesthetic form ("an actual present," "alive"). Thus "flickering between" fiction and nonfiction, 10:04 seeks to escape the constraints of form—of art's unmistakable difference from the world at large—in order to imagine a literary work that has become indistinguishable from the world, inseparable from "actual" life, invested in fortifying a "'fantasy of coeval readership'" (93). The formlessness of 10:04 resides in this turn from "fabrication" (art) to actuality, a turn away from the enclosed or total form of the artwork and toward the world in which the work is being read. Thus, while 10:04 describes itself as "a novel" (a word emblazoned, for instance, on the book's title page), it might be more accurate to think of it as a novel that desperately wants to not be a novel—not just because it would prefer to be an essay or a poem (both of which are likely true in the case of 10:04), but also because it would prefer not to be forced to mark off the formal or aesthetic boundary between literature and "actual" life. The novel that is "totaled in advance" appears here as the novel that has crashed into the world.
Of course, considered in these terms, a truly formless novel is an impossibility; the very phrase "formless novel," an oxymoron. The literalism of literature only goes so far. 10:04 thus remains, despite itself, a novel; and as a novel, it is impossible to mistake it for "any of the other objects" one might "encounter in the world." Literary formlessness is, at bottom, a contradiction in terms. The narrator himself knows this, writing (interestingly enough, in verse) of Whitman's Specimen Days: "It's among the greatest poems and fails / because it wants to become real and can / only become prose" (194). But what, in that case, is the point of 10:04's own impossible desire to "become real," its own fruitlessly prosaic pursuit of formlessness?
If literary formlessness is a contradiction, then what one discovers through it must be its opposite. The truth at the heart of the impossibility of literary formlessness is thus not hard to state: it is literary form. And if form is what the novel ends up affirming precisely through the work of attempting to negate it, then we can finally understand why the novel as a whole is so formally committed to contradiction—to oscillation, vacillation, and ambivalence. While the political dilemma of 10:04 is how to reconcile two opposing beliefs (are co-ops morally trivial or aren't they? is family a model of collectivity or a retreat from it?), the book's aesthetic dilemma is how to distinguish two seemingly identical forms: the compromised story from the subsumed one, damaged art from redeemed art. But where contradiction is clearly a problem for the narrator's compromised politics, it ultimately turns out to be a solution to the novel's damaged aesthetic. That's because the problem of indistinguishability is simply what contradiction looks like after it's been resolved. The narrator's self-contradictory politics are thus, in the end, also a lesson in the dialectical power of art—in art's capacity to resolve contradiction even (or especially) as art itself emerges out of the material, social circumstances of contradiction. The two identical photographs (one of our world, one of the world to come); the two indistinguishable stories (the first a sign of capitulation to the market, the second an attempt to "subsume" that capitulation): to learn to tell the difference between them is to learn to see the seemingly imperceptible difference between the resolution of contradiction and the mere unawareness of it. Contradiction is a process. That process takes time—and that time makes all the difference. It's the difference, above all, between "art before…capital" and art "after" it (134). For Lerner, for now, it is the special power of aesthetic form—the identical stories, the doubled photographs—to show us contradiction in the process of making itself disappear. The more dialectical lesson of 10:04's damaged form, though, is that the disappearance of contradiction is not the same as—is, in fact, exactly the opposite of—its never having existed at all.
How can market concession be a version of art's autonomy?4 How can damage be a version of repair? How can formlessness be a kind of form? These paradoxes only make sense once we see contradiction itself as a formal strategy for resolving contradiction. This is what it means for a novel that seems essentially to be one long concession to market forces to be at the same time a thought experiment in the eventual overthrow of those forces; for a novel that "wants to become real" also to be a treatise on why novels never can. Like the emphatic "no" that the narrator knows was "just a moment in the dialectic of her yes" (57), formlessness is just a moment in 10:04's dialectic of form: a rejection of form that is itself transformed or sublated back into form—the same kind of form, but now also a little bit different. And so, too, is inevitable damage (capitulation to the market) simply a moment in the dialectic of damage's aesthetic formulation: its subsumption—as Ashton puts it—into intended form. To say that formless = form, that damage = restoration, is not to say something illogical. It is—as Fredric Jameson does when, at the end of The Political Unconscious, he imagines "an imperative to thought in which the ideological would be grasped as somehow at one with the Utopian, and the Utopian at one with the ideological"—to say something dialectical.5 The risk of this dialectical strategy for a writer like Lerner is that, in the end, we won't be able to tell the difference between the two: between ideology and utopia, damaged art and redeemed art, a revolutionary novel (Ashton's 10:04) and a reactionary one (my 10:04). The reward, though, is that it teaches us to see indistinguishability itself as the product of a deeper antagonism, the sign of some underlying but unequivocal contradiction. It reminds us that, under capitalism, there is nothing besides contradiction. Thus, while Ashton sees this dialectical process as being fully realized by the novel ("10:04 presents itself as the achievement of this work"), I remain more ambivalent on the topic. More precisely, I view the novel's formal ambivalence as a sign of what it knows it can't achieve; and I see the novel's commitment to contradiction as an unsuccessful but not necessarily unhopeful way of figuring the distance between the ambivalence it can't escape and the revolutionary politics it sincerely wants but most definitely does not have. The contradictory form of 10:04—its narcissistic politics, its bad collectivities, its misguided formlessness, its intentional damage—is, at best, a reflection of the contradictions that structure life under capitalism; an ambivalent but not imprecise depiction of what it means to live in a world that both is and isn't on the cusp of change.
1.Jennifer Ashton, "Totaling the Damage: Revolutionary Ambition in Recent American Poetry," nonsite.org #18.↑
2.Ben Lerner, 10:04 (New York: Faber and Faber, 2014), 4. Subsequent citations appear in text.↑
3.For a more sustained reading of this mantra, see Nicholas Brown, "Art after Art after Art," nonsite.org #18. For Brown, Lerner's repeated invocation of the world to come (a world that will be "the same, just a little different") ultimately becomes a way to pinpoint the slight but central difference between the novel and its commodity character; that is, between what the novel "is trying to do" on its own aesthetic terms (namely, "assert its autonomy") and what the novel has to do to "meet certain market conditions." My argument in the present essay, by contrast, is that the difficulty of distinguishing between these two options is in fact a serious and central formal problem for the novel, and that the reason 10:04 makes indistinguishability such a central problem is that it functions as a dialectical counterpoint to both the political contradictions that plague the narrator and the aesthetic contradictions that haunt the novel as a whole.↑
4.In Literature and the Creative Economy (Stanford: Stanford University Press, 2014), Sarah Brouillette smartly grapples with an inverted form of this question: what happens if an assertion of art's autonomy is itself what the market wants? "Even when it is critique, [art's] anti-instrumental imperatives are exemplary, marketable, consumable, and often articulated in a way that anticipates their suitability to what they contest. It is of course against this very backdrop of the marketable antimarket gesture, of recognition of the service art can do to what it contests, that the ideal of aesthetic autonomy becomes not a dead issue … but rather a vital concern for cultural producers all over again" (208).↑
5.Fredric Jameson, The Political Unconscious: Narrative as a Socially Symbolic Act (Ithaca: Cornell University Press, 1981), 286-287.↑
Theodore Martin is assistant professor of English at the University of Wisconsin, Milwaukee. His work has appeared in Modern Language Quarterly and Novel, and is forthcoming in Postmodern/Postwar and After. He has just completed a book titled Contemporary Drift: Genre, Historicism, and the Problem of the Present.
Category: Feature, Issue #18 | RSS 2.0	| Give a Comment | trackback | | Print

redpajama_set_name: RedPajamaCommonCrawl


text:
John Bull's Adventures in the Fiscal Wonderland  is a novel by Charles Geake and Francis Carruthers Gould, written in 1904 and published by Methuen & Co. of London. It is a political parody of Lewis Carroll's two books, Alice's Adventures in Wonderland (1865) and Through the Looking-Glass (1871).

The book features 48 drawings by Gould, after the originals by John Tenniel.

It is critical of the economic politics of the day, which John Bull tries to make sense of. A number of notable British politicians are identified in the book. Joseph Chamberlain is the Prefferwense, the Mad Hatter, the Cheshire Cat, and the Knave of Hearts; Arthur Balfour is the March Hare and Humpy Dumpy; the Earl of Rosebery is Tweedle-R., Henry Campbell-Bannerman is Twee-C.-B., Jesse Collings is the White Rabbit, and the Duke of Devonshire is the Dormouse.

Editions
1904 original edition available at the Internet Archive
1904 edition in html format
2010 modern edition Evertype

Notes

1904 British novels
1904 fantasy novels
Books based on Alice in Wonderland
British fantasy novels
Methuen Publishing books
Henry Campbell-Bannerman

redpajama_set_name: RedPajamaWikipedia


text:
Research team rewriting book on photosynthesis
PULLMAN — Toss out that old biology book you were hanging onto for reference. The chapter on photosynthesis is about to be rewritten, thanks to work by David Kramer and his research team at Washington State University's Institute of Biological Chemistry.
Using instruments they designed and built themselves, Kramer and his group have discovered that plants adjust their light intake to match their metabolic needs by regulating the level of protons within sealed chambers in the chloroplasts.
Traditionally, plant physiologists have focused on the role of electrons in photosynthesis. Kramer's lab has changed the field by looking at what's happening with the protons inside chloroplasts in intact living leaves.
"That's a whole half of the way that the plant works that had never really been observed in a living plant," he said.
Their work also provides the first clear demonstration of a new link between what have usually been called the light reactions and dark reactions. Kramer's latest work, co-authored by Thomas Avenson, Jeffrey Cruz and Atsuko Kanazawa, was reported in the June 20 edition of the Proceedings of the National Academy of Sciences.
Green plants use energy from light to split water molecules into protons, electrons and oxygen. Textbook diagrams of the process show the electrons skimming through a series of chlorophylls and other photosynthetic pigments, eventually producing an energy-storing compound called NADPH. At the same time, the protons move into sealed compartments within the chloroplasts. The resulting energy gradient powers an enzyme called ATP synthase, which makes the energy-storing compound ATP. These are the "light reactions," so called because they can only occur in the presence of light.
The "dark reactions," which may occur either in light or dark, use the energy stored in NADPH and ATP to drive the assembly of carbon dioxide into sugars. Together, the light and dark reactions create nourishment for the plant and, through the food chain, human beings and all other animals as well.
Despite their key role in supporting most forms of life on the planet, the link between light reactions and dark reactions has eluded scientists. A plant somehow balances them so it incorporates energy at a rate suited to its needs. If a plant takes in too much light energy, it could die; if it takes in too little, it will not thrive as well as it could.
Kramer's team showed that a plant achieves that balance by changing the level of protons within the sealed compartments. When the compartments fill to a certain level, the photosynthetic machinery becomes less responsive to light.
Kramer likens the process to filling a bathtub. You can vary the water level by turning the faucet or by opening or closing the drain. His research shows that plants do it at the drain; and the drain, in this case, is the enzyme ATP synthase.
"It is as if the hole just gets smaller," said Kramer. "The system is still working fine, but it's like plugging up a little bit of your bathtub drain."
Their hand-made instruments allowed Kramer and his colleagues to deliver a pulse of light to a leaf and then sense slight changes in the color of light emitted by the leaf in response. That let them track protons moving in and out of the compartments.
They found that when a plant needs to slow its rate of photosynthesis, its ATP synthase lets fewer protons leave the compartments. The protons pile up, and photosynthesis slows or stops. When they looked at a mutant strain of the plant that didn't down-regulate well, they found that protons didn't accumulate to the level needed to trigger a shut-down.
"That's a big story, because we think we now understand what the connection is between what we call the light reactions and the dark reactions," Kramer said. "Nobody knew. We've found maybe the major regulatory connection between the light and dark reactions."
Categories: Research
biological chemistry, David Kramer, IBC, photosynthesis
Arsenic-contaminated water associated with antibiotic resistance in children
A WSU-led study found a higher prevalence of antibiotic-resistant E.coli in rural areas of Bangladesh with high arsenic contamination levels in drinking water compared to areas with less contamination.
Five WSU faculty named to Highly Cited Researchers list
Dogan Gursoy, Kris Kowdley, Dan "Annie" Du, Yuehe Lin, and Nathan McDowell are on this year's list recognizing researchers whose publications rank among the top 1% most cited in the world.


text:
Chess Player's Chronicle/Paul Morphy A Sketch from the Chess World
< Chess Player's Chronicle
Paul Morphy. A Sketch from the Chess World
This article appeared in Chess Player's Chronicle (Third Series) 1859, p289-90.[1]
123825Paul Morphy. A Sketch from the Chess World
SUCH is the title of the work before us. It was published some months ago at Leipzig by Messrs. VEIT & Co. in two volumes ; the author is Herr MAX LANGE, the well-known editor of the Berlin Schachzeitung. We have received, at the same time, the proof-sheets of a translation of the above, by Herr ERNEST FALKBEER, which is to be presented to the public in a few days by the Chess publisher Mr. J. H. STARIE, of Rathbone Place, London.
The German edition is divided into fifteen chapters, of which the first volume contains ten, the second five; to each chapter are added notes, which the learned author, in contra-distinction to the notes of the games, calls glossaries. The style is entirely German, and perhaps too didactic for a Chess work, but on the whole, much valuable information may be gathered from it. The games, one hundred and twenty in number, are given in the German notation, and are very correctly printed. The notes to the games are generally to the purpose, and often elaborately disquisite. Particular care seems to have been bestowed on the analysis of the games in the match between ANDEHSSEN and MORPHY, which fills half of the second volume. This section is analyzed in so masterly a manner, that we strongly suspect the German champion himself has furnished the necessary data to the notes.
In order to give our readers a more distinct idea of the work, we will here cite the contents of the different chapters, of which, however, the glossaries generally form the greater part.
Chapter I. treats of the chivalrous nature of the game. Chapter II. compares PAOLO BOI with PAUL MORPHY. Chapter III. gives five games played by MORPHY in his youth. Chapter IV. speaks of the Chess Congress in New York in 1857. Chapter V. relates MORPHY'S triumphs in New York. Chapter VI. represents the American champion on his return to New Orleans. Chapter VII. narrates his voyage to, and his arrival in England. Chapter VIII. enumerates his victories over his English opponents. Chapter IX. is devoted to the match with Löwenthal, and the last Chapter of the first volume refers to the Chess Meeting at Birmingham.
In the second volume, the first chapter describes the youthful hero's reception and blindfold performance in the Café de la Régence in Paris.
The second contains the games with his French opponents. The third and fourth present a detailed account of the respective matches with HAHRWITZ and ANDERSSEN. The concluding chapter of the second volume expresses the author's own reflections on the American champion's extraordinary triumphs and future prospects.
These are the elements of which this work is composed, the merits of which we fully acknowledge ; we cannot, however, conclude these remarks without animadverting to some points which must obtrude themselves to the English reader. First and foremost among them is the way in which the German author speaks of Morphy in the first volume; his praises sound like an apotheosis, no Roman poet ever addressed more flattering or high sounding epithets to his Cassars. Even Caissa's crown of glory seems to grow pale before this new-born light. No living Chess player, nay, even none of our dead celebrities could be compared to him, whose rising reputation is, as yet, in its first phase, and whom, when in his full glory, no Pantheon could hold, nor Westminster Abbey enshroud. Far less high sounding, however, are the praises meted out to the youthful hero in the cantos of the second volume, at the end of which there seems to be even an inkling of the possibility that some player may yet be found whose lance may not be shivered upon the unconquered breast-plate of this fearful paladin. Similar discrepancies on other points may be found in the two volumes. Although we fully agree with MAX LANGE in the estimate of ANDERSSEN'S play, and think with him that in his match with MORPHY he has played considerably below his strength, we cannot but disagree with him in his estimate of HARRWITZ'S play, who, as he seems to infer, has played up to his strength. Whoever peruses the games between MORPHY and HARRWITZ must easily see that in the latter part of the match, HARRWITZ laboured under some inimical influence, moral or physical, whatever that might have been, we leave it to him to explain.
In the glossaries we have found much interesting matter, and many details with which we were, as yet, unacquainted, especially as to the match between ANDERSSEN and MORPHY; as to these, however, we must refer our reader to the book itself, or the English translation by Herr ERNEST FALKBEER, which has united the two volumes into one, added a goodly number of games to the original edition, and enriched it with translator's notes, which, considering Herr FALKBEER'S skill as a Chess player, must considerably augment the intrinsic value of the work. In comparing the English text with the German, we were struck with the correctness of the translation, especially in the notes, where redundant phrases and periphrastic style are the prevalent characteristics of the original, thus making an exact translation doubly difficult.
This work is in the public domain in the United States because it was published before January 1, 1924. It may be copyrighted outside the U.S. (see Help:Public domain).
Retrieved from "https://en.wikisource.org/w/index.php?title=Chess_Player%27s_Chronicle/Paul_Morphy_A_Sketch_from_the_Chess_World&oldid=3795044"


The Bombadils. What a treat!
A rapt audience was glad that Canadians, The Bombadils graced Tintenbar Hall last Sunday afternoon. The combination of instruments (guitar, mandolin, violin, banjo, flute, harmonica), voices, personalities and stories blended to make a wonderful experience. The musicianship was of such a high standard it felt at times a prairie wind or a babbling mountain stream was flowing through the music. Each instrument weaving its magic to form a coherent whole.
Among others, the Bombadils performed songs from their new CD Grassy Roads, Wandering Feet. Bluegrass tunes, (Rocky Mountain Path) traditional sailors songs, (Heave Away) old standards like Black Is The Colour Of My True Love's Hair.
Their original tunes were outstanding: the nostalgic Portrait, the heart warming Nova Scotia Goodbye, the romantic Hour of The Blue Snow.
A couple of Audience comments: Wow what amazing young talent !!! And how fabulous were the audience! Brilliant afternoon !
And from Mavis, who has been coming to Tintenbar Hall for over 80 years: I can't believe I'm in Tintenbar!"
Those in the hall last Sunday fervently hope The Bombadils make Tintenbar part of their next Australian tour itinerary. We all left lighthearted and feeling a deep affinity for our Canadian cousins.
Liora Claff opened for the Bombadils with a bracket of finely sung and played folk tunes. She too received warm applause.
There are more pics in the gallery.


text:
Video Tweet
Ubisoft - E3 2015 Press Conference
#Showcase #Presentation #Ubisoft
Log in to your PixlBit account in the bar above or join the site to leave a comment.
EarthBound Review Rewind
Remember that silly old trope in teen movies where the nerdy "plain Jane" character magically becomes super attractive just by removing her glasses? It was a bit like that with EarthBound- an RPG that was largely ignored when released on the Super NES in its twilight years. Nintendo's ill-conceived ad campaign telling the player that "this game stinks" didn't do it any favors either. The game's overtly humorous kid-friendly facade was unusual for its time and quickly written off as ugly and boring by all but the most accepting of RPG fans. But thanks to the positive praise in gaming forums years later, it picked up steam and eventually became lauded as one of the best on the Super NES. Much like the bespectacled character in every 90s teen comedy, it was as if EarthBound removed its glasses. Nothing was different about it- just our perception of what was already there.
E3 Expo 2015
Combat of Giants: Dinosaurs 3D Preview on 03/14/2011 at 10:22 PM
Killer Freaks From Outer Space Preview on 06/09/2011 at 07:19 PM
ZombiU Hands On Preview on 10/25/2012 at 12:00 PM
Assassin's Creed IV: Black Flag Preview on 03/13/2013 at 10:00 AM


COCONUT CREEK, Fla. - Food for the Poor is packing up donations to help islands in the Caribbean that were affected by Hurricane Maria.
Three 40-foot containers were loaded up Monday morning and are already on the way to the islands.
"None of those islands are wealthy enough or prepared for a devastation of this magnitude," Food for the Poor's executive director Angel Aloma said. "First Irma came by and started it, then Maria came by and finished it."
Boxes of supplies were loaded onto pallets and placed in containers at Food for the Poor's warehouse in Coconut Creek Monday morning.
"It's all the stuff they would need immediately -- clean water for drinking, canned foods, blankets, hygiene items, tarps, buckets to start the clean-up process," Aloma said.
Three containers were sent to the port Monday morning, and from there will sail to Dominica, Antigua and St. Lucia.
"Through our partners in St. Lucia we're helping St. Martin and the Virgin Islands, and through our partner in Antigua, we are helping Barbuda," Aloma said. "Barbuda was declared inhabitable -- that's how terribly it was destroyed."
Aloma said Food for the Poor will be helping out for the long haul.
"It's great to help out at the beginning when all the news media takes care of it, but then eventually it becomes old news, but the devastation does not become old news for them," Aloma said.
The next shipment will include things like gas chainsaws to help with the cutting away of trees, and generators to provide some electricity.


Painting the walls of the swimming pool with a waterproof coating such as epoxy paints in order to prevent water from penetrating the shell of the pool and damaging is a great option. This is because epoxy paint does not have a rough surface which makes it easy on the skin of the swimmer, resists cracking more efficiently and can be cleaned much easily. The various features of epotec pool paint have been described here: Solid The epotec pool paint does not have solvents or thinners which gives it a completely solid coating. This leads to retention of the entire applied thickness with no shrinking unlike conventional paints used for houses which have only 30-40% of solids. The solid nature of the epotec pool paint is the reason behind its good strength. Thickness Epotec becomes inert when cured. Thus, it cannot be removed by chemicals and can only get affected by wear and tear. Consequently, a thick coat of the paint needs to be applied for long lasting effect. Thick coat of paint means that it will last long. The epotec pool paint does not have thinners but its special characteristics allow it to be easily applied using brush or rollers. The standard thickness achieved with epotec pool paint is 320 microns. Life With 7-14 years of life, the longevity of epotec epoxy pool paint Perth is impressive. This implies that once applied using thick coat, the paint doesn't need recoating for this time period. Upgrading swimming pools Epotec paint is widely used for upgrading swimming pools such as during fiberglass pool repairs, fiberglass lined concrete swimming pools, marblesheen pools and other concrete pools. This is because epotec paint prevents general wear and tear of the pool walls and protects them from chlorine and other chemicals commonly found in swimming pools. Epotec paint also gives protection from hydrostatic water pressure. The column of liquid water in the swimming pool exerts a force known as hydrostatic pressure. This force causes the liquid water to penetrate the concrete. Permeable concrete is more prone to water penetration. Epotec paint eliminates this problem by forming a barrier between the concrete and the water because hydrostatic pressure is created only when water comes in direct contact with the concrete of the swimming pool. Water can also pass through the concrete in the form of gas. The epotec paint being impermeable to both liquid and gas molecules, is also capable of preventing the passage of water through concrete in the form of gas molecules. Types of colour Epotec pool paint is available in a number of colour shades. The standard shades have been listed here.
There are a number of factors which come into play at the time of selecting a suitable colour such as the colour of the surroundings like tiles, vegetation, etc. However, a lighter colour is recommended for epoxy paint as it tends to get yellower with time.


text:
\section{Conclusion}
In this paper, we have proposed three novel approaches to solve the optimization problem for quantizing the network with both low-precision weights and activations. We first propose a two-stage approach to quantize the weights and activations in a two-step manner. We also observe that continuously quantize from high-precision to low-precision is also beneficial to the final performance. To better utilize the knowledge from the full-precision model, we propose to jointly learn the low-precision model and its full-precision counterpart to optimize the gradient problem during training. Using 4-bit weights and activations for all layers, we even outperform the performance of the 32-bit model on ImageNet and Cifar100 with general frameworks.
\section{Methods}
\begin{figure*}[!t]
	\centering
	\resizebox{0.9\linewidth}{!}
	{
		\begin{tabular}{c}
			\includegraphics{pdf/1.pdf}
		\end{tabular}
	}
	\caption{Demonstration of the guided training strategy. We use the residual network structure for illustration.}
	\label{fig:knowledge_transfer}
\end{figure*}
In this section, we will first revisit the quantization function in the neural network and the way to train it. Then we will elaborate our three methods in the subsequent sections.
\subsection{Quantization function revisited} \label{sec:baseline}

A common practise in training a neural network with low-precision weights and activations is to introduce a quantization function. Considering the general case of $k$-bit quantization as in~\cite{zhou2016dorefa}, we define the quantization function $Q(\cdot)$ to be
\begin{equation}
	{z_q} = Q({z_r}) = \frac{1}{{{2^k} - 1}}round(({2^k} - 1){z_r})
\end{equation}
where ${z_r} \in [0,1]$ denotes the full-precision value and ${z_q} \in [0,1]$ denotes the quantized value. With this quantization function, we can define the weight quantization process and the activation quantization process as follows:

\noindent \textbf{Quantization on weights}:
\begin{equation}\label{eq:quan-weigtht}
	{w_q} = Q(\frac{{\tanh (w)}}{{2\max (\left| {\tanh (w)} \right|)}} + \frac{1}{2}).
\end{equation}In other words, we first use $\frac{{\tanh (w)}}{{2\max (\left| {\tanh (w)} \right|)}} + \frac{1}{2}$ to obtain a normalized version of $w$ and then perform the quantization, where $\tanh(\cdot)$ is adopted to reduce the impact of large values.

\noindent \textbf{Quantization on activations}:

 Same as \cite{zhou2016dorefa}, we first use a clip function $f(x) = clip(x,\,0,1)$ to bound the activations to $[0, 1]$. After that, we conduct quantize the activation by applying the quantization function $Q(\cdot)$ on $f(x)$.
\begin{equation} \label{eq:quan-activations}
	{x_q} = Q(f(x)).
\end{equation}

\noindent \textbf{Back-propagation with quantization function}: In general, the quantization function is non-differentiable and thus it is impossible to directly apply the back-propagation to train the network. To overcome this issue, we adopt the straight-through estimator \cite{zhou2016dorefa, hubara2016binarized, bengio2013estimating} to approximate the gradients calculation. Formally, we approximate the partial gradient $\frac{{\partial {z_q}}}{{\partial {z_r}}}$ with an identity mapping, namely $\frac{{\partial {z_q}}}{{\partial {z_r}}} \approx 1$.  Accordingly, $\frac{{\partial l}}{{\partial {z_r}}}$ can be approximated by
\begin{equation}
	\frac{{\partial l}}{{\partial {z_r}}} = \frac{{\partial l}}{{\partial {z_q}}}\frac{{\partial {z_q}}}{{\partial {z_r}}} \approx \frac{{\partial l}}{{\partial {z_q}}}.
\end{equation}

\subsection{Two-stage optimization}\label{sec:two-stage}
With the straight-through estimator, it is possible to directly optimize the low-precision network. However, the gradient approximation of the quantization function inevitably introduces noisy signal for updating network parameters. Strictly speaking, the approximated gradient may not be the right updating direction. Thus, the training process will be more likely to get trapped at a poor local minima than training a full precision model. Applying the quantization function to both weights and activations further worsens the situation.

To reduce the difficulty of training, we devise a two-stage optimization procedure: at the first stage, we only quanitze the weights of the network while setting the activations to be full precision. After the converge (or after certain number of iterations) of this model, we further apply the quantization function on the activations as well and retrain the network. Essentially, the first stage of this method is a related subproblem of the target one. Compared to the target problem, it is easier to optimize since it only introduces quantization function on weights. Thus, we are more likely to arrive at a good solution for this sub-problem. Then, using it to initialize the target problem may help the network avoid poor local minima which will be encountered if we train the network from scratch.
Let $M_{low}^{K}$ be the high-precision model with $K$-bit. We propose to learn a low-precision model $M_{low}^{k}$ in a two-stage manner with $M_{low}^{K}$ serving as the initial point, where $k<K$.
 The detailed algorithm is shown in Algorithm \ref{algo:two-stage}.
\begin{algorithm}[]
	\KwIn{Training data $\{ ({{\bf{x}}_i},y_i)\}_{i=1}^N$; A $K$-bit precision model $M_{low}^K$.}
	\KwOut{A low-precision deep model $M^k_{low}$ with weights ${{\bf{W}}_{low}}$ and activations being quantized into $k$-bit.}

	\textbf{Stage 1}: Quantize ${{\bf{W}}_{low}}$:\\
	\For{ $\mathrm{epoch} = 1,...,L$}
	{
		\For{ $t = 1,...T$}
		{
			Randomly sample a mini-batch data;\\
			Quantize the weights ${{\bf{W}}_{low}}$ into $k$-bit by calling some quantization methods with $K$-bit activations\;
		}
	}
	\textbf{Stage 2}: Quantize activations:\\
	Initialize ${{\bf{W}}_{low}}$ using the converged $k$-bit weights from \textbf{Stage 1} as the starting point; \\
	\For{ $\mathrm{epoch} = 1,...,L$}
	{
		\For{ $t = 1,...T$}
		{
			Randomly sample a mini-batch data;\\
			Quantize the activations into $k$-bit  by calling some quantization methods while keeping the weights to $k$-bit;
		}
	}
	\caption{Two-stage optimization for $k$-bit quantization}
	\label{algo:two-stage}
\end{algorithm}

\subsection{Progressive quantization} \label{sec:progressive}

The aforementioned two-stage optimization approach suggests the benefits of using a related easy optimized problem to find a good initialization. However, separating the quantization of weights and activations is not the only solution to implement the above idea. In this paper, we also propose another solution which progressively lower the bitwidth of the quantization during the course of network training.
Specifically, we progressively conduct the quantization from higher precisions to lower precisions (\eg, 32-bit $\to$ 16-bit $\to$ 4-bit $\to$ 2-bit). The model of higher precision will be used the the starting point of the relatively lower precision, in analogy with annealing.

Let $\{{b_1},...,{b_n}\}$ be a  sequence precisions, where  $b_n<b_{n-1}, ..., b_2<{b_1}$, $b_n$ is the target precision and $b_1$ is set to 32 by default. The whole progressive optimization procedure  is summarized in as Algorithm~\ref{algo:progressive optimization}.
 Let $M_{low}^{k}$ be the low-precision model with $k$-bit and $M_{full}$ be the full precision model. In each step, we propose to learn $M_{low}^{k}$, with the solution in the $(i-1)$-th step, denoted by $M_{low}^{K}$, serving as the initial point, where $k<K$.

\begin{algorithm}[]
	\KwIn{Training data $\{ ({{\bf{x}}_j},y_j)\}_{j=1}^N$; A pre-trained 32-bit full-precision  model ${M_{full}}$ as baseline; the precision sequence $\{{b_1},...,{b_n}\}$ where $b_n<b_{n-1}, ..., b_2<{b_1} = 32$.}
	\KwOut{A low-precision deep model $M_{low}^{b_n}$.}
	Let $M_{low}^{b_1}=M_{full}$, where $b_1 = 32$\;
	\For{ $i = 2,...n$}
	{
		Let $k = b_i$ and $K=b_{i-1}$\;
		Obtain $M_{low}^{k}$ by calling some quantization methods with $M_{low}^{K}$  being the input\;
		%
		%
		%
	}
	\caption{Progressive quantization for accurate CNNs with low-precision weights and activations}
	\label{algo:progressive optimization}
\end{algorithm}

\subsection{Guided training with a full-precision network}\label{sec:mutual}
The third method proposed in this paper is inspired by the success of using information distillation ~\cite{romero2014fitnets, hinton2015distilling, parisotto2016actor, zagoruyko2016paying, ba2014deep} to train a relatively shallow network. Specifically, these methods usually use a teacher model (usually a pretrained deeper network) to provide guided signal for the shallower network. Following this spirit, we propose to train the low-precision network alongside another guidance network. Unlike the work in \cite{romero2014fitnets, hinton2015distilling, parisotto2016actor, zagoruyko2016paying, ba2014deep}, the guidance network shares the same architecture as the target network but is pretrained with full-precision weights and activations.

However, a pre-trained model may not be necessarily optimal or may not be suitable for quantization. As a result, directly using a fixed pretrained model to guide the target network may not produce the best guidance signals. To mitigate this problem, we do not fix the parameters of a pretrained full precision network as in the previous work \cite{zhang2017deep}.

By using the guidance training strategy, we assume that there exist some full-precision models with good generalization performance, and an accurate low-precision model can be obtained by directly performing the quantization on those full-precision models. In this sense, the feature maps of the learned low-precision model should be close to that obtained by directly doing quantization on the full-precision model. To achieve this, essentially, in our learning scheme, we can jointly train the full-precision and low-precision models. This allows these two models adapt to each other. We even find by doing so the performance of the full-precision model can be slightly improved in some cases.

Formally, let ${{\bf{W}}_{full}}$ and ${{\bf{W}}_{low}}$ be the full-precision model and low-precision model, respectively. Let $\mu ({\bf{x}};{{\bf{W}}_{{full}}})$ and $\nu ({\bf{x}};{{\bf{W}}_{{low}}})$ be the nested feature maps (e.g., activations) of the full-precision model and low-precision model, respectively. To create the guidance signal, we may require that the nested feature maps from the two models should be similar. However,  $\mu ({\bf{x}};{{\bf{W}}_{{full}}})$ and $\nu ({\bf{x}};{{\bf{W}}_{{low}}})$  is usually not directly comparable since one is full precision and the other is low-precision.

To link these two models,  we can directly quantize the weights and activations of the full-precision model by equations (\ref{eq:quan-weigtht}) and (\ref{eq:quan-activations}). For simplicity, we denote the quantized feature maps by  $Q(\mu ({\bf{x}};{{\bf{W}}_{{full}}}))$. Thus, $Q(\mu ({\bf{x}};{{\bf{W}}_{{full}}}))$ and  $\nu ({\bf{x}};{{\bf{W}}_{{low}}})$ will become comparable. Then we can define the guidance loss as:
\begin{equation}
	R({{\bf{W}}_{full}},{{\bf{W}}_{low}}) = \frac{1}{2}\parallel Q(\mu ({\bf{x}};{{\bf{W}}_{{full}}})) - \nu ({\bf{x}};{{\bf{W}}_{{low}}}){\parallel^2},
\end{equation}
where $\parallel\cdot\parallel$ denotes some proper norms.

Let ${L_{{\theta _1}}}$ and ${L_{{\theta _2}}}$ be the cross-entropy classification losses for the full-precision and low-precision model, respectively. The guidance loss will be added to ${L_{{\theta _1}}}$ and ${L_{{\theta _2}}}$, respectively, resulting in two new objectives for the two networks, namely
\begin{equation} \label{eq:objective1}
	L_1({{\bf{W}}_{full}})  = {L_{{\theta _1}}} + \lambda R({{\bf{W}}_{full}},{{\bf{W}}_{low}}).
\end{equation}
and
\begin{equation} \label{eq:objective2}
	L_2({{\bf{W}}_{low}})  = {L_{{\theta _2}}} +  \lambda R({{\bf{W}}_{full}},{{\bf{W}}_{low}}).
\end{equation}
where $\lambda$ is a balancing parameter. Here, the guidance loss $R$ can be considered as some regularization on ${L_{{\theta _1}}}$ and ${L_{{\theta _2}}}$.

In the learning procedure, both ${{\bf{W}}_{full}}$ and ${{\bf{W}}_{low}}$ will be updated by minimizing $L_1({{\bf{W}}_{full}})$ and $L_2({{\bf{W}}_{low}})$ separately, using a mini-batch stochastic gradient descent method. The detailed algorithm is shown in Algorithm \ref{algo:one-mutual learning}. A high-bit precision model $M_{low}^K$ is used as an initialization of $M_{low}^k$, where $K>k$. Specifically, for the full-precision model, we have $K=32$. Relying on $M_{full}$, the weights and activations of $M_{low}^k$ can be initialized by equations (\ref{eq:quan-weigtht}) and (\ref{eq:quan-activations}), respectively.

Note that the training process of the two networks are different.
When updating ${{\bf{W}}_{low}}$ by minimizing $L_2({{\bf{W}}_{low}})$, we use full-precision model as the initialization and apply the forward-backward propagation rule in Section \ref{sec:baseline}  to fine-tune the model. When updating ${{\bf{W}}_{full}}$ by minimizing $L_1({{\bf{W}}_{full}})$, we use conventional forward-backward propagation to fine-tune the model.

\begin{algorithm}[]
	\KwIn{Training data $\{ ({{\bf{x}}_i},y_i)\}_{i=1}^N$; A pre-trained 32-bit full-precision model $M_{full}$; A $k$-bit precision model $M_{low}^k$.}
	\KwOut{A low-precision deep model $M^k_{low}$ with weights and activations being quantized into $k$ bits.}
	Initialize $M_{low}^k$ based on $M_{full}$;\\
	\For{ $\mathrm{epoch} = 1,...,L$}
	{
		\For{ $t = 1,...T$}
		{
			Randomly sample a mini-batch data;\\
			Quantize the weights ${{\bf{W}}_{low}}$  and activations into $k$-bit by minimizing $L_2({{\bf{W}}_{low}})$\;
			Update $M_{full}$ by minimizing $L_1({{\bf{W}}_{full}})$\;
		}

	}
	\caption{Guided training with a full-precision network for $k$-bit quantization}
	\label{algo:one-mutual learning}
\end{algorithm}

\subsection{Remark on the proposed methods}
The proposed three approaches tackle the difficulty in training a low-precision model with different strategies. They can be applied independently. However, it is also possible to combine them together. For example, we can apply the progressive quantization to any of the steps in the two-stage approach; we can also apply the guided training to any sub-step in the progressive training. Detailed analysis on possible combinations will be experimentally evaluated in the experiment section.

\subsection{Implementation details} \label{sec:implementation}

In all the three methods, we quantize the weights and activations of all layers except that the input data are kept to 8-bit. Furthermore, to promote convergence, we propose to add a scalar layer after the last fully-connected layer before feeding the low-bit activations into the softmax function for classification. The scalar layer has only one trainable small scalar parameter and is initialized to 0.01 in our approach.

During training, we randomly crop 224x224 patches from an image or its horizontal flip, with the per-pixel mean subtracted. We don't use any further data augmentation in our implementation. We adopt batch normalization (BN)~\cite{ioffe2015batch} after each convolution before activation. For pretraining the full-precision baseline model, we use Nesterov SGD and batch size is set to 256. The learning rate starts from 0.01 and is divided by 10 every 30 epochs. We use a weight decay 0.0001 and a momentum 0.9. For weights and activations quantization, the initial learning rate is set to 0.001 and is divided by 10 every 10 epochs. We use a simple single-crop testing for standard evaluation. Following~\cite{zagoruyko2016paying}, for ResNet-50, we add only two guidance losses in the 2 last groups of residual blocks. And for AlexNet, we add two guidance losses in the last two fully-connected layers.

	 \input{experiment.tex}

	 \input{conclusion.tex}

\small
\bibliographystyle{ieee}

\section{Experiment}

To investigate the performance of the proposed methods, we conduct experiments on Cifar100 and ImageNet datasets. Two representative networks, different precisions AlexNet and ResNet-50 are evaluated with top-1 and top-5 accuracy reported. We use a variant of AlexNet structure~\cite{krizhevsky2012imagenet} by removing dropout layers and add batch normalization after each convolutional layer and fully-connected layer. This structure is widely used in previous works~\cite{zhou2016dorefa, zhu2016trained}.  We analyze the effect of the guided training approach, two-stage optimization and the progressive quantization in details in the ablation study. Seven methods are implemented and compared:

\begin{enumerate}
\item ``\textbf{Baseline}'': We implement the baseline model based on DoReFa-Net as described in Section~\ref{sec:baseline}.

\item ``\textbf{TS}'': We apply the two-stage optimization strategy described in  Sec.~\ref{sec:two-stage} and Algorithm~\ref{algo:two-stage} to quantize the weights and activations. We denote the first stage as \textbf{Stage1} and the second stage as \textbf{Stage2}.

\item ``\textbf{PQ}'': We apply the progressive quantization strategy described in  Sec.~\ref{sec:progressive} and Algorithm~\ref{algo:progressive optimization} to continuously quantize weights and activations simultaneously from high-precision (\ie, 32-bit) to low-precision.

\item ``\textbf{Guided}'': We implement the guided training approach as described in Sec.~\ref{sec:mutual} and Algorithm~\ref{algo:one-mutual learning} to independently investigate its effect on the final performance.

\item ``\textbf{PQ+TS}'': We further combine \textbf{PQ} and \textbf{TS} together to see whether their combination can improve the performance.

\item ``\textbf{PQ+TS+Guided}'': This implements the full model by combining \textbf{PQ}, \textbf{TS} and \textbf{Guided} modules together.

\item ``\textbf{PQ+TS+Guided**}'': Based on PQ\-+\-TS\-+\-Guided, we use full-precision weights for the first convolutional layer and the last fully-connected layer following the setting of~\cite{zhu2016trained, zhou2016dorefa} to investigate its sensitivity to the proposed method.
\end{enumerate}

\subsection{Evaluation on ImageNet}

We further train and evaluate our model on ILSVRC2012~\cite{russakovsky2015imagenet}, which includes over 1.2 million images and 50 thousand validation images. We report 4-bit and 2-bit precision accuracy for both AlexNet and ResNet-50. The sequence of bit-width precisions are set as $\{32, 8, 4, 2\}$. The results of INQ~\cite{zhou2017incremental} are directly cited from the original paper. We did not use the sophisticated image augmentation and more details can be found in Sec.~\ref{sec:implementation}. We compare our model to the 32-bit full-precision model, INQ, DoReFa-Net and the baseline approach described in Sec.~\ref{sec:baseline}. For INQ, only the weights are quantized. For DoReFa-Net, the first convolutional layer uses the full-precision weights and the last fully-connected layer use both full-precision weights and activations.

\emph{\textbf{Results on AlexNet:}}
The results for AlexNet are listed in Table~\ref{tab:AlexNet}. Compared to competing approaches, we achieve steadily improvement for 4-bit and 2-bit settings. This can be attributed to the effective progressive optimization and the knowledge from the full-precision model for assisting the optimization process. Furthermore, our 4-bit full model even outperforms the full-precision reference by 0.7\% on top-1 accuracy. This may be due to the fact that on this data, we may not need a model as complex as the full-precision one. However, when the expected bit-width decrease to 2-bit, we observe obvious performance drop compared to the 32-bit model while our low-bit model still brings 2.8\% top-1 accuracy increase compared to the \emph{Baseline} method.

\emph{\textbf{Results on ResNet-50:}}
The results for ResNet-50 are listed in Table~\ref{tab:ResNet-50}. For the full-precision model, we implement it using Pytorch following the re-implementation provided by Facebook\footnote{\url{https://github.com/facebook/fb.resnet.torch}}.
Comparatively, we find that the performance are approximately consistent with the results of AlexNet. Similarly, we observe that our 4-bit full model is comparable with the full-precision reference with no loss of accuracy. When decreasing the precision to 2-bit, we achieve promising improvement over the competing \emph{Baseline} even though there's still an accuracy gap between the full-precision model. Similar to the AlexNet on ImageNet dataset, we find our 2-bit full model improves more comparing with the 4-bit case. This phenomenon shows that when the model becomes more difficult to optimize, the proposed approach turns out to be more effective in dealing with the optimization difficulty.
To better understand our model, we also draw the process of training for 2-bit ResNet-50 in Figure~\ref{fig:resnet_2bits} and more analysis can be referred in Sec.~\ref{sec:ablation}.

\begin{table*}[!tbp]
		\centering
		\scalebox{0.7}
		{
			\begin{tabular}{c c c | c c c | c c c }
				\hline
                 Accuracy & Full precision &5-bit (INQ) &4-bit (DoReFa-Net)  & 4-bit (Baseline)  & 4-bit (PQ+TS+Guided) &2-bit (DoReFa-Net) &2-bit (Baseline) & 2-bit (PQ+TS+Guided)\\\hline
                 Top1 &57.2\%  &57.4\%  &56.2\%  &56.8\%  &\bf{58.0}\%  &48.3\%  &48.8\%  &\bf{51.6}\% \\
                 Top5 &80.3\%  &80.6\%  &79.4\% &80.0\% &\bf{81.1}\% &71.6\%  &72.2\%  &\bf{76.2}\%  \\\hline

			\end{tabular}}
			\caption{Top1 and Top5 validation accuracy of AlexNet on ImageNet.}
			\label{tab:AlexNet}
\end{table*}

\begin{table*}[!tbp]
	\centering
	\scalebox{0.7}
	{
		\begin{tabular}{c c c| c c c | c c c }
			\hline
			Accuracy & Full precision &5-bit (INQ) &4-bit (DoReFa-Net) & 4-bit (Baseline)  & 4-bit (PQ+TS+Guided) &2-bit (DoReFa-Net) &2-bit (Baseline) & 2-bit (PQ+TS+Guided)\\\hline
			Top1 &75.6\%  &74.8\%  &74.5\%  &75.1\%  &\bf{75.7}\%  &67.3\%  &67.7\%  &\bf{70.0}\%  \\
			Top5 &92.2\%  &91.7\%  &91.5\%  &91.9\%  &\bf{92.0}\% &84.3\% &84.7\%  &\bf{87.5}\%  \\\hline

		\end{tabular}}
		\caption{Top1 and Top5 validation accuracy of ResNet-50 on ImageNet.}
		\label{tab:ResNet-50}
	\end{table*}

\subsection{Evaluation on Cifar100}
Cifar100 is an image classification benchmark containing images of size 32x32 in a training set of 50,000 and
a test set of 10,000. We use the AlexNet for our experiment. The quantitative results are reported in  Table~\ref{tab:cifar100_AlexNet}. From the table, we can observe that the proposed approach steadily outperforms the competing method DoReFa-Net. Interestingly, the accuracy of our 4-bit full model also surpasses its full precision model. We speculate that this is due to 4-bit weights and activations providing the right model capacity and preventing overfitting for the networks.

\begin{table*}[!tbp]
	\centering
	\scalebox{0.76}
	{
		\begin{tabular}{c c |c  c c | c c c}
			\hline
			Accuracy & Full precision &4-bit (DoReFa-Net)  & 4-bit (Baseline) & 4-bit (PQ+TS+Guided) &2-bit (DoReFa-Net) &2-bit (Baseline) &2-bit (PQ+TS+Guided)\\\hline
			Top1 &65.4\%  &64.9\% &65.0\%   &\bf{65.8}\%  &63.4\% &63.9\%  &\bf{64.6}\%  \\
			Top5 &88.3\% &88.5\%  &88.5\%   &\bf{88.6}\% &87.5\%  &87.6\%  &\bf{87.8}\% \\\hline

		\end{tabular}}
		\caption{Top1 and Top5 validation accuracy of AlexNet on Cifar100.}
		\label{tab:cifar100_AlexNet}
	\end{table*}

\subsection{Ablation study} \label{sec:ablation}
In this section, we analyze the effects of different components of the proposed model.

\vspace{1mm}
\noindent\textbf{\emph{Learning from scratch vs. Fine-tuning:}}
To analyze the effect, we perform comparative experiments on Cifar100 with AlexNet using learning from scratch and fine-tuning strategies. The results are shown in Figure~\ref{fig:cifar}, respectively. For convenience of exposition, this comparison study is performed based on method \emph{TS}.
First, we observe that the overall accuracy of fine-tuning from full-precision model is higher than that of learning from scratch. This indicates that the initial point for training low-bitwidth model is crutial for obtaining good accuracy.
In addition, the gap between the \emph{Baseline} and \emph{TS} is obvious (\ie, 2.7 \% in our experiment) with learning from scratch. This justifies that the two-stage optimization strategy can effectively help the model converge to a better local minimum.
\begin{figure}
	\centering
	\resizebox{0.9\linewidth}{!}
	{
		\begin{tabular}{c}
			\includegraphics{pdf/cifar100.pdf}
		\end{tabular}
	}
	\caption{Validation accuracy of 4-bit AlexNet on Cifar100 using (a): the fine-tuning strategy; (b): learning from scratch strategy. \emph{Stage2+Guided} means we combine the methods \emph{Stage2} and \emph{Guided} together during optimization to investigate the effect of the guided training on the final performance.}
	\label{fig:cifar}
\end{figure}

\begin{table}[!tbp]
	\centering
	\scalebox{1.0}
	{
		\begin{tabular}{c c c}
			\hline
			Method &top-1  &top-5 \\\hline
			 4-bit (TS) &57.7\% &81.0\%\\
			 4-bit (PQ)  &57.5\%  &80.8\%\\
			 4-bit (PQ+TS) &57.8\% &80.8\% \\
			 4-bit (Guided) &57.3\% &80.4\%\\
			 4-bit (PQ+TS+Guided)  &58.0\% &81.1\%\\
			 4-bit (PQ+TS+Guided**)  &\bf{58.1}\% &\bf{81.2}\% \\\hline
			 2-bit (TS)  &50.7\% &74.9\% \\
			 2-bit (PQ)  &50.3\% &74.8\% \\
			 2-bit (PQ+TS)  &50.9\% &74.9\% \\
			 2-bit (Guided) &50.0\%  &74.1\% \\
			 2-bit (PQ+TS+Guided)  &51.6\% &76.2\% \\
			 2-bit (PQ+TS+Guided**) &\bf{52.5}\% &\bf{77.3}\% \\\hline

		\end{tabular}}
		\caption{Evaluation of different components of the proposed method on the validation accuracy with AlexNet on ImageNet.}
		\label{tab:AlexNet_ablation}
	\end{table}

\begin{table}[!tbp]
	\centering
	\scalebox{1.0}
	{
		\begin{tabular}{c c c}
			\hline
			Method &top-1  &top-5 \\\hline
			4-bit (TS) &75.3\% &91.9\%\\
			4-bit (PQ)  &75.4\%  &91.8\%\\
			4-bit (PQ+TS) &75.5\% &92.0\% \\
			4-bit (Guided) &75.3 \% &91.7\% \\
			4-bit (PQ+TS+Guided)  &75.7\% &92.0\%\\
			4-bit (PQ+TS+Guided**)  &\bf{75.9}\% &\bf{92.4}\% \\\hline
			2-bit (TS)  &69.2\% &87.0\% \\
			2-bit (PQ)  &68.8\% &86.9\% \\
			2-bit (PQ+TS)  &69.4\% &87.0\% \\
		    2-bit (Guided) &69.0\%  & 86.8\%\\
			2-bit (PQ+TS+Guided)  &70.0\% &87.5\% \\
			2-bit (PQ+TS+Guided**) &\bf{70.8}\% &\bf{88.3}\% \\\hline

		\end{tabular}}
			\caption{Evaluation of different components of the proposed method on the validation accuracy with ResNet-50 on ImageNet.}
			\label{tab:ResNet-50_ablation}
\end{table}

\vspace{1mm}
\noindent\textbf{{\emph{The effect of quantizing all layers:}}}
This set of experiments is performed to analyze the influence for quantizing the first convolutional layer and the last fully-connected layer. Several previous works~\cite{zhu2016trained} argue to keep these two layers precision as 32-bit floating points to decrease accuracy loss. By comparing the results of \emph{PQ+TS+Guided**} and \emph{PQ+TS+Guided} in Table~\ref{tab:AlexNet_ablation} and Table~\ref{tab:ResNet-50_ablation}, we notice that the accuracy gap between the two settings is not large, which indicates that our model is not sensitive to the precision of these two layers. It can be attributed to two facts. On one hand, fine-tuning from 32-bit precision can drastically decrease the difficulty for optimization. On the other hand, the progressive optimization approach as well as the guided training strategy further ease the instability during training.

\vspace{1mm}
\noindent\textbf{{\emph{The effect of the two-stage optimization strategy:}}}
We further analyze the effect of each stage in the \emph{TS} approach in Figure~\ref{fig:cifar} and Figure~\ref{fig:resnet_2bits}.
We take the 2-bitwidth ResNet-50 on ImageNet as an example. In Figure~\ref{fig:resnet_2bits}, \emph{Stage1} has the minimal loss of accuracy. As for the \emph{Stage2}, although it incurs apparent accuracy decrease in comparison with that of the \emph{Stage1}, its accuracy is consistently better than the results of \emph{Baseline} in every epoch. This illustrates that progressively seeking for the local minimum point is crutial for final better convergence. We also conduct additional experiments on Cifar100 with 4-bit AlexNet. Interestingly, taking the model of \emph{Stage1} as the initial point, the results of \emph{Stage2} even have relative increase using two different training strategies as mentioned above. This can be interpreted by that further quantizing the activations impose more regularization on the model to overcome overfitting.
Overall, the two-step optimization strategy still performs steadily better than the Baseline method which proves the effectiveness of this simple mechanism.
\begin{figure}[!htb]
	\centering
	\resizebox{0.9\linewidth}{!}
	{
		\begin{tabular}{c}
			\includegraphics{pdf/resnet_2bits.pdf}
		\end{tabular}
	}
	\caption{Validation accuracy of 2-bit ResNet-50 on ImageNet. \emph{Stage2+Guided} means we combine the methods \emph{Stage2} and \emph{Guided} together during training.}
	\label{fig:resnet_2bits}
\end{figure}

\vspace{1mm}
\noindent\textbf{{\emph{The effect of the progressive quantization strategy:}}} What's more, we also separately explore the progressive quantization (\ie, \emph{PQ}) effect on the final performance.
In this experiment, we apply AlexNet on the ImageNet dataset.
We continuously quantize both weights and activations simultaneously from 32-bit$\to$8-bit$\to$4-bit$\to$2-bit and explictly illustrate the accuracy change process for each precision in Figure~\ref{fig:progressive}. The quantitative results are also reported in Table~\ref{tab:AlexNet_ablation}  and Table~\ref{tab:ResNet-50_ablation}. From the figure we can find that for the 8-bit and 4-bit, the low-bit model has no accuracy loss with respect to the full precision model. However, when quantizing from 4-bit to 2-bit, we can observe significant accuracy drop.
Despite this, we still observe $1.5\%$ relative improvement by comparing the top-1 accuracy over the 2-bit baseline, which proves the effectiveness of the proposed strategy. It is worth noticing that the accuracy curves become more unstable when quantizing to lower bit. This phenomenon is reasonable since the precision becomes lower, the value will change more frequently during training.

\begin{figure}[!htb]
	\centering
	\resizebox{0.9\linewidth}{!}
	{
		\begin{tabular}{c}
			\includegraphics{pdf/progressive.pdf}
		\end{tabular}
	}
	\caption{Validation accuracy of the progressive quantization approach using AlexNet on ImageNet.}
	\label{fig:progressive}
\end{figure}

\vspace{1mm}
\noindent\textbf{{\emph{The effect of the jointly guided training:}}}
We also investigate the effect of the guided joint training approach explained in Sec.~\ref{sec:mutual}. By comparing the results in Table~\ref{tab:AlexNet_ablation} and Table~\ref{tab:ResNet-50_ablation}, we can find that \emph{Guided} method steadily improves the \emph{baseline} method by a promising margin. This justifies the low-precision model can always benefit by learning from the full-precision model.
What's more, we can find \emph{PQ+TS+Guided} outperforms \emph{PQ+TS} in all settings. This shows that the guided training strategy and the progressive learning mechanism can benefit from each other for further improvement.

\vspace{1mm}
\noindent\textbf{{\emph{Joint vs. without joint:}}}
We further illustrate the joint optimization effect on guided training in Figure~\ref{fig:mutual}. For explaning convenience, we implement it based on the method \emph{Stage2+Guided} and report the 2-bit AlexNet top-1 validation accuracy on ImageNet. From the figure, we can observe that both the full-precision model and its low-precision counterpart can benefit from learning from each other. In contrast, if we keep the full-precision model unchanged, apparent performance drop is observed. This result strongly supports our assumption that the high-precision and the low-precision models should be jointly optimized in order to obtain the optimal gradient during training. The improvement on the full-precision model may due to the ensemble learning with the low-precision model and similar observation is found in~\cite{zhang2017deep} but with different task.
\begin{figure}[!htb]
	\centering
	\resizebox{0.9\linewidth}{!}
	{
		\begin{tabular}{c}
			\includegraphics{pdf/mutual.pdf}
		\end{tabular}
	}
	\caption{The effect of the joint training strategy using AlexNet on ImageNet.}
	\label{fig:mutual}
\end{figure}

\section{Introduction}
The state-of-the-art deep neural networks ~\cite{krizhevsky2012imagenet, simonyan2014very, he2016deep} usually involve millions of parameters and need billions of FLOPs during computation. Those memory and computational cost can be unaffordable for mobile hardware device or especially implementing deep neural networks on chips. To improve the computational and memory efficiency, various solutions have been proposed, including pruning network weights \cite{han2015learning, han2015deep}, low rank approximation of weights \cite{kim2015compression, zhang2016accelerating}, and training a low-bit-precision network \cite{zhou2017incremental, courbariaux2015binaryconnect, zhu2016trained, zhou2016dorefa}. In this work, we follow the idea of training a low-precision network and our focus is to improve the training process of such a network. Note that in the literature, many works adopt this idea but only attempt to quantize the weights of a network while keeping the activations to 32-bit floating point~\cite{zhou2017incremental, courbariaux2015binaryconnect, zhu2016trained}. Although this treatment leads to lower performance decrease comparing to its full-precision counterpart, it still needs substantial amount of computational resource requirement to handle the full-precision activations. Thus, our work targets the problem of training network with \textit{both low-bit quantized weights and activations.}

The solutions proposed in this paper contain three components. They can be applied independently or jointly. The first method is to adopt a two-stage training process. At the first stage, only the weights of a network is quantized. After obtaining a sufficiently good solution of the first stage, the activation of the network is further required to be in low-precision and the network will be trained again. Essentially, this progressive approach first solves a related sub-problem, i.e., training a network with only low-bit weights and the solution of the sub-problem provides a good initial point for training our target problem. Following the similar idea, we propose our second method by performing progressive training on the bit-width aspect of the network. Specifically, we incrementally train a serial of networks with the quantization bit-width (precision) gradually decreased from full-precision to the target precision. The third method is inspired by the recent progress of mutual learning \cite{zhang2017deep} and information distillation \cite{romero2014fitnets, hinton2015distilling, parisotto2016actor, zagoruyko2016paying, ba2014deep}. The basic idea of those works is to train a target network alongside another guidance network. For example, The works in \cite{romero2014fitnets, hinton2015distilling, parisotto2016actor, zagoruyko2016paying, ba2014deep} propose to train a small student network to mimic the deeper or wider teacher network. They add an additional regularizer by minimizing the difference between student's and teacher's posterior probabilities~\cite{hinton2015distilling} or intermediate feature representations~\cite{ba2014deep, romero2014fitnets}. It is observed that by using the guidance of the teacher model, better performance can be obtained with the student model than directly training the student model on the target problem.  Motivated by these observations, we propose to train a full-precision network alongside the target low-precision network. Also, in contrast to standard knowledge distillation methods, we do not require to pre-train the guidance model. Rather, we allow the two models to be trained jointly from scratch since we discover that this treatment enables the two nets adjust better to each other.

Compared to several existing works that achieve good performance when quantizing both weights and activations~\cite{wu2016quantized, zhou2016dorefa, hubara2016binarized, rastegari2016xnor}, our methods is more considerably scalable to the deeper neural networks \cite{he2016deep, he2016identity}. For example, some methods adopt a layer-wise training procedure \cite{wu2016quantized}, thus their training cost will be significantly increased if the number of layers becomes larger. In contrast, the proposed method does not have this issue and we have experimentally demonstrated that our method is effective with various depth of networks (\ie, AlexNet, ResNet-50).

\section{Related work}
\label{sec:related_work}

Several methods have been proposed to compress deep models and accelerate inference during testing. We can roughly summarize them into four main categories: limited numerial percision, low-rank approximation, efficient architecture design and network pruning.

\textbf{Limited numerical precision}
When deploying DNNs into hardware chips like FPGA, network quantization is a must process for efficient computing and storage. Several works have been proposed to quantize only parameters with high accuracy~\cite{courbariaux2015binaryconnect, zhu2016trained,  zhou2017incremental}. Courbariaux {\em et al.\/}\,  \cite{courbariaux2015binaryconnect} propose to constrain the weights to binary values (\ie, -1 or 1) to replace multiply-accumulate operations by simple accumulations. To keep a balance between the efficiency and the accuracy, ternary networks~\cite{zhu2016trained} are proposed to keep the weights to 2-bit while maintaining high accuracy. Zhou {\em et al.\/}\,  \cite{zhou2017incremental} present incremental network quantization (INQ) to efficiently convert any pre-trained full-precision CNN model into low-precision whose weights are constrained to be either powers of two or zero. Different from these methods, a mutual knowledge transfer strategy is proposed to jointly optimize the full-precision model and its low-precision counterpart for high accuracy. What's more, we propose to use a progressive optimization approach to quantize both weights and activations for better performance.

\textbf{Low-rank approximation}
Among existing works, some methods attempt to approximate low-rank filters in pre-trained networks~\cite{kim2015compression, zhang2016accelerating}. In~\cite{zhang2016accelerating}, reconstruction error of the nonlinear responses are minimized layer-wisely, with subject to the low-rank constraint to reduce the computational cost.
Other seminal works attempt to restrict filters with low-rank constraints during training phrase~\cite{novikov2015tensorizing, tai2015convolutional}. To better exploit the structure in kernels, it is also proposed to use low-rank tensor decomposition approaches~\cite{denton2014exploiting, novikov2015tensorizing} to remove the redundancy in convolutional kernels in pretrained networks.

\textbf{Efficient architecture design}
The increasing demand for running highly energy efficient neural networks for hardware devices have motivated the network architecture design. GoogLeNet~\cite{szegedy2015going} and SqueezeNet~\cite{iandola2016squeezenet} propose to replace 3x3 convolutional filters with 1x1 size, which tremendously increase the depth of the network while decreasing the complexity a lot. ResNet~\cite{he2016deep} and its variants~\cite{zagoruyko2016wide, he2016identity} utilize residual connections to relieve the gradient vanishing problem when training very deep networks. Recently, depthwise separable convolution employed in Xception~\cite{chollet2016xception} and  MobileNet~\cite{howard2017mobilenets} have been proved to be quite effective. Based on it, ShuffleNet~\cite{zhang2017shufflenet} generalizes the group convolution and the depthwise separable convolution to get the state-of-the-art results.

\textbf{Pruning and sparsity}
Substantial effort have been made to reduce the storage of deep neural networks in order to save the bandwidth for dedicated hardware design. Han~{\em et al.\/}\,  \cite{han2015learning, han2015deep} introduce ``deep compression'', a three stage pipeline: pruning, trained quantization and Huffman coding to effectively reduce the memory requirement of CNNs with no loss of accuracy. Guo~{\em et al.\/}\,  \cite{guo2016dynamic} further incorporate connection slicing to avoid incorrect pruning. More works~\cite{wen2016learning, lebedev2016fast, liu2015sparse} propose to employ structural sparsity for more energy-efficient compression.

redpajama_set_name: RedPajamaArXiv


Bomis was a Internet company created in 1996 by Jimmy Wales, Tim Shell, and Michael Davis. Bomis is best known for having supported the creation of the projects Nupedia and Wikipedia.
Bomis was popular for hosting entertainment, sports and adult sites. When Nupedia started, Larry Sanger was hired to be the editor-in-chief for the project. But Nupedia resulted too slow to grow and too expensive to maintain, and Wikipedia started as a side project to help Nupedia. Finally Wikipedia became more popular than Nupedia and it was transferred to a foundation in 2003. Bomis was shut down in 2006.
This page was last changed on 15 February 2019, at 10:17.


O'Brien Construction Company, Inc, (OCI) is a full-service client based construction and project management company.
OCI is committed to each and every client. We understand that each project is unique and deserving of quality service.
OCI's integrity has allowed us to build a solid reputation. We pride ourselves in maintaining a high rate of repeat business.