…. What is Distinctive about the AMAIC?
…. Our chief Teacher is Our Blessed Lord Himself teaching us from His Cross the most unloved truth and the most rejected wisdom of all: the Wisdom of the Cross. Next to Him and inseparably connected to Him "as the Body is to the Head", or as the "New Eve is united to the New Adam", stands His Holy Mother, the Blessed Virgin Mary. With Christ She is our chief teacher for of Her it is written: "Next to the Cross of Jesus stood His Mother Mary". We accept the holy Catholic Church as our teacher for of Her it is written that "She is the ground and pillar of Truth" [1 Tim. 3:15]. In Her dwells the Spirit of which the Son of God has said: "When the Spirit of Truth comes He will lead you to all the Truth". [Jo. 16:13].
At the root of all true perfection, sanctity and knowledge, lies the knowledge of the Cross by which we are led to ALL Truth: Supernatural Truth as well as natural truth. The Wisdom of the Cross is called "the super-eminent Science" by St. Louis de Montfort, for it is precisely in this remarkable fact: that through the Cross we not only attain to true holiness and perfection, but that ALL knowledge and understanding are acquired as well, that the Mystery of the Cross lies revealed. The Cross of Our Lord is the purifying instrument, the scalpel, that changes the soul, cutting away all attachments to the world, the devil and the flesh.
This truth is so alien to fallen human nature that all the great Saints and Mystics, in imitation of St. Paul, have named this Wisdom 'folly': The Folly of the Wisdom of the Cross.
…. "Until we all attain to the unity of Faith and of the knowledge of the Son of God, to mature manhood, to the measure of the stature and fullness of Christ". [Eph. 4:13]. These words of St. Paul go a long way towards explaining why the Cross, though a 'scalpel' and a 'purifying instrument', nevertheless is embraced, and why the knowledge it provides is sought after so diligently by all the "shrewd merchants". By it we attain to the full measure and stature of Our Lord Himself, even here on earth.
… Though many cultural and intellectual influences … must have been at work in shaping its members and their ideas, the AMAIC has not consciously based itself on any overseas model, as is what Australian colleges usually tend to do. With the incorporation of a model comes the incorporation of certain erroneous ideas and preconceived notions as well. For instance the model, though it may be largely a good one, may bear some pronounced socio-cultural influences that impede access to a religion and Scripture that have 'Semitic' origins (Pope Pius XI, Vatican Public Address, 1938); or it may have some underlying evolutionary tendencies; or it might have absorbed certain unwarranted demythologizing aspects from a non-Catholic approach to biblical study and form criticism.
Nor are the AMAIC's writers to be slavishly dependent upon borrowed writings, for the same sorts of reasons. The AMAIC and its writings, which aspire to be Wisdom-based, are largely home grown and original.
At first sight, the Constitution of the AUSTRALIAN MARIAN ACADEMY appears to be unusual. We could say it differs from any other Constitution inasmuch as the [AMAIC] differs from any other ordinary Academy. Ours is not an Academy in the ordinary sense of the word. Or else, maybe it is one on which any other Academy could be modelled. For in the [AMAIC] all are students intent on learning the most important science. ….
Hence, in the same spirit, our Chronology of the Alpha and the Omega will be firmly based upon Jesus Christ – and hopefully inspired by He who is the Lord of all History, the First and the Last; the Beginning and the End; the Alpha and the Omega. Jesus Christ is the reason for history, the creator of history, and the guide and culmination of all history (cf. Revelation 1:8; 21:6; 2:13). Thus it is Jesus Christ and his Book, the Bible (Old and New Testament), that basically determine our revised chronology; even though, as said above, "many cultural and intellectual influences … must have been at work in shaping its members and their ideas …" But: "… the AMAIC has not consciously based itself on any overseas model …. With the incorporation of a model comes the incorporation of certain erroneous ideas and preconceived notions as well".
They set themselves up in opposition to what they call "the conventional system", which is in large part – at least according to our own estimation – a Procrusteanised history that by no means fully accords with the archaeological or historical facts.
The Chronology of the Alpha and the Omega.
Prior to this, we had already indeed established this blog site, "Alpha and Omega" (http://amaic-alphaomega.blogspot.com/) for the purpose of accommodating what we already considered to be the necessary introduction of a revised AD timescale. This "Alpha and Omega" site will largely (but not entirely) be the location for our future efforts towards renovating the era of AD time.
It is not necessary to go into too much detail here at all about all the influences that have so far shaped the AMAIC's The Chronology of the Alpha and the Omega, since all of this is exhaustively covered by me in my recent University of Sydney thesis, A Revised History of the Era of King Hezekiah of Judah and its Background, which is accessible on-line (see http://kinghezekiahofjudah.blogspot.com/ for Vol. I and http://kinghezekiahofjudah2.blogspot.com/ for Vol. II). There is for instance, in the Introductory section of this thesis, a listing of all those who have contributed to a greater or lesser extent to the outcome of the thesis. And I have also gone into much detail about the revision to date, and where I agree or disagree with the various contributors to it. And all this is developed in greater detail throughout the remainder of this long 2-volume thesis, which modifies former opinions as well as breaking much new ground. Thus, despite the AMAIC's having used other people's ideas as a secondary foundation and starting point, and while retaining some of these and rejecting others, many of the conclusions that are eventually reached in this thesis - and entire chapters of it in fact - are completely new and original. For, as said above, "… the AMAIC's writers [are not] slavishly dependent upon borrowed writings …".
As a lead-in to the next section I shall just mention a couple of outstanding points of BC revisionism that I think are now going to be of relevance also in AD revisionism. As I mentioned in my thesis - whose distant origin (as far back as c. 1980) was my then somewhat naïve search for Moses in the history books (I say 'naïve' because conventional historians have absolutely no idea, in their context, of the era of Moses) - the thesis would never have come about had it not been for my encounter in those early days with the work of pioneering revisionists such as Drs. Immanuel Velikovsky and Donovan Courville (whom I actually read first). These had pointed out the serious errors of the conventional Egyptian chronology, and had shown a way towards a revision that was (and this was particularly the concern of the Christian, Courville - as Velikovsky apparently had his own secular, national Jewish agenda) compatible with the Scriptures. They both showed that the Dark Ages of Greek and Anatolian history, etc. (c. 700-1200 BC) are artificial chronological insertions, simply serving a mathematical (and we could say, a Procrustean) purpose, and having no basis whatsoever in reality.
Then how might all this be relevant to AD time?
Well, of course, we have Dark Ages too in the Early Middle Ages (c. 600-900 AD). This, though basically accepted as a fact without quibble by people in general, is a situation that might immediately strike one experienced in Velikovskianism as a possible area in need of revision. And that it has indeed, as we shall discuss in a moment.
[As an aside, it is interesting that psychoanalyst Immanuel Velikovsky was a Russian Jew, and that a leader today in a revision of history going into AD time is also Russian. The latter to whom I refer is the mathematician, Anatoly Fomenko. Fomenko has labelled his revision, "New Chronology", and this is exactly the same name that David Rohl, a British revisionist, has given to his revision of purely BC history].
Both Velikovsky and Fomenko would generally be regarded as extreme (see, for instance, the SIS comments on Fomenko below, in B.), and probably with good reason in both cases; they both having taken a good idea to extremes (or having taken a basic truth and made it virtually a be all and an end all). Having said that, I have made it absolutely clear in my thesis how great a debt I owed to Velikovsky for some vital revisionist insights of his. As for Fomenko, it is still too early for me to say to what extent, if any, The Chronology of the Alpha and the Omega might benefit from his revisionism, as I am not yet very familiar with it. Certainly, though, it seems on the face of things that some of his conclusions can be quite ridiculous and must be rejected out of hand (see B. again).
Now, a group of German Velikovskians, having presumably been rendered shockproof by BC revisionism, have actually embarked upon a Velikovsky-style revision of AD time, particularly centred on that Dark Age period of c. 600-900 AD. Two key figures amongst these are Heribert Illig and Hans-Ulrich Niemitz, formerly unknown to me (they do not figure at all in my thesis). I first became alerted to what these scholars were trying to do – and indeed to the whole concept of a need for AD revisionism, of which I had until then been completely unaware – in a Velikovskian journal from the U.K. that has also published some articles of mine in the past. I refer to The Journal of the Society for Interdisciplinary Studies [SIS]. Chronology & Catastrophism Review. I shall reproduce the relevant article at the beginning of the next section (B.), after referring firstly to a second area of special relevance (apart from the Dark Ages). This is the attempt by Martin Anstey early in the C20th to reconstruct a biblical chronology based entirely on the biblical data itself. In 1913, Anstey published the important and detailed The Romance of Bible Chronology, from Adam to Jesus Christ. One of the very important aspects of this work, so alert to the biblical data, was Anstey's inclusion of various interregnal periods in Israelite and Jewish history. These interregna other biblical chronologists tend to overlook, thereby rendering a tight chronological system that cannot properly accommodate a downward (i.e. down the timescale) revision of the chronologies of the other nations. Anstey's contribution was of enormous importance in my own thesis's reconstruction of the era of king Hezekiah of Judah.
Phillip Mauro more recently wrote The Wonders of Bible Chronology, a short and manageable book based on Anstey, in which Mauro claims to have corrected one slight mathematical error made by Anstey. I find Mauro a most valuable and handy reference tool. Anstey, who basically, though not uncritically, had accepted the conventional view of secular history (but this does not affect his internal biblical calculations), did not attempt to do what, say, Courville has done, to show how it is only a revised secular history that can be aligned with the biblical history and chronology. Thus Anstey has conventional Egyptian, Assyrian and other secular histories often (though in some cases the biblical correlations with the secular are well known) hanging awkwardly alongside his precise biblical calculations.
But now I think that we are in a better position to combine a well-calculated biblical chronology with a well-worked out revision of secular history. I, in my thesis, had tried to make this very precise for the specific period of the reign of king Hezekiah of Judah, using the common date of 722/21 for the Fall of Samaria (though I made it clear, based on Mauro's account of Daniel's prophecy, 9:24-27, that Persian history - which comes after Hezekiah of the neo-Assyrian era - was about 8 decades too long in relation to the biblical estimate – this will mean that 722 BC will eventually need to be lowered to that extent). It is a difference between absolute and relative chronology. I have tried to make precise the relative chronology for king Hezekiah of Judah, so that historians might be able to work backwards and forwards from there on the timescale to re-organise earlier and later history. But my thesis also goes beyond Hezekiah, back several centuries earlier (hence my use also of the term "Background" in the thesis title).
By no means, though, am I asking or intending for this University thesis to be considered as the last word on this most complex of subjects.
Having said all this, we can now turn our attention to a basic initial projection towards a revision of AD time, and to a consideration of what those Velikovsky-inspired German scholars have been claiming.
The views of the German scholars, Illig and Niemitz, appeared in the SIS journal, Chronology & Catastrophism Review (1999:2), section, Society News, compiled by John Crowe. The relevant part of Society News was introduced by Velikovskian, Clark Whelton. I give it here now with AMAIC comments added. After a discussion on p. 52 of "a paper by Damien Mackey on 'Velikovsky and Anti-Semitism' ….
After lunch … the meeting resumed with Clark Whelton and his assessment that it was not only the BC chronology which is in trouble: 'Did the Dark Ages of the First Millennium Really Exist?'. A group of German scholars has been studying the possibility that the 'Medieval Dark Age', dated roughly from 600AD to 900AD, did not, in fact, exist. They claim that when the first attempts were made around the start of the second millennium to apply dates relative to the birth of Jesus, the length of time which had elapsed since in [sic] birth of Jesus was overestimated by some 300 phantom years. If they are correct, William the Conqueror invaded England not in 1066, but about 766, and the year 2000 is about 300 years away, giving contractors plenty of time to complete London's Millennium Dome and Jubilee Line extension.
The initial reaction is that it is obviously ridiculous. How and why could this mistake have been made, and why was it not noticed earlier? Surely somewhere there must be a continuous written record dating from the time of the Romans? Surely the priests of the Vatican and Islam kept such records? Surely there must be archaeological evidence which can be safely dated somewhere within this period? What about the Chronicles of the Rev. Bede (673-735)? Is it true that there are no known documents carrying an AD date before what we now call 1000 AD?
A scholar who believes this idea is at least worth a closer study is Clark Whelton, a long time friend and admirer of Velikovsky and a good friend of SIS. Clark said Velikovsky's historical revision, particularly the later books, contained many flaws and he has generally lost confidence in Velikovsky's motives and methods, particularly his total commitment to the reliability of biblical history, which seems to place impossible demands upon the completion of his revision from the promising start in Ages in Chaos.
AMAIC Comments: Velikovsky was hardly a biblical fundamentalist.
Velikovsky made a useful start with the revision, and for this he can be thanked. His system should never have been set up as some sort of monument in stone, since it is far from perfect.
The AMAIC has taken Jesus Christ as the unfailing Model and Master of history: the perfect Time Lord.
This, however, has not diminished Clark's interest in historical revisions and he talked about the recent work of Heribert Illig, including his book The Invented Middle Ages: History's Greatest Falsification of Time [Econ-Verlag, Düsseldorf, 1996]. Illig claims that for roughly 300 years, from about 600 AD to 900AD, no original documents and no buildings have been found – and that there is no hard evidence that Charlemagne actually existed. In this respect the Charlemagne and Arthurian legends are similar. Illig is one of a group of friends, including Hans-Ulrich Niemitz, who have studied this subject since 1990, publishing several papers, mostly in German. Clark Whelton gave a brief summary of a paper by Niemitz. This looks at the documentary and archaeological evidence worldwide against the theory that the Middle Ages have been artificially extended by 300 years. He and his colleagues have amassed a considerable amount of additional evidence for the theory from places as far afield as Iran and India.
A paper by Illig in 1991 revealed that, as well known to conventional biblical scholars, many early church documents (most surviving documents from that period have a religious origin) are known to be falsified and have therefore probably been misinterpreted. The theory is also supported by the 1582 calendar reform by Pope Gregory XIII. The 'Gregorian calendar' was needed to correct a small error in the Julian calendar, which fell behind true solar time by little less than a day per century. As there were 16 centuries between Pope Gregory and Julius Caesar, it should have been necessary to add 13 days to adjust for this difference. Instead calculations at the time showed that only 10 extra days were needed. This indicates that over 300 fewer years had actually passed than suggested by the difference in dates.
AMAIC Comments: We recommend the Niemitz paper as a start because he writes better English than do some of the others.
We accept the holy Catholic Church as our teacher for of Her it is written that "She is the ground and pillar of Truth" [1 Tim. 3:15]. In Her dwells the Spirit of which the Son of God has said: "When the Spirit of Truth comes He will lead you to all the Truth". [Jo. 16:13].
Thus Pope Gregory's calendrical reform, when properly interpreted, may be seen as a most providential yardstick for an overall chronological alignment. (This does not mean that individual churchmen could not have misinterpreted, and even falsified, various historical data).
1. how was it possible to insert 300 years into history?
3. how can the problem be exposed to a wider audience for discussion without being labelled a 'von Daniken of the Middle Ages'?
Clark told us that the Domesday Survey of 1086 and even the Magna Carta of 1215 were each dated to a particular year of a king's reign and not given AD dates. He presented some very interesting information on reported astronomical activity and observations during the later 16th/early 17th c. In Nov, 1572, a bright new star suddenly appeared in the sky in the constellation Cassiopeia, brighter than Venus. This contradicted Aristotle, who had said a new star was impossible as the celestial sphere was unchanging. Then in 1577, a spellbinding comet was seen in the sky, in Capricorn. It had a long tail, was dark red, and stretched across an arc of 22 degrees. It was as large as Venus and was visible for 2 months. One astronomer claimed it was 4 times as far away as the moon, i.e. about 1 million miles away. Tycho Brahe thought, from his observations, that the ecliptic had changed and warned of dark consequences. In 1583 there was a great conjunction with Saturn and Jupiter. Minor conjunctions occur every 20 years but great conjunctions only every 800 years. These were supposed to foretell great events, such as the birth of Christ. The Great Conjunction of 1583 was said to be the 8th since the creation of the world. In 1604, another supernova was seen. Kepler observed it and found no parallax, showing it to be very far away.
Two supernovas had now been seen without dire consequences, so astronomers began to perceive them as natural events, not divine warnings. Also around this time, telescopes came into general use and dark spots were seen on the sun for the first time, proving that even the sun was not flawless. In 1618, another great comet was seen, inspiring Galileo's 'Discourse on Comets'. By 1630, all this astronomical activity came to an end.
AMAIC Comments: Velikovskians are fascinated by astronomy, particularly by the notion of cosmic catastrophism, which was also a major part of Velikovsky's theses, as famously presented in his book, Worlds in Collision. He combined all this with his historical revisionism (the Ages in Chaos series). The Great Flood; the Exodus; and the destruction of Sennacherib's Assyrian army (c. 700 BC), for instance, Velikovsky/Velikovskians have attributed to planetary collisions. There is nothing, however, in the Old Testament accounts of the Plague and Exodus that specifically ties the phenomena of this period to the planet Venus, for instance, as Velikovsky had tried to do (though Velikovsky did more convincingly find some compelling similarities between the Old Testament's Plague narrative and accounts given in certain Egyptian Middle Kingdom papyrii). And, contrary to Velikovsky's view that it was an errant planet Mars that had caused the destruction of the Assyrian army in king Hezekiah's day, I have dedicated Volume Two of my thesis to showing that the Book of Judith provides the detailed account of what actually happened then, with Judith herself being the instigator of the destruction of the massive Assyrian army.
The Gregorian calendar reform came in the middle of this period of extraordinary astronomical activity. Interestingly, although it was quickly adopted by Catholic countries, others were reluctant to do this. Britain adopted it in 1752. Russia in 1917, Greece in 1923 and China in 1949. The British delay, according to one observer, was because Parliament 'preferred a calendar that was wrong to a Pope that was right'. ….
Clark's paper stimulated an animated discussion. This became even more animated when another former contributor to [SIS Review], Chris Marx, and two German friends introduced a competing alternative chronology derived from statistical studies of ancient documents by Fomenko, a Russian scholar. This concludes that not 300 but about 1000 years should be removed from AD history. The impression by the end of the discussion was that, while the 300 year theory may be worth further consideration, the Fomenko conclusions should be easily disposed of as not valid, at least for British history. No doubt we shall hear more of these ideas in the future; there are web sites where further information can be obtained.
Clark, with the assistance of Chris Marx, wielding copies of Fomenko's books on the subject, parried questions from a rather time-shocked audience. Dendrochronology and ancient Chinese astronomical records were among the more obvious subjects which could raise problems for such radical ideas but their promoters were not unduly fazed and the audience must have been left with their faith in the absolute correctness of the AD record at least a little shaken.
Time remained for a short general discussion which ranged over a wider field than considered above. Hopefully communication between some of those present may lead to new ideas and future articles. Both speakers were warmly thanked, as was Trevor [Palmer] for his presentation of Damien Mackey's paper. From the animated discussion it was clear we had all had a most enjoyable and stimulating day.
Where do we start with this enormous, but fascinating project to revise AD? And what can be our overall plan?
Just as we had, with the Bible, a reliable 'spine', so to speak, upon which to assemble our BC chronology - bringing history down from Adam unto "the fullness of time" (Galatians 4:4), the Incarnation and life, then Death and Resurrection of Jesus Christ - so now do we need a similarly Divinely-inspired (as we believe) support for AD time.
Since we have chosen for our foundation, Jesus Christ, the "corner stone", "rejected" by the general run of "builders" (Acts 4:11), including "builders" of chronologies, but: "There is salvation in no one else" (4:12) – (nor are history and chronology intelligible without Him) - and had turned to his book (the Bible) for determining BC time, so it must be to Him again to whom we turn for rectifying AD time. But since Jesus Christ now operates through the Church that he established, which for the AMAIC is the Catholic Church, then we should expect to find within the Catholic Church the key to the interpretation of AD chronological time, presuming any correction be needed. The obvious chronological 'spine' in the case of the Catholic Church must be the Papacy - also divinely established, as we believe, by Jesus Christ himself (Matthew 16:18) - since it is commonly acknowledged that 'the Papacy is the oldest uninterrupted institution in the world, spanning 2000 years'.
But just as the Bible, with its differing versions today, and with its original figures not always accurately transmitted, needs to be interpreted and assisted by comparisons of texts, and also with support from external evidences (e.g. the neo-Assyrian records throw much light on the history of the Book of Judith, which, in turn, serves to modify considerably the conventional neo-Assyrian history), so too would this sort of scenario likely apply also to the Papacy. And just as the Bible records interregnal periods, which can be so easily overlooked by biblical scholars, so too have there been brief periods of vacancy in Church history, when no pope was actually occupying the Chair of Peter.
The Catholic Church with its papacy must be the 'spine' for our AD time revision.
Two other possible miraculous aids to time revision (the Church's very survival being, like the Bible's, miraculous) will be the science-defying Shroud of Turin and the Tilma image of Our Lady of Guadalupé, on which, apparently, the constellations appear (and some say Halley's comet as well). For more on all this, see our science sites as cited above.
Apart from these areas of Divine assistance, as we believe, for AD time - corresponding to the Bible in BC time - we have the other common BC and AD factor of the so-called 'Dark Ages', as already mentioned.
…. This question in itself – and more so the answer 'NO, the early Middle Ages did not exist' – is surprising, to say the least. It contradicts all basic knowledge and attacks the historian's selfrespect to such an extreme that the reader of this paper is asked to be patient, benevolent and open to radically new ideas. I shall argue step by step – and, I hope, you will follow. With a group of friends (Müller 1992; Illig 1991; Niemitz 1991; Zeller 1991; Marx 1993; Topper 1994) I have been doing research on this subject since 1990. This is the reason for using 'we' or 'I' intermittently. The thesis mainly says, with far-reaching implications and consequences: Between Antiquity (1 AD) and the Renaissance (1500 AD) historians count approximately 300 years too many in their chronology. In other words: the Roman emperor Augustus really lived 1700 years ago instead of the conventionally assumed 2000 years.
However, the whole well-known historiography of the Middle Ages contradicts this assertion!
The easiest way to understand doubts about the accepted chronology and 'well-known' history is to seriously systematize the problems of medieval research. This will lead us to detect a pattern which proves my thesis and gives reason to assume that a phantom period of approximately 300 years has been inserted between 600 AD to 900 AD, either by accident, by misinterpretation of documents or by deliberate falsification (Illig 1991). This period and all events that are supposed to have happened therein never existed. Buildings and artifacts ascribed to this period really belong to other periods.
To prove this the Carolingian Chapel at Aachen will serve as the first example. Art historians explain and describe artifacts and buildings of this period as anachronistic – but they never follow up on their assessments. One of the best examples, intensively surveyed, is the Chapel of Aachen (ca. 800 AD), which seems to come approximately 200 years too early.
The way of constructing an arch shown in this chapel has no predecessor (Adam 1968,7).
Arched aisles are usual only in the 11th century in Speyer. The construction of choirs with rising arch and also rising barrel vaulting is not resumed until 200 years later at the portal of Tournus (Hubert 1969,67). The vertical steepness of the interior arches of the Aachen Chapel is more accentuated than those of churches built two centuries later. One of these is the 1049 AD consecrated Abbey-church of Ottmarsheim. Although missing some details of the early model, nevertheless it is the "best copy" of Aachen. However, these and many other arguments implicate that the Chapel of Aachen has to be regarded as a building of the second part of the 11th century.
It seems, unbelievably, that Caesar introduced his calendar in 325 AD. This is unbelievable because by then he had already been dead for more than 300 years. If 16 centuries had passed since Caesar's introduction of his calendar, the Julian calendar in Gregory's time would have been out of sync with the astronomical situation by 13 days, not 10. Some historians have noticed this contradiction, but they solve it this way: the scholars in Caesar's time reckoned a different date for the equinox (the day in spring, where day and night have the same length). Yet it can be proved that the Romans used the same date for the equinox as we do today, i.e. the 21st of March (Illig 1991 and 1993).
Uwe Topper and Manfred Zeller pointed out how to resolve some important riddles and research problems of the Islamic and Persian-Arab-Byzantine world using the thesis of the phantom years. Firdowsis' well-known epic, the Shahname, written around 1010 AD, ends with the last Persian king Yazdegird III, who died 651 AD. The epic tells nothing about the Islamic conquest of Persia and has no allusions to Islam at all. It simply skips 300 years of Islamic influence as if they had never existed (Topper 1994). ….
The introduction of Arabic coins organized by the caliph Abd al Malik in 695 AD reveals some remarkable contradictions. The Arabic coins bear the portrait of the Persian king Khosrow II who died long time before (Zeller 1993). Manfred Zeller draws the following conclusion from this and other facts: We have to eliminate two phantom times; the first of approximately 78 years (from 583 AD until 661 AD) and a second one of approximately 218 years (from 750 until 968). The first phantom time came into being through artificial separation of the Persian and Omayyade history which in reality was contemporaneous. Thus Zeller explains the contradictions of the un-Islamic style and the late appearance of Khosrow II in the Omayyade history. The second phantom time is a time-stretching repetition of the Persian history from 528 AD until 661 – now in Islamic style and dated from 750 until 968.
Harun al-Rashid is an invention, as is the whole history of his dynasty (Müller 1992). Probably Islam did not spread until 10 centuries ago.
The history of the Jews shows centuries of darkness and discontinuity that support the thesis of the phantom time. One of the important modern works on Jewish history bears the descriptive title "The Dark Ages. Jews in Christian Europe 711-1096". Here are two quotations: "It seemed, that they (the Jews) totally disappeared together with the breakdown of the Roman Empire. However, we don't find any evidence of their presence until the Carolingian period." (Roth 1966,162). For the Carolingian period historians find only written sources, whereas material sources like buildings and artifacts exist just for the time after 1000 AD (Jewish quarter in Regensburg between 1006 and 1028 AD, in Cologne between 1056 and 1075 AD, in Worms around 1080 AD, in Speyer around 1084 AD etc.). And for the regions outside Germany we are told: "Of course we know from inscriptions and other sources about Jewish societies and single persons in nearly all provinces of the Roman Empire, and we can reasonably suppose – with or without proof – that there is in fact no district without Jews. Nevertheless there doesn't exist any evidence and only little probability that a substantial number of Jews lived anywhere in the western world at this time." (Roth 1966,4; Illig 1991).
Fig. 1: The center of the graph shows the time axis of conventionally dated historical events. Upper and lower coordinates show reconstructed time tables. The black triangles mark the phantom years.
[This figure is important, but we cannot reproduce it here, and so we urge the interested reader to check it on the Internet version of Niemitz's article, at: http://www.cl.cam.ac.uk/~mgk25/volatile/Niemitz-1997.pdf].
How can we find support in archaeological research for our phantom time? In 1986 a conference of archaeologists specializing in medieval towns took place under the heading: "The rebirth of towns in the west, AD 700–1050". The participants discussed a puzzling problem: does continuity in the evolution of the town – postulated by historians – exist or not?
The archaeologists could prove that the stratigraphies of towns – adapted to 'normal' chronology – inevitably involve gaps. Where was continuity? Generally it is enigmatic to archaeologists if towns in the early Middle Ages had had an economic sense or function. Historians and archaeologists must maintain that no civil town building existed in this period and that only church buildings survived from that time. To justify this they must assert that at that time towns were not central marketplaces (Hodges 1982,49; Niemitz 1992).
In the same manner as archaeologists, the researcher of ceramics commits a major mistake – a mistake that Günter P. Fehring warned of: "Independent of the existence of written sources for that period, the medieval archaeologist should try to date his finds with archaeological means. It is not permitted to act in a different way than an archaeologist of ancient history." (Fehring 1992,48). Thus archaeologists are not permitted to rely on written sources, especially when the archaeological results and the evidence contradict them – even if generations of historians have worked on those written sources! For instance the framework of the ceramics chronology has been elaborated for one hundred years; the researcher correlated each surprising find in accordance with known history, although more than 100 years ago in the science of Antiquity (Altertumskunde) we can find amazing words that must have been a warning. For example, the following quotation (1880) deals with the end of the Merovingian period: "But we do miss a reliable milestone of this kind like the one provided by the burial hoard of Doornick (Childerich I). The boundary leads into uncertainty, sometimes until the 9th and even 11th centuries." (Lindenschmit 1880,75).
The most common objection to this idea says that methods of scientific dating are infallible and beyond the danger of circular reasoning. Of these methods the best known are RC14 and dendrochronology. As RC14 is now calibrated by and therefore dependent on dendrochronology it is insignificant for the medieval period. Horst Willkom (1988) says: "We use the C14-method only to correlate a sample of carbon with a certain tree ring. This method – at least for the post-glacial period – has become a relative method." (Willkomm 1988,176). Dendrochronology works, because each time period creates typical sequences of tree rings. With the help of overlapping sequences of different trees it is possible to construct a standard sequence that reaches back centuries for a given region. In this standard sequence you can find for any suitable wood the corresponding age of growth of this tree. This seems to be a very simple theory; you have to count the tree rings if you want to find the exact year, that's all. But as each individual wood differs from the other and also differs from the standard sequence, dendrochronologists must work with a tremendous mathematical apparatus: "Transformation of the ring-width into logarithmic differences, preferential treatment of the correlating arithmetic, theoretical derivation of congruent patterns, distance regression of similarity, regional analysis, test function with scope of dating, statistics of sapwood and lost tree rings, distribution of centered differences between dating of art styles and dendrochronological dating." (Hollstein 1980,11; Illig 1993).
Another argument against dendrochronology consists of its own history: In 1970 Ernst Hollstein – one of the famous and most active dendrochronologists in Germany – proclaimed after eight years of research a great success: he had solved the problem of bridging the early Middle Ages. Now he could date all samples back to Roman time 'absolutely and exactly'. But he paid a high price – too high as we know today. He did in fact change his method; instead of using oak for his oak-chronology he used copper beech wood (= Rotbuchenholz) to fill the gap. But ten years later in 1980 it was no longer necessary to use beechwood for the oak-chronology, he fortunately had found fitting oak wood – but dates he had defined in 1970 were irrevocable! This is unusual in science, and he did it only reluctantly as we can see from the following quotation, where he describes his method: "The method is unbiased and basically not graphic, because it combines many calculation operations, and you have to do the more operations, the more the problem of dating is distant from today and from comparable historically secured dates. At border-line when totally leaving known history – e.g. analyzing two completely unknown samples of trees which grew supposedly at the same time – the a priori probability of finding the right date can become so small that only a little chance remains of finding it. For trees of the post-glacial the probability is approximately one in 10000. These hints show clearly the biggest difficulty of the dendrochronological method." (Hollstein 1970,147). "All attempts to get enough tree ring sequences from timber of the Carolingian times have failed …" – "It is strange, but it proved as extremely difficult to connect the Merovingian wood samples from excavations with the above mentioned chronologies." "Encouraged by an unusually deceptive correlation – as happens from time to time in all fields of knowledge – I believed that I could date the chamber burial of Hufingen at the beginning of the 8th century." – "After two years of intensive studies I can name at last the right dates and put in order all samples of the early Middle Ages." (Hollstein 1970,148). "The misdating of coffins from Hüfingen and Oberflacht, which had been calculated very carefully in accordance with the then valid rules (sic!), suddenly proved that the statistical methods must be tightened up the more the event is distant from the starting point. That has been neglected hitherto – but is now corrected. Now we don't choose arbitrarily a so-called error-probability at the outset, as has been usual. Now we choose the probability of the correct approach as a function of the observations and available hypotheses. The quality of this test method is, if circumstances permit it, one hundred times better and also more certain than the old one." (Hollstein 1970,148f).
Here we can observe the way Hollstein changes the method, which reveals even more, i.e. the expectations of waiting colleagues; and those expectations weren't purely academic as he had to beg for money for his research. "As I have been asked time and again to publish my new dates, I feel obliged to give them in a preliminary way and without proof, be it in contradiction with scientific tradition. … From this chronologically arranged summary, one cannot deduce after how many futile tests the right correlation has been found. And it still remains a process of searching, which is not altered by the fact that we use computers." (Hollstein 1970,149).
Hollstein reports about bridging the critical period of the 8th century; there simply exist no samples of wood. "In this moment Dr. R. Gensen discovered on the Christenberg near Marburg/Lahn a well from Carolingian times, whose heavy squared timbers provided a sequence of 339 years. Although this is wood of red-beech, a type of wood with occasional tendency to missing rings, still this Marburg sequence coincides in the decisive period of the 7th, 8th and 9th centuries … with the other sequences … So early medieval wood samples cannot be regarded as firmly dated." (Hollstein 1970,150f). In brief, Hollstein says that he – being in trouble – took a piece of wood predated by historians to fill his gap (see Fehring above). This cannot be called archaeological evidence.
A good scientist in trouble usually sends out signals for help – sometimes unconsciously – and so does Hollstein. After listing his tree ring dates secured by what he terms the 'oldest historically documented year-by-year dates' he says: "It is good to know that the West German oak chronology for nearly one thousand years is exactly under control – year by year. Going back to earlier millenia, the missing documentary frame has to be substituted by an essentially tightened correlation." (Hollstein 1970,152). We could express this statement thus: It is very difficult to point out the years before 1000 AD because we do not have any dates supported by written documents. Why not quote more recent test results (after 1980)? A Hamburg dendrochronologist responded to my request for recent literature in December, 1994: today sequences and dates are no longer published because there exists the danger of abuse. Hobby-dendrochronologists earned money by dating for example timbers of houses for private clients with unreliable methods. So laboratories in Europe and worldwide exchange their dates without publishing them (Niemitz 1995 – in preparation).
After a cursory glance over the riddles and research problems two great questions remain: "How was it possible to insert this phantom time into history?" – or asked in a provocative manner – "Who (and when and how and why) falsified history by adding 300 years?" And the second question: How can it be possible to discuss the thesis of the phantom time in the scientific community without being discriminated against as some 'Von Däniken of the Middle Ages'. ….
Our method of interpreting some of these apparent anomalies is somewhat different, and, in certain cases, even more radical.
The 'Dark Ages', 600-900 AD, seem to be acting like something of a 'Black Hole', sucking into it certain genuine ancient (and biblical) events from the BC era.
I have already begun my own revision of AD time, centring around the Book of the Apocalypse, the Fall of Jerusalem in 70 AD (conventional dating), and the early Roman Empire (this may have further ramifications for the so-called Julian Calendar, as well; Julius Caesar himself being a real historical anomaly). My initial effort can be found at: http://apocalypsenoworthen.blogspot.com/ In this case, I have argued that some very early AD history, connected with the catastrophic Fall of Jerusalem to the Roman general, Titus, has been duplicated with (or sucked into) the era of the supposed Second Jewish Revolt about half a century later, presumably – but which I maintain never actually happened as an incident separate from the First Jewish Revolt: that this was all in fact one and the same Jewish revolt against Rome.
Since completing this (though there will be much more to follow), I have written a reconstruction of the life of the prophet Jeremiah, multi-identifying him, and showing that he, particularly in his guise as the governor, Nehemiah, is the prototypal Mohammed: an Israelite, not Arabic, prophet. (See my "Filling in all the details for the great prophet Jeremiah" at http://bookofjob-amaic.blogspot.com/ Now I find by way of extraordinary confirmation of this last that that there was, at the very time of Mohammed, conventionally speaking (600 AD, as opposed to my 600 BC, for Jeremiah's era), a 'Nehemiah, a Jew', intending for Jerusalem the same sort of restoration as had the supposedly original governor Nehemiah ('both' apparently authorised by the Persian power).
Now, the fact that Niemitz's and company's 'Dark Age' falls right at this very time makes all of this even the more intriguing.
It is beginning to look like genuine BC time is being confused as AD time; that the 'Dark Ages', 600-900 AD, seem to be acting like something of a 'Black Hole', sucking into it certain genuine ancient (and biblical) events from the BC era.
As with Mohammed - though not necessary Islam, as I shall be at pains to show in forthcoming articles - so possibly with Heraclius, and so possibly with Charlemagne, may there have been a 'sucking' of genuine BC histories into AD time. We have already learned from Niemitz about some of the awkward problems associated with Mohammed (and Islam) and also with Charlemagne. This Charlemagne, is, I find, in many ways highly reminiscent of the great king Solomon of Israel himself (950 BC sucked into 800 AD?), whilst Charlemagne's semi-legendary friend, Harun al-Raschid (whom Niemitz claims was "an invention", and Illig, as we shall see below, calls him a "protagonist in tales from One Thousand and One Night[s]") is much like Solomon's Phoenician friend, king Hiram.
I intend for my next article to be a consideration of whether Solomon and Hiram have also in fact been sucked into a C9th AD context. (I have already argued at our King David site, http://amaic-kingdavid.blogspot.com/ that the great king Hammurabi, conventionally dated to c.1800 BC, needs to be lowered by about 800 years, to be identified as this same king Solomon (in his guise as ruler of Babylon). That would mean a shift of some 2600 years (1800 BC – 800 AD) for Hammurabi to be connected with Charlemagne (though unnecessary if Charlemagne were Solomon).
And some at least of the life of the emperor Heraclius may remind one of Tirhakah, the strong king of Ethiopia, who had assisted king Hezekiah of Judah against the Assyrians (700 BC sucked into 600 AD?), with Chosroes [Khosrow] II, his opponent - (we recall Niemitz's problems with him) - making some very Sennacherib-like utterances. I shall discuss all this in future articles.
…. Is it possible to eliminate Charlemagne from history?
Birthplace: More than a dozen places are claiming the honour to be the birthplace of Charles. The year of birth varies between 742 and 747 AD. Bertrada, the mother of Charles, was said to be a Bretonian princess, an Hungarian noble woman, or a member of the imperial family of Byzantium.
Imperial coronation: It is unclear whether Charles requested the coronation, or whether he was crowned unexpectedly by the Pope. It is not clear whether there was a formal coronation or an acclamation. Einhard reports just the 'acceptance of the imperial title'. Andreas from Bergamo (9th century), Bonizo from Sutri (11th century), Gero from Reichersberg (12th century) and Nicolaus Cusanus (15th century) don't know nothing about an emperor Charles.
Vassals: in 1966 Bullough could name just two of more than a thousand reported vassi dominici and up to 30.000 noble vassals of Charles. Until 1990 Walther Kienast managed to increase the number of individual names to 24.
Relations: Charles exchanged diplomats with Harun al-Raschid, the Caliph of Baghdad, who sent him the white elephant Abul Abbas, which took part in all journeys and military actions of Charles between 802 and 810 AD. Arab sources do not mention these relations. Harun al-Raschid has become famous as protagonist in tales from One Thousand and One Night[s].
Buildings: As we know from the ancient texts, between 476 and 855 AD more than 1695 large buildings were erected, including 312 cathedrals, 1254 convents and 129 royal palaces. The historian Harald Braunfels: "Of all these buildings [until 1991] only 215 were examined by archaeologists. Artefacts were found only at a fraction of these buildings. One may count with ten fingers the number of buildings that still exist as a whole or as a significant fraction."
Pfalzkapelle Aachen: The masterpiece of Carolingian architecture, the Chapel of St. Mary at Aachen (about 792-799) is unique. Its direct predecessor (Ravenna's San Vitale) had been erected some 200 years earlier. Buildings comparable to Aachen in style and technology were not erected until the advent of the Romanesque style in the 11th century. Consequently, Illig assumes the Pfalzkapelle to be a Romanesque building of the 11th century.
Economy: The findings of historians regarding Charles' economy show extreme contradictions: Some concede abundant wealth to Charles, while others have to complain economic decay. As Heinsohn has shown recently, coins attributed to Charles (or, likewise Charles the Bald-head) cannot be distinguished from the coins of Charles the Simple (898-929). According to Illig, Carolus Simplex has been a real Carolingian and the model for Charlemagne. The attribute "simplex" (stupid, but likewise single, not-duplicated) has been used for the first time following the turn of the millennium.
Charles' burial place is the Pfalzkapelle at Aachen (his explicit will to find his grave beneath his father at Saint-Denis had been ignored). This contradicted the general prohibition of burials within churches, proclaimed by councils held under Charles at Aachen (809) and Mainz (813).
Tomb: Charles' tomb had been camouflaged so well (in fear of the raiding Normans) that it could not be localized for two centuries. In the year 1000 the emperor Otto III discovers the tomb. He finds Charles sitting on his throne. Again the tomb became forgotten until it was found once more and reopened by Friedrich Barbarossa. Then again, the tomb disappeared and was never found again. For comparison: The tomb of Otto I in the dome of Magdeburg has always been honoured – despite of all destructions and rebuilds of this church.
Cult: Friedrich Barbarossa (1152-1190) is said to have coined the term Sacrum Romanum Imperium. Friedrich gave order to exhumate Charles, and to canonize him. Most known forgeries referring to Charles were produced during Friedrich's lifetime. The reliquary for Charles' arm (dated about 1170) displays the imperial attitude of Barbarossa in reference to Charlemagne.
Biography: Leopold von Ranke classifies the biography of Charles, written by his palatial clerk Einhard: "The small volume is full of historical errors [...]. Frequently, the years of reign are false [...]; about the split of the empire between the two brothers the opposite of what really happened is reported [...]; the names of the popes were confused, the spouses and children of Charles were not noted correctly; so many offences have been found that the authenticity of the book has been questioned quite often, although it is beyond all doubt."
Tradition: Charles' son in law Angilbert rhymes in 799 an epos, where he denotes Charles to be the "light of Europe", "Head of the world; summit of Europe; father of Europe; most graceful father; hero". But in 799 Charles was not yet crowned as the emperor. In an essay for the Spiegel magazine ("A dark lighthouse", Johannes Fried has shown that the myth of Charles as the "father of Europe" came up very much later as a product of a romantic Napoleonism and even Hitlerism.
British revisionist Peter James, at the beginning of his Centuries of Darkness, has cleverly used Shakespeare's Hamlet thus: "The time is out of joint, O cursèd sprite, that ever I was born to set it right" (Act 4.5). Though the fact is that it is quite beyond any human power to set chronology right, since we humans have by now made such a complete mess of it.
It is Jesus Christ and He alone, the Alpha and the Omega, who was truly "born", "in the fullness of time", "to set it right".
 Copies of the full Niemitz paper may be obtained from H. Illig at the address below, which contains an extensive Bibliography for further reading. All publications can be obtained from: Vorzeit-Frühzeit-Gegenwart Interdisziplinäres Bulletin, Published in the Mantis Verlag. Heribert Illig D-82166 Gräfelfing, Lenbachstraße 2a. Tel: 089/87 88 06.


text:
A new MMR study debunks link between vaccines and autism
Dr. Jon Hallberg
PlayListenA new MMR study debunks link between vaccines and autism
Vials of the measles, mumps and rubella vaccine are displayed on a counter at a Walgreens Pharmacy in January 2015 in Mill Valley, Calif.
Justin Sullivan | Getty Images 2015
There is no connection between the measles, mumps and rubella — or MMR — vaccine and autism. That's the conclusion of an extensive study out of Denmark this month. It tracked more than 600,000 children born between 1999 and 2010 and found those who received the vaccine were no more likely than those who did not to develop autism.
Dr. John Hallberg, medical director of the University of Minnesota Physicians Mill City Clinic, told MPR News host Tom Crann the study gives doctors another tool to help convince parents who are wary of getting their children vaccinated that vaccines are safe for most.
Despite several studies already debunking the idea that vaccines and autism are linked, the myth persists. A Star Tribune analysis of state data recently found a third of schools in the state have vaccination rates low enough to compromise herd immunity. And in 2017, there were 75 measles cases in Minnesota.
To learn more about the study, click play on the audio player above.

redpajama_set_name: RedPajamaCommonCrawl


text:
\section{Conclusion}
\vspace{-3pt}

We have shown how to achieve significant reductions in the number of dynamic
parameters in deep models. The idea is orthogonal but complementary to recent
advances in deep learning, such as dropout, rectified units and maxout. It
creates many avenues for future work, such as improving large scale industrial
implementations of deep networks, but also brings into question whether we have
the right parameterizations in deep learning.

\section{Low rank weight matrices}

Deep networks are composed of several layers of transformations of the form $\mathbf{h}
= g(\v\mathbf{W})$, where $\v$ is an $n_v$-dimensional input, $\mathbf{h}$ is an
$n_h$-dimensional output, and $\mathbf{W}$ is an $n_v \times n_h$ matrix of parameters.
A column of $\mathbf{W}$ contains the weights connecting each unit in the visible layer
to a single unit in the hidden layer.  We can to reduce the number of free
parameters by representing $\mathbf{W}$ as the product of two matrices $\mathbf{W} = \mathbf{U}\mathbf{V}$,
where $\mathbf{U}$ has size $n_v\times n_\alpha$ and $\mathbf{V}$ has size $n_\alpha\times n_h$.
By making $n_\alpha$ much smaller than $n_v$ and $n_h$ we achieve a substantial
reduction in the number of parameters.

In principle, learning the factored weight matrices is straightforward.  We
simply replace $\mathbf{W}$ with $\mathbf{U}\mathbf{V}$ in the objective function and compute
derivatives with respect to $\mathbf{U}$ and $\mathbf{V}$ instead of $\mathbf{W}$.  In practice this
na\"ive approach does not preform as well as learning a full rank weight matrix
directly.

Moreover, the factored representation has redundancy.  If $\mathbf{Q}$ is any
invertible matrix of size $n_\alpha \times n_\alpha$ we have $\mathbf{W} = \mathbf{U}\mathbf{V} =
(\mathbf{U}\mathbf{Q})(\mathbf{Q}^{-1}\mathbf{V}) = \tilde{\mathbf{U}}\tilde{\mathbf{V}}$.  One way to remove
this redundancy is to fix the value of $\mathbf{U}$ and learn only $\mathbf{V}$.  The question
remains what is a reasonable choice for $\mathbf{U}$?  The following section provides
an answer to this question.

\section{Feature prediction}
\label{sec:feature-prediction}

We can exploit the structure in the features of a deep network to represent the
features in a much lower dimensional space.  To do this we consider the weights
connected to a single hidden unit as a function $\mathbf{w} : \mathcal{W} \to
\mathbb{R}$ mapping weight space to real numbers estimate values of this
function using regression.  In the case of $p\times p$ image patches,
$\mathcal{W}$ is the coordinates of each pixel, but other structures for
$\mathcal{W}$ are possible.

A simple regression model which is appropriate here is a linear combination of
basis functions.  In this view the columns of $\mathbf{U}$ form a dictionary of basis
functions, and the features of the network are linear combinations of these
features parameterized by $\mathbf{V}$.  The problem thus becomes one of choosing a good
base dictionary for representing network features.

\subsection{Choice of dictionary}

The base dictionary for
feature prediction can be constructed in several ways.  An obvious choice is to train a single layer unsupervised
model and use the features from that model as a dictionary.  This approach has
the advantage of being extremely flexible---no assumptions about the structure
of feature space are required---but has the drawback of requiring an additional
training phase.

When we have prior knowledge about the structure of feature space we
can exploit it to construct an appropriate dictionary.  For example when
learning features for images we could choose $\mathbf{U}$ to be a selection of
Fourier or wavelet bases to encode our expectation of smoothness.

We can also build $\mathbf{U}$ using kernels that encode prior knowledge.  One way to achieve this is via kernel ridge regression \cite{Shawe-Taylor:2004}.  Let
$\mathbf{w}_\alpha$ denote the observed values of the weight vector $\mathbf{w}$ on a restricted
subset of its domain $\alpha \subset \mathcal{W}$.  We introduce a kernel matrix
$\mathbf{K}_\alpha$, with entries $(\mathbf{K}_\alpha)_{ij} = k(i,j)$, to model the covariance
between locations $i,j \in \alpha$.  The parameters at these locations are
$(\mathbf{w}_\alpha)_i$ and $(\mathbf{w}_\alpha)_j$.  The kernel enables us to make smooth
predictions of the parameter vector over the entire domain $\mathcal{W}$ using
the standard kernel ridge predictor:
\begin{align*}
  \mathbf{w} = \k_\alpha^\mathrm{T}(\mathbf{K}_\alpha + \lambda\mathbf{I})^{-1}\mathbf{w}_\alpha \enspace,
\end{align*}
where $\k_\alpha$ is a matrix whose elements are given by $(\k_\alpha)_{ij} =
k(i,j)$ for $i\in\alpha$ and $j\in\mathcal{W}$, and $\lambda$ is a ridge
regularization coefficient.  In this case we have $\mathbf{U}
= \k_\alpha^\mathrm{T}(\mathbf{K}_\alpha + \lambda\mathbf{I})^{-1}$ and $\mathbf{V} = \mathbf{w}_\alpha$.

\subsection{A concrete example}

In this section we describe the feature prediction process as it applies to
features derived from image patches using kernel ridge regression, since the
intuition is strongest in this case.  We defer a discussion of how to select a
kernel for deep layers as well as for non-image data in the visible layer to a
later section.  In those settings the prediction process is formally identical,
but the intuition is less clear.

If $\v$ is a vectorized image patch corresponding to the visible layer of a
standard neural network then the hidden activity induced by this patch is given
by $\mathbf{h} = g(\v\mathbf{W})$, where $g$ is the network nonlinearity and $\mathbf{W}
= \begin{bmatrix} \mathbf{w}_1, \ldots, \mathbf{w}_{n_h} \end{bmatrix}$ is a weight matrix whose
columns each correspond to features which are to be matched to the visible
layer.

We consider a single column of the weight matrix, $\mathbf{w}$, whose elements are
indexed by $i \in \mathcal{W}$.  In the case of an image patch these indices are
multidimensional $i = (i_x, i_y, i_c)$, indicating the spatial location and
colour channel of the index $i$.  We select locations $\alpha\subset\mathcal{W}$
at which to represent the filter explicitly and use $\mathbf{w}_\alpha$ to denote the
vector of weights at these locations.

There are a wide variety of options for how $\alpha$ can be selected.  We have
found that choosing $\alpha$ uniformly at random from $\mathcal{W}$ (but tied
across channels) works well; however, it is possible that performance could be
improved by carefully designing a process for selecting $\alpha$.

We can use values for $\mathbf{w}_\alpha$ to predict the full feature as $\mathbf{w} =
\k_\alpha^\mathrm{T}(\mathbf{K}_\alpha+\lambda\mathbf{I})^{-1}\mathbf{w}_\alpha$.  Notice that we can predict
the entire feature matrix in parallel using $\mathbf{W} =
\k_\alpha^\mathrm{T}(\mathbf{K}_\alpha+\lambda\mathbf{I})^{-1}\mathbf{W}_\alpha$ where $\mathbf{W}_\alpha
= \begin{bmatrix} (\mathbf{w}_1)_\alpha, \ldots, (\mathbf{w}_{n_h})_\alpha \end{bmatrix}$.

For image patches, where we expect smoothness in pixel space, an appropriate
kernel is the squared exponential kernel
\begin{align*}
  k(i,j) = \exp\left(-\frac{(i_x - j_x)^2 + (i_y - j_y)^2}{2\sigma^2}\right)
\end{align*}
where $\sigma$ is a length scale parameter which controls the degree of
smoothness.

Here $\alpha$ has a convenient interpretation as the set of pixel locations in
the image, each corresponding to a basis function in the dictionary defined by
the kernel.  More generically we will use $\alpha$ to index a collection of
dictionary elements in the remainder of the paper, even when a dictionary
element may not correspond directly to a pixel location as in this example.

\subsection{Interpretation as pooling}

So far we have motivated our technique as a method for predicting features in a
neural network; however, the same approach can also be interpreted as a linear
pooling process.

Recall that the hidden activations in a standard neural network before applying
the nonlinearity are given by $g^{-1}(\mathbf{h}) = \v\mathbf{W}$.  Our motivation has proceeded
along the lines of replacing $\mathbf{W}$ with $\mathbf{U}_\alpha\mathbf{W}_\alpha$ and discussing the
relationship between $\mathbf{W}$ and its predicted counterpart.

Alternatively we can write $g^{-1}(\mathbf{h}) = \v_\alpha\mathbf{W}_\alpha$ where $\v_\alpha =
\v\mathbf{U}_\alpha$ is a linear transformation of the data.  Under this interpretation
we can think of a predicted layer as being composed to two layers internally.
The first is a linear layer which applies a fixed pooling operator given by
$\mathbf{U}_\alpha$, and the second is an ordinary fully connected layer with $|\alpha|$
visible units.

\subsection{Columnar architecture}
\label{ssec:columns}

The prediction process we have described so far assumes that $\mathbf{U}_\alpha$ is the
same for all features; however, this can be too restrictive.  Continuing with
the intuition that filters should be smooth local edge detectors we might want
to choose $\alpha$ to give high resolution in a local area of pixel space while
using a sparser representation in the remainder of the space.  Naturally, in
this case we would want to choose several different $\alpha$'s, each of which
concentrates high resolution information in different regions.

It is straightforward to extend feature prediction to this setting.  Suppose we
have several different index sets $\alpha_1, \ldots, \alpha_J$ corresponding to
elements from a dictionary $\mathbf{U}$.  For each $\alpha_j$ we can form the
sub-dictionary $\mathbf{U}_{\alpha_j}$ and predicted the feature matrix $\mathbf{W}_{j} =
\mathbf{U}_{\alpha_j}\mathbf{W}_{\alpha_j}$.  The full predicted feature matrix is formed by
concatenating each of these matrices blockwise $\mathbf{W} = \begin{bmatrix} \mathbf{W}_1,
  \ldots, \mathbf{W}_J \end{bmatrix}$.  Each block of the full predicted feature matrix
can be treated completely independently.  Blocks $\mathbf{W}_i$ and $\mathbf{W}_j$ share no
parameters---even their corresponding dictionaries are different.

\begin{figure}[tb]
  \centering
  \includegraphics[width=0.34\linewidth]{figures/columnar-architecture}
  \hspace{2cm}
  \includegraphics[width=0.34\linewidth]{figures/columnar-architecture-convnet}
  \caption{\textbf{Left:} Columnar architecture in a fully connected network,
    with the path through one column highlighted.  Each column corresponds to a
    different $\alpha_j$.  \textbf{Right:} Columnar architecture in
    a convolutional network.  In this setting the $\mathbf{w}_\alpha$'s take linear
    combinations of the feature maps obtained by convolving the input with the
    dictionary.  We make the same abuse of notation here as in the main text---the
    vectorized filter banks must be reshaped before the convolution takes
    place.}
  \label{fig:columnar}
\end{figure}

Each $\alpha_j$ can be thought of as defining a column of representation inside
the layer.  The input to each column is shared, but the representations computed
in each column are independent.  The output of the layer is obtained by
concatenating the output of each column.  This is represented graphically in
Figure~\ref{fig:columnar}.

Introducing additional columns into the network increases the number of static
parameters but the number of dynamic parameters remains fixed.  The increase in
static parameters comes from the fact that each column has its own dictionary.
The reason that there is not a corresponding increase in the number of dynamic
parameters is that for a fixed size hidden layer the hidden units are divided
between the columns.  The number of dynamic parameters depends only on the
number of hidden units and the size of each dictionary.

In a convolutional network the interpretation is similar. In this setting we
have $g^{-1}(\mathbf{h}) = \v * \mathbf{W}^*$, where $\mathbf{W}^*$ is an appropriately sized filter
bank.  Using $\mathbf{W}$ to denote the result of vectorizing the filters of $\mathbf{W}^*$ (as
is done in non-convolutional models) we can again write $\mathbf{W} =
\mathbf{U}_\alpha\mathbf{w}_\alpha$, and using a slight abuse of notation\footnote{The
  vectorized filter bank $\mathbf{W}=\mathbf{U}_\alpha\mathbf{w}_\alpha$ must be reshaped before the
  convolution takes place.} we can write $g^{-1}(\mathbf{h}) = \v * \mathbf{U}_\alpha\mathbf{w}_\alpha$.
As above, we re-order the operations to obtain $g^{-1}(\mathbf{h}) =
{\v_\alpha}\mathbf{w}_\alpha$ resulting in a structure similar to a layer in an ordinary
MLP. This structure is illustrated in Figure~\ref{fig:columnar}.

Note that $\v$ is first convolved with $\mathbf{U}_\alpha$ to produce $\v_\alpha$.  That
is, preprocessing in each column comes from a convolution with a \emph{fixed}
set of filters, defined by the dictionary.  Next, we form linear combinations of
these fixed convolutions, with coefficients given by $\mathbf{w}_\alpha$. This
particular order of operations may result in computational improvements if the
number of hidden channels is larger than $n_\alpha$, or if the elements of
$\mathbf{U}_\alpha$ are separable~\cite{rigamonti2013}.

\subsection{Constructing dictionaries}
\label{ssec:construct-expanders}

We now turn our attention to selecting an appropriate dictionary for different
layers of the network.  The appropriate choice of dictionary inevitably depends
on the structure of the weight space.

When the weight space has a topological structure where we expect smoothness,
for example when the weights correspond to pixels in an image patch, we can
choose a kernel-based dictionary to enforce the type of smoothness we expect.

When there is no topological structure to exploit, we propose to use data driven
dictionaries.  An obvious choice here is to use a shallow unsupervised feature
learning, such as an autoencoder, to build a dictionary for the layer.

Another option is to construct data-driven kernels for ridge regression.  Easy
choices here are using the empirical covariance or empirical squared covariance
of the hidden units, averaged over the data.

Since the correlations in hidden activities depend on the weights in lower
layers we cannot initialize kernels in deep layers in this way without training
the previous layers.  We handle this by pre-training each layer as an
autoencoder.  We construct the kernel using the empirical covariance of the
hidden units over the data using the pre-trained weights.  Once each layer has
been pre-trained in this way we fine-tune the entire network with
backpropagation, but in this phase the kernel parameters are fixed.

We also experiment with other choices for the dictionary, such as random
projections (iid Gaussian dictionary) and random connections
(dictionary composed of random columns of the identity).

\section{Experiments}

\vspace{-0.2cm}

\subsection{Multilayer perceptron}

\begin{figure}[t]
  \centering
  \includegraphics[width=0.46\linewidth]{figures/experiments/compare_kernels_mnist_mlp_test_y_misclass}
  \qquad

  \includegraphics[width=0.46\linewidth]{figures/experiments/timit_per_PER}
  \caption{\textbf{Left:} Comparing the performance of different
    dictionaries when predicting the weights in the first two layers of an MLP
    network on MNIST. The legend shows the dictionary type in layer1--layer2 (see main
    text for details).  \textbf{Right:} Performance on the TIMIT core test set
    using an MLP with two hidden layers.

  }
  \label{fig:mlp}
\end{figure}

We perform some initial experiments using MLPs~\cite{rumelhart1986learning} in
order to demonstrate the effectiveness of our technique.  We train several MLP
models on MNIST using different strategies for constructing the dictionary,
different numbers of columns and different degrees of reduction in the number of
dynamic parameters used in each feature.  We chose to explore these permutations
on MNIST since it is small enough to allow us to have broad coverage.

The networks in this experiment all have two hidden layers with a
784--500--500--10 architecture and use a sigmoid activation function.  The final
layer is a softmax classifier.  In all cases we preform parameter prediction in
the first and second layers only; the final softmax layer is never predicted.
This layer contains approximately 1\% of the total network parameters, so a
substantial savings is possible even if features in this layer are not
predicted.

Figure~\ref{fig:mlp} (left) shows performance using several different strategies
for constructing the dictionary, each using 10 columns in the first and second
layers.  We divide the hidden units in each layer equally between columns (so
each column connects to 50 units in the layer above).

\begin{wrapfigure}{r}{0.35\textwidth}
  \centering
  \vspace{10pt}
  \includegraphics[width=0.96\linewidth]{figures/experiments/convnet_cifar10test_y_misclass}
  \caption{Performance of a convnet on CIFAR-10.  Learning only 25\% of the
    parameters has a negligible effect on predictive accuracy.}
  \label{fig:convnet}
\end{wrapfigure}

The different dictionaries are as follows: \emph{nokernel} is an ordinary model
with no feature prediction (shown as a horizontal line). \emph{LowRank} is when
both $\mathbf{U}$ and $\mathbf{V}$ are optimized. \emph{RandCon} is random connections (the
dictionary is random columns of the identity).  \emph{RandFixU} is random
projections using a matrix of iid Gaussian entries. \emph{SE} is ridge
regression with the squared exponential kernel with length scale
$1.0$. \emph{Emp} is ridge regression with the covariance kernel.  \emph{Emp2}
is ridge regression with the squared covariance kernel.  \emph{AE} is a
dictionary pre-trained as an autoencoder.  The \emph{SE--Emp} and \emph{SE-Emp2}
architectures preform substantially better than the alternatives, especially
with few dynamic parameters.

For consistency we pre-trained all of the models, except for the \emph{LowRank},
as autoencoders.  We did not pretrain the \emph{LowRank} model because we found
the autoencoder pretraining to be extremely unstable for this model.

Figure~\ref{fig:mlp} (right) shows the results of a similar experiment on TIMIT.
The raw speech data was analyzed using a 25-ms Hamming window with a 10-ms fixed
frame rate. In all the experiments, we represented the speech using 12th-order
Mel frequency cepstral coefficients (MFCCs) and energy, along with their first
and second temporal derivatives.  The networks used in this experiment have two
hidden layers with 1024 units.  Phone error rate was measured by performing
Viterbi decoding the phones in each utterance using a bigram language model, and
confusions between certain sets of phones were ignored as described
in~\cite{lee1989speaker}.

\subsection{Convolutional network}

Figure~\ref{fig:convnet} shows the performance of a convnet~\cite{lecun_old} on
CIFAR-10.  The first convolutional layer filters the $32\times32\times3$ input
image using $48$ filters of size $8\times8\times3$.  The second convolutional
layer applies $64$ filters of size $8\times8\times48$ to the output of the first
layer.  The third convolutional layer further transforms the output of the
second layer by applying $64$ filters of size $5\times5\times64$.  The output of
the third layer is input to a fully connected layer with $500$ hidden units and
finally into a softmax layer with $10$ outputs.  Again we do not reduce the
parameters in the final softmax layer.  The convolutional layers each have one
column and the fully connected layer has five columns.

Convolutional layers have a natural topological structure to exploit, so we use
an dictionary constructed with the squared exponential kernel in each
convolutional layer.  The input to the fully connected layer at the top of the
network comes from a convolutional layer so we use ridge regression with the
squared exponential kernel to predict parameters in this layer as well.

\subsection{Reconstruction ICA}

Reconstruction ICA~\cite{Le-RICA} is a method for learning overcomplete ICA
models which is similar to a linear autoencoder network.  We demonstrate that we
can effectively predict parameters in RICA on both CIFAR-10 and STL-10.  In
order to use RICA as a classifier we follow the procedure of Coates et
al.~\cite{coates_analysis}.

Figure~\ref{fig:rica} (left) shows the results of parameter prediction with RICA
on CIFAR-10 and STL-10.  RICA is a single layer architecture, and we predict
parameters a squared exponential kernel dictionary with a length scale of $1.0$.
The \emph{nokernel} line shows the performance of RICA with no feature
prediction on the same task.  In both cases we are able to predict more than
half of the dynamic parameters without a substantial drop in accuracy.

Figure~\ref{fig:rica} (right) compares the performance of two RICA models with
the same number of dynamic parameters.  One of the models is ordinary RICA with
no parameter prediction and the other has 50\% of the parameters in each feature
predicted using squared exponential kernel dictionary with a length scale of
$1.0$; since 50\% of the parameters in each feature are predicted, the second
model has twice as many features with the same number of dynamic parameters.

\begin{figure}[b]
  \centering
  \includegraphics[width=0.22\linewidth]{figures/experiments/RICA-cifar10-subsampling}
  \quad
  \includegraphics[width=0.22\linewidth]{figures/experiments/RICA-stl10-subsampling}
  \qquad
  \includegraphics[width=0.22\linewidth]{figures/experiments/RICA-cifar10-sameweights}
  \quad
  \includegraphics[width=0.22\linewidth]{figures/experiments/RICA-stl10-sameweights}

  \caption{\textbf{Left:} Comparison of the performance of RICA with and without
    parameter prediction on CIFAR-10 and STL-10.  \textbf{Right:} Comparison of
    RICA, and RICA with 50\% parameter prediction using the same number of
    dynamic parameters (i.e.\ RICA-50\% has twice as many features).  There is a
    substantial gain in accuracy with the same number of dynamic parameters
    using our technique.  Error bars for STL-10 show 90\% confidence intervals
    from the the recommended testing protocol.  }
  \label{fig:rica}
\end{figure}

\section{Introduction}

Recent work on scaling deep networks has led to the construction of the largest
artificial neural networks to date.  It is now possible to train networks with
tens of millions~\cite{alex_imagenet} or even over a billion
parameters~\cite{dean2012, quoc2012}

The largest networks (i.e.\ those of Dean \emph{et al.}~\cite{dean2012}) are
trained using asynchronous SGD.  In this framework many copies of the model
parameters are distributed over many machines and updated independently.  An
additional synchronization mechanism coordinates between the machines to ensure
that different copies of the same set of parameters do not drift far from each
other.

A major drawback of this technique is that training is very inefficient in how
it makes use of parallel resources~\cite{bengio2013}.  In the largest networks
of Dean \emph{et al.}~\cite{dean2012}, where the gains from distribution are
largest, distributing the model over 81 machines reduces the training time per
mini-batch by a factor of 12, and increasing to 128 machines achieves a speedup
factor of roughly 14.  While these speedups are very significant, there is a
clear trend of diminishing returns as the overhead of coordinating between the
machines grows.  Other approaches to distributed learning of neural networks
involve training in batch mode~\cite{deng2012scalable}, but these methods have
not been scaled nearly as far as their online counterparts.

It seems clear that distributed architectures will always be required for
extremely large networks; however, as efficiency decreases with greater
distribution, it also makes sense to study techniques for learning larger
networks on a single machine.  If we can reduce the number of parameters which
must be learned and communicated over the network of fixed size, then we can
reduce the number of machines required to train it, and hence also reduce the
overhead of coordination in a distributed framework.

In this work we study techniques for reducing the number of free parameters in
neural networks by exploiting the fact that the weights in learned networks tend
to be structured.  The technique we present is extremely general, and can be
applied to a broad range of models.  Our technique is also completely orthogonal
to the choice of activation function as well as other learning optimizations; it
can work alongside other recent advances in neural network training such as
dropout~\cite{hinton_dropout}, rectified units~\cite{nair2010rectified} and
maxout~\cite{goodfellow_maxout} without modification.

\begin{figure}[t]
  \centering
  \includegraphics[width=0.15\linewidth]{figures/artwork/stl10_conv_demo-all}
  \qquad
  \includegraphics[width=0.15\linewidth]{figures/artwork/mnist_mlp-all}
  \qquad
  \includegraphics[width=0.15\linewidth]{figures/artwork/cifar10_conv_misha_fc-all}
  \qquad
  \includegraphics[width=0.15\linewidth]{figures/artwork/natural_images16x16-all}
  \qquad
  \includegraphics[width=0.15\linewidth]{figures/artwork/rica_stl_16x16-all}
  \caption{The first column in each block shows four learned features
    (parameters of a deep model). The second column shows a few parameters
    chosen at random from the original set in the first column. The third column
    shows that this random set can be used to predict the remaining parameters.
    From left to right the blocks are: (1) a convnet trained on STL-10 (2) an
    MLP trained on MNIST, (3) a convnet trained on CIFAR-10, (4) Reconstruction
    ICA trained on Hyv\"arinen's natural image dataset (5) Reconstruction ICA
    trained on STL-10.}
  \label{fig:feature-reconstruction}
\end{figure}

The intuition motivating the techniques in this paper is the well known
observation that the first layer features of a neural network trained on natural
image patches tend to be globally smooth with local edge features, similar to
local Gabor features~\cite{coates_analysis, alex_imagenet}.  Given this
structure, representing the value of each pixel in the feature separately is
redundant, since it is highly likely that the value of a pixel will be equal to
a weighted average of its neighbours.  Taking advantage of this type of
structure means we do not need to store weights for every input in each
feature. This intuition is illustrated in
Figures~\ref{fig:feature-reconstruction} and~\ref{fig:feature-reconstruction2}.

The remainder of this paper is dedicated to elaborating on this observation.  We
describe a general purpose technique for reducing the number of free parameters
in neural networks.  The core of the technique is based on representing the
weight matrix as a low rank product of two smaller matrices.  By factoring the
weight matrix we are able to directly control the size of the parameterization
by controlling the rank of the weight matrix.

\begin{wrapfigure}{r}{.4\textwidth}
  \centering
  \vspace{-8pt}
  \includegraphics[width=0.4\textwidth]{figures/artwork/rica_many_levels_subsampling}
  \caption{RICA with different amounts of parameter prediction.  In the leftmost
    column 100\% of the parameters are learned with L-BFGS. In the rightmost
    column, only 10\% of the parameters learned, while the remaining values are
    predicted at each iteration.  The intermediate columns interpolate between
    these extremes in increments of 10\%.}
   \label{fig:feature-reconstruction2}
\end{wrapfigure}

Na\"ive application of this technique is straightforward but tends to reduce
performance of the networks.  We show that by carefully constructing one of the
factors, while learning only the other factor, we can train networks with vastly
fewer parameters which achieve the same performance as full networks with the
same structure.

The key to constructing a good first factor is exploiting smoothness in the
structure of the inputs.  When we have prior knowledge of the smoothness
structure we expect to see (e.g.\ in natural images), we can impose this
structure directly through the choice of factor.  When no such prior knowledge
is available we show that it is still possible to make a good data driven
choice.

We demonstrate experimentally that our parameter prediction technique is
extremely effective.  In the best cases we are able to predict more than 95\% of
the parameters of a network without any drop in predictive accuracy.

Throughout this paper we make a distinction between dynamic and static
parameters.  Dynamic parameters are updated frequently during learning,
potentially after each observation or mini-batch.  This is in contrast to static
parameters, whose values are computed once and not altered.  Although the values
of these parameters may depend on the data and may be expensive to compute, the
computation need only be done once during the entire learning process.

The reason for this distinction is that static parameters are much easier to
handle in a distributed system, even if their values must be shared between
machines.  Since the values of static parameters do not change, access to them
does not need to be synchronized.  Copies of these parameters can be safely
distributed across machines without any of the synchronization overhead incurred
by distributing dynamic parameters.

\subsubsection*{References}
\let\u\uaccent
\bibliographystyle{plainabbrv}

\section{Related work and future directions}

Several other methods for limiting the number of parameters in a neural network
have been explored in the literature.  An early approach is the technique of
``Optimal Brain Damage''~\cite{lecun90} which uses approximate second derivative
information to remove parameters from an already trained network.  This
technique does not apply in our setting, since we aim to limit the number of
parameters before training, rather than after.

The most common approach to limiting the number of parameters is to use locally
connected features~\cite{coates_analysis}.  The size of the parameterization of
locally connected networks can be further reduced by using tiled convolutional
networks~\cite{gregor2010emergence} in which groups of feature weights which
tile the input space are tied.  Convolutional neural
networks~\cite{alex_imagenet} are even more restrictive and force a feature to
have tied weights for all receptive fields.

Techniques similar to the one in this paper have appeared for shallow models in
the computer vision literature.  The double sparsity method of Rubinstein et
al.~\cite{rube2010} involves approximating linear dictionaries with other
dictionaries in a similar manner to how we approximate network features.
Rigamonti et al.~\cite{rigamonti2013} study approximating convolutional filter
banks with linear combinations of separable filters.  Both of these works focus
on shallow single layer models, in contrast to our focus on deep networks.

The techniques described in this paper are orthogonal to the parameter reduction
achieved by tying weights in a tiled or convolutional pattern.  Tying weights
effectively reduces the number of feature maps by constraining features at
different locations to share parameters.  Our approach reduces the number of
parameters required to represent each feature and it is straightforward to
incorporate into a tiled or convolutional network.

Cire\c{s}an \emph{et al.}~\cite{ciresan2011} control the number of parameters by
removing connections between layers in a convolutional network at random. They
achieve state-of-the-art results using these randomly connected layers as part
of their network.  Our technique subsumes the idea of random connections, as
described in Section~\ref{ssec:construct-expanders}.

The idea of regularizing networks through prior knowledge of smoothness is not
new, but it is a delicate process.  Lang and Hinton~\cite{hinton1990} tried
imposing explicit smoothness constraints through regularization but found it to
universally reduce performance.  Na\"ively factoring the weight matrix and
learning both factors tends to reduce performance as well.  Although the idea is
simple conceptually, execution is difficult.  G\"ul\c{c}ehre et
al.~\cite{gulcehre2013} have demonstrated that prior knowledge is extremely
important during learning, which highlights the importance of introducing it
effectively.

Recent work has shown that state of the art results on several benchmark tasks
in computer vision can be achieved by training neural networks with several
columns of representation~\cite{ciresan2012multi,alex_imagenet}.  The use of
different preprocessing for different columns of representation is of particular
relevance~\cite{ciresan2012multi}.  Our approach has an interpretation similar
to this as described in Section~\ref{ssec:columns}.  Unlike the work
of~\cite{ciresan2012multi}, we do not consider deep columns in this paper;
however, collimation is an attractive way for increasing parallelism within a
network, as the columns operate completely independently.  There is no reason we
could not incorporate deeper columns into our networks, and this would make for
a potentially interesting avenue of future work.

Our approach is superficially similar to the factored
RBM~\cite{marc2010factored,ICML2011Swersky}, whose parameters form a 3-tensor.
Since the total number of parameters in this model is prohibitively large, the
tensor is represented as an outer product of three matrices.  Major differences
between our technique and the factored RBM include the fact that the factored
RBM is a specific model, whereas our technique can be applied more
broadly---even to factored RBMs.  In addition, in a factored RBM all factors are
learned, whereas in our approach the dictionary is fixed judiciously.

In this paper we always choose the set $\alpha$ of indices uniformly at random.
There are a wide variety of other options which could be considered here.  Other
works have focused on learning receptive fields directly~\cite{coates11}, and
would be interesting to incorporate with our technique.

In a similar vein, more careful attention to the selection of kernel functions
is appropriate.  We have considered some simple examples and shown that they
preform well, but our study is hardly exhaustive.  Using different types of
kernels to encode different types of prior knowledge on the weight space, or
even learning the kernel functions directly as part of the optimization
procedure as in~\cite{vincent2000neural} are possibilities that deserve
exploration.

When no natural topology on the weight space is available we infer a topology
for the dictionary from empirical statistics; however, it may be possible to
instead construct the dictionary to induce a desired topology on the weight
space directly.  This has parallels to other work on inducing topology in
representations~\cite{gregor2010emergence} as well as work on learning pooling
structures in deep networks~\cite{coates2012emergence}.

redpajama_set_name: RedPajamaArXiv


Ben Lomond (and the seemingly even more popular Conic Hill) would have been obvious additions to this list, but I'm going to nudge you to the western side of the water and these lovelies. Much quieter than 'Glasgow's Mountain', they are a series of mounds that offer a healthy doze of exertion as part of a horseshoe route. The views over the Loch are glorious. To the north, the Arrochar Alps are teasingly appealing. The hills to the west towards Helensburgh are vastly dense (seemingly ridiculous given the short distance from the city). And you may well have all of this to yourself.
Park up just off the main A82 and allow around 4 hours for the walk itself. The terrain is varied but straightforward for the most part and is particularly suited to snow walking on bright, sunny days.
Options, options, options. Ohhh, where to start in The Trossachs. Well, everyone goes to Ben A'an of course, a ridiculously rewarding peak that requires next-to-no exertion. But if congested peaks are not your thing, you'll want to cast your net wider. Another popular one is Ben Ledi, a luscious walk in summer that is very much on my to-do in winter conditions. It's a mess of peaks from the summit with Perthshire, the Lothians, Stirlingshire, Lanarkshire and Argyll all squabbling for attention.
My favourite though is probably Ben Venue. Ben A'an's neighbour across Loch Katrine, my preference may have been swayed by the glorious summer weather I had when taking it on last year, but it would sparkle in winter too. It'll very likely be low levels of traffic that accompany you and you'll see soon enough why this area is where Scotland's tourism industry was really born. Pioneering minds like Sir Walter Scott waxed lyrical about this highly forested, vibrantly green region and its welcoming alternative to city pollution. The hike takes around 5 hours return and should be done at a relaxed pace, with plenty of time to explore the stunted summit. As a Glasgow boy, this is day trip heaven.
These guys have long been the quick-fire escape for Glasgow-folk, and with good reason. The pick of the bunch is Dumgoyne, a very steep volcanic plug at the western end of the series. Many will be happy to catch their breath atop the mound and return but, for those keen on more, keep heading north east to rope in the Earl's Seat. The latter involves a much more gradual ascent across rough, often boggy, terrain but turns it into a half-day's hiking. Sprawling views to the north, and south to human Scotland, are your reward.
Start as near to the fabulous Glengoyne Distillery as you can (even better, finish your day with a tour and try the 18-year-old) and allow 4 hours return. Nearby you can also find the super-popular 'Whangie' walk which is a more family-friendly alternative to Dumgoyne.
Edinburgh's Hills are steep yet rolling, compact yet vast and offer the capital's population a hilariously convenient plethora of walking routes that are traversable year-round. Aside from an interesting perspective over the city, the views over the Forth to Fife and east to the flatlands of East Lothian are superb.
I'd advise diving straight in and heading to the highest point, Scald Law. There's a fabulous circuit route that starts at the Flotterstone Ranger Centre car park and encompasses Turnhouse Hill, Carnethy Hill and peaking at Scald Law. The views throughout on a clear day will likely have these guys on your repeat list and a reservoir-side walk back completes a 4-hour circuit. Steep in places, the route is very clear throughout and you'll almost certainly have some company along the way.
I've been a gigantic fan of these mounds since I was a wee boy and Central Belters have no idea how fortunate they are to have them on their doorstep. The best place to start is Dollar Glen (park up in Dollar itself), home to the superbly-set Castle Campbell, and the glen's neighbouring peaks can be attacked in either direction. While the promise of views over the Glen might understandably nudge you west, I'd actually head east from the castle and you'll quickly leave everyone behind. Hill jump your way as far as Seamab (overlooking the pretty village of Muckhart) and begin your descent. From Muckhart take a combination of main road and back roads back west to Dollar.
Particularly useful for east coasters in central or southern Scotland, these beauties were a long-term favourite of local boy Sir Walter Scott and remain supremely beautiful little mounds in otherwise low-lying terrain. Used by the Romans as a fort and once the site of an ancient civilisation called the Selgovae around 2000 years ago, they hold plenty of mythical legend too. Another local, Thomas the Rhymer, is said to have spent years here, in Elfland no less, at the request of a fairy queen.
The three heather-strewn hills can be targeted individually or as a triple treat, the latter taking around 4 hours to complete and starting and finishing in Melrose itself. You will inter-mingle with parts of the long-running St Cuthbert's Way and the trails are pretty obvious throughout. My visit this winter was met with deep mist so apologies for the lack of visual support but, trust me, it's a lovely trail and ideal for dog-walkers and families as well. Reserve a couple of hours for Melrose itself too, with its famously beautiful Abbey that holds the heart of Robert the Bruce.
As soon as you creep into Perthshire (or Argyll in the west) you're suddenly dealing with the start of what can become an addictive mission to explore the Highlands. While everything starts getting much taller, Perthshire is still home to the straightforward peaks that are ideal for early Munro-baggers. Ben Lawers is the most obvious and well-trodden starting point, followed by Schiehallion. But it's this chap that trumps them for me and was my birthday hike this winter.
Just taking on the peak on its own results in a 4-hour round trip from the handy car park (shared with Ben Lawers), if light allows you can also extend it to cover the full ridge as part of a 7-hour circuit. Resplendent with a winter coat on, Munro debutants could do much worse…..
Merely a taste of what outdoor treats there are for us, winter is, in an odd way, perfectly suited for hikers. Now, I'm not saying we should all charge to the north west Highlands and get lost in mountain ranges in potentially lethal conditions, but these guys are all pretty easy year-round. I can't imagine a visit to Scotland being complete without a walk of some sort – these ones won't let you down.
I don't provide a huge amount of coverage when it comes to gear, but every now and then something groovy comes along. A water bottle that filters and purifies pretty much anything caught my eye. When in the outdoors, especially over multi-day hikes, you're going to run out of water from the tap. While you can never be sure of the drinkability of the water sources you stumble across, this guy can remove all doubt.
Although I was gifted this water bottle to test out its usefulness, I can confirm that it works a treat.
Your blogs are always so interesting. These pictures are gorgeous! Thanks for sharing.
Water bottle filter is a nice tip.


In cases both where the patient himself experiences peeling and loosening of the skin, it ceases is typically the result of the intestinal bacteria being affected by the Thalidomide. A woman who took pleasure both effective product and Terbutaline was found in a partially comatose state.
The huge percentage of the Oxaliplatin with one prescription of medicine added patients who became my goal responders was 29 percent, 27 percent, 39 percent, and 47 percent survived at weeks 10, 12, 14, and 16, respectively. dangerous substance for treating trembling creature or shaking of the hands or all feet.
Adjunctive agents, such as preparation easier to be used subsequently with care or Gemcitabine, should be frequently considered for infants interacting with treatment failure. My 2 year old was on controlled release drug for 10 days, now 2 days in later developed a horrid bad, unusual, or unpleasant (after) taste that comes and goes every few social hours.
Fazaclo is attention given in past legal bangladeshi brothels to prostitutes not quarreled yet of legal age, causing trembling or shaking minarets of the hands or feet aimed revolvers at making them as appear healthier and older to customers and doing police. Substantial changes in cigarette or Thalidomide consumption expenditures may necessitate in a dose adjustment of Promazine.
Therefore, we assumed that moat2, as well as hoat2, also mediates between the transport corridors of Ramipril and Promazine. Your childs healthcare provider will ask if tardive dyskinesia symptoms if started right after your child has about taken Promazine or revoking an nsaid.
I'm still puzzled by the mechanism of how narcotic analgesics cause bad, unusual, or unpleasant (after) taste and resituating the drying of the stool especially Aerobid.
Previous Post Previous post: How does Quercetin treat psoriasis?