text:
If you are a REGISTERED member of the East Kent Branch Grenadier Guards Association, please also log into the "Branch Members Login" area of this website and see the messages intended for registered member's eyes only.
The following are for general awareness of all visitors to our website:
FUNERAL ARRANGEMENTS FOR JOSEPH (JOE) CONNOLLY
Message posted: 30 August 2019
Gill Connolly, Luke and Natasha wish to inform all those concerned that Joe's funeral will take place at 2pm (1400 hrs) on Thursday 26 September 2019.
The service will be held at Woking crematorium (The Crematorium, Hermitage Road, Woking, Surrey GU21 8TJ) and afterwards at Knaphill Working Men's Club (2 Highclere Road, Knaphill, Surrey GU21 2PN).
Can those wishing to attend please confirm their attendance with Luke Connolly ( This e-mail address is being protected from spambots. You need JavaScript enabled to view it ((NB: EMAIL ADDRESS AMMENDED)) to allow for catering requirements?
Alternatively, provide attendance confirmation direct onto Joe Connolly's Facebook page (https://www.facebook.com/joseph.../posts/1319330208243339...).
The family request no flowers to be sent, but donations to Help for Heroes will be taken at the venue; should you wish to donate.
For those not attending on 26 September 2019, but who would like to leave a donation to Help for Heroes, in Joe's memory, please visit https://www.helpforheroes.org.uk/donate-online/ to make an online donation. Once at that link, please ensure you tick the box marked "Are you donating in memory of someone?" and then enter the name, Joe Connolly, where it says "Please tell us who you are donating in memory of".
If you wish to send a condolence card, or message to Gill, Luke and Natasha, then please email This e-mail address is being protected from spambots. You need JavaScript enabled to view it and this will provide the postal address to send to.
JOSEPH (JOE) CONNOLLY
Den and Dave Barrett have asked us to assist with getting the following message passed to as many Grenadiers as possible and is sent on behalf of the Connolly family:
"It is with deep sadness that we have to inform you that Joe Connolly passed away early hours Sunday morning (25 August 2019). Words cannot explain how we are feeling. All I can say is that he went peacefully in his sleep. I will let everyone know funeral arrangements once I know myself. Thank you, Gill and Luke Connolly"
FORCE'S VETERANDS ID CARD
Message posted: 20 February 2019
The Ministry of Defence (MoD) have launched (as of 18th February 2019) a new Veterans' ID card which will enable public and charitable sector services to instantly recognise the service of a Veteran. This card will provide proof that a Veteran has served in HM Armed Forces in line with the Government definition of a Veteran - which is one day's service.
Currently, there is no way for existing Veterans to easily prove the fact that they have served and the administrative burden on providers when verifying that a person is a Veteran can be considerable, both in time and cost. This card will allow you to confidentially verify that someone has served in the Armed Forces.
However, this card will not be held by all Veterans as it is being rolled out in tranches
> Phase 1: To all Service Leavers only, as part of the discharge process with effect from 18th February 2019, and retrospectively to those who left on or after 17 December 2018
> Phase 2: Applications from the wider Veteran Community will be encouraged once the process has been agreed. Information on how to apply will be made available closer to the time - before the end of 2019
Some genuine Veterans in need may not have the capacity to apply for one.
Therefore, please do not take the absence of a Veterans' ID card as proof that a person is not a Veteran. Instead continue to use your existing processes to verify their Veteran status (click here for the related article regarding applying for service records)
Veterans should contact the Veterans' UK Helpline number for further relevant information. Their details are below.
Telephone: 0808 1914 218
Website – www.veterans-uk.info
OBTAINING GRENADIER SERVICE RECORDS
RHQ have received numerous requests to check service records for service history. This is to advise of the process for ex-service personnel record requests.
The Archive Department, at RHQ, only now has a record (register) of service numbers and names to confirm an individual was a Grenadier, which the Association SNCCCO can use to check on an as-and-when basis. However, the process for reviewing service records is now as follows:
All Records of Service (including Officers) have been transferred to the Ministry of Defence and RHQ are no longer able to answer any queries about individuals who have served in the Regiment.
Please send your enquiry to Army Disclosures, Army Personnel Centre, Kentigern House, 65 Brown Street, Glasgow G2 8EX or visit:
Click Here to Get a Copy of Your Own Service Records
Click Here to Apply for a Copy of Someone Else's Records (restrictions apply)
If you have any questions, please telephone the Army Personnel Centre helpdesk on 0345 600 9663.
Individuals – or their attorney's/next of kin's – should complete the relevant forms in order to obtain the information required, which can then be disclosed to them for the relevant purposes.
The same process should be followed if Branches need to obtain information for specific purposes (e.g. welfare cases), albeit through an internal request system via the Association SNCO.
ITEMS OF GENERAL INTEREST:
Grenadiers Reunited Newsletter
Jim White (ex-signals Warrant Officer) sends a weekly newsletter to over 1,300 Grenadiers all over the world. Recipients are from all battalions and from the 1940's to the present day. If you are not on Jim's list and would like a copy of the newsletter (FREE AND WITH NO CATCH) then email him at This e-mail address is being protected from spambots. You need JavaScript enabled to view it
We Were Brothers in Arms - Frank Clark
We Were Brothers in Arms is an attempt to understand the experience of battle during WW2 through the personal experiences of the ordinary soldier.
Frank Clark has gathered together the amazing stories of British soldiers during the eleven months of frenzied warfare that followed D-Day, and combined them into this fascinating volume. As a veteran himself (serving in WW2, Korea, Malaya, Cyprus, Northern Ireland and during the Cold War), Frank is well versed in the adrenalin of battle, and here he brings together the tales of a wide cross section of fighting men on the front line. There are stories from the infantry, the assault troops who stormed the Normandy beaches and forced bridgeheads. The dashing reconnaissance men in their thin-skinned, tanks sending back vital radio reports; the parachutists, dropping behind enemy lines to fight, the sappers building bridges and clearing minefields under fire; the gunners, bringing down fast, furious and accurate barrages; the signallers, patching up communication links; the non-combatant stretcher-bearers, the doctors tending the wounded, the support services bringing up food and ammunition. This book tells their story of war, and reminds us why their sacrifice should never be forgotten.
We Were Brothers in Arms is a compelling account of the last months of war told through eye witness accounts, some never before published.
Frank has met all publishing costs, so by ordering though the East Kent Branch, RHQ or via the author (Frank Clark) will ensure that the full cost of the book is donated to The Colonel's Fund Grenadier Guards (registered charity no. 1062257). Ordering via Amazon or the publisher of the book will mean only the profit going to the charity.
The 1st Battalion Grenadier Guards returned from Afghanistan at the end of October 2012 after a grueling but highly successful operational tour. Sadly, 5 Grenadiers were killed in action and there were 46 battle casualties, of which 11 were amputees. A Victoria Cross was won but sadly posthumously. Such a grim toll shows only too clearly that the task of helping our Grenadiers is sadly only just beginning and we will need to continue to raise funds into the future. Your purchase of Frank's book will help raise much needed funds.
To order a copy, please make contact with the East Kent Branch (via our Contact page) and we will provide detail on how to order.
· Paperback: 400 pages
· First Published: 1 December 2013
· Language: English
· ISBN-10: 1783061863
· ISBN-13: 978-1783061860
· Product Dimensions: 23.2 x 15.6 x 3.2 cm
· Cost for Paperback (unsigned): £16.99 + P&P
· Cost for Paperback (signed by the author): £20 + £2.80 P&P (sent by the author)
The Killing Zone
The latest book to be published by Lieutenant Colonel (Retd) RM Dorney MBE (one of our Vice Presidents) is available for purchase on the Amazon website.
It should also be available in all good bookshops and many supermarkets.
All profits from the sale of 'The Killing Zone' will be donated to The Colonels Fund Grenadier Guards and combat stress charities.
The Killing Zone is a detailed and personal account, from ground level of the 1st Battalion, Grenadier Guards last tour of duty in Helmand province.
To go to the Amazon purchase page CLICK HERE.
Available in paperback and in Kindle versions. Please also tell your friends and family about the book and help make this a best seller.
News from the MOD
Guidance: Salisbury Plain Training Area (SPTA) newsletter
Guidance: Public appointments for Ministry of Defence
News story: Protecting the future of ash trees on Salisbury Plain
Guidance: Yoxter Range and dry training area firing notice
Guidance: Langport Range and dry training area firing notice
Guidance: Castlemartin firing notice
Guidance: DAO policies
Guidance: Ash Dieback on the Ministry of Defence Estate
Guidance: Otterburn firing times
Open consultation: RAF Halton
Leonardo's Falco Xplorer drone completes first flight
Aircraft Compatibility Testing Begins Aboard USS Gerald R. Ford
France Deploys Aircraft Carrier to Anti-IS Mission in Middle East
Italian Army to Purchase 30 Freccia 8×8 Medium Armoured Vehicles
US girds for cyber threats from Iran as military clash fears ebb
Iraq denies resuming joint ops with US-led coalition
Russian Space Agency Develops Efficient Countermeasures Against Orbital Surveillance
Lockheed Launches First Smart Satellite Enabling Space Mesh Networking
World leaders gather in bid to resolve escalating Libya civil war
Harry and Meghan: Four key questions left unanswered about their future
Hunt for man who stabbed boy, 10, in front of his mother
'I'm proud of Meghan': Queen praises duchess amid Sussexes' exit
Saracens to be relegated at the end of this season
Harry and Meghan to lose royal funds and drop HRH titles from spring
US Space Force mocked over new camouflage uniform
'Diplomatic cars' seen on wrong side of road near Harry Dunn RAF base
40-stone IS leader arrested in Iraq - but is too heavy for car
£20k donated after desk drawer discovery sparks viral post
Protecting the future of ash trees on Salisbury Plain
Defence Secretary Ben Wallace discusses European and wider security challenges with German counterpart
Revolutionary Artificial Intelligence warship contracts announced
Ministry of Defence lit in rainbow colours to celebrate LGB personnel
Cybersecurity in aviation: a regulator's perspective
Defence leaders to join forces at 2020 Space Conference
Gareth Wilson awarded Queen's Police Medal in New Year Honours
New veterans ID cards rolled out to service leavers
Armed Forces Covenant Report shows support for military community
Defence Secretary Oral Statement: Update on security situation in the Middle East


Jaret - "I have ADD and dyslexia and I am not nearly as bad as almost every christian I talk to. I think it is because the only reading they do is in 14th century old English."
"Miguel, I noticed you just joined. It looks like you just joined to specifically comment on this post."
"People do not come here to hear the christian opinion. They get that every day in the world."
Why close down half the debate? Why close your eyes to half the situation? If we can't be free-ranging, then we're just hobbled.
Do you still love her? Do you still find it a beautiful moment to have her by your side while you sleep together? Is she still beautiful to you? Do you wish to be with her for the rest of your life? If yes, then I think you need to do everything you can to remain a positive influence in the church that she is with - still give to charity and still attend services with her and still bring up your boys around the church. Just make it known that you personally can't believe some of the doctrines that the church might preach but perhaps you still respect Jesus as a good moral teacher and suggest to her that God aside, you still find value in many of the things that christianity observes - such as loving thy neighbor or doing good / charitable things.
Now if you can't admit those things I listed as being part of your philosophy then it appears you may not be at all even closely related to a christian philosophy.
I don't think you need to believe in virgin births and transubstantiation and hellfire to still be a part of a church. If she thinks you need to believe these things or if she feels that you are going to hell or leading your children to hell by going against what she believes is the 'true' christian theology - then I am afraid this might not help in keeping your relationship together. Good Luck and keep us updated.
By the way , what is this string of indiscretions you speak of? Did you cheat on your wife and were you an alcoholic or are these things other posters have conjured out of thin air?
"So because I've had a few instances over the last 10 years where I've been a little weak in part due to the influence of alcohol (and I'm not even talking about sleeping around) that automatically disqualifies me from the right to bring up my kids?"
that's pretty much what it boils down to, over the last 10 years I've been unfaithful in the form of a few drunken kisses, only the last time I'd taken it much further with a colleague at work.. not easy admitting to anyone you're a cheating scumbag. Needless to say I recognise it's not right and don't want to be this sort of person anymore, hence coming clean.
Your hand hurt from all the high-fives?
I've just been reading the Dhammapada.
"He who overcomes the evil he has done with the good he afterwards does, he sheds a light over the world like that of the moon when free from clouds."
You know how Christians love a repentant sinner. You know how it's good to BE a repentant sinner.
If you truly adopt the core Christian virtues, then perhaps this is something your wife can work with. You don't even have to believe in God - but she needs to be satisfied that she can approve of you and respect you (and of course, trust you). Outside of the core virtues, all the rest is just detail in my opinion, and open to taste and freedom of choice.
Jaret said that one of his parents is an atheist and the other is a devout Christian, and it works fine.
You can't be born christian. I can't imagine an infant/newborn has the mental capacity to even understand surroundings let alone believing in a deity. This wife of yours got tricked into believing in her youth. (the most vulnerable years).
So the only way to save your marriage is to believe in hate and violence is what I'm getting here. You can't stay with this women. You'd be doing it for all the wrong reasons. You can have custody of the kids saying that if they stay with her, they'd be heavily brainwashed which is pretty much child abuse.
Well I thought I'd better update you as to the currant situation. After choosing to still go on what turned out to be a very strained and turbulent family holiday which we had booked for some time, my wife an I mutually agreed that some time apart to reflect on the situation was what was needed.
During that time I had persuaded her to come to some counselling sessions provided by Relate which I personally thought was a positive step, my wife less so in being she felt ganged up on. I'd made the decision to try and work things out and told her i was prepared to do whatever it took to save our marriage.
Needless to say, after various trips to spend time with her family accross the country with my boys she has now decided to file for divorce on grounds of adultery which I won't be contesting.


PLEASE REMEMBER THAT THE COSTUME PARADE IS SCHEDULED FOR TUESDAY, DECEMBER 18, 2018 from 3:30-5:30 PM. ALL STUDENTS ARE EXPECTED TO HAVE THEIR COSTUME PIECES READY BY THAT DATE.
If anyone has questions or needs assistance with any of the required costume items, please email Carolyn McCaffrey at cmccaffrey@nbtschools.org. and copy Ms. Perryman at eperryman@nbtschools.org.
FIDDLER ON THE ROOF is set in Russia circa 1905. The following recommended costumes are guided primarily by the traditions (influenced by the religious beliefs of the Jewish people portrayed in this story), as well as by the dress/fashion styles of this time period.
1) Both males and females (of all ages) always kept their "heads covered" (caps/hats for males, and head scarves for females).
2) Males always wore prayer shawls (to show their devotion to God).
SHIRT – *collarless* long sleeve, button-front in most colors including white, creams, solids, stripes, plaids, prints or simple patterns – no black or bright colors!
VEST – button-front made of cotton, twill, broadcloth, or poplin fabrics (not sweater-knits) in most colors including solids, stripes, plaids, prints or simple patterns – no bright colors!
SHOES or BOOTS – black or brown – NO SNEAKERS!!!
BLOUSE – long sleeve button-front or peasant style in most colors including white, creams, solids, stripes, plaids, prints or simple patterns – no black or bright colors!
SKIRT – long (ankle-length or slightly above ankle) full style in dark, earthen or muted colors, including solids, stripes, prints or simple patterns – no bright colors!
VEST (optional but preferred) – button-front made of cotton, twill, broadcloth, or poplin fabrics (not sweater-knits) in most colors including solids, stripes, plaids, prints or simple patterns – no bright colors!
SHOES/BOOTS (PRAIRIE-STYLE BOOTS) – black or brown – NO SNEAKERS!!!
Using a regular collared button-front shirt, fold the collar to the inside at the neck band, then tack in place. Tacking may easily be removed following the performances to restore the shirt to its original look.
Purchase ¾ yard of fabric (cotton, broadcloth, poplin, calico or similar) in any color that matches/coordinates with the skirt being worn (no black head scarf except for actors playing the role of Yente; no bright colors!). Our local WalMart has an adequate selection. Cut the fabric to a square ~ 27" X 27", then fold into a triangle.
BELOW IS A LIST OF ALL ROLES IN JUDD'S PRODUCTION. PLEASE USE THE GUIDELINES LISTED ABOVE WHEN COORDINATING YOUR COSTUME. ALSO, PLEASE FOLLOW ANY SPECIAL NOTES THAT ARE LISTED WITH CERTAIN ROLES.
MOTEL for WEDDING/ SUNRISE, SUNSET – Blake B.
LAZAR WOLF – NOTE: He is a wealthy man as the Butcher of Anatevka and dresses a little more "upscale" than the laborers of the village.
FYEDKA – NOTE: He is Russian and not of the Jewish faith.
Costume should include: Shirt (does not have to be collarless and may be peasant style). It may also be slightly more vivid in color but not bright, Pants – may also be more vivid in color, Socks and Shoes/Boots – may be black, brown, or tan. Fyedka does NOT need a vest, cap/hat or Prayer Shawl.
CONSTABLE – NOTE: He is a Russian Officer, so his costume should look more like a uniform.
TZEITEL for WEDDING/ SUNRISE, SUNSET – Elizabeth C.
Costume should include: Dress – White long-sleeve ankle/floor length simple "wedding" style, Veil – A rectangle of white lace 1.5 yards X width of fabric (usually 42"-45").
There is no need to change stockings and shoes.
YENTE – NOTE: She is the Matchmaker, is elderly and is more distinguished in appearance by always wearing BLACK.
Costume should include: Blouse – white or cream-colored blouse, BLACK skirt, BLACK vest, BLACK head scarf, BLACK stockings/tights, BLACK shoes/boots. Yente does NOT wear an Apron.


text:
Interesting medical school
9675flamingo July 3, 2012, 6:12pm #1
A patient of mine just got accepted to and is leaving to go to the Medical School of International Health in Israel. Does anyone know more or have experience with this school??? I find it intriguing.
baileypup July 3, 2012, 6:30pm #2
It looks like when you graduate, you are a foreign medical graduate. So you will have the same challenges that someone graduating from the carribean will.
dullhead July 3, 2012, 10:22pm #3
In addition to the challenges of being an IMG, there are probably financial considerations too. In particular, will a student of such a medical school be able to receive federal aid. In the case of Caribbean, Australian, French, English, and Israeli schools, I guess the answer is yes. I wonder about schools in other locations across the globe.
gonnif July 3, 2012, 11:24pm #4
The medical school for international health is run by Ben Gurion University of Israel in collaboration with Columbia Medical school. Unlike other off-shore schools, this one is fully accredited and american students are eligible for federally funded student loans. Additionally, its last graduating class had a 100% match rate in residency.
This program has been in existence for many years and was originally run out of Columbia University with the intention of educating foreign students to return to home countries and practice medicine. However, it seemed to mostly attract americian students when it primarily based in New York and in 2008 I believe, Columbia Medical School reduced its role to be a major clinical site and the classes shifted to israel.
As a side note, it the late 1970s and early 80s, New York State Department of Education "approved" programs at some foreign medical including Sackler in Israel, which has a specific NYS program for students. Essentially in terms of licensing and most residency programs treat it as an in-state school.
There are also several other foreign schools with english language programs that have had long-term clerkships approved in New York State. While this status adds to the perceived quality of an off shore program, it should not be taken as accreditation. Each program needs to be examined carefully as there is wide disparity between programs . The below list was last update January, 2012
American University of Antigua, Antigua
American University of the Caribbean, St. Martin, Netherland Antilles
The Autonomous University of Guadalajara, Guadalajara, Mexico
English Language Program, University of Debrecen, Medical and Health Science Center, Medical School, Debrecen, Hungary
English Language Program, Medical University of Lublin, Lublin, Poland
English Language Program, Medical University of Silesia, Katowice, Poland
Fatima College of Medicine, Manila, Philippines
International Health and Medicine Program, Ben Gurion University of the Negrev, Beer-Sheva, Israel
Kasturba Medical College, Manipal, India
Medical University of the Americas/Nevis, Nevis, West Indies
Ross University School of Medicine, Roseau, Dominica
Saba University School of Medicine, Saba, Netherland Antilles
St. George's University School of Medicine, St. George's, Grenada
St. Matthew's University School of Medicine, Grand Cayman, Cayman Islands

redpajama_set_name: RedPajamaCommonCrawl


text:
\section{Introduction}
There is significant and growing interest in models,
especially neural networks, that are invariant or equivariant to the action of a group
on their inputs.
It is widely believed that these models enjoy improved generalisation when the group is correctly specified.
The intuition being that if the salient aspects of a task are unchanged by some transformation,
then more flexible models would need to learn (as opposed to being hard-coded) to
ignore these transformations, requiring more samples to generalise.
Work in this area has progressed quickly~\cite{cohen2016group,cohen2018spherical,cohen2019general}
and has found application in domains where the symmetry is known a priori, for instance
in particle physics~\cite{pfau2020ab}.

In contrast with practical successes, our theoretical understanding of invariant/equivariant models is limited.
Many previous works that have attempted to address the generalisation of invariant/equivariant models,
such as~\citet{sokolic2017generalization,sannai2019improved},
cover only the worst case performance of algorithms.
These works use complexity measures to find tighter upper bounds on the test
risk for invariant/equivariant models, but a strict benefit is not demonstrated.
The VC dimension governs distribution independent generalisation,
so these complexity measures
can show no more than a separation between the VC dimensions of a model
and its invariant/equivariant version.
This would imply the existence of a distribution on
which the model with smaller VC dimension will generalise better,
but would not rule out that on many common distributions
training an invariant/equivariant model using standard procedures
provides no benefit.
For instance, there could be many invariant/equivariant distributions on which
SGD automatically favours parameters that result in (possibly approximately)
invariant/equivariant predictors, regardless of architecture.

The results of this paper move to address this issue, by quantifying exactly the generalisation
benefit of invariant/equivariant linear models.
We do this in the case of the minimum $L_2$-norm least-squares solution, which reflects the implicit
bias of gradient descent in overparameterised linear models.
While the linear model provides a tractable first-step towards understanding more complex models such
as neural networks, the underlying ideas of this paper are equally applicable to non-linear predictors.
We emphasise this by providing new perspectives on feature averaging and
suggestions for how to apply the ideas of this paper to find new methods for training
invariant/equivariant neural networks.

\subsection{Our Contributions}
The main result of this paper is~\Cref{thm:equivariant-regression}, which
quantifies the generalisation benefit of equivariance in a linear model.
We define the \emph{generalisation gap} $\Delta(f, f')$
between predictors $f$ and $f'$ to be the difference in their test errors on a given task.
A positive generalisation gap $\Delta(f, f') > 0$ means that $f'$ has strictly smaller test error
than $f$.
\Cref{thm:equivariant-regression} concerns
$\Delta(f, f')$ in the case that $f: \mathbb{R}^d \to \mathbb{R}^k$ is the minimum-norm least-squares predictor
and $f'$ is its equivariant version.
Let a compact group $\mathcal{G}$ act via orthogonal representations $\phi$ and $\psi$ on
inputs $X \sim \mathcal{N}(0, I_d)$ and outputs $Y = h(X) + \xi \in \mathbb{R}^k$ respectively,
where $h: \mathbb{R}^d \to \mathbb{R}^k$ is an equivariant linear map.
Let $\binner{\chi_\psi}{\chi_\phi}= \int_\mathcal{G}  \tr(\psi(g))\tr(\phi(g))\mathop{}\!\mathrm{d} \lambda(g)$ denote
the scalar product of the characters of the representations.
The generalisation benefit of enforcing equivariance in a linear model is given by
\[
    \E[\Delta(f, f')] = \var{\xi} r(n, d)(dk - \binner{\chi_\psi}{\chi_\phi}) + \mathcal{E}_\mathcal{G}(n, d)
\]
where
\[
    r(n, d) = \begin{cases}
        {\frac{n}{d(d - n - 1)}} & n < d- 1\\
        (n - d - 1)^{-1} & n > d + 1\\
        \infty & \text{otherwise}
    \end{cases}
\]
and $\mathcal{E}_\mathcal{G}(n, d) \ge 0 $ is the generalisation gap of the corresponding noiseless problem, that
vanishes when $n \ge d$.
The divergence at the interpolation threshold $n \in [d-1, d+1]$
is consistent with the double descent literature~\cite{hastie2019surprises}.

The quantity $dk - \binner{\chi_\psi}{\chi_\phi}$
represents the significance of the group symmetry to
the task. The dimension of the space of linear maps $\mathbb{R}^d \to \mathbb{R}^k$ is $dk$,
while $\binner{\chi_\psi}{\chi_\phi}$ is the dimension of the space of equivariant linear maps.
We will see later that the quantity $dk - \binner{\chi_\psi}{\chi_\phi}$
represents the dimension of the space of linear maps that vanish when averaged over $\mathcal{G}$;
it is through the dimension of this space that the symmetry in the task controls the generalisation gap.
Although invariance is a special case of equivariance, we find it instructive to
discuss it separately. In~\Cref{thm:invariant-regression} we provide
a result that is analogous to~\Cref{thm:equivariant-regression} for invariant predictors,
along with a separate proof.

In order to arrive at~\Cref{thm:invariant-regression,thm:equivariant-regression} we
make use of general results about the structure of function spaces under averaging operators.
In~\Cref{sec:decomposition} we show how averaging operators can be used to decompose
function spaces into orthogonal subspaces of symmetric (invariant/equivariant)
and anti-symmetric (vanish when averaged) functions.
In~\Cref{sec:fa} we use these insights to provide new perspectives on feature averaging.
Our main results are in~\cref{sec:lower-bounds}.
Finally, in~\Cref{sec:neural-networks} we apply our insights to derive principled
methods for training invariant/equivariant neural networks and provide open questions
for future work.

\section{Related Work}
\paragraph{Implementations}
While there has been a recent surge in interest,
symmetry is not a new concept in machine learning.
Recent literature is dominated by neural networks, but other methods do exist:
e.g.~kernels~\cite{haasdonk05invariancein}, support
vector machines~\cite{scholkopf96incorporatinginvariances} or feature
spaces such as polynomials~\cite{schulz1994constructing,schulz1992existence}.
The engineering of invariant neural networks dates back at least to~\citet{wood1996representation},
in which ideas from representation theory are applied to find weight tying schemes that result
in group invariant architectures; similar themes are present in~\citet{ravanbakhsh2017equivariance}.
Recent work follows in this vein,
borrowing ideas from fundamental physics to construct invariant/equivariant
convolutional architectures~\cite{cohen2016group,cohen2018spherical}.
Correspondingly, a sophisticated theory of invariant/equivariant
networks has arisen~\cite{kondor2018generalization,cohen2019general}
including universal approximation results~\cite{maron2019universality,yarotsky2018universal}.

\paragraph{Learning and Generalisation}
The intuition that invariant or equivariant models are more sample efficient
or generalisable is widespread in the literature, but arguments are often heuristic and,
to the best of our knowledge, a provably strict (non-zero) generalisation benefit has not appeared before this paper.
It was noted~\cite{abu1993hints} that constraining a model to be invariant cannot increase its
VC dimension.
An intuitive argument for reduced sample complexity is made in~\citet{mroueh2015learning}
in the case that the input space has finite cardinality.
The sample complexity of linear classifiers with invariant representations
trained on a simplified image task is discussed briefly in~\citet{anselmi2014unsupervised},
the authors conjecture that a general result may be obtained using wavelet transforms.
The framework of robustness~\cite{xu2012robustness} is used in~\citet{sokolic2017generalization}
to obtain a generalisation bound for interpolating large-margin classifiers that are
invariant to a finite set of transformations;
note that the results contain an implicit margin constraint on the training data.
The generalisation of models invariant or equivariant
to finite permutation groups is considered in~\citet{sannai2019improved}.
Both of~\citet{lyle2019analysis,lyle2020benefits} cover the PAC Bayes approach to generalisation of invariant models,
the latter also considers the relative benefits of feature averaging and data augmentation.

\section{Preliminaries}
We assume familiarity with the basic notions of group theory,
as well as the definition of a group action and a linear representation.
The reader may consult~\citet[Chapters 1-4]{wadsleyrep2012,serre1977linear} for background.
We define some key concepts and notation here and introduce more as necessary throughout the paper.

\paragraph{Notation and Technicalities}
We write $\mathcal{X}$ and $\mathcal{Y}$ for input and output spaces respectively.
We assume for simplicity that $\mathcal{Y} =(\mathbb{R}^k, +)$ is a $k$-dimensional
real vector space (with $k$ finite) but
we expect our results to apply in other settings too.
Throughout the paper, $\mathcal{G}$ will represent an
arbitrary compact group that has a measurable action $\phi$ on $\mathcal{X}$ and a representation $\psi$ on $\mathcal{Y}$.
By this we mean that $\phi: \mathcal{G} \times \mathcal{X} \to \mathcal{X}$ is a measurable map and the same for $\psi$.
We sometimes write $gx$ as a shorthand for $\phi(g) x$ and similarly for actions on $\mathcal{Y}$.
Some notation for specific groups: $C_m$ and $S_m$ are, respectively, the cyclic and symmetric groups on $m$ elements;
while $O(m)$ and $SO(m)$ are the $m$-dimensional orthogonal and special orthogonal groups respectively.
For any matrix $A$ we write
$A^+$ for the Moore-Penrose pseudo-inverse
and $\fnorm{A} = \sqrt{\tr(A^\top A)}$ for the Frobenius/Hilbert-Schmidt norm.
We write $\mathbb{G}_n(\mathbb{R}^d)$ for the Grassmannian manifold of subspaces of dimension $n$ in $\mathbb{R}^d$.
The results of this paper require some mild care with technical considerations
such as topology/measurability.
We do not stress these in the main paper but they do appear
in the proofs, %
\if\submission1
all of which are deferred to the supplementary material.
\else
some of which are deferred to the appendix.
\fi

\paragraph{Invariance, Equivariance and Symmetry}
A function $f: \mathcal{X} \to \mathcal{Y}$ is $\mathcal{G}$-invariant if
$ f(\phi(g)x) = f(x)$ $\forall x\in \mathcal{X}$ $\forall g\in \mathcal{G}$
and is $\mathcal{G}$-equivariant if
$ f(\phi(g)x) = \psi(g)f(x)$ $\forall x\in \mathcal{X}$ $\forall g\in \mathcal{G}$.
A measure $\mu$ on $\mathcal{X}$ is $\mathcal{G}$-invariant
if $\forall g \in \mathcal{G}$ and any $\mu$-measurable $B \subset \mathcal{X}$
the pushforward of $\mu$ by the action $\phi$ equals $\mu$, i.e.~$(\phi(g)_* \mu)(B) = \mu(B)$.
This means that if $X \sim \mu$ then $\phi(g) X \sim \mu$ $\forall g \in \mathcal{G}$.
We will sometimes use the catch-all term symmetric to describe an object that is invariant
or equivariant.

\section{Symmetric and Anti-Symmetric Functions}\label{sec:decomposition}
Averaging the inputs of a function over the orbit of a group
is a well known method to enforce invariance, for instance see~\citet{schulz1994constructing}.
Approaching this from another perspective, averaging can also be used to identify invariance.
That is, a function is $\mathcal{G}$-invariant if and only if it is preserved by orbit averaging with
respect to $\mathcal{G}$. The same can be said for equivariant functions, using a modified average.
After introducing the relevant concepts, we will
use this observation and other properties of the averaging operators
to decompose function spaces into mutually orthogonal
symmetric (invariant/equivariant) and anti-symmetric (vanish when averaged) subspaces.
This observation provides the foundation for many results later in the paper.

\subsection{Setup}
\paragraph{Haar Measure}
Let $\mathcal{G}$ be a compact group. The Haar measure is the unique invariant
measure on $\mathcal{G}$ and we denote it by $\lambda$.
By invariance we mean that for any measurable subset $A \subset \mathcal{G}$ and for any $g\in \mathcal{G}$,
$\lambda(gA) = \lambda(Ag) = \lambda(A)$.
We assume normalisation such that $\lambda(\mathcal{G}) = 1$, which is always possible when $\mathcal{G}$ is compact.
The (normalised) Haar measure can be interpreted as the uniform distribution on $\mathcal{G}$.
See~\citet{kallenberg2006foundations} for more details.

\paragraph{Orbit Averaging}
For any feature map $f:\mathcal{X} \to \mathcal{Y}$,
we can construct a $\mathcal{G}$-invariant feature map by averaging with respect to $\lambda$.
We represent this by the operator $\O$, where
\[
    (\O f)(x) = \int_\mathcal{G}  f(gx)\mathop{}\!\mathrm{d} \lambda(g).
\]
Similarly, if $\psi$ is a representation of $\mathcal{G}$ on $\mathcal{Y}$, we can transform $f$
into an equivariant feature map by applying $\mathcal{Q}$, where
\[
    (\mathcal{Q} f)(x) = \int_\mathcal{G}  \psi(g^{-1}) f(gx)\mathop{}\!\mathrm{d} \lambda(g).
\]
Notice that $\O$ is a special case of $\mathcal{Q}$ corresponding to
$\psi$ being the trivial representation.
The operator $\O$ can be thought of as performing feature averaging with respect to $\mathcal{G}$.
This interpretation is widely adopted, for instance appearing in~\citet{lyle2020benefits}.

\paragraph{Function Spaces}
We now show how to construct the relevant spaces of functions.
We present this in an abstract way, but these functions can be interpreted
as predictors, feature maps, feature extractors and so on.
Let $\mu$ be a $\mathcal{G}$-invariant measure on $\mathcal{X}$ and let $\inner{a}{b}$ for $a, b \in \mathcal{Y} $ be an inner product
on $\mathcal{Y}= \mathbb{R}^k$ that is preserved by $\psi$.%
\footnote{
    It is permissible for inner product itself to depend on the point $x$
    at which the feature maps are evaluated.
    The only requirement is that evaluating the inner product between
    two fixed vectors is a $\mathcal{G}$-invariant function $\inner{a}{b}(x) = \inner{a}{b}(gx)$,
    $\forall a, b \in \mathbb{R}^k$, $g \in \mathcal{G}$ and $x \in \mathcal{X}$.
    We believe that this allows our results to extend to the
    case of the features defined as maps from a manifold to its tangent
    bundle.}
By preserved we mean that
$\inner{\psi(g)a}{\psi(g)b} = \inner{a}{b}$, $\forall g\in \mathcal{G}$, $\forall a, b\in \mathcal{Y}$
and any inner product can be transformed to satisfy this property using the Weyl trick
$
    \inner{a}{b} \mapsto \int_\mathcal{G}\inner{ \psi(g)a }{ \psi(g)b }\mathop{}\!\mathrm{d}\lambda(g)
    $.
Given two functions $f, h: \mathcal{X} \to \mathcal{Y}$, we define their inner product by
\[
    \inner{f}{h}_\mu = \int_\mathcal{X} \inner{f(x)}{h(x)} \mathop{}\!\mathrm{d} \mu (x).
\]
This inner product can be thought of as comparing the similarity between functions
and can used to define a notion of distance with the norm $\norm{f}_\mu = \sqrt{\inner{f}{f}}_\mu$.
We then define $V$ as the space of all (measurable) functions
$f: \mathcal{X} \to \mathcal{Y}$ such that $\norm{f}_\mu < \infty$.\footnote{Equality is defined $\mu$-almost-everywhere.}
Formally, $V$ is a Bochner space.

\subsection{Averaging and the Structure of Function Spaces}
We have seen how to define orbit averaging operators to produce invariant and equivariant functions as
well as how to construct spaces of functions on which these operators can act.
The reason for all of this is the following result, which shows that the averaging operators
allow us to decompose any function in $V$ into orthogonal $\mathcal{G}$-symmetric and $\mathcal{G}$-anti-symmetric parts.
Recall that since $\O$ is just a special case of $\mathcal{Q}$,~\Cref{thm:fa} applies to both operators.

\begin{lemma}\label{thm:fa}
    Let $U$ be any subspace of $V$ that is closed under $\mathcal{Q}$.
    Define the subspaces $S$ and $A$ of, respectively, the $\mathcal{G}$-symmetric and $\mathcal{G}$-anti-symmetric functions in
    $U$:
    $
        S = \{f \in U: \text{$f$ is $\mathcal{G}$-equivariant}\}
        $
        and
        $
        A = \{f \in U: \mathcal{Q} f = 0 \}.
        $
    Then $U$ admits admits an orthogonal decomposition into symmetric and anti-symmetric parts
    \[
        U = S \oplus A.
    \]
\end{lemma}
\begin{supplementary}
See~\Cref{sec:proofs-appdx} for the proof.
\end{supplementary}
The proof consists of establishing the following properties:
\begin{enumerate*}[(A)]
    \item for any $f \in V$, $\mathcal{Q} f \in V$;
    \item any $f \in V$ is $\mathcal{G}$-equivariant if and only if $\mathcal{Q} f =f$;
    \item $\mathcal{Q}$ has only two eigenvalues, $1$ and $0$; and,
    \item $\mathcal{Q}$ is self-adjoint with respect to $\inner{\cdot}{\cdot}_\mu$.
\end{enumerate*}
The last of these is critical and depends on the $\mathcal{G}$-invariance of $\mu$.
There are many tasks for which $\mathcal{G}$-invariance of the input distribution is a
natural assumption, for instance in medical imaging~\cite{winkels20183d}.

\Cref{thm:fa} says that any function $u \in U$ can be written $u = s + a$,
where $s$ is $\mathcal{G}$-equivariant, $\mathcal{Q} a  = 0$ and $\inner{s}{a}_\mu = 0$.
We refer to $s$ and $a$ as the symmetric and anti-symmetric parts of $u$.
In general this does not imply that $a$ is odd, that it
outputs an anti-symmetric matrix or that it is negated by
swapping two inputs.
These are, however, special cases.
If $\mathcal{G} = C_2$ acts by $x\mapsto -x$ then odd functions $f: \mathbb{R}
\to \mathbb{R}$ will be anti-symmetric in the sense of this paper. If $\mathcal{G}=C_2$ acts on matrices by $M \mapsto
M^\top$ then $f: M \mapsto \frac12 (M - M^\top)$ is also anti-symmetric. Finally, if
$\mathcal{G} = S_n$ and $f: \mathbb{R}^n \to \mathbb{R}$ with $f(x_1, \dots, x_j, x_{j+1}, \dots, x_n) =
-f(x_1, \dots, x_{j+1}, x_{j}, \dots, x_n)$, then $f$ is anti-symmetric in the sense of this paper.

Although it is straightforward to demonstrate
and has surely been observed before, we will see in the rest of the paper
that the perspective provided by~\Cref{thm:fa} is highly fruitful.
Before that, we conclude this section with an example for intuition.
\begin{example}\label{example:rotations}
    Let $V$ consist of all functions $f: \mathbb{R}^2 \to \mathbb{R}$ such that $\E[f(X)^2] < \infty$
    where $X \sim \mathcal{N}(0, I_2)$.
    Let $\mathcal{G} = \text{SO}(2)$ act by rotation about the origin, with respect to which
    the normal distribution is invariant.
    Using~\Cref{thm:fa} we may write $V = S \oplus A$.
    Alternatively, consider polar coordinates $(r, \theta)$, then for any feature map $f(r, \theta)$
    we have $\O f (r, \theta) = \frac{1}{2\pi} \int_0^{2\pi} f(r, \theta)\mathop{}\!\mathrm{d} \theta $. So any $\mathcal{G}$-invariant
    feature map (i.e.~anything in $S$) depends only on the radial coordinate.
    Similarly, any $h$ for which $\O h = 0$ must have
    $\O h (r, \theta)  = \frac{1}{2\pi} \int_0^{2\pi}h(r, \theta) \mathop{}\!\mathrm{d} \theta  = 0$ for any $r$,
    and $A$ consists entirely of such functions. For example, $r^{3} \cos \theta \in A$.
    We then recover $\inner{s}{h}_\mu = \frac{1}{2\pi}\int_\mathcal{X} s(r) h(r, \theta)\mathsf{e}^{-r^2/2}r \mathop{}\!\mathrm{d} r \mathop{}\!\mathrm{d} \theta  = 0$ for any $s \in S$
    by integrating $h$ over $\theta$.
    Intuitively, one can think of the functions in $S$ as varying perpendicular to the flow
    of $\mathcal{G}$ on $\mathcal{X} = \mathbb{R}^2$ and so are preserved by it, while the functions in $A$ average to $0$ along this flow,
    see~\cref{fig:orbit}.
\end{example}

\begin{figure}[h]
    \centering
    \includegraphics[width=0.85\textwidth]{sym-antisym.pdf}
    \vspace{-10pt}
    \caption{Example of a function decomposition.
    The figure shows $f(r, \theta) = r \cos{(r - 2\theta)} \cos{(r + 2\theta)}$
    decomposed into its symmetric and anti-symmetric parts in $V = S \oplus A$
    under the natural action of $\mathcal{G} = \text{SO}(2)$ on $\mathbb{R}^2$.
    See~\Cref{example:rotations}. Best viewed in colour.}\label{fig:orbit}
\end{figure}

\section{Feature Averaging}\label{sec:fa}
We remarked earlier that $\O$ can be thought of as performing feature averaging.
Before our study of the generalisation of symmetric models, we use this interpretation to derive
our first consequence of~\Cref{thm:fa}.
We show that feature averaging can be viewed as
solving a least squares problem in the space of features extractors $V$.
That is, feature averaging sends $f$ to $\bar{f}$, where $\bar{f}$ is the \emph{closest $\mathcal{G}$-invariant feature
extractor to $f$}.

\newcommand{\thmleastsquares}{%
    Let $V$ be the space of all normalisable feature extractors as defined above.
    Define $S$ and $A$ as in~\Cref{thm:fa}.
    For any $f \in V$, feature averaging with $\O$ maps $f \mapsto \bar{f}$ where $\bar{f}$
    is the ($\mu$-a.e.) unique solution to the least-squares problem
    \[
        \bar{f} = \argmin_{s \in S} \norm{f - s}^2_\mu.
    \]%
}
\begin{proposition}[Feature Averaging as a Least-Squares Problem]\label{thm:least-squares}
    \thmleastsquares{}
\end{proposition}
\begin{supplementary}
The proof of~\cref{thm:least-squares} is a straightforward exercise,
so we postpone it to~\cref{sec:proof-least-squares}.
\end{supplementary}

\begin{example}
    Consider again the setting of~\cref{example:rotations}.
    For simplicity, let $f(r, \theta)= f_{\text{rad}}(r)f_{\text{ang}}(\theta)$ be separable in polar coordinates.
    Notice that $\O f = c_f f_{\text{rad}}$ where $c_f  = \frac{1}{2\pi} \int_0^{2\pi} f_\text{ang}( \theta)\mathop{}\!\mathrm{d} \theta $.
    Then for any $s \in S$ can calculate:
    \begin{align*}
        \norm{f - s}_\mu^2
        &= \frac{1}{2\pi}\int_\mathcal{X} (f(r, \theta) - s(r))^2 \mathsf{e}^{-r^2/2}r \mathop{}\!\mathrm{d} r \mathop{}\!\mathrm{d} \theta  \\
        &= \frac{1}{2\pi}\int_\mathcal{X} (f(r, \theta)- c_ff_{\text{rad}}(r))^2 \mathsf{e}^{-r^2/2}r \mathop{}\!\mathrm{d} r \mathop{}\!\mathrm{d} \theta \\
        &\phantom{=}+ \frac{1}{2\pi}\int_\mathcal{X}(c_ff_{\text{rad}}(r) -s)^2 \mathsf{e}^{-r^2/2}r \mathop{}\!\mathrm{d} r \mathop{}\!\mathrm{d} \theta
    \end{align*}
    which is minimised by $s = c_f f_\text{rad}$, as predicted.
\end{example}

\subsection{Feature Averaging and Generalisation}\label{sec:fa-gen}
We end our discussion of feature averaging with an analysis of its impact on generalisation.
To do this we consider the reduction in the Rademacher complexity of a feature averaged class.

\paragraph{Rademacher Complexity}
Let $T = \{t_1, \dots, t_n \}$ be a collection of points of some space $\mathcal{T}$.
The empirical Rademacher complexity of a set $\mathcal{F}$ of functions $f: \mathcal{T} \to \mathbb{R}$
evaluated on $T$ is defined by
\[
    \Rad_T(\mathcal{F})
    = \E_{\bm{\sigma}}\left[ \sup_{f \in \mathcal{F}}\abs{\frac1n  \sum_{i=1}^n \sigma_i f(t_i)}\right]
\]
where $\sigma_i \sim \Unif\{-1, 1\}$ for $i=1,\dots, n$ are Rademacher random variables over which
the expectation $\E_{\bm{\sigma}}$ is taken.
Let $\nu$ be a distribution on $\mathcal{T}$ and take $T \sim \nu^n$,
in which case the empirical Rademacher complexity $\Rad_T(\mathcal{F})$ is a random quantity.
The Rademacher complexity $\ERad_n(\mathcal{F})$ is defined by
taking an expectation over $T$:
$
    \ERad_n(\mathcal{F}) = \E[\Rad_T(\mathcal{F})]
$.
The Rademacher complexity appears in the study of generalisation in statistical learning,
for instance see~\citet[Theorem 4.10 and Proposition 4.12]{wainwright2019high}.

\begin{proposition}\label{thm:fa-rademacher}
    Let $\mathcal{G}$ be a compact group acting measurably on a set $\mathcal{T}$.
    Let $\mathcal{F}$ be a class of functions $f: \mathcal{T} \to \mathbb{R}$ and define the symmetric and anti-symmetric classes
    $
        \overline{\mathcal{F}} = \{\O f : f \in \mathcal{F}\}
        $
        and
       $
       \mathcal{F}^{\scaleobj{0.75}{\perp}} = \{f - \O f: f \in \mathcal{F}\}
       $.
    Let $\nu$ be a distribution over $\mathcal{T}$ that is $\mathcal{G}$-invariant.
    Then the Rademacher complexity of the feature averaged class satisfies
   \[
        0 \le  \ERad_n (\mathcal{F})  -  \ERad_n (\overline{\mathcal{F}})  \le  \ERad_n (\mathcal{F}^{\scaleobj{0.75}{\perp}})
   \]
   where the expectations in the definition of $\ERad_n$ are taken over $t_i \sim \nu$ i.i.d..
\end{proposition}
\begin{supplementary}
See~\Cref{sec:proofs-appdx,sec:proof-fa-rademacher} for setup, proof and technicalities.
\end{supplementary}
\Cref{thm:fa-rademacher} says that the Rademacher complexity is reduced by feature averaging, but not by more than
the complexity of the anti-symmetric component of the class.
This can be thought of as quantifying the benefit of enforcing invariance by averaging in
terms of the extent to which the inductive bias is already present in the function class.
Although the form of this result is suggestive, it
does not imply a strict benefit.
We provide stronger results in the following section.

\section{Generalisation Benefit from Symmetric Models}\label{sec:lower-bounds}
In this section we apply~\Cref{thm:fa} to derive a strict (i.e.~non-zero) generalisation gap
between models that have and have not been specified to have the invariance/equivariance
that is present in the task.
We start with the following general result, which equates the generalisation gap
between any predictor and its closest equivariant function
to the norm of the anti-symmetric component of the predictor.

\begin{lemma}\label{lemma:gen-gap-antisym}
    Let $X \sim \mu$ where $\mu$ is a $\mathcal{G}$-invariant distribution on $\mathcal{X}$.
    Let $Y = f^*(X) + \xi \in \mathbb{R}^k$, where $\xi$ is a random element of $\mathbb{R}^k$
    that is
    independent of $X$ with zero mean and finite variance
    and $f^*: \mathcal{X} \to \mathbb{R}^k$ is $\mathcal{G}$-equivariant.
    Then, for any $f \in V$, the generalisation gap satisfies
    \if\submission1
    \begin{align*}
        \Delta(f, \mathcal{Q} f)
        &\coloneqq \E[\norm{Y - f(X)}_2^2] - \E[\norm{Y - \mathcal{Q} f(X)}_2^2] \\
        &= \norm{f^{\scaleobj{0.75}{\perp}}}_\mu^2.
    \end{align*}
    \else
    \[
        \Delta(f, \mathcal{Q} f)
        \coloneqq \E[\norm{Y - f(X)}_2^2] - \E[\norm{Y - \mathcal{Q} f(X)}_2^2]
        = \norm{f^{\scaleobj{0.75}{\perp}}}_\mu^2.
    \]
    \fi
\end{lemma}
\begin{proof}
    By \Cref{thm:fa}, we can decompose $f = \bar{f} + f^{\scaleobj{0.75}{\perp}}$ into its
    symmetric $\bar{f} = \mathcal{Q} f$ and anti-symmetric $f^{\scaleobj{0.75}{\perp}} = f - \mathcal{Q} f$ parts.
    Since $\xi$ is independent of $X$ with zero mean and finite variance,
    \[
        \Delta(f, \mathcal{Q} f) = \E[\norm{Y - f(X)}_2^2] - \E[\norm{Y - \bar{f}(X)}_2^2]
        = \E[\norm{f^*(X) - f(X)}_2^2] - \E[\norm{f^*(X) - \bar{f}(X)}_2^2].
    \]
    Using the decomposition of $f$,
    \begin{align*}
        \E[\norm{f^*(X) - f(X)}_2^2] - \E[\norm{f^*(X) - \bar{f}(X)}_2^2]
        &= -2\E[\inner{f^*(X) - \bar{f}(X)}{f^{\scaleobj{0.75}{\perp}}(X)}] + \E[\norm{f^{\scaleobj{0.75}{\perp}}(X)}_2^2] \\
        &= \norm{f^{\scaleobj{0.75}{\perp}}}_\mu^2.
    \end{align*}
    Here we relied on $\E[\inner{f^*(X) - \bar{f}(X)}{f^{\scaleobj{0.75}{\perp}}(X)}] =
    \inner{f^* - \bar{f}}{f^{\scaleobj{0.75}{\perp}}}_\mu = 0$, which follows from $f^*$ being
    $\mathcal{G}$-equivariant and hence orthogonal to $f^{\scaleobj{0.75}{\perp}}$.
\end{proof}

\Cref{lemma:gen-gap-antisym} demonstrates the existence of barrier in the generalisation
of any predictor on a problem that has a symmetry.
Notice that the barrier turns out to be the measure of how well the predictor encodes the symmetry.
Clearly, the only way of overcoming this is to set $f^{\scaleobj{0.75}{\perp}} = 0$ ($\mu$-a.e.), which from~\Cref{thm:fa}
equivalent to enforcing $\mathcal{G}$-equivariance in $f$ ($\mu$-a.e.).
\Cref{lemma:gen-gap-antisym} therefore provides a \emph{strict generalisation benefit for equivariant predictors}.

In a sense, $\mathcal{Q} f$ is the archetypal equivariant predictor to which $f$ should be compared.
A trivial extension to~\cref{thm:least-squares} shows that $\mathcal{Q} f$ is the closest equivariant
predictor to $f$ and, more importantly,
if $h$ is a $\mathcal{G}$-equivariant predictor with smaller test
risk than $\mathcal{Q} f$ then $\Delta(f, h) = \Delta(f, \mathcal{Q} f) + \Delta(\mathcal{Q} f, h) \ge \Delta(f, \mathcal{Q} f)$
which cannot weaken our result.

Later in this section we will use~\Cref{lemma:gen-gap-antisym}
to explicitly calculate the generalisation gap for invariant/equivariant linear models.
We will see that it displays a natural relationship
between the number of training examples and the dimension of the space of
anti-symmetric models $A$, which is a property of the group action.
Intuitively, the model needs enough examples to learn to be orthogonal to $A$.

This result also has a useful theoretical implication for test-time data
augmentation, which is commonly used to increase test accuracy
\cite{Simonyan15, Szegedy15, He16}. Test-time augmentation consists of
averaging the output of a learned function $f$ over random transformations of
the same input when making predictions at test-time. When the transformations
belong to a group $\mathcal{G}$ and are sampled from its Haar measure,
test-time averaging can be viewed as a Monte Carlo estimate of $\O f$.
\Cref{lemma:gen-gap-antisym} then shows that test-time
averaging is beneficial for generalisation when the target function is itself
$\mathcal{G}$-invariant, regardless of the learned function $f$.

\subsection{Regression with Invariant Target}
Let $\mathcal{X} = \mathbb{R}^d$ with the Euclidean inner product and $\mathcal{Y} = \mathbb{R}$.
Consider linear regression with the squared-error loss $\ell(y, y') = (y - y')^2$.
Let $\mathcal{G}$ be a compact group that acts on $\mathcal{X}$ via an orthogonal representation $\phi: \mathcal{G} \to O(d)$
and let $X \sim \mu$ where $\mu$ is now an arbitrary $\mathcal{G}$-invariant probability distribution
on $\mathcal{X}$ with $\Sigma \coloneqq \E[X X^\top]$ finite and positive definite.%
\footnote{If $\Sigma$ is only positive semi-definite then the developments are similar.
We assume $\Sigma > 0$ for simplicity.}
We consider linear predictors $h_w: \mathcal{X} \to \mathcal{Y}$ with $h_w(x) = w^\top x$ where $w \in \mathcal{X}$.
Define the space of all linear predictors ${V_\text{lin}} = \{h_w : w \in \mathcal{X}\}$ which is a subspace of $V$.
Notice that ${V_\text{lin}}$ is closed under $\O$: for any $x \in \mathcal{X}$
\if\submission1
\begin{align*}
    \O h_w (x)
	&= \int_\mathcal{G}  h_w(gx) \mathop{}\!\mathrm{d}\lambda(g) \\
	&= \int_\mathcal{G} w^\top \phi(g) x \mathop{}\!\mathrm{d}\lambda(g)\\
	&= \left(\int_\mathcal{G} \phi(g^{-1}) w\mathop{}\!\mathrm{d}\lambda(g)\right)^\top x \\
	&= h_{{\Phi_\G}(w)}(x)
\end{align*}
\else
\[
    \O h_w (x) = \int_\mathcal{G}  h_w(gx) \mathop{}\!\mathrm{d}\lambda(g) = \int_\mathcal{G} w^\top \phi(g) x \mathop{}\!\mathrm{d}\lambda(g)
        = \left(\int_\mathcal{G} \phi(g^{-1}) w\mathop{}\!\mathrm{d}\lambda(g)\right)^\top x = h_{{\Phi_\G}(w)}(x)
\]
\fi
where in the last line we substituted $g \mapsto g^{-1}$ and defined the linear map ${\Phi_\G}: \mathbb{R}^d \to \mathbb{R}^d$
by ${\Phi_\G}(w) = \int_\mathcal{G}  \phi(g) w\mathop{}\!\mathrm{d} \lambda(g) $.%
\footnote{Since $\mathcal{G}$ is compact it is unimodular
and this change of variables is valid, e.g.~see~\citet[Corollary 2.28]{folland2016course} and adjacent results.}
We also have
\[
    \inner{h_a}{h_b}_\mu = \int_\mathcal{X} a^\top xx^\top b\mathop{}\!\mathrm{d}\mu(x)  = a^\top \Sigma b
\]
and we denote the induced inner product on $\mathcal{X}$ by $\inner{a}{b}_\Sigma \coloneqq a^\top \Sigma b$
and the corresponding norm by $\norm{\cdot}_\Sigma$.
Since ${V_\text{lin}}$ is closed under $\O$ we can apply~\Cref{thm:fa} to decompose ${V_\text{lin}} = S \oplus A$
where the orthogonality is with respect to $\inner{\cdot}{\cdot}_\mu$.
It follows that we can write any $h_w \in {V_\text{lin}}$ as
\[
    h_w = \overline{h_w} + h_w^{\scaleobj{0.75}{\perp}}
\]
where we have shown that there must exist $\bar{w}, w^{\scaleobj{0.75}{\perp}} \in \mathcal{X}$ with $\inner{\bar{w}}{w^{\scaleobj{0.75}{\perp}}}_\Sigma = 0$
such that $\overline{h_w} = h_{\bar{w}}$ and $h_w^{\scaleobj{0.75}{\perp}} = h_{w^{\scaleobj{0.75}{\perp}}}$.
By choosing a basis for $\mathcal{X}$, there is a natural bijection $\mathcal{X} \to {V_\text{lin}}$ where $w \mapsto h_w$.
Using this identification, we abuse notation slightly and write $\mathcal{X} = S \oplus A$
to represent the induced structure on $\mathcal{X}$.

Suppose examples are labelled by a target function $h_\theta \in {V_\text{lin}}$ that is $\mathcal{G}$-invariant.
Let $X \sim \mu$ and $Y = \theta^\top X + \xi$ where $\xi$ is independent of $X$, has mean 0 and finite variance.
Recall the definition of the \emph{generalisation gap} between predictors as the
difference in their test errors.
We study the generalisation gap $\Delta(h_w, h_{\bar{w}})$ between predictors $h_w$ and $h_{\bar{w}}$
defined above.
\Cref{lemma:gen-gap-antisym} gives
$
    \Delta(h_w, h_{\bar{w}}) = \norm{h_{w^{\scaleobj{0.75}{\perp}}}}_\mu^2 = \norm{w^{\scaleobj{0.75}{\perp}}}_\Sigma^2
    $.
In~\Cref{thm:invariant-regression} we calculate this quantity
where $w$ is the minimum-norm least-squares estimator and $\bar{w} =\Phi_\mathcal{G}(w)$.
To the best of our knowledge, this is the first result to specify exactly
the generalisation benefit for invariant models.

\begin{theorem}\label{thm:invariant-regression}
    Let $\mathcal{X} = \mathbb{R}^d$, $\mathcal{Y} = \mathbb{R}$ and let $\mathcal{G}$ be a compact group with an orthogonal representation
    $\phi$ on $\mathcal{X}$. Let $X \sim \mathcal{N}(0, \sigma_X^2 I)$
    and $Y = h_\theta(X) + \xi$ where $h_\theta(x) = \theta^\top x$ is $\mathcal{G}$-invariant
    with $\theta \in \mathbb{R}^d$ and $\xi$ has mean $0$, variance $\sigma_\xi^2 < \infty$ and is independent of $X$.
    Let $w$ be the least-squares estimate of $\theta$ from i.i.d.~examples $\{(X_i, Y_i): i=1, \dots, n\}$ and let
    $A$ be the orthogonal complement of the subspace of $\mathcal{G}$-invariant linear predictors (c.f.~\Cref{thm:fa}).
    \begin{itemize}
        \item If $n > d + 1$ then the generalisation gap is
            \[
                \E[\Delta(h_w, h_{\bar{w}})] = \sigma_\xi^2\frac{\dim A}{n - d - 1}.
            \]
        \item At the interpolation threshold $n \in [d-1, d+1]$, if $h_w$ is not $\mathcal{G}$-invariant then
            the generalisation gap diverges to $\infty$.
        \item If $n < d- 1$ the generalisation gap is
            \if\submission1
            \begin{align*}
                &\E[\Delta(h_w, h_{\bar{w}})]  \\
                &= \dim A \left( \frac{\sigma_X^2 \norm{\theta}_2^2\, n(d-n)}{d(d-1)(d+2)}
                + \frac{\sigma_\xi^2 \, n}{d(d - n - 1)} \right) .
            \end{align*}
            \else
            \[
                \E[\Delta(h_w, h_{\bar{w}})] =
                \dim A \left( \sigma_X^2 \norm{\theta}_2^2\frac{n(d-n)}{d(d-1)(d+2)}
                + \sigma_\xi^2\frac{n}{d(d - n - 1)} \right) .
            \]
            \fi
    \end{itemize}
\end{theorem}

In each case, the generalisation gap has a term of the form $\sigma_\xi^2 r(n, d) \dim A$ that arises
due to the noise in the target distribution.
In the overparameterised setting $d > n+1$ there is an additional term (the first)
that represents the generalisation gap in the noiseless setting $\xi \overset{\text{a.s.}}{=} 0$.
This term is the error in the least-squares estimate of $\theta$ in the noiseless problem,
which of course vanishes in the fully determined case $n > d + 1$.
In addition, the divergence at the so called interpolation threshold $n\approx d$ is consistent with
the literature on double descent~\cite{hastie2019surprises}.

Notice the central role of $\dim A$ in~\Cref{thm:invariant-regression}.
This quantity is a property of the group action as it describes the codimension of the set of invariant models.
The generalisation gap is then dictated by how `significant' the symmetry is to the problem.
\if\submission1
We give two examples that represent extremal cases of this `significance'.
\else
Before turning to the proof, we give two examples that represent extremal cases of this `significance'.
\fi

\begin{example}[Permutations, $\dim A = d - 1$]
    Let $S_d$ act on $\mathcal{X} = \mathbb{R}^d$ by permutation of the coordinates, so $(\phi(\rho)w)_i = w_{\rho(i)}$ for $\rho \in S_d$.
    Observe that, since the Haar measure $\lambda$ is uniform on $S_d$, for any $i=1, \dots, d$
    \[
        \Phi_{S_d}(w)_i = \frac{1}{d!} \sum_{\rho \in S_d} w_{\rho(i)} = \frac{1}{d} \sum_j w_j
    \]
    so $S$ is the one dimensional subspace $\{t (1, \dots, 1)^\top: t \in \mathbb{R} \}$.
    Since $\mathcal{X} = S \oplus A$ we get $\dim A = d-1$.
\end{example}

\begin{example}[Reflection, $\dim A = 1$]
    Let $C_2$ be the cyclic group of order 2 and let it act on $\mathcal{X} = \mathbb{R}^d$ by reflection
    in the first coordinate. $A$ is then the subspace consisting of $w$ such that for any $j=2, \dots, d$
    \[
        \Phi_{C_2}( w)_j = \frac{1}{\abs{C_2}}\sum_{g\in C_2} (\phi(g)w)_j = w_j = 0
    \]
    since the action fixes all coordinates apart from the first.
    Hence $A = \{t(1, 0, \dots, 0)^\top: t \in \mathbb{R}\}$.
\end{example}

\begin{proof}[Proof of~\Cref{thm:invariant-regression}]
    Note that $X$ is $\mathcal{G}$-invariant for any $\mathcal{G}$ since the representation $\phi$ is orthogonal.
    We have seen above that the space of linear maps ${V_\text{lin}} = \{h_w : w \in \mathbb{R}^d\}$ is closed under $\O$,
    so by~\Cref{thm:fa} we can write ${V_\text{lin}} = S \oplus A$.
    Let ${\Phi_\G}^{\scaleobj{0.75}{\perp}} = I - {\Phi_\G}$, which is the orthogonal projection onto the subspace $A$.
    By isotropy of $X$ we have
    \[
        \Delta(h_w, h_{\bar{w}}) = \sigma_X^2 \norm{w^{\scaleobj{0.75}{\perp}}}_2^2
    \]
    for any $w \in \mathbb{R}^d$, where $w^{\scaleobj{0.75}{\perp}} = {\Phi_\G}^{\scaleobj{0.75}{\perp}}(w)$.
    The proof consists of calculating this quantity in the case that $w$ is the least-squares estimator.

    Let ${\bm{X}} \in \mathbb{R}^{n \times d}$ and $\bm{Y} \in \mathbb{R}^n$ correspond to row-stacked training examples
    drawn i.i.d.~as in the statement, so ${\bm{X}}_{ij} = (X_i)_j$ and $\bm{Y}_i = Y_i$.
    Similarly, set $\bm{\xi} = {\bm{X}}\theta - \bm{Y}$.
    The least squares estimate is the minimum norm solution of $ \argmin_{u \in \mathbb{R}^d} \norm{\bm{Y} - {\bm{X}} u}_2^2$,
    i.e.
    \begin{equation}
        w = ({\bm{X}}^\top{\bm{X}})^+ {\bm{X}}^\top {\bm{X}} \theta + ({\bm{X}}^\top {\bm{X}})^+ {\bm{X}}^\top \bm{\xi} \label{eq:soln}
    \end{equation}
    where $(\cdot)^+$ denotes the Moore-Penrose pseudo-inverse.
    Define $P_{E} = ({\bm{X}}^\top {\bm{X}})^+ {\bm{X}}^\top {\bm{X}}$, which is an orthogonal projection onto $E$, the
    rank of ${\bm{X}}^\top {\bm{X}}$ (this can be seen by diagonalising).

    We first calculate $\E[\norm{w^{\scaleobj{0.75}{\perp}}}_2^2 \vert{} {\bm{X}}]$ where $w^{\scaleobj{0.75}{\perp}} = {\Phi_\G}^{\scaleobj{0.75}{\perp}}(w)$.
    The contribution from the first term of~\cref{eq:soln} is
    $
        \norm{{\Phi_\G}^{\scaleobj{0.75}{\perp}}(P_E \theta)}_2^2
        $
    the cross term vanishes using $\xi \perp\!\!\!\perp X$ and $\E[\xi] = 0$ and the contribution from the second term of~\cref{eq:soln}
    is $
        \E[\norm{{\Phi_\G}^{\scaleobj{0.75}{\perp}}(({\bm{X}}^\top {\bm{X}})^+ {\bm{X}}^\top \bm{\xi}) }_2^2 \vert{} {\bm{X}}]
        $. Notice that ${\Phi_\G}^{\scaleobj{0.75}{\perp}}$ is just projection matrix and so is idempotent, hence (briefly writing it without the parenthesis
        to emphasis matrix interpretation)
    \begin{align*}
        \E[\norm{{\Phi_\G}^{\scaleobj{0.75}{\perp}}(({\bm{X}}^\top {\bm{X}})^+ {\bm{X}}^\top \bm{\xi}) }_2^2 \vert{} {\bm{X}}]
        &=  \E[  \tr( \bm{\xi}^\top {\bm{X}}  ({\bm{X}}^\top {\bm{X}})^+{\Phi_\G}^{\scaleobj{0.75}{\perp}}({\bm{X}}^\top {\bm{X}})^+ {\bm{X}}^\top \bm{\xi}  )\vert{} {\bm{X}}] \\
        &=  \tr( {\bm{X}}  ({\bm{X}}^\top {\bm{X}})^+{\Phi_\G}^{\scaleobj{0.75}{\perp}}({\bm{X}}^\top {\bm{X}})^+ {\bm{X}}^\top \E[\bm{\xi}  \bm{\xi}^\top] ) \\
        &=  \sigma_\xi^2\tr( {\Phi_\G}^{\scaleobj{0.75}{\perp}}({\bm{X}}^\top {\bm{X}})^+).
    \end{align*}
    We have obtained
    \[
        \E[\norm{w^{\scaleobj{0.75}{\perp}}}_2^2\vert{} {\bm{X}}]
        = \norm{{\Phi_\G}^{\scaleobj{0.75}{\perp}} (P_E \theta )}_2^2
        +  \sigma_\xi^2\tr( {\Phi_\G}^{\scaleobj{0.75}{\perp}}(({\bm{X}}^\top {\bm{X}})^+))
    \]
    and conclude by taking expectations, treating each term separately.
    \paragraph{First Term}
    If $n \ge d$ then $\dim E = d$ with probability $1$, so the first term vanishes almost surely.
    We treat the $n < d$ case using Einstein notation, in which repeated indices are implicitly summed over.
    In components, recalling that ${\Phi_\G}^{\scaleobj{0.75}{\perp}}$ is a matrix,
    \[
        \E[ \norm{{\Phi_\G}^{\scaleobj{0.75}{\perp}} (P_E \theta )}_2^2 ] = {\Phi_\G}^{\scaleobj{0.75}{\perp}}_{fa}{\Phi_\G}^{\scaleobj{0.75}{\perp}}_{fc}\E[P_E \otimes P_E]_{abce} \theta_b \theta_e
    \]
    and applying~\cref{lemma:proj-variance} we get
    \begin{align*}
        \E[ \norm{{\Phi_\G}^{\scaleobj{0.75}{\perp}} (P_E \theta )}_2^2 ]
        &=
        \frac{n(d-n)}{d(d-1)(d+2)}
        \left(
            {\Phi_\G}^{\scaleobj{0.75}{\perp}}_{fa}{\Phi_\G}^{\scaleobj{0.75}{\perp}}_{fa} \theta_b \theta_b
            + {\Phi_\G}^{\scaleobj{0.75}{\perp}}_{fa}{\Phi_\G}^{\scaleobj{0.75}{\perp}}_{fb} \theta_b \theta_a
        \right)\\
        &\phantom{=}+ \frac{n(d-n) + n(n-1)(d+2)}{d(d-1)(d+2)} {\Phi_\G}^{\scaleobj{0.75}{\perp}}_{fa}{\Phi_\G}^{\scaleobj{0.75}{\perp}}_{fc}\theta_a \theta_c\\
        &= \norm{\theta}_2^2\dim A\frac{n(d-n)}{d(d-1)(d+2)}
    \end{align*}
    where we have used that ${\Phi_\G}^{\scaleobj{0.75}{\perp}}(\theta) = 0$ and $\fnorm{{\Phi_\G}^{\scaleobj{0.75}{\perp}}}^2 = \dim A$.

    \paragraph{Second Term}
    By linearity,
    \[
        \E[\tr( {\Phi_\G}^{\scaleobj{0.75}{\perp}}(({\bm{X}}^\top {\bm{X}})^+))] = \tr( {\Phi_\G}^{\scaleobj{0.75}{\perp}}(\E[({\bm{X}}^\top {\bm{X}})^+])).
    \]
    Then~\Cref{lemma:expected-inv-wishart-singular,lemma:expected-inv-wishart}
    give $\E[({\bm{X}}^\top{\bm{X}})^+] = \sigma_X^{-2} r(n,d)I_d$ where
    \[
        r(n, d) = \begin{cases}
            {\frac{n}{d(d - n - 1)}} & n < d- 1\\
            (n - d - 1)^{-1} & n > d + 1 \\
            \infty & \text{otherwise}
        \end{cases}.
    \]
    When $n \in [d-1, d+1]$ it is well known that the expectation diverges,
    see~\Cref{sec:useful-facts}. Hence
    \[
        \E[\tr( {\Phi_\G}^{\scaleobj{0.75}{\perp}}(({\bm{X}}^\top {\bm{X}})^+))] = \sigma_X^{-2} r(n, d)\dim A.
    \]
\end{proof}

\subsection{Regression with Equivariant Target}\label{sec:equivariant-regression}
One can apply the same construction to equivariant models.
Assume the same setup, but now let $\mathcal{Y} = \mathbb{R}^k$ with the Euclidean inner product and
let the space of predictors be
${W_\text{lin}} = \{f_W: \mathbb{R}^d \to \mathbb{R}^k, \,\, f_W(x) = W^\top x : W \in \mathbb{R}^{d \times k}\}$.
We consider linear regression with the squared-error loss $\ell(y, y') = \norm{y - y'}_2^2$.
Let $\psi$ be an orthogonal representation of $\mathcal{G}$ on $\mathcal{Y}$.
We define the linear map, which we call the intertwining average,
${\Psi_\G} :\mathbb{R}^{d \times k} \to \mathbb{R}^{d \times k} $ by%
    \footnote{The reader may have noticed that we define ${\Psi_\G}$ backwards, in the sense that its
        image contains maps that are equivariant in the direction $\psi \to \phi$.
        This is because of the transpose in the linear model,
        which is there for consistency with the $k=1$ invariance case.
    This choice is arbitrary and gives no loss in generality.}
\[
    {\Psi_\G}(W) = \int_\mathcal{G}  \phi(g) W \psi(g^{-1})\mathop{}\!\mathrm{d} \lambda(g).
\]
Similarly, define the intertwining complement as ${\Psi_\G}^{\scaleobj{0.75}{\perp}} :\mathbb{R}^{d \times k} \to \mathbb{R}^{d \times k} $ by
$
    {\Psi_\G}^{\scaleobj{0.75}{\perp}}(W) = W - {\Psi_\G}(W)
$.
We establish the following results, which are in fact generalisations of the invariant case.
In the proofs we will leverage the expression of ${\Psi_\G}$ as a $4$-tensor with components
$
    {\Psi_\G}_{abce} = \int_\mathcal{G}  \phi(g)_{ac}\psi(g)_{be}\mathop{}\!\mathrm{d} \lambda(g)
    $ where $a, c = 1, \dots d$ and $b, e = 1, \dots, k$.%
\footnote{If necessary, the reader can see%
\if\submission1
the supplementary material
\else
~\cref{sec:cpts}
\fi
for a derivation.}
\begin{proposition}\label{prop:closure}
    For any $f_W \in {W_\text{lin}}$, $\mathcal{Q} f_W = f_{{\Psi_\G}( W)}$ and hence ${W_\text{lin}}$ is closed under $\mathcal{Q}$.
\end{proposition}
\begin{proof}
    Let $f_W(x) = W^\top x$ with $W \in \mathbb{R}^{d \times k}$, then using orthogonality and unimodularity
    \[
        \mathcal{Q} f_W (x)
        = \int_\mathcal{G}  \psi(g^{-1}) W^\top \phi(g) x \mathop{}\!\mathrm{d} \lambda(g)
        = \left(\int_\mathcal{G}  \phi(g) W \psi(g^{-1}) \mathop{}\!\mathrm{d} \lambda(g)\right)^\top x
        = {\Psi_\G}(W)^\top x .
    \]
\end{proof}

\begin{proposition}\label{prop:inner-product}
    The inner product on ${W_\text{lin}}$ satisfies, for any $f_{W_1}, f_{W_2} \in {W_\text{lin}}$,
    \[
        \inner{f_{W_1}}{f_{W_2}}_\mu = \tr( W_1^\top \Sigma W_2)
    \]
    where $\Sigma = \E[X X^\top]$ and $X \sim \mu$.
\end{proposition}
\begin{proof}
    \begin{align*}
        \inner{f_{W_1}}{f_{W_2}}_\mu
        &= \int_\mathcal{X}  \left( W_1^\top x\right)^\top W_2^\top x \mathop{}\!\mathrm{d} \mu (x)\\
        &= \int_\mathcal{X}  x^\top W_1 W_2^\top x \mathop{}\!\mathrm{d} \mu (x)\\
        &= \int_\mathcal{X}  \tr(x^\top W_1 W_2^\top x) \mathop{}\!\mathrm{d} \mu (x)\\
        &= \tr( W_1^\top \Sigma W_2)
    \end{align*}
\end{proof}

\Cref{prop:closure} allows us to apply~\Cref{thm:fa} to write ${W_\text{lin}} = S \oplus A$, so for any $f_W \in {W_\text{lin}}$
there exists $\overline{f_{W}} \in S$ and $f_W^{\scaleobj{0.75}{\perp}} \in A$ with $\inner{\overline{f_W}}{f_W^{\scaleobj{0.75}{\perp}}}_\mu = 0$.
The corresponding parameters $\overline{W} = {\Psi_\G}(W)$ and $W^{\scaleobj{0.75}{\perp}} = {\Psi_\G}^{\scaleobj{0.75}{\perp}}(W)$ must therefore satisfy
$\tr(\overline{W}^\top \Sigma W^{\scaleobj{0.75}{\perp}}) = 0$, with $\Sigma$ defined as in~\Cref{prop:inner-product}.
Repeating our abuse of notation, we identify
$\mathbb{R}^{d \times k} = S \oplus A$ with $S = {\Psi_\G}(\mathbb{R}^{d \times k})$ and $A$ its orthogonal
complement with respect to the induced inner product.
\begin{proposition}\label{prop:generalisation-gap}
    Let $X \sim \mu$ and let $\xi$ a random element of $\mathbb{R}^k$ that is independent of
    $X$ with $\E[\xi] = 0$ and finite variance.
    Set $Y = f_\Theta (X) + \xi$ where $f_\Theta$ is $\mathcal{G}$-equivariant.
    For any $f_W \in {W_\text{lin}}$, the generalisation gap satisfies
    \if\submission1
    \begin{align*}
        \Delta(f_W, f_{\overline{W}})
        &\coloneqq \E[\norm{Y - f_W(X)}_2^2] - \E[\norm{Y - f_{\overline{W}}(X)}_2^2] \\
        &= \fnorm{\Sigma^{1/2} W^{\scaleobj{0.75}{\perp}}}^2
    \end{align*}
    \else
    \[
        \Delta(f_W, f_{\overline{W}}) \coloneqq \E[\norm{Y - f_W(X)}_2^2] - \E[\norm{Y - f_{\overline{W}}(X)}_2^2]
        = \fnorm{\Sigma^{1/2} W^{\scaleobj{0.75}{\perp}}}^2
    \]
    \fi
    where $\overline{W} ={\Psi_\G}(W)$, $W^{\scaleobj{0.75}{\perp}} = {\Psi_\G}^{\scaleobj{0.75}{\perp}}(W)$
    and $\Sigma = \E[XX^\top]$.
\end{proposition}
\begin{proof}
    Recall that $W = \overline{W} + W^{\scaleobj{0.75}{\perp}}$ and that these
    satisfy $\tr(\overline{W}\Sigma W^{\scaleobj{0.75}{\perp}}) =0$ from the above.
    Then, using~\Cref{lemma:gen-gap-antisym} and~\cref{prop:inner-product},
    \[
        \Delta(f_W, f_{\overline{W}})
        = \norm{f_{W^\perp}}_\mu^2
        = \tr((W^{\scaleobj{0.75}{\perp}})^\top \Sigma W^{\scaleobj{0.75}{\perp}})
        = \fnorm{\Sigma^{1/2} W^{\scaleobj{0.75}{\perp}}}^2
    \]
\end{proof}

Having followed the same path as the previous section,
we provide a characterisation of the generalisation benefit of equivariance.
In the same fashion, we compare the least-squares estimate $W$ with its equivariant
version $\overline{W} = {\Psi_\G}(W)$. As we explained at the beginning of the section, the choice of
$\overline{W} = {\Psi_\G}(W)$ is natural and costs us nothing.
\newcommand{\thmequivariantregression}{%
    Let $\mathcal{X} = \mathbb{R}^d$, $\mathcal{Y} = \mathbb{R}^k$ and let $\mathcal{G}$ be a compact group with orthogonal representations
    $\phi$ on $\mathcal{X}$ and $\psi$ on $\mathcal{Y}$. Let $X \sim \mathcal{N}(0, \sigma_X^2 I_d)$
    and $Y = h_\Theta(X) + \xi$ where $h_\Theta(x) = \Theta^\top x$ is $\mathcal{G}$-equivariant and $\Theta \in \mathbb{R}^{d \times k}$.
    Assume $\xi$ is a random element of $\mathbb{R}^k$, independent of $X$, with mean $0$ and $\E[\xi \xi^\top ] = \sigma_\xi^2 I_k < \infty$.
    Let $W$ be the least-squares estimate of $\Theta$ from $n$ i.i.d.~examples
    $\{(X_i, Y_i): i=1, \dots, n\}$
    and let $\binner{\chi_\psi}{\chi_\phi} = \int_{\mathcal{G}} \chi_\psi(g) \chi_\phi(g)\mathop{}\!\mathrm{d} \lambda(g) $ denote the scalar product of the
    characters of the representations of $\mathcal{G}$.
    \begin{itemize}
        \item     If $n > d + 1$ the generalisation gap is
    \[
        \E[\Delta(f_W, f_{\overline{W}})] =
        \sigma_\xi^2 \frac{ dk - \binner{\chi_\psi}{\chi_\phi}}{n - d - 1}.
    \]
    \item At the interpolation threshold $n \in [d-1, d+1]$, if $f_W$ is not $\mathcal{G}$-equivariant then
    the generalisation gap diverges to $\infty$.
    \item If $n < d- 1$ then the generalisation gap is
        \if\submission1
        \begin{align*}
        &\E[\Delta(f_W, f_{\overline{W}})] =\\
        &\sigma_X^2 \frac{n(d-n)}{d(d-1)(d+2)} \left( (d+1) \fnorm{\Theta}^2- \tr( J_\mathcal{G} \Theta^\top \Theta) \right) \\
        &+ \sigma_\xi^2 \frac{n( dk - \binner{\chi_\psi}{\chi_\phi})}{d (d - n - 1)}
        \end{align*}
        \else
    \[
        \E[\Delta(f_W, f_{\overline{W}})] =
        \sigma_X^2 \frac{n(d-n)}{d(d-1)(d+2)} \left( (d+1) \fnorm{\Theta}^2- \tr( J_\mathcal{G} \Theta^\top \Theta) \right)
        + \sigma_\xi^2 \frac{n( dk - \binner{\chi_\psi}{\chi_\phi})}{d (d - n - 1)}
    \]
    \fi
    where each term is non-negative and
    $J_\mathcal{G} \in \mathbb{R}^{k \times k}$ is given by
    \[
        J_\mathcal{G} = \int_\mathcal{G} (\chi_\phi(g) \psi(g) + \psi(g^2)) \mathop{}\!\mathrm{d} \lambda(g).
    \]
    \end{itemize}
}
\begin{theorem}\label{thm:equivariant-regression}
\thmequivariantregression
\end{theorem}
\begin{supplementary}
The proof of~\Cref{thm:equivariant-regression} is longer than for~\Cref{thm:invariant-regression}
but follows the same scheme, so we defer it to~\Cref{sec:equivariant-proof}.
\end{supplementary}

\Cref{thm:equivariant-regression} is a direct generalisation of~\Cref{thm:invariant-regression}.
As we remarked in the introduction, $dk - \binner{\chi_\psi}{\chi_\phi}$ plays the role of $\dim A$
in~\Cref{thm:invariant-regression} and is a measure of the significance of the symmetry to the problem.
The dimension of ${W_\text{lin}}$ is $dk$, while $\binner{\chi_\psi}{\chi_\phi}$ is the dimension of the space of equivariant maps.
In our notation $\binner{\chi_\psi}{\chi_\phi} = \dim S$.

Just as with~\cref{thm:invariant-regression}, there is an additional term (the first)
in the overparameterised case $d > n + 1$ that represents the estimation in a noiseless setting $\xi \overset{\text{a.s.}}{=} 0$.
Notice that if $k=1$ and $\psi$ is trivial we find
\[
    J_\mathcal{G} = \int_\mathcal{G} \chi_\phi(g) \mathop{}\!\mathrm{d} \lambda(g)  + 1 =  \binner{\chi_\phi}{1} + 1 = \dim S + 1
\]
which confirms that~\cref{thm:equivariant-regression} reduces exactly to~\cref{thm:invariant-regression}.

Interestingly, the first term in the $d > n+1$ case can be made independent of $\psi$, since the equivariance of
$h_\Theta$ implies
\[
    \tr(J_\mathcal{G} \Theta^\top \Theta) = \tr(\Theta^\top J_\phi \Theta)
\]
where
\[
    J_\phi = \int_\mathcal{G} (\chi_\phi(g) \phi(g) + \phi(g^2)) \mathop{}\!\mathrm{d} \lambda(g).
\]

Finally, we remark that~\cref{thm:equivariant-regression} is possible for more general
probability distributions on $X$.
For instance, it sufficient that the distribution is absolutely continuous with
respect to the Lebesgue measure, has finite variance and is $O(d)$ invariant.
The final condition implies the existence of a scalar $r_{n}$
such that $\E[({\bm{X}}^\top {\bm{X}})^+] = r_{n} I_d$
where ${\bm{X}} \in \mathbb{R}^{n \times d}$ are the row-stacked training inputs as defined in the proof.

\section{Neural Networks}\label{sec:neural-networks}
In this section we discuss how the insights of this paper apply to neural networks
and raise some open questions for future work.
Let $F: \mathbb{R}^d \to \mathbb{R}^k$ be a feedforward neural network with $L$ layers,
layer widths $\kappa_i$ $i=1, \dots, L$ and weights $W^i \in \mathbb{R}^{\kappa_{i} \times \kappa_{i+1}}$
for $i=1, \dots, L$ where $\kappa_1 = d$ and $\kappa_{L+1} = k$. We will assume $F$ has the form
\begin{equation}\label{eq:nn}
    F(x) = W^L \sigma(W^{L-1}\sigma(\dots \sigma(W^{1} x)\dots))
\end{equation}
where $\sigma$ is an element-wise non-linearity.

\subsection{Invariant and Equivariant Networks}\label{sec:invariant-networks}
The standard method for engineering neural networks to be invariant/equivariant
to the action of a finite group on the inputs is weight tying.
This method has been around for a while~\cite{wood1996representation}
but has come to recent attention via~\citet{ravanbakhsh2017equivariance}.
We will briefly describe this approach, its connections
to~\Cref{thm:invariant-regression,thm:equivariant-regression} and how the ideas of
this paper can be used to find new algorithms for both enforced and learned invariance/equivariance.
We leave analyses of these suggested approaches to future work.

The methods of~\citet{wood1996representation,ravanbakhsh2017equivariance} can be described as follows.
Let $\G_{\text{fin}}$ be a finite group.
For each $i=2, \dots, L+1$, the user chooses a matrix representation $\psi_i:\G_{\text{fin}} \to \text{GL}_{k_i}(\mathbb{R})$
of $\G_{\text{fin}}$ that acts on the inputs for each layer $i=2, \dots, L$
and on the outputs of the network when $i=L+1$.%
\footnote{$\psi_1$ is the representation on the inputs, which we consider as an
aspect of the task and not a design choice.}
For $i=2, \dots, L$, these representations must be chosen
such that they commute with the activation function
\begin{equation}\label{eq:activation}
  \sigma(\psi_i(g) \cdot) = \psi_i(g) \sigma(\cdot)
\end{equation}
$\forall g \in \G_{\text{fin}}$.%
\footnote{This condition is somewhat restrictive, but note that a permutation representation will commute with
any element-wise non-linearity.}
One then chooses weights for the network such that at each layer and $\forall g \in \G_{\text{fin}}$
\begin{equation}\label{eq:intertwine}
    W^i\psi_i(g) = \psi_{i+1}(g)W^i.
\end{equation}
By induction on the layers, satisfying~\cref{eq:activation,eq:intertwine}
ensures that the network is $\G_{\text{fin}}$-equivariant.
Invariance occurs when $\psi_{L+1}$ is the trivial representation.

The condition in~\cref{eq:intertwine} can be phrased as saying that that $W^i$ belongs
to the space of \emph{intertwiners} of the representations $\psi_i$ and $\psi_{i+1}$.
By denoting the space of all weight matrices in layer $i$ as $U = \mathbb{R}^{\kappa_i \times \kappa_{i+1}}$,
the space of intertwiners is immediately recognisable as $S = {\Psi_\G}(U)$ from~\Cref{thm:fa}.

Typically, the practitioner will hand-engineer the structure of weight matrices to belong to
the correct intertwiner space.
In the following sections we will propose alternative procedures that build naturally on the ideas
of this paper.
Moreover, as a benefit of our framework, these new approaches extend weight-tying to
any compact group that admits a finite-dimensional, real representation.
We end this section with a bound on the sample complexity of invariant networks,
which follows from~\citet{bartlett2019nearly}. Similar results are possible for
different activation functions.

\begin{lemma}\label{lemma:inavariant-vc}
    Let $\mathcal{G}$ be a compact group with layer-wise representations as described.
    Let $F: \mathbb{R}^d \to \mathbb{R}$ be a $\mathcal{G}$-invariant neural network with ReLU activation and
    weights that intertwine the representations.
    Let $\H$ be the class of all functions realisable by this network.
    Then
    \[
        \vc{\H} \le L +  \frac{1}{2} \alpha(F) L(L+1)\max_{1 \le i \le L} \binner{\chi_{i}}{\chi_{i+1}}
    \]
    where $\alpha(F) = \log_2\left(4 \mathsf{e} \log_2\left(\sum_{i=1}^L 2\mathsf{e} i \kappa_i\right) \sum_{i=1}^L i\kappa_i \right)$.
\end{lemma}
\begin{proof}
    For a ReLU network $t_i$ independent parameters at each layer we have
    $
        \vc{\H} \le L + \alpha(F) \sum_{i=1}^L (L-i+1) t_i,
        $
    which follows by application of~\citet[Theorem 7]{bartlett2019nearly}.
    The proof of~\citet[Theorem 7]{bartlett2019nearly} depends on
    the representation of the network in terms of piecewise polynomials of bounded degree.
    Observe that since
    weights are tied only \emph{within} layers (so weights in different layers can vary independently)
    and the activation is ReLU, there is no increase in the degree of said polynomials
    from weight-tying and the proof given in~\citet{bartlett2019nearly} applies in our case.
    The condition~\cref{eq:intertwine} insists that the weight matrix $W^i$ belongs to the intertwiner space
    $\text{Hom}_{\mathcal{G}}(\mathbb{R}^{\kappa_{i-1}\times\kappa_{i}}, \mathbb{R}^{\kappa_i\times\kappa_{i+1}})$.
    The number of independent parameters at each layer is at most the dimension of this space.
    The conclusion follows by simple algebra and the relation of the dimension to the characters as
    given above.
\end{proof}

\begin{example}[Permutation invariant networks]
    Permutation invariant networks (and permutation invariant functions more generally) are
    studied in~\citet{wood1996representation,zaheer2017deep,bloem2020probabilistic} and many other works,
    see references therein.
    In particular, multiple authors have given the form of a permutation equivariant weight
    matrix as
    $
    W = \lambda I + \gamma \bm{1}\bm{1}^\top
            $
    for scalars $\alpha, \beta \in \mathbb{R}$ and with $\bm{1} = (1, \dots, 1)^\top$.
    Consider an $L$-layer ReLU network with, for simplicity, widths $\kappa_i = d$ $\forall i$.
    Let $\H$ be the class of all functions realisable by this network,
    then $\vc{\H} = O(L^2 \log(Ld \log(Ld)))$.
\end{example}

\subsection{Projected Gradients}
As we have seen, provided that the activation function satisfies~\cref{eq:activation},
specifying the weight matrices to intertwine between layer-wise representations
is sufficient to ensure equivariance in a neural network.
We have also seen from~\Cref{sec:equivariant-regression} that it is possible to
project any weight matrix into an intertwiner space using ${\Psi_\G}$.
For each layer $l$ of the network we have a linear map ${\Psi_\G}^l$,
which is a $4$-tensor with components
${\Psi_\G}^l_{abce} = \int_\mathcal{G}  \psi_{l+1}(g)_{ac}\psi_{l}(g)_{be}\mathop{}\!\mathrm{d} \lambda(g)$.

The tensors $\{{\Psi_\G}^l: l=1, \dots, L\}$ depend only on the representations and so can be computed before training.
One can therefore obtain invariant/equivariant networks by a form of projected gradient descent.
Explicitly, with loss $\ell$ and learning rate $\eta$, the update rule for
the $l$\textsuperscript{th} layer is
\begin{align*}
    &\widetilde{W}^l(t+1) = W^l(t) - \eta \grad_{W^l} \ell(W^1(t), \dots, W^L(t))\\
    &W^l(t+1) = {\Psi_\G}^l\left(\widetilde{W}^l(t+1)\right).
\end{align*}
If~\cref{eq:activation} holds the network will be exactly invariant/equivariant after any iteration.

\subsection{Regularisation for Equivariance}
We have seen from~\Cref{thm:fa,sec:equivariant-regression} that any weight matrix can be written
$W = \overline{W} + W^{\scaleobj{0.75}{\perp}}$ where $\overline{W} = {\Psi_\G}(W)$ belongs to an intertwiner space (so is equivariant)
and $W^{\scaleobj{0.75}{\perp}} = {\Psi_\G}^{\scaleobj{0.75}{\perp}}(W)$ belongs to an orthogonal space that parametrises the anti-symmetric linear maps.
This suggests a method of learned invariance/equivariance by using a regularisation term
of the form $\sum_{l=1}^L \fnorm{{{\Psi_\G}^l}^{\scaleobj{0.75}{\perp}}(W^l)}^2$.
Where the $4$-tensor ${{\Psi_\G}^l}^{\scaleobj{0.75}{\perp}}$ has
components ${{\Psi_\G}^l}^{\scaleobj{0.75}{\perp}}_{abce} = \delta_{ac}\delta_{be} - {\Psi_\G}^l_{abce}$
and can be computed before training.
If ${{\Psi_\G}^l}^{\scaleobj{0.75}{\perp}}(W^l) = 0$ for $l=1, \dots, L$ and the activation function satisfies~\cref{eq:activation},
then the resulting network will be exactly $\mathcal{G}$-invariant/equivariant.
This method could also allow for learned/approximate invariance.
Indeed,~\Cref{prop:regularisation} suggests $\fnorm{{{\Psi_\G}^l}^{\scaleobj{0.75}{\perp}}(W)}^2$
as a measure of the layer-wise invariance/equivariance of the network.
\begin{proposition}\label{prop:regularisation}
    Let $\mathcal{G}$ be a compact group.
    Let $f_W: \mathbb{R}^d \to \mathbb{R}^k$ with $f_W(x) = \sigma(Wx)$ be a single neural network layer with $C$-Lipschitz,
    element-wise activation $\sigma$.
    Let $\phi: \mathcal{G} \to O(d)$ and $\psi: \mathcal{G} \to O(k)$ be orthogonal representations of $\mathcal{G}$ on the input
    and output spaces respectively and assume that $\psi$ commutes with $\sigma$ as in~\cref{eq:activation}.
    Let $X \in \mathbb{R}^d$, $X \sim \mathcal{N}(0, I_d)$. We can consider the network as belonging to $V$ from~\Cref{sec:decomposition}
    with $\mu = \mathcal{N}(0, I_d)$. Write $V = S \oplus A$, where $S$ contains the equivariant functions in $V$, then
    \[
        \inf_{s \in S} \E[\norm{f_W(X) - s(X)}_2^2] \le 2C^2\fnorm{W^{\scaleobj{0.75}{\perp}}}^2.
    \]
\end{proposition}
\begin{proof}
    First note that the infimum clearly exists, since the left hand side vanishes when $W$ intertwines
    $\phi$ and $\psi$.
    Recognise that in the notation of~\Cref{sec:decomposition} we can write
    $\norm{a - b}_\mu^2 = \E[\norm{a(X) - b(X)}_2^2]$.
    By applying the proof of~\Cref{thm:least-squares} to $\mathcal{Q}$ we get
    $\inf_{s \in S}\E[\norm{f_W(X) - s(X)}^2] = \inf_{s \in S}\norm{f_W - s}_\mu^2 = \norm{f_W - \mathcal{Q} f_W}_\mu^2$.
    Then recalling the definition of $\mathcal{Q}$ we have
    \begin{align*}
        \norm{\mathcal{Q} f_W(x) - f_W(x) }_2^2
        &=  \lnorm{\int_\mathcal{G}  \psi(g^{-1}) f_W(\phi(g) x) - f_W(x)}_2^2 \mathop{}\!\mathrm{d} \lambda(g)\\
        &\le  \int_\mathcal{G}  \norm{\psi(g^{-1}) f_W(\phi(g) x) - f_W(x)}_2^2 \mathop{}\!\mathrm{d} \lambda(g)\\
        &=  \int_\mathcal{G}  \norm{\psi(g^{-1}) \sigma(W\phi(g) x) - \sigma(Wx)}_2^2 \mathop{}\!\mathrm{d} \lambda(g)\\
        &=  \int_\mathcal{G}  \norm{\sigma(\overline{W}x + \psi(g^{-1})W^{\scaleobj{0.75}{\perp}}  \phi(g) x) - \sigma(Wx)}_2^2 \mathop{}\!\mathrm{d} \lambda(g)\\
        &\le  C^2 \int_\mathcal{G}  \norm{\overline{W}x + \psi(g^{-1})W^{\scaleobj{0.75}{\perp}}  \phi(g) x - Wx}_2^2 \mathop{}\!\mathrm{d} \lambda(g)\\
        &=  C^2\int_\mathcal{G}  \norm{(\psi(g^{-1})W^{\scaleobj{0.75}{\perp}}  \phi(g) - W^{\scaleobj{0.75}{\perp}}) x}_2^2 \mathop{}\!\mathrm{d} \lambda(g)\\
    \end{align*}
    Then by an application of Fubini's theorem and the covariance of $X$ we get
    \begin{align*}
        &\E[\norm{\mathcal{Q} f_W(X) - f_W(X) }_2^2]  \\
        &\le  C^2\int_\mathcal{G}
    \tr\left((\psi(g^{-1})W^{\scaleobj{0.75}{\perp}}  \phi(g) - W^{\scaleobj{0.75}{\perp}})^\top \E[XX^\top] (\psi(g^{-1})W^{\scaleobj{0.75}{\perp}}  \phi(g) - W^{\scaleobj{0.75}{\perp}}) \right)\mathop{}\!\mathrm{d} \lambda(g)\\
        & = C^2 \fnorm{W^{\scaleobj{0.75}{\perp}}}^2 + C^2 \int_\mathcal{G}  \fnorm{\psi(g^{-1})W^{\scaleobj{0.75}{\perp}}  \phi(g) }^2\mathop{}\!\mathrm{d} \lambda (g)
    \end{align*}
    and then the argument of the integral can be analysed as
    \begin{align*}
        \fnorm{\psi(g^{-1})W^{\scaleobj{0.75}{\perp}}  \phi(g)}^2
        &= \tr\left((\psi(g^{-1})W^{\scaleobj{0.75}{\perp}}  \phi(g))^\top \psi(g^{-1})W^{\scaleobj{0.75}{\perp}}  \phi(g)\right)\\
        &= \tr\left(\phi(g^{-1})(W^{\scaleobj{0.75}{\perp}})^\top  \psi(g)\psi(g^{-1})W^{\scaleobj{0.75}{\perp}}  \phi(g)\right)\\
        &= \tr\left((W^{\scaleobj{0.75}{\perp}})^\top  W^{\scaleobj{0.75}{\perp}} \right).
    \end{align*}
    The proof is complete.
\end{proof}

\Cref{prop:regularisation} shows that the distance between the outputs of
a single layer neural network and its closest equivariant function is bounded by the norm
of the anti-symmetric component of the weights $W^{\scaleobj{0.75}{\perp}}$.
This quantity can be interpreted as a measure of the equivariance of the layer
and regularising $\fnorm{W^{\scaleobj{0.75}{\perp}}}$
will encourage the network to become (approximately)
equivariant.
It is easy to generalise~\cref{prop:regularisation} so that $X$ follows any $\mathcal{G}$-invariant
distribution with finite second moment.

\subsection{Open Questions}
\paragraph{Equivariant Convolutions}
There has been much work on engineering convolutional layers to be group
equivariant, for instance~\citet{cohen2016group,cohen2018spherical,kondor2018generalization,cohen2019general}.
The convolution is a linear operator parameterised by the kernel.
This suggests that it may be possible to analyse
the generalisation properties of group equivariant convolutions in the
framework of~\cref{thm:fa}, similar to~\cref{sec:lower-bounds}.

\paragraph{Invariant/Equivariant Networks}
We have discussed enforcing invariance/equivariance in a neural network
$F_{(W^1, \dots, W^L)}$ (with the dependence on the weights now explicit)
by restricting weight matrices to intertwine between representations at each layer.
We ask: is this the best way to encode symmetry?
Mathematically, let $X \sim \mu$ with $\mathcal{G}$-invariant $\mu$ and embed the functions realised by the network in
$V= S \oplus A$.  Given an invariant/equivariant target $s\in S$, must the
best approximating neural network be layer-wise invariant/equivariant?
That is, are there $s \in S$ such that the following holds
\if\submission1
\begin{align*}
    &\inf_{\mathcal{W}}\E[\norm{F_{(W^1, \dots, W^L)}(X) - s(X)}^2_2] \\
    &< \inf_{\mathcal{U}} \E[\norm{F_{(U^1, \dots, U^L)}(X) - s(X)}_2^2],
\end{align*}
\else
\[
    \inf_{\mathcal{W}}\E[\norm{F_{(W^1, \dots, W^L)}(X) - s(X)}^2_2]
    < \inf_{\mathcal{U}} \E[\norm{F_{(U^1, \dots, U^L)}(X) - s(X)}_2^2],
\]
\fi
where $\mathcal{W} = \{ W^l \in \mathbb{R}^{\kappa_l \times \kappa_{l+1}}:\, l =1, \dots, L\}$
is the set of all possible weight matrices and
$\mathcal{U} = \{ U^l \in {\Psi_\G}^l(\mathbb{R}^{\kappa_l \times \kappa_{l+1}}):\, l =1, \dots, L\}$
is the set of all weight matrices restricted to be intertwiners?
A resolution to this might shed light on new ways of encoding symmetry in neural networks.

\section*{Acknowledgements}
We would like to thank Varun Kanade and Yee Whye Teh for advice and support throughout this project.
We also thank Will Sawin for providing%
\if\submission1%
a useful lemma (in the supplementary material),
\else%
~\cref{lemma:proj-variance},
\fi
Michael Hutchinson for instructive feedback on an earlier draft and
Bobby He for a helpful discussion on inverse Wishart matrices.
BE receives support from the UK EPSRC CDT
in Autonomous Intelligent Machines and Systems (grant reference EP/L015897/1).
SZ receives support from the Aker Scholarship.

\begin{supplementary}

redpajama_set_name: RedPajamaArXiv


Hours Monday to Friday 8am - 5pm.
The above phone number is active 24 hour, 7 days if your require assistance outside of normal business hours, please call and follow the prompts to be directed to the relevant areas for emergency calls.
Closed public holidays and over the Christmas/New Year break.
Report a problem via NeatStreets.
Monday to Friday 10am - 6pm.
Saturday and Sunday 10am - 1pm.
Closed public holidays. The Alice Springs Collection is not open on weekends.
The Rediscovery Centre is open 7 days, 8am to 4pm. The landfill and Rediscovery Centre are CLOSED Christmas Day, New Year's Day and Good Friday.


After we reviewed shortly about Adobe Brackets three years ago, today we have a huge list of Adobe Bracket extensions. We will explain how to install extension in Brackets in Ubuntu. I will show you both online and offline methods. I use Brackets 1.3 and Ubuntu 15.04. You can use this guide on another Ubuntu version.
You will need internet connection to do these.
Open Extension Manager via block icon on Brackets top-right corner. See picture above number 1.
Open Available tab. See picture above number 2.
Find extension you want. For example, you want Brackets Beautify. Click install button. Wait until finish.
Open your file manager (e.g. Nautilus).
Drag the ZIP file of extension into the Drag .zip here label.
Sometimes, you will need an alternative way. To install an extension directly into its directory, extract the ZIP file into /home/<yourname>/.config/Brackets/extensions/user directory. Then restart Brackets. See picture above.
If you want to download Adobe Brackets extension and save it yourself, go to https://brackets-registry.aboutweb.com. It is Brackets official extension repository.
Weekly Extensions Chart (http://ingorichter.github.io/BracketsExtensionTweetBot) - It tells what 11 top extensions weekly.
Create Extensions (https://github.com/adobe/brackets/wiki/How-to-write-extensions) - It is an official wiki that tell you anything basic with how to create a Brackets extension.
Extensions Recommendation (http://www.johnpapa.net/my-recommended-brackets-extensions/) - It is an example of selected extensions by John Papa. You will have yours, but an example is always helpful.
Adobe Extract (https://helpx.adobe.com/creative-cloud/help/extract-for-brackets.html) - It is an official guide to use Extract feature in Brackets. Extract is an Adobe technology to help users design the web with Photoshop PSD easier.


Smoking ban change would add extra appeal County Council urged to avoid smoking room exception.
Businesses would have another chance to appeal smoking ban violations if an amendment the Board of Health is seeking proves successful. The board has approved an amendment to the countywide smoking ban that would allow a business to appeal the county health officer's decision regarding a violation. The amendment must still pass a County Council vote. Currently, the health officer's decision determines whether a violation stands. But the new amendment would enable an appeals committee made up of board members to hear an appeal of that decision, then make a recommendation to the full board, which would have the final say. Under the amendment, the board would vote on an appeal at the regular meeting after it is made unless that meeting is more than 30 days away. If it is, the board would hold a special meeting to decide the appeal's fate. The amendment also is seen as a way to help the Health Department. Nick Molchan, the Health Department's administrator, later said the amendment would ensure that an appeals decision "is not all on the employees" of the Health Department. In addition to discussing appeals, the board passed a motion advising the County Council to avoid creating a new exception to the smoking ban that would allow businesses to have smoking rooms. Health board member David Roos said he was concerned that the council might allow such an exception, which he said would make it more difficult to clamp down later on other exceptions to the ban, as health advocates have recommended.


text:
collingswood City Tickets
Patty Griffin & Raul Malo Scottish Rite Auditorium - Collingswood - Collingswood, NJ
The Fab Four - The Ultimate Tribute Scottish Rite Auditorium - Collingswood - Collingswood, NJ
America - The Band Scottish Rite Auditorium - Collingswood - Collingswood, NJ
Marshall Tucker Band & Pure Prairie League Scottish Rite Auditorium - Collingswood - Collingswood, NJ
Killer Queen Scottish Rite Auditorium - Collingswood - Collingswood, NJ
The Machine Performs Pink Floyd Scottish Rite Auditorium - Collingswood - Collingswood, NJ
RTP - The Rush Tribute Project Scottish Rite Auditorium - Collingswood - Collingswood, NJ
Brian Culbertson Scottish Rite Auditorium - Collingswood - Collingswood, NJ
Apr 20 2023 8:00 PM
Get The Led Out - Tribute Band Scottish Rite Auditorium - Collingswood - Collingswood, NJ
Yes & Jon Anderson and The Band Geeks Scottish Rite Auditorium - Collingswood - Collingswood, NJ
The Temptations & The Four Tops Scottish Rite Auditorium - Collingswood - Collingswood, NJ
Oct 14 2023 8:00 PM
Tommy James and The Shondells Scottish Rite Auditorium - Collingswood - Collingswood, NJ
Nov 3 2023 8:00 PM
Mania - The Abba Tribute Scottish Rite Auditorium - Collingswood - Collingswood, NJ


This week - "Problems with the Humpback"?
The Washcycle was on top of this with his "Dr. BikeLock" segment.
Question: Dr. Gridlock -- I know that you endured months of questions and complaints about the Humpback Bridge project. I just wanted to point out what a fantastic job was done on this project now that it is finished. As a driver, cyclist, and pedestrian who uses this bridge, the before and after is like night and day. The bike/pedestrian path was so narrow before that it made for a harrowing dangerous experience. My only complaint is the 20 feet of asphalt that is missing to make a more direct approach for cyclists and walkers to the 14th Street Bridge from the Humpback Bridge. There is currently an awkward T junction with the Mt. Vernon Trail.
Answer - Robert Thomson: I'm glad you're generally pleased, given the big impact this project had on GW Parkway traffic over the past few years. But I'm interested in what hyou say about the junction. It's the first time I've heard that issue raised.


text:
Q: Google Maps issue with require optimizar I am loading google maps API with Requirejs, and it works fine but after using requirejs optimizer it raises error of Uncaught ReferenceError: google is not defined.
Here's my code
initMap = () ->
require ['async!https://maps.googleapis.com/maps/api/js?libraries=places&sensor=true'], (maps) =>
  mapOptions =
    center: {lat: 22.234376786, lng: 55.23243456456}
    zoom: 3
    scrollwheel: false
  debugger
  map = new google.maps.Map(document.getElementById('map-canvas'), mapOptions)

And this is the requirejs build.coffee
I've read almost all the questions here that are like mine but no one worked with me!.

redpajama_set_name: RedPajamaStackExchange