Mannheimer Swartling cares about privacy and protecting the personal data processed by the firm. All personal data is processed in accordance with applicable data protection legislation.
When parties such as clients or their representatives, business partners and consultants, foreign representatives, witnesses, counterparties, representatives or counsel of counterparties are in contact with us or occur in connection with our engagements, this entails personal data being provided to us or obtained by us. When individuals visit our offices we normally register their information in a visitor management system. Our support organisation also collects and processes personal data about contact persons of suppliers and other external parties. Below we describe how we collect, process and share personal data in such cases. Further down we also provide information about the rights of data subjects in relation to us as controller as well as our contact details.
Much of our communication takes place by telephone and email, which essentially always entails the processing of personal data. Emailing or phoning us generally means that personal data that can be linked to individuals are provided.
We collect the personal data provided to us in connection with our engagements or that are otherwise processed during the preparation or administration of an engagement. We primarily collect personal data directly from the individuals concerned. However during engagements we sometimes receive information about individuals involved without the information being provided directly from them. We may also supplement the personal data provided by obtaining information from private and public records and sources. Normally, there is no obligation to provide us with personal data. However, if we do not receive certain personal data, we will not be able to accept an engagement, since we will not be able to comply with our obligations, inter alia, to perform conflict of interest and money laundering checks. Our support organisation also collects and processes personal data when in contact with e.g. suppliers and other external parties.
The personal data we process may consist of contact details (e.g. name, title, work address, telephone number and email address), identification details (e.g. passport details and date of birth /ID number), as well as invoicing information (e.g. account number and tax details). In specific engagements the personal data may also comprise other information, e.g. course of events or other circumstances, or other information relevant to the engagement. In connection with visits at our offices, we normally register the visitor's name and employer (or equivalent).
What are the purposes of our processing of personal data?
We process personal data provided or obtained in connection with engagements so we can fulfil our obligations and safeguard our clients' interests, and also for administration in connection with engagements, as well as obligations which follow from law or the rules issued by the Swedish Bar Association. Before we accept engagements, we must also perform compulsory conflict of interest and money laundering checks.
Our support organisation also processes personal data so that we can manage and administrate our relationships with suppliers and other external parties, as well as visits to our offices.
We may also use personal data as a basis for our market and client analyses, business and methodology development, as well as for statistical purposes, risk management, and marketing purposes.
What is the legal basis for our processing?
In relation to information about clients who are private individuals, the legal basis for processing personal data is performance of the contract which governs the engagement. In relation to clients' representatives, business partners and consultants, foreign representatives, representatives and counsel of counterparties, etc., our processing of personal data is normally based on a balancing of interests. This entails that we consider it necessary to process the personal data for the purposes that concern our client's or, where applicable, our legitimate interests and that these outweigh any opposing interests or fundamental rights and freedoms.
The processing of personal data in connection with the conflict of interest and money laundering checks and archiving of documents after an engagement has been concluded are based on our duty to comply with our legal obligations (e.g. under accounting and anti-money laundering legislation, as well as the Swedish Bar Association Code of Conduct).
We may have additional grounds for the processing in connection with the various engagements we have accepted.
Processing of personal data relating to suppliers or their representatives and other external parties is based on our legitimate interest in administrating the relationship and performing our contractual obligations.
In relation to the information registered in connection with visits at our offices the legal basis for our processing is – in addition to the fact that the processing may be covered by the preceding paragraphs – our legitimate interest in managing the visits, and/or our duty to comply with our legal obligations.
When we process personal data in order to analyse and develop our business, and for marketing communication, processing is based on our legitimate interest in improving and marketing our business.
Who has access to the personal data that we process?
We employ appropriate technical and organisational security measures to help protect the personal data we process from loss and to guard against, inter alia, access from unauthorised persons.
Transfers of data outside the EU/EEA are made in line with applicable data protection laws and for the purposes specified above. When personal data is transferred to the firm's international offices, the data may be transferred to countries outside the EU/EEA. Transfers of this type are normally based on the EU Commission's standard contractual clauses. Transfers to countries outside the EU/EEA may also occur within the scope of a given engagement insofar as is necessary to establish, exercise or defend our client's legal claims.
(v) it is otherwise permitted by law.
We do not save the personal data longer than necessary given the purpose of the processing, unless otherwise required or permitted by law.
The personal data that may be processed before and during the performance of an engagement are subsequently saved in accordance with Mannheimer Swartling's obligations under the Swedish Bar Association Code of Conduct after the engagement has been concluded. This means that the personal data are saved for at least ten years from and including the date the engagement was concluded, or for a longer period as required by the nature of the engagement.
Anyone who does not wish to receive invitations to our events or marketing material from us can unregister by contacting us at dataprotection@msa.se.
Mannheimer Swartling Advokatbyrå AB, Reg. No. 556399-4499, having the address Norrlandsgatan 21, SE-111 87, Stockholm, is the controller of the personal data processing as described above. This means that we are responsible for ensuring that the personal data are processed correctly and in accordance with applicable data protection laws.
Data subjects have – unless this is prevented by the duty of confidentiality set by the Swedish Bar Association – the right to know what personal data we process about them. A data subject also has the right to request that we rectify or erase inaccurate or incomplete personal data about them (e.g. if the personal data are no longer needed for the purpose or if the consent is withdrawn). Data subjects are also entitled to object to specific processing of personal data and request that processing of personal data be restricted. Finally, data subjects have the right to receive, in machine-readable format, personal data they have provided, and have the data transferred to another party responsible for data processing.
Note that the above rights may be limited by the duty of confidentiality and archiving obligation applying to members of the Swedish Bar Association. Restriction or erasure of personal data may mean that we are unable to meet our commitments.
Anyone who is dissatisfied with how we process their personal data is entitled to report this to the Swedish Data Inspection Board (Sw. Datainspektionen), which is the supervisory authority for our processing of personal data.
If you have any questions or complaints about how we process your personal data or wish to exercise any of your rights set out above, you are welcome to contact us by email at dataprotection@msa.se or by post to the address above.


text:
Commuters to face travel misery after union members vote for rail strikes
COMMUTERS in Times Territory are set to face travel misery after members of a leading rail union voted to go on strike over possible job cuts. The National Union of Rail, Maritime and Transport Workers (RMT)
COMMUTERS in Times Territory are set to face travel misery after members of a leading rail union voted to go on strike over possible job cuts.
The National Union of Rail, Maritime and Transport Workers (RMT) yesterday (Tuesday) announced its members at train company First Capital Connect (FCC) had voted by more than three to one in favour of industrial action.
Workers at National Express East Anglia have also voted to go on strike.
A date for any walk-out has yet to be confirmed but it is expected to take place within the next month.
RMT general secretary Bob Crow said: "Our members at First Capital Connect and National Express East Anglia have struck a blow against selfish employers who are seeking to slash jobs solely in order to maintain profits and dividends.
"The RMT executive will meet shortly to decide on the shape of action our members will take if the employers fail to withdraw these damaging and completely unnecessary cuts.
9 Two people rescued after flood warnings issued
10 The latest court results for Welwyn Hatfield and Potters Bar

redpajama_set_name: RedPajamaCommonCrawl


Country Style Furnishings Country Style Furnishings Country Style Living Room Furniture With. Country Style Furnishings The French Country Style 33 Furnishing Examples. Country Style Furnishings Country Style Home Furnishings Home Style. Country Style Furnishings Country Style Furnishings Country Style Dining Room Furniture Living. Country Style Furnishings Country Style Chair French Country Dining Tables And Chairs Stunning. Country Style Furnishings What Is French Country Style? Home, Furniture, Furnishings, French. Country Style Furnishings Country Style Furnishings Classic English Country Style Decor Ideas.


text:
\section{Background and motivation}
\label{background}

Submission to ICML 2022 will be entirely electronic, via a web site
(not email). Information about the submission process and \LaTeX\ templates
are available on the conference web site at:
\begin{center}
\textbf{\texttt{http://icml.cc/}}
\end{center}

The guidelines below will be enforced for initial submissions and
camera-ready copies. Here is a brief summary:
\begin{itemize}
\item Submissions must be in PDF\@.
\item \textbf{New to this year}: If your paper has appendices, submit the appendix together with the main body and the references \textbf{as a single file}. Reviewers will not look for appendices as a separate PDF file. So if you submit such an extra file, reviewers will very likely miss it.
\item Page limit: The main body of the paper has to be fitted to 8 pages, excluding references and appendices; the space for the latter two is not limited. For the final version of the paper, authors can add one extra page to the main body.
\item \textbf{Do not include author information or acknowledgements} in your
    initial submission.
\item Your paper should be in \textbf{10 point Times font}.
\item Make sure your PDF file only uses Type-1 fonts.
\item Place figure captions \emph{under} the figure (and omit titles from inside
    the graphic file itself). Place table captions \emph{over} the table.
\item References must include page numbers whenever possible and be as complete
    as possible. Place multiple citations in chronological order.
\item Do not alter the style template; in particular, do not compress the paper
    format by reducing the vertical spaces.
\item Keep your abstract brief and self-contained, one paragraph and roughly
    4--6 sentences. Gross violations will require correction at the
    camera-ready phase. The title should have content words capitalized.
\end{itemize}

\subsection{Submitting Papers}

\textbf{Paper Deadline:} The deadline for paper submission that is
advertised on the conference website is strict. If your full,
anonymized, submission does not reach us on time, it will not be
considered for publication.

\textbf{Anonymous Submission:} ICML uses double-blind review: no identifying
author information may appear on the title page or in the paper
itself. \cref{author info} gives further details.

\textbf{Simultaneous Submission:} ICML will not accept any paper which,
at the time of submission, is under review for another conference or
has already been published. This policy also applies to papers that
overlap substantially in technical content with conference papers
under review or previously published. ICML submissions must not be
submitted to other conferences and journals during ICML's review
period.
Informal publications, such as technical
reports or papers in workshop proceedings which do not appear in
print, do not fall under these restrictions.

\medskip

Authors must provide their manuscripts in \textbf{PDF} format.
Furthermore, please make sure that files contain only embedded Type-1 fonts
(e.g.,~using the program \texttt{pdffonts} in linux or using
File/DocumentProperties/Fonts in Acrobat). Other fonts (like Type-3)
might come from graphics files imported into the document.

Authors using \textbf{Word} must convert their document to PDF\@. Most
of the latest versions of Word have the facility to do this
automatically. Submissions will not be accepted in Word format or any
format other than PDF\@. Really. We're not joking. Don't send Word.

Those who use \textbf{\LaTeX} should avoid including Type-3 fonts.
Those using \texttt{latex} and \texttt{dvips} may need the following
two commands:

{\footnotesize
\begin{verbatim}
dvips -Ppdf -tletter -G0 -o paper.ps paper.dvi
ps2pdf paper.ps
\end{verbatim}}
It is a zero following the ``-G'', which tells dvips to use
the config.pdf file. Newer \TeX\ distributions don't always need this
option.

Using \texttt{pdflatex} rather than \texttt{latex}, often gives better
results. This program avoids the Type-3 font problem, and supports more
advanced features in the \texttt{microtype} package.

\textbf{Graphics files} should be a reasonable size, and included from
an appropriate format. Use vector formats (.eps/.pdf) for plots,
lossless bitmap formats (.png) for raster graphics with sharp lines, and
jpeg for photo-like images.

The style file uses the \texttt{hyperref} package to make clickable
links in documents. If this causes problems for you, add
\texttt{nohyperref} as one of the options to the \texttt{icml2022}
usepackage statement.

\subsection{Submitting Final Camera-Ready Copy}

The final versions of papers accepted for publication should follow the
same format and naming convention as initial submissions, except that
author information (names and affiliations) should be given. See
\cref{final author} for formatting instructions.

The footnote, ``Preliminary work. Under review by the International
Conference on Machine Learning (ICML). Do not distribute.'' must be
modified to ``\textit{Proceedings of the
$\mathit{39}^{th}$ International Conference on Machine Learning},
Baltimore, Maryland, USA, PMLR 162, 2022.
Copyright 2022 by the author(s).''

For those using the \textbf{\LaTeX} style file, this change (and others) is
handled automatically by simply changing
$\mathtt{\backslash usepackage\{icml2022\}}$ to
$$\mathtt{\backslash usepackage[accepted]\{icml2022\}}$$
Authors using \textbf{Word} must edit the
footnote on the first page of the document themselves.

Camera-ready copies should have the title of the paper as running head
on each page except the first one. The running title consists of a
single line centered above a horizontal rule which is $1$~point thick.
The running head should be centered, bold and in $9$~point type. The
rule should be $10$~points above the main text. For those using the
\textbf{\LaTeX} style file, the original title is automatically set as running
head using the \texttt{fancyhdr} package which is included in the ICML
2022 style file package. In case that the original title exceeds the
size restrictions, a shorter form can be supplied by using

\verb|\icmltitlerunning{...}|

just before $\mathtt{\backslash begin\{document\}}$.
Authors using \textbf{Word} must edit the header of the document themselves.

\section{Format of the Paper}

All submissions must follow the specified format.

\subsection{Dimensions}

The text of the paper should be formatted in two columns, with an
overall width of 6.75~inches, height of 9.0~inches, and 0.25~inches
between the columns. The left margin should be 0.75~inches and the top
margin 1.0~inch (2.54~cm). The right and bottom margins will depend on
whether you print on US letter or A4 paper, but all final versions
must be produced for US letter size.
Do not write anything on the margins.

The paper body should be set in 10~point type with a vertical spacing
of 11~points. Please use Times typeface throughout the text.

\subsection{Title}

The paper title should be set in 14~point bold type and centered
between two horizontal rules that are 1~point thick, with 1.0~inch
between the top rule and the top edge of the page. Capitalize the
first letter of content words and put the rest of the title in lower
case.

\subsection{Author Information for Submission}
\label{author info}

ICML uses double-blind review, so author information must not appear. If
you are using \LaTeX\/ and the \texttt{icml2022.sty} file, use
\verb+\icmlauthor{...}+ to specify authors and \verb+\icmlaffiliation{...}+ to specify affiliations. (Read the TeX code used to produce this document for an example usage.) The author information
will not be printed unless \texttt{accepted} is passed as an argument to the
style file.
Submissions that include the author information will not
be reviewed.

\subsubsection{Self-Citations}

If you are citing published papers for which you are an author, refer
to yourself in the third person. In particular, do not use phrases
that reveal your identity (e.g., ``in previous work \cite{langley00}, we
have shown \ldots'').

Do not anonymize citations in the reference section. The only exception are manuscripts that are
not yet published (e.g., under submission). If you choose to refer to
such unpublished manuscripts \cite{anonymous}, anonymized copies have
to be submitted
as Supplementary Material via CMT\@. However, keep in mind that an ICML
paper should be self contained and should contain sufficient detail
for the reviewers to evaluate the work. In particular, reviewers are
not required to look at the Supplementary Material when writing their
review (they are not required to look at more than the first $8$ pages of the submitted document).

\subsubsection{Camera-Ready Author Information}
\label{final author}

If a paper is accepted, a final camera-ready copy must be prepared.
For camera-ready papers, author information should start 0.3~inches below the
bottom rule surrounding the title. The authors' names should appear in 10~point
bold type, in a row, separated by white space, and centered. Author names should
not be broken across lines. Unbolded superscripted numbers, starting 1, should
be used to refer to affiliations.

Affiliations should be numbered in the order of appearance. A single footnote
block of text should be used to list all the affiliations. (Academic
affiliations should list Department, University, City, State/Region, Country.
Similarly for industrial affiliations.)

Each distinct affiliations should be listed once. If an author has multiple
affiliations, multiple superscripts should be placed after the name, separated
by thin spaces. If the authors would like to highlight equal contribution by
multiple first authors, those authors should have an asterisk placed after their
name in superscript, and the term ``\textsuperscript{*}Equal contribution"
should be placed in the footnote block ahead of the list of affiliations. A
list of corresponding authors and their emails (in the format Full Name
\textless{}email@domain.com\textgreater{}) can follow the list of affiliations.
Ideally only one or two names should be listed.

A sample file with author names is included in the ICML2022 style file
package. Turn on the \texttt{[accepted]} option to the stylefile to
see the names rendered. All of the guidelines above are implemented
by the \LaTeX\ style file.

\subsection{Abstract}

The paper abstract should begin in the left column, 0.4~inches below the final
address. The heading `Abstract' should be centered, bold, and in 11~point type.
The abstract body should use 10~point type, with a vertical spacing of
11~points, and should be indented 0.25~inches more than normal on left-hand and
right-hand margins. Insert 0.4~inches of blank space after the body. Keep your
abstract brief and self-contained, limiting it to one paragraph and roughly 4--6
sentences. Gross violations will require correction at the camera-ready phase.

\subsection{Partitioning the Text}

You should organize your paper into sections and paragraphs to help
readers place a structure on the material and understand its
contributions.

\subsubsection{Sections and Subsections}

Section headings should be numbered, flush left, and set in 11~pt bold
type with the content words capitalized. Leave 0.25~inches of space
before the heading and 0.15~inches after the heading.

Similarly, subsection headings should be numbered, flush left, and set
in 10~pt bold type with the content words capitalized. Leave
0.2~inches of space before the heading and 0.13~inches afterward.

Finally, subsubsection headings should be numbered, flush left, and
set in 10~pt small caps with the content words capitalized. Leave
0.18~inches of space before the heading and 0.1~inches after the
heading.

Please use no more than three levels of headings.

\subsubsection{Paragraphs and Footnotes}

Within each section or subsection, you should further partition the
paper into paragraphs. Do not indent the first line of a given
paragraph, but insert a blank line between succeeding ones.

You can use footnotes\footnote{Footnotes
should be complete sentences.} to provide readers with additional
information about a topic without interrupting the flow of the paper.
Indicate footnotes with a number in the text where the point is most
relevant. Place the footnote in 9~point type at the bottom of the
column in which it appears. Precede the first footnote in a column
with a horizontal rule of 0.8~inches.\footnote{Multiple footnotes can
appear in each column, in the same order as they appear in the text,
but spread them across columns and pages if possible.}

\begin{figure}[ht]
\vskip 0.2in
\begin{center}
\centerline{\includegraphics[width=\columnwidth]{icml_numpapers}}
\caption{Historical locations and number of accepted papers for International
Machine Learning Conferences (ICML 1993 -- ICML 2008) and International
Workshops on Machine Learning (ML 1988 -- ML 1992). At the time this figure was
produced, the number of accepted papers for ICML 2008 was unknown and instead
estimated.}
\label{icml-historical}
\end{center}
\vskip -0.2in
\end{figure}

\subsection{Figures}

You may want to include figures in the paper to illustrate
your approach and results. Such artwork should be centered,
legible, and separated from the text. Lines should be dark and at
least 0.5~points thick for purposes of reproduction, and text should
not appear on a gray background.

Label all distinct components of each figure. If the figure takes the
form of a graph, then give a name for each axis and include a legend
that briefly describes each curve. Do not include a title inside the
figure; instead, the caption should serve this function.

Number figures sequentially, placing the figure number and caption
\emph{after} the graphics, with at least 0.1~inches of space before
the caption and 0.1~inches after it, as in
\cref{icml-historical}. The figure caption should be set in
9~point type and centered unless it runs two or more lines, in which
case it should be flush left. You may float figures to the top or
bottom of a column, and you may set wide figures across both columns
(use the environment \texttt{figure*} in \LaTeX). Always place
two-column figures at the top or bottom of the page.

\subsection{Algorithms}

If you are using \LaTeX, please use the ``algorithm'' and ``algorithmic''
environments to format pseudocode. These require
the corresponding stylefiles, algorithm.sty and
algorithmic.sty, which are supplied with this package.
\cref{alg:example} shows an example.

\begin{algorithm}[tb]
   \caption{Bubble Sort}
   \label{alg:example}
\begin{algorithmic}
   \STATE {\bfseries Input:} data $x_i$, size $m$
   \REPEAT
   \STATE Initialize $noChange = true$.
   \FOR{$i=1$ {\bfseries to} $m-1$}
   \IF{$x_i > x_{i+1}$}
   \STATE Swap $x_i$ and $x_{i+1}$
   \STATE $noChange = false$
   \ENDIF
   \ENDFOR
   \UNTIL{$noChange$ is $true$}
\end{algorithmic}
\end{algorithm}

\subsection{Tables}

You may also want to include tables that summarize material. Like
figures, these should be centered, legible, and numbered consecutively.
However, place the title \emph{above} the table with at least
0.1~inches of space before the title and the same after it, as in
\cref{sample-table}. The table title should be set in 9~point
type and centered unless it runs two or more lines, in which case it
should be flush left.

\begin{table}[t]
\caption{Classification accuracies for naive Bayes and flexible
Bayes on various data sets.}
\label{sample-table}
\vskip 0.15in
\begin{center}
\begin{small}
\begin{sc}
\begin{tabular}{lcccr}
\toprule
Data set & Naive & Flexible & Better? \\
\midrule
Breast    & 95.9$\pm$ 0.2& 96.7$\pm$ 0.2& $\surd$ \\
Cleveland & 83.3$\pm$ 0.6& 80.0$\pm$ 0.6& $\times$\\
Glass2    & 61.9$\pm$ 1.4& 83.8$\pm$ 0.7& $\surd$ \\
Credit    & 74.8$\pm$ 0.5& 78.3$\pm$ 0.6&         \\
Horse     & 73.3$\pm$ 0.9& 69.7$\pm$ 1.0& $\times$\\
Meta      & 67.1$\pm$ 0.6& 76.5$\pm$ 0.5& $\surd$ \\
Pima      & 75.1$\pm$ 0.6& 73.9$\pm$ 0.5&         \\
Vehicle   & 44.9$\pm$ 0.6& 61.5$\pm$ 0.4& $\surd$ \\
\bottomrule
\end{tabular}
\end{sc}
\end{small}
\end{center}
\vskip -0.1in
\end{table}

Tables contain textual material, whereas figures contain graphical material.
Specify the contents of each row and column in the table's topmost
row. Again, you may float tables to a column's top or bottom, and set
wide tables across both columns. Place two-column tables at the
top or bottom of the page.

\subsection{Theorems and such}
The preferred way is to number definitions, propositions, lemmas, etc. consecutively, within sections, as shown below.
\begin{definition}
\label{def:inj}
A function $f:X \to Y$ is injective if for any $x,y\in X$ different, $f(x)\ne f(y)$.
\end{definition}
Using \cref{def:inj} we immediate get the following result:
\begin{proposition}
If $f$ is injective mapping a set $X$ to another set $Y$,
the cardinality of $Y$ is at least as large as that of $X$
\end{proposition}
\begin{proof}
Left as an exercise to the reader.
\end{proof}
\cref{lem:usefullemma} stated next will prove to be useful.
\begin{lemma}
\label{lem:usefullemma}
For any $f:X \to Y$ and $g:Y\to Z$ injective functions, $f \circ g$ is injective.
\end{lemma}
\begin{theorem}
\label{thm:bigtheorem}
If $f:X\to Y$ is bijective, the cardinality of $X$ and $Y$ are the same.
\end{theorem}
An easy corollary of \cref{thm:bigtheorem} is the following:
\begin{corollary}
If $f:X\to Y$ is bijective,
the cardinality of $X$ is at least as large as that of $Y$.
\end{corollary}
\begin{assumption}
The set $X$ is finite.
\label{ass:xfinite}
\end{assumption}
\begin{remark}
According to some, it is only the finite case (cf. \cref{ass:xfinite}) that is interesting.
\end{remark}

\subsection{Citations and References}

Please use APA reference format regardless of your formatter
or word processor. If you rely on the \LaTeX\/ bibliographic
facility, use \texttt{natbib.sty} and \texttt{icml2022.bst}
included in the style-file package to obtain this format.

Citations within the text should include the authors' last names and
year. If the authors' names are included in the sentence, place only
the year in parentheses, for example when referencing Arthur Samuel's
pioneering work \yrcite{Samuel59}. Otherwise place the entire
reference in parentheses with the authors and year separated by a
comma \cite{Samuel59}. List multiple references separated by
semicolons \cite{kearns89,Samuel59,mitchell80}. Use the `et~al.'
construct only for citations with three or more authors or after
listing all authors to a publication in an earlier reference \cite{MachineLearningI}.

Authors should cite their own work in the third person
in the initial version of their paper submitted for blind review.
Please refer to \cref{author info} for detailed instructions on how to
cite your own papers.

Use an unnumbered first-level section heading for the references, and use a
hanging indent style, with the first line of the reference flush against the
left margin and subsequent lines indented by 10 points. The references at the
end of this document give examples for journal articles \cite{Samuel59},
conference publications \cite{langley00}, book chapters \cite{Newell81}, books
\cite{DudaHart2nd}, edited volumes \cite{MachineLearningI}, technical reports
\cite{mitchell80}, and dissertations \cite{kearns89}.

Alphabetize references by the surnames of the first authors, with
single author entries preceding multiple author entries. Order
references for the same authors by year of publication, with the
earliest first. Make sure that each reference includes all relevant
information (e.g., page numbers).

Please put some effort into making references complete, presentable, and
consistent, e.g. use the actual current name of authors.
If using bibtex, please protect capital letters of names and
abbreviations in titles, for example, use \{B\}ayesian or \{L\}ipschitz
in your .bib file.

\section*{Accessibility}
Authors are kindly asked to make their submissions as accessible as possible for everyone including people with disabilities and sensory or neurological differences.
Tips of how to achieve this and what to pay attention to will be provided on the conference website \url{http://icml.cc/}.

\section*{Software and Data}

If a paper is accepted, we strongly encourage the publication of software and data with the
camera-ready version of the paper whenever appropriate. This can be
done by including a URL in the camera-ready copy. However, \textbf{do not}
include URLs that reveal your institution or identity in your
submission for review. Instead, provide an anonymous URL or upload
the material as ``Supplementary Material'' into the CMT reviewing
system. Note that reviewers are not required to look at this material
when writing their review.

\section*{Acknowledgements}

\textbf{Do not} include acknowledgements in the initial version of
the paper submitted for blind review.

If a paper is accepted, the final camera-ready version can (and
probably should) include acknowledgements. In this case, please
place such acknowledgements in an unnumbered section at the
end of the paper. Typically, this will include thanks to reviewers
who gave useful comments, to colleagues who contributed to the ideas,
and to funding agencies and corporate sponsors that provided financial
support.

\nocite{langley00}

\subsection{PRE-PROCESSING \& FEATURE ENGINEERING}
\textbf{\ul{Question 1: Did you perform any data pre-processing methods?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.8\textwidth]{fig/Q1.png}
    \caption{Pre-processing methods include: Standard Scaler, Feature Scaling, One-hot Encoding, extracting the 90\% convergence time and final performances.}
    \label{fig:my_label}
\end{figure}

\textbf{\ul{Question 2: Did you perform feature engineering methods?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.8\textwidth]{fig/Q2.png}
    \caption{Feature engineering methods include: Clustering; Polynomial features of train\_num, feat\_num; Multi-step forecasting for data augmentation.}
    \label{fig:my_label}
\end{figure}

\subsection{DATA USED FOR LEARNING}

\textbf{\ul{Question 3: Did you use all points on the learning curves or only some of them?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.9\textwidth]{fig/Q3.png}
    \caption{Most of the methods use all points on the learning curves for learning.}
    \label{fig:my_label}
\end{figure}

\textbf{\ul{Question 4: Did you make use of meta-features of datasets?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.65\textwidth]{fig/Q4.png}
    \caption{All methods took advantage of meta-features of datasets.}
    \label{fig:my_label}
\end{figure}

\textbf{\ul{Question 5: Did you implement a Hyperparameter Optimization component for your agent using the provided hyperparameters of algorithms?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.65\textwidth]{fig/Q5.png}
    \caption{Some HPO tools were used, such as Hyperband
and FSBO.}
    \label{fig:my_label}
\end{figure}

\textbf{\ul{Question 6: In case you used either or both meta-features of datasets and algorithms, did it improve the performance of your method?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=1\textwidth]{fig/Q6.png}
    \caption{More experiments need to be done to confirm whether meta-features of datasets and algorithms help.}
    \label{fig:my_label}
\end{figure}

\textbf{\ul{Question 7: In any case, did you find the meta-features useful in our meta-learning setting?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=1\textwidth]{fig/Q7.png}
    \caption{Not all participants find the provided meta-features useful.}
    \label{fig:my_label}
\end{figure}

\subsection{POLICY CHARACTERISTICS}

\textbf{\ul{Question 8: Does your agent learn a policy from datasets in the meta-training phase?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{fig/Q8.png}
    \caption{Most of the agents use a learned policy from the meta-training phase.}
    \label{fig:my_label}
\end{figure}

\ul{\textbf{Question 9: How does your agent manage the exploration-exploitation trade-offs (revealing a known good algorithm's learning curve vs. revealing a new algorithm candidate's learning curve )?}}

\begin{enumerate}
    \item With an $\epsilon$ greedy policy, in a Reinforcement Learning framework, only in meta- training we create different Q-matrices. In the meta-testing phase, we perform the choice of the new algorithm with the computed Q-matrices.

    \item We are very restrictive with switching the explored algorrithm. We preselect the single best performing algorithm from the validation learning curves statically and only explore other algorithms, if its learning curve is stale. So we strongly emphasize exploiting the single best algorithm.

    \item A modified Round Robin on the top-k performing algorithms. The incumbent i.e. the value of the top performing algorithm on the test dataset is challenged by zero budget allocated algorithms. Since the training on the validation data allows for immediate look up in the test dataset.

    \item Bayesian Optimization

    \item We find the best algorithm by a learning curve predictor.
\end{enumerate}

\ul{\textbf{Question 10: Does your agent switch between learning curves during an episode (i.e. switching between training different algorithms on the dataset at hand)?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.9\textwidth]{fig/Q10.png}
    \caption{All agents switch between learning curves to find the best algorithm for the task at hand.}
    \label{fig:my_label}
\end{figure}

\ul{\textbf{Question 11: Does your agent leverage partially revealed learning curves on the dataset at hand?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.8\textwidth]{fig/Q11.png}
    \caption{Some agents do not take into account information of partial learning curves on the task at hand for deciding their actions.}
    \label{fig:my_label}
\end{figure}

\ul{\textbf{Question 12: Did you make any assumptions about the shapes of the learning curves?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{fig/Q12.png}
    \caption{Only one team made an assumption that the learning curves are \textit{monotonically increasing}.}
    \label{fig:my_label}
\end{figure}

\ul{\textbf{Question 13: Does your agent predict unseen points on a learning curve?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{fig/Q13.png}
    \caption{3 out of 5 agents make decisions based on predicted performance scores.}
    \label{fig:my_label}
\end{figure}

\ul{\textbf{Question 14: Does your agent perform pairwise comparisons of algorithms' learning curves?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=1\textwidth]{fig/Q14.png}
    \caption{Pairwise comparisons of algorithms' learning curves have been exploited in 60\% of the agents.}
    \label{fig:my_label}
\end{figure}

\ul{\textbf{Question 15: How does your agent spend the given time budgets (i.e. choosing delta\_t at each step)?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=1\textwidth]{fig/Q15.png}
    \caption{Participants use either or both hard-coded policy and learned policy to distribute a given time budget (no one does it randomly).}
    \label{fig:my_label}
\end{figure}

\ul{\textbf{Question 16: How does your agent choose which algorithm to contribute to its learning curve at each step (i.e. choosing A\_star)?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=1\textwidth]{fig/Q16.png}
    \caption{Most of the agents choose the best algorithm so far as $A^*$, with only one exception that uses Sequential Model Based Optimisation (SMBO) with a Gaussian Process Surrogate.}
    \label{fig:my_label}
\end{figure}

\ul{\textbf{Question 17: Which phase did you focus on more to improve your agent's performance?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.9\textwidth]{fig/Q17.png}
    \caption{Participants give a slightly higher importance weight to meta-training than meta-testing.}
    \label{fig:my_label}
\end{figure}

\ul{\textbf{Question 18: Did you build an algorithm ranking?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{fig/Q18.png}
    \caption{More than half of the participants build an algorithm ranking and use it as a tool for selecting algorithms.}
    \label{fig:my_label}
\end{figure}

\ul{\textbf{Question 19: Did you use Reinforcement Learning to train your agent?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{fig/Q19.png}
    \caption{Only one participant applies Reinforcement Learning to train the agent.}
    \label{fig:my_label}
\end{figure}

\subsection{METHOD IMPLEMENTATION}

\ul{\textbf{Question 20: What is the percentage of originality of your method/implementation?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=1\textwidth]{fig/Q20.png}
    \caption{The percentage of originality of their methods/implementations ranges from 60\% to 80\%.}
    \label{fig:my_label}
\end{figure}

\ul{\textbf{Question 21: Did you use a pre-trained agent?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{fig/Q21.png}
    \caption{Only one participant pre-train their agent using all the provided datasets in meta-training.}
    \label{fig:my_label}
\end{figure}

\ul{\textbf{Question 22: Is your method strongly based on an existing solution? Which one(s)?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=1\textwidth]{fig/Q22.png}
    \caption{Q-learning and starting kit baselines served as bases for 2 methods, while the other methods were built from scratch.}
    \label{fig:my_label}
\end{figure}

\ul{\textbf{Question 23: Did you use Neural Networks for your agent?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{fig/Q23.png}
    \caption{Neural Networks were implemented in 3 out of 5 methods.}
    \label{fig:my_label}
\end{figure}

\ul{\textbf{Question 24: Do you find the provided libraries / packages / frameworks sufficient?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{fig/Q24.png}
    \caption{The provided libraries/packages/frameworks were sufficient for most of the participants.}
    \label{fig:my_label}
\end{figure}

\ul{\textbf{Question 25: Check all Python packages/frameworks you used.}}
\begin{figure}[H]
    \centering
    \includegraphics[width=1\textwidth]{fig/Q25.png}
    \caption{Scikit-learn and Pytorch are the most used packages by participants.}
    \label{fig:my_label}
\end{figure}

\ul{\textbf{Question 26: Did you use any specific AutoML / Meta-learning / Hyperparameter Optimization libraries?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{fig/Q26.png}
    \caption{Only one participant uses SMAC for  hyperparameter optimization.}
    \label{fig:my_label}
\end{figure}

\ul{\textbf{Question 27: Was it difficult for you to deal with the provided data format of the learning curves and meta-features?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.8\textwidth]{fig/Q27.png}
    \caption{40\% of participants struggled with the provided data format. Some comments include: rather than nested dictionaries create a single dictionary with a tuple identifier (dataset, algorithm).}
    \label{fig:my_label}
\end{figure}

\ul{\textbf{Question 28: How much time did you spend developing your agents?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.9\textwidth]{fig/Q28.png}
    \caption{All participants completed their solutions within 4 weeks.}
    \label{fig:my_label}
\end{figure}

\ul{\textbf{Question 29: What's the difficulty induced by the computation resource (memory, time budget, etc) constraints?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.8\textwidth]{fig/Q29.png}
    \caption{The provided computation resource was reasonable to participants.}
    \label{fig:my_label}
\end{figure}

\subsection{USER EXPERIENCE}

\ul{\textbf{Question 30: Was the challenge duration enough for you to develop your methods?}}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.9\textwidth]{fig/Q30.png}
    \caption{The challenge duration was enough for the participants.}
    \label{fig:my_label}
\end{figure}

\ul{\textbf{Question 31: Your evaluation on the starting kit}}
\begin{figure}[H]
    \centering
    \includegraphics[width=1\textwidth]{fig/Q31.png}
    \caption{Some improvements on the starting kit need to be made (see below).}
    \label{fig:my_label}
\end{figure}

\ul{\textbf{Question 32: Which improvements can be made for the starting kit? You are welcome to list all issues and bugs you ran into.}}
\begin{enumerate}
    \item The algorithm meta features were non-informative, the dataset meta features' categorical columns at some point were non informative and the categories were not entirely represented in the validation dataset. Also the environment allows to exploit 0 budgets to improve the agent's performance in a non realistic way. Also we wanted to use many libraries, that were not available (e.g pytorch geometric or networkx) or with other versions (e.g. sklearn's quantile regression) was not available initially. The validation \& test set (on server) were slightly different. In particular, the distribution of timestamps was dramatically different.

    \item Return the whole learning curve after a suggestion, not only the last observed budget and the last observed performance.

    \item It may be useful to query the learning curve at specific iterations or explain clearly the meaning of the timestamps of the observed samples. If they are random or there is not an underlying pattern or structure, it is difficult to predict the next budget.

    \item The final rank should be obtained by running the methods on other meta-train dataset. In the current set-up, the test curve is highly correlated to the validation curve (seen in the development stage), therefore simply overfitting the latter will probably help to obtain good results in the former.

    \item Decreasing the budget when the suggested algorithm+budget tuple is not enough to query a new point in the learning curve is unrealistic. In the real world, once we decide to run some algorithm for a given time-range or number of epochs, we will obtain some change in the accuracy (unless the learning curve is in a plateau).
\end{enumerate}

\ul{\textbf{Question 33: Your evaluation on the challenge website:\\ https://codalab.lisn.upsaclay.fr/competitions/753}}
\begin{figure}[H]
    \centering
    \includegraphics[width=1\textwidth]{fig/Q33.png}
    \caption{One comment for the challenge website is: the way the ALC is explained is not very straightforward.}
    \label{fig:my_label}
\end{figure}
\section{Background and motivation}
\label{background}
\textbf{Meta-learning} has been playing an increasingly important role in Automated Machine Learning.
While it is a natural capability of living organisms, who constantly transfer acquired knowledge across tasks to quickly adapt to changing environments, artificial learning systems are still in their meta-learning "infancy". They are only capable, so far, to transfer knowledge between very similar tasks. At a time when society is pointing fingers at AI for being wasteful with computational resources, there is an urgent need for learning systems, which \textbf{recycle their knowledge}. To achieve that goal, some research areas have been widely studied, including few-shot learning \cite{Wang2020}, transfer learning \cite{Zhuang2020}, representation learning \cite{Bengio2013}, continual learning \cite{Delange2021}, life-long learning \cite{Chen2018}, and meta-learning \cite{Vanschoren2018}.
However, \textbf{meta-learning from learning curves}, an essential sub-problem in meta-learning \cite{mohr2022learning}, is still under-studied. This motivates the design of this new challenge series we are proposing.

Learning curves evaluate algorithm incremental performance improvement, as a function of training time, number of iterations, and/or number of examples. Our challenge design builds on top of previous challenges and work, which considered other aspects of the problem: meta-learning as a recommendation problem, but not from learning curves \cite{Guyon2019, liu_winning_2020} and few-shot learning \cite{metadl2021}. Analysis of past challenges revealed that top-ranking methods often involve switching between algorithms during training, including "freeze-thaw" Bayesian techniques \cite{Swersky2014}. However, previous challenge protocols did not allow evaluating such methods separately, due to inter-dependencies between various heuristics. Furthermore, we want to study the potential benefit of \textbf{learned policies}, as opposed to applying hand-crafted black-box optimization methods.

Our challenge took inspiration from MetaREVEAL {\cite{nguyen2021}}, ActivMetal \cite{SunHosoya2018}, REVEAL \cite{sunhosoya2019}, and Freeze-Thaw Bayesian Optimization \cite{Swersky2014}. A meta-learner needs to learn to solve two problems at the same time: \textbf{algorithm selection} and \textbf{budget allocation}. We are interested in meta-learning strategies that leverage information on \textbf{partially trained algorithms}, hence reducing the cost of training them to convergence. We offer pre-computed learning curves as a function of time, to facilitate benchmarking. Meta-learners must "pay" a cost emulating computational time for revealing their next values. Hence, meta-learners are expected to learn the \textbf{exploration-exploitation trade-offs} between continuing "training" an already tried good candidate algorithm and checking new candidate algorithms.

In this paper, we first recap design of the first round of our challenge. Then, we present the first round results and compare the participants' solutions with our baselines. After the end of the first round, we have identified some limitations in our design and proposed a new design and a new meta-dataset for the second round.

\section {Design of the first round}
In this section, we describe the design of the first round of our challenge, including: data, challenge protocol, and evaluation. More details can be found on our competition page \footnote{{\url{https://codalab.lisn.upsaclay.fr/competitions/753}}}.

\subsection{Learning curve data}
Despite the fact that learning curves are widely used, there were not many learning curve meta-datasets available in the Machine Learning community at the time of organizing this challenge. Taking advantage of 30 cross-domain datasets used in the {AutoML challenge \cite{Guyon2019}}, we computed {\em de novo} learning curves (both on the validation sets and the test sets) for 20 algorithms, by submitting them to the {AutoML challenge \cite{Guyon2019}}\ as post-challenge submissions. These algorithms are created from two base algorithms, Random Forest and Gradient Boosting, but with different values of the \textit{max\_features} hyperparameter.
We also provided meta-features of each dataset, such as type of learning task, evaluation metric, time budget, etc. In this first round, we considered a \textbf{learning curve as a function of time}. Each point on the learning curve corresponds to the algorithm performance at a certain time.

In addition to the aforementioned real-world meta-dataset, we synthetically generated 4000 learning curves for the participants to practice and get familiar with our starting kit. The points on each synthetic learning curve are sampled from a parameterized sigmoid function with its hyperparameters generated from matrix factorizations. This allows us to create some hidden relationships between algorithms and datasets (i.e. some algorithms perform well particularly for some datasets). Details of how this synthetic meta-dataset was created can be found in {\cite{nguyen2021}}.

\subsection{Challenge protocol}

We organized a novel two-phase competition protocol:
\begin{itemize}
    \setlength{\itemsep}{0pt}%
    \setlength{\parskip}{0pt}
    \item \textbf{Development phase}: participants make as many submissions as they want \footnote{up to a comfortable limit that was never reached in our competition. The limit also aims to avoid overfitting the validation sets, though it has been shown that participants are usually aware not to overfit them \cite{overfitting-kaggle}},

    which are evaluated on the \textit{validation learning curves}.

    \item \textbf{Final test phase}: no new submissions are accepted in this phase. The last submission of each participant in the Development phase is transferred automatically to this phase. It is evaluated on the \textit{test learning curves}.
\end{itemize}

Note that the meta-datasets are never exposed to the participants in neither phase, because this is a challenge with code submission (only the participants' agent sees the data).

This setting is novel because it is uncommon in reinforcement learning to have a separate phase for agent ``development'' and agent ``testing''. Validation learning curves are used during the development phase and test learning curves during the final phase, to prevent agent from overfitting. Moreover, we implemented the k-fold meta-cross-validation (with k=6) to reduce variance in the evaluations of the agents (i.e. 25 datasets for meta-training and 5 datasets for meta-testing). The final results are averaged over datasets in the test folds.

During meta-training, learning curves and meta-data collected on 25 datasets are passed to the agent for meta-learning in any possible ways implemented by the agent. Then during meta-testing, one dataset is presented to the agent at a time. The agent interacts back and forth with an ``environment'', similarly to a Reinforcement Learning setting. It keeps suggesting to reveal algorithms' validation learning curves and choosing the current best performing algorithm based on observations of the partially revealed learning curves.

\subsection{Evaluation}
In this challenge series, we want to search for agents with high ``any-time learning" capacities, which means the ability to have good performances if they were to be stopped at any point in time. Hence, the agent is evaluated by the Area under the agents' Learning Curve (ALC) which is constructed using the learning curves of the best algorithms chosen at each time step (validation learning curves in the Development phase, and the test learning curves in the Final phase). The computation of the ALC is explained in {\cite{nguyen2021}}. The results will be averaged over all meta-test datasets and shown on the leaderboards. The final ranking is made according to the average test ALC.

As indicated in the competition rules, participants should make efforts to guarantee the reproducibility of their methods (e.g. by fixing all random seeds involved). In the Final Phase, all submissions were run \textbf{three times} with the same seed, and the run with the \textbf{worst performance} is used for the final ranking\footnote{The output of each run can be found in this Google Drive folder: {\href{https://drive.google.com/drive/folders/1HrMZNQeR1kDOmLqjRGgcozut2M_k-jHs?usp=sharing}{[link]}}}. This penalizes any team who did not fix their own seeds.

On each dataset $\mathcal{D}_i$, we evaluated an agent $\mathcal{A}_j$ by the Area under the Learning curve $ALC_i^j$ of the agent on the dataset. In the final ranking, the agent is ranked based on its average ALC over all datasets ($\mathcal{N}=30$ datasets):

\begin{equation}
    \mu_j =  \frac{\sum_{i=1}^{\mathcal{N}} ALC_i^j}{\mathcal{N}}
\end{equation}

To measure the variability of an agent $\mathcal{A}_j$, we computed the standard deviation of ALC scores obtained by the agent over all datasets:

\begin{equation}
    \sigma_j = \sqrt{\frac{\sum_{i=1}^{\mathcal{N}} (ALC_i^j - \mu_j)^2}{\mathcal{N}}}
\end{equation}

\section{Analyses of the first round results}
Results, final rankings, and prizes in the Final phase of the first round are shown in Table \ref{table:summary}. The 1st, 2nd, and 3rd ranked teams qualifying for prizes, as per the challenge rules\footnote{{\url{https://codalab.lisn.upsaclay.fr/competitions/753}}}, were awarded prizes of 500\$, 300\$, and 200\$, respectively. In the remainder of this section, we give a more in-depth view of how each team performed on each dataset in this round, compared to our baselines.

In this round, the participants are asked to solve two tasks simultaneously: \textit{algorithm selection} and \textit{budget allocation}. Both of them are crucial to achieving our goal of maximizing the area under an agent's learning curve. We found approaches submitted by the participants using a wide range of methods, from simple (e.g. using algorithm ranking and pre-defined values for $\Delta t$) to more sophisticated (e.g. predicting scores and timestamps of unseen learning curve points). The results first indicate that using both learned policies (models) for choosing algorithms and spending time budget (used by team \textit{{MoRiHa}} and \textit{{neptune}}) yields better ALC scores than hard-coded ones (e.g. using a fixed pre-defined list of $\Delta t$ in \textit{{AIpert}} and our \textit{DDQN} baseline).

According to Table \ref{table:result-details}, team \textit{{MoRiHa}} obtained the highest average ALC of 0.43 in the final phase.
It succeeded in achieving the highest ALC score on 21 out of 30 datasets. In addition, it performed notably better than other teams in some datasets, such as \textit{tania}, \textit{robert}, \textit{newsgroups}, and \textit{marco}. They are datasets of either multi-label or multi-class classification tasks with a very high number of features. Moreover, most of them are sparse datasets, which is often seen as a challenge for learners. Further investigation will be done in our future work to understand the outperformance of {MoRiHa}'s method on these particular datasets. We will also encourage team {MoRiHa} to join the second round to study the robutness of their results.
Team \textit{{neptune}} has a score of 0.42 which is very close to the winner's, followed by team \textit{{AIpert}} with a score of 0.40. Team \textit{{automl-freiburg}}, which was ranked 4th, achieved a slightly lower score (0.37) than our DDQN baseline (0.38), and so did team \textit{{automl-hannover}} (0.32).

The successes of the top-ranked team can be explained by the strategies they implemented. Team \textit{{MoRiHa}}, which finished in the 1st place, uses a simple yet efficient approach that explores the most promising algorithms and avoids wasting time switching between too many different algorithms. Interestingly, team \textit{{neptune}} learns a policy for allocating time budget using learning curve convergence speed. The Reinforcement Learning-based approach of team \textit{{AIpert}} is very intuitive as our competition borrows the RL framework to formulate our problem, which also explains our baseline choice of DDQN. However, by complementing it with the K-means clustering method, \textit{{AIpert}}'s approach achieved higher performance than our baseline. Both \textit{{AIpert}} and \textit{{automl-freiburg}} share the same idea of suggesting algorithms based on dataset similarity.

\begin{table*}[]
\centering
\caption{\textbf{Final phase ranking of the first round}. Teams are ranked based on their average ALC scores, which were recorded on the worst run among three runs. Team \textit{{MoRiHa}} was ranked 1st, but not qualified for a monetary prize. Teams \textit{{neptune}}, \textit{{AIpert}}, and \textit{{automl-freiburg}} are qualified for the prizes.}
\label{table:summary}
\begin{tabular}{|c|c|c|c|l|}
\hline
\rowcolor[HTML]{FFFFFF}
\textbf{Rank} & \textbf{\begin{tabular}[c]{@{}c@{}}Team\\ (username)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}ALC \\ score\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}Monetary Prize\\ qualification\end{tabular}} & \multicolumn{1}{c|}{\cellcolor[HTML]{FFFFFF}\textbf{\begin{tabular}[c]{@{}c@{}}Comments/\\source code\end{tabular}}}                                                                       \\ \hline
\rowcolor[HTML]{FFFFFF}
1             & \begin{tabular}[c]{@{}c@{}}\textbf{{MoRiHa}}\\ (username: jmhansel)\end{tabular}                                                  & 0.43                                                          &   NO            & \begin{tabular}[c]{@{}l@{}}This team is not qualified for\\ a monetary prize due to close\\ relation with the organizers\\ {\href{https://github.com/fmohr/meta-learn-from-LC-2022}{[CODE URL]} \href{https://drive.google.com/file/d/1CdLOEuM-7C94WdTSwHHgelGAOfZ-xCNW/view?usp=sharing}{[FACTSHEET]}} \end{tabular} \\ \hline
\rowcolor[HTML]{FFFFFF}
2             & \begin{tabular}[c]{@{}c@{}}\textbf{{neptune}}\\ (username: {neptune})\end{tabular}                                                   & 0.42                                                          & OK         &  \begin{tabular}[c]{@{}l@{}} {\href{https://github.com/neptuneai/MetaDL}{[CODE URL]} \href{https://drive.google.com/file/d/1eAI0GD-0cYwI3XND23pkX5t0nvWVRy6t/view?usp=sharing}{[FACTSHEET]}} \end{tabular}  \\ \hline
\rowcolor[HTML]{FFFFFF}
3             & \begin{tabular}[c]{@{}c@{}}\textbf{{AIpert}}\\ (username: {AIpert})\end{tabular}                                                     & 0.40                                                          & OK         &   {\href{https://github.com/EleGo9/Meta-Learning-Curve_AIpert}{[CODE URL]} \href{https://drive.google.com/file/d/1gMSTlcD2__EkFKW3qd7bV9WYb51PWs1O/view?usp=sharing}{[FACTSHEET]}}                                                                                \\ \hline

\rowcolor[HTML]{FFFFFF}
4             & \begin{tabular}[c]{@{}c@{}}\textbf{{automl-freiburg}}\\(username: {automl-freiburg})\end{tabular}                                           & 0.37                                                          & OK          &   {\href{https://github.com/sebastianpinedaar/meta-learn-from-LC}{[CODE URL]} \href{https://drive.google.com/file/d/1y8mwxa5jVt-BM5iLPXJHVHwxpbW0OIt-/view?usp=sharing}{[FACTSHEET]}}                                                                                \\ \hline
\rowcolor[HTML]{FFFFFF}
5             & \begin{tabular}[c]{@{}c@{}}\textbf{{automl-hannover}} \\ (username: {amsks})\end{tabular}                                                     & 0.32                                                          &                & {\href{https://github.com/timruhkopf/Meta_challenge}{[CODE URL]} \href{https://drive.google.com/file/d/1bABinDdTtQ524QtHMyte0oYL336JvCgk/view?usp=sharing}{[FACTSHEET]}}                                                                                \\ \hline
\rowcolor[HTML]{FFFFFF}
6             &\begin{tabular}[c]{@{}c@{}}\textbf{{pprp}} \\ (username: {pprp}) \end{tabular}                                                         & 0.24                                                          &                &                                                                                \\ \hline
\rowcolor[HTML]{FFFFFF}
7             & \begin{tabular}[c]{@{}c@{}}\textbf{{arushsharma24}} \\ (username: {arushsharma24}) \end{tabular}                                                    & 0.23                                                          &                &                                                                                \\ \hline
\rowcolor[HTML]{FFFFFF}
8             & \begin{tabular}[c]{@{}c@{}}\textbf{{Xavier}} \\ (username: {Xavier}) \end{tabular}                                                           & 0.17                                                          &                &                                                                                \\ \hline
\end{tabular}
\end{table*}

\renewcommand{\arraystretch}{1.2}

\begin{table*}
\centering
\caption{\textbf{ALC scores of the top 5 methods: {MoRiHa}~(AT01), {neptune}~(AT02), {AIpert}~(AT03), {automl-freiburg}~(AT04), {automl-hannover}~(AT05), and our baselines: Double Deep Q Network (DDQN), Best on Samples (BOS), Freeze-Thaw BO (FT), Average Rank (AR), Random Search (RS)}. The reported scores correspond to the worst of 3 runs for each method. The last row shows the average ALC scores (in descending order, from left to right) over 30 datasets.}
\label{table:result-details}
\includegraphics[width=0.9\textwidth]{fig/1st-round-heatmap.png}
\end{table*}

\section{Winning solutions}
In this section, we briefly introduce strategies of the winning solutions. More details on their implementations can be found in our factsheet summary (Appendix \ref{sec:factsheet}) or in individual factsheets provided by the winning teams in Table \ref{table:summary}.

\subsection{Team \textit{{MoRiHa}} (1st place)}
According to team \textit{{MoRiHa}}, they focus on ``doing the right thing at the right time". Their agent is very goal-oriented (i.e. maximizing the area under the learning curve) and does not rely on complex models or expensive computations. They emphasize the importance of having an accurate schedule regarding the invested time (choosing $\Delta t$ for each query) in order to avoid wasting time budget. One of their key findings is that switching between algorithms during exploration is very costly and it is rarely beneficial to switch the explored algorithm more than once. They build a \textbf{time model} for each algorithm to predict the time when the \textbf{first point} on the algorithm's learning curve is available. In addition, they keep a \textbf{list of algorithms ranked descending} based on their ALC in the meta-training phase. In meta-testing their algorithm scheduler explores algorithms in an order starting from the best algorithm. If an algorithm's learning curve stales, the next algorithm will be chosen. Time allocation is done using the time models and a \textbf{heuristic} procedure. This is the only team having an assumption that the learning curves are monotonically increasing.

\subsection{Team \textit{{neptune}} (2nd place)}
As described by team \textit{{neptune}}, they train a learning curve predictor to \textbf{predict unseen points} on the learning curves (i.e. by interpolating the original scores) in order to find the best algorithm for a new dataset. In addition, they train an algorithm classifier to categorize algorithms into three groups based on their \textbf{learning curve convergence speed}: Fast/Median/Slow. Different budget allocation strategies will be selected according to the algorithm's convergence type. Regarding their implementation, they train MLP networks to perform both tasks: learning curve prediction and algorithm classification.

\subsection{Team \textit{{AIpert}} (3rd place)}
According to team \textit{{AIpert}}, their method aims at uncovering good algorithms as fast as possible, using a low-computational cost and simple process. The novelty lies in the combination of an off-policy Reinforcement Learning method (\textbf{Q-learning}) and \textbf{K-means clustering} model. As similar datasets usually have the same \textit{learning behavior}, organizing similar datasets into groups based on their meta-features is essential. They thus build a Q-learning matrix for each dataset cluster (12 clusters in total). In meta-training, each dataset is seen multiple times and the corresponding Q-learning matrix is updated. This allows the agents to be exposed to more situations (different observations and rewards) on a dataset. In meta-testing, they first determine which cluster the given dataset belongs to. Then, the Q-learning matrix associated with the cluster is utilized as a policy to guide the agent. $\Delta t$ is not chosen as an absolute value but from \textbf{a fixed list of time portions} of the total time budget.

\subsection{Team \textit{{automl-freiburg}} (4th place)}
As described by team \textit{{automl-freiburg}}, their algorithm selection policy is based on a \textbf{Deep Gaussian Processes} (DGP) surrogate, in a similar way to FSBO \cite{fsbo}. The surrogate aims to predict the performance of an algorithm at time $t + \Delta t$, with $\Delta t$ chosen by a \textbf{trained budget predictor}. The DGP is trained during the meta-training phase, and fine-tuned in the meta-testing phase. During meta-training, they store the best algorithm for each dataset to be used as the first algorithm to query during meta-testing. The ``best" algorithm is defined as the one that has the highest $\frac{y_0}{t_0}$, which means they favor algorithms that achieve \textbf{high performance early}. Given a new dataset in meta-testing, the best algorithm of the \textbf{closest dataset} (previously seen in meta-training, based on the euclidean
distance) will be selected. During meta-testing, they keep updating the observed algorithms and fine-tune the DGP surrogate.

\section{Lessons learned and new design}
In this section, we describe the limitations of our original design and how we addressed them in the second round.

First, one set of limitations arises because we precomputed learning curves (performance as a function of time) and therefore have a fixed predetermined sampled time points. In our first challenge edition, we opted to interpolate in between time points with the last recorded performance. The criticism of participants was that in real life, if they chose an in between time point, they would get newer information. To mitigate that, in the new challenge round, the participants cannot choose any time point (thus no need to interpolate), they have to choose the next pre-computed learning curve point.

We thus provide a new type of learning curve for the second round: learning curve as a function of training data size, as opposed to learning curves as a function of time. The time budget for querying a point on a learning curve will be returned by the environment and not chosen by the agent, which means that the agent has to pay whatever it costs.

Second, the test learning curves were highly correlated with the validation curves. Therefore, one could overfit the former by simply overfitting the latter. In the second round, the agent will always be evaluated using the test learning curves but on a completely different set of datasets in each phase (Development phase and Final phase).

The second round of our challenge was accepted at the {AutoML-Conf 2022}. Participation in the first round is not a prerequisite for the second round. The second round comes with some \textbf{new features}, including:

\begin{itemize}
    \item \textbf{Learning curve}: we focus on learning curves as functions of training data size. We thus collected a new large meta-dataset of such learning curves.
    \item \textbf{Competition protocol}: Given a portfolio of algorithms, an agent suggests which algorithm and the amount of training data to evaluate the algorithm on a new task (dataset) efficiently. The agent observes information on both the \textit{training learning curves} and \textit{validation learning curves} to plan for the next step. Test learning curves, which are kept hidden, will be used for evaluating the agent.
    \item \textbf{Data split}: We use half of the meta-dataset for the Development phase and the other "fresh" half to evaluate the agent in the Final phase.
\end{itemize}

The second round is split into three phases:
\begin{itemize}
    \item \textbf{Public phase (1 week)}: participants practice with the given starting kit and sample data
    \item \textbf{Development phase (6 weeks)}: participants submit agents that are meta-trained and meta-tested on the platform. 15 datasets will be used in this phase.
    \item \textbf{Final phase (1 week)}: no further submissions are made in this phase. Your last submission in the Development phase will be forwarded automatically to this phase and evaluated on 15 fresh datasets (not used in the Development phase).
\end{itemize}

Like in the first round, this is a competition with code submission and the participants do not see the data in either phase (only their submitted agent is exposed to the meta-datasets).

We created a new meta-dataset of pre-computed learning curves of 40 algorithms with different hyperparameters on 30 datasets used in the AutoML challenge. The algorithms are created from four base algorithms: K-Nearest Neighbors (KNN), Multilayer Perceptron (MLP), Adaboost, Stochastic Gradient Descent (SGD), but with different values of hyperparameters. We added meta-features of datasets and hyperparameters of algorithms. We respected the data split of the AutoML challenge to produce three sets of learning curves for each task, from the \textbf{training, validation, and test sets}. The type of metric used to compute the learning curves of the meta-dataset is provided in the meta-features of the dataset. We also generated a new synthetic meta-dataset that contains 12000 learning curves, in a similar way used in the first round, but with a new type of learning curve as explained above (learning curve as a function of training data size).

In meta-training, the following data is given to the agent to meta-learn: meta-features of datasets, hyperparameters of algorithms, training learning curves, validation learning curves, and test learning curves. While in meta-testing, the agent interacts with an environment in a Reinforcement Learning style. Given a portfolio of algorithms, an agent suggests which algorithm and the amount of training data to evaluate the algorithm on a new task (dataset) efficiently. The agent observes information on both the training learning curve and validation learning curve to plan for the next step. An episode ends when the given time budget is exhausted. The following two lines of code demonstrate the interactions between the agent and the environment:

\begin{center}
    $action = trained\_agent.suggest(observation)$

$observation, done = env.reveal(action)$
\end{center}

with:

\textbf{observation} : a tuple of\\ $(A, p, t, R\_train\_A\_p, R\_validation\_A\_p)$, with:

\begin{itemize}
    \item $A$: index of the algorithm provided in the previous action,
    \item $p$: decimal fraction of training data used, with the value of p in [0.1, 0.2, 0.3, ..., 1.0]
    \item $t$: the amount of time it took to train A with training data size of p, and make predictions on the training/validation/test sets.
    \item $R\_train\_A\_p$: performance score on the training set
    \item $R\_validation\_A\_p$: performance score on the validation set
\end{itemize}

\textbf{action}: a tuple of $(A, p)$, with:
\begin{itemize}
    \item $A$: index of the algorithm to be trained and tested
    \item $p$: decimal fraction of training data used, with the value of p in [0.1, 0.2, 0.3, ..., 1.0]
\end{itemize}

The scoring program automatically chooses the best algorithm at each time step (i.e. the algorithm with the highest validation score found so far, which is different from the first round where it was chosen by the agent) to compute the agent's test learning curve (as a function of time spent). The metric used for ranking on the leaderboard is the Area under the agent's Learning Curve (ALC).

\subsection{Baseline results of the second round}
We re-use our baselines from the first round, including: Double Deep Q Network, Best on Samples, Freeze-Thaw Baysian Optimization, Average Rank, and Random Search agents. Their implementations are taken from the first round (\textit{Anonymous-paper}) with some modifications in order to work with our new protocol and meta-dataset. We also provide them (except the Random Search agent) a smarter policy for choosing the training data size in each step. Each agent keeps track of the algorithms ($A$) and the training data size ($p$) tested on the algorithms. Every time it re-selects an algorithm, it should increase $p$ by 0.1 to go forward on the algorithm learning curve.

We run each method three times and report the worst run of each of them in Table \ref{table:2nd-round-baselines}. Similar to the first round, among the baselines, \textbf{DDQN} still performs best in the second round with an average ALC of 0.38. It is the winner (co-winner) of 16 out of 30 datasets with the highest performance difference compared to other baseline methods seen on the dataset \textit{tania}. \textbf{BOS} and \textbf{FT} are not far behind with average scores of 0.37 and 0.36, respectively. The improvement of \textbf{BOS} compared to the first round can be explained by the adaptation of its strategy in this round. It tries each method on a small subset of training data first (e.g. 10 percent of training samples), instead of spending a fixed amount of time as in the previous round. This should help it not waste a time budget since choosing an adequate amount of time to get a new point on learning is no longer necessary in the second round. The success of \textbf{BOS} suggests that information on the first point of the learning curve is crucial to decisions on selecting algorithms.

\textbf{AR} and \textbf{RS} perform poorly with the same score of 0.28. Although focusing only on one algorithm as what \textbf{AR} method does can bring benefits in some cases (e.g. on dataset \textit{cadata} and \textit{didonis}), a dataset-dependent policy for selecting algorithms is still necessary to be successful on multiple cross-domain tasks.

\begin{table*}
\centering
\caption{\textbf{Baseline results in the second round.} We show ALC scores of Double Deep Q Network (DDQN), Best on Samples (BOS), Freeze-Thaw Bayesian Optimization (FT), Average Rank (AR), and Random Search (RS) on each dataset, in descending order of the average ALC scores (on the last row), from left to right. The first 15 datasets are used in the Development phase, the rest is for the Final phase.}
\label{table:2nd-round-baselines}
\includegraphics[width=0.7\textwidth]{fig/2nd-round-baselines-heatmap.png}
\end{table*}

\section{Conclusion}

The first round results of our challenge have revealed that agents that learn both
policies for selecting algorithms and allocating time budget are more successful in our challenge setting. Team \textit{{MoRiHa}}, who finished in 1st place, outperformed all other teams and our baselines on two-thirds of the datasets. We propose a novel setting and a new meta-dataset for the second round and present baseline results. \textbf{DDQN} maintains its advantages in the second round, while an intuitive and simple baseline as \textbf{BOS} can work quite well in the second round. We are looking forward to see whether the findings of the first round will be reinforced and whether participants' solutions can outperform our baselines significantly in the second round.

For our future work, we want to perform more post-challenge analyses to verify whether progress was made in meta-learning from learning curves. First, we would like to do a point-by-point comparison of the winning methods in the first round, based on their fact sheets. Second, to investigate further the winning methods and see what contributed the most to their success, we want to perform systematic experiments in collaboration with the winners. More concretely, we will build a common workflow and ask participants to conduct ablation studies. Lastly, we are also interested in examining
the effect of changes in our reward function hyper-parameters on participants' performances.

\section*{Acknowledgment}
We would like to thank challenge participants for providing feedback and sharing their methods. We are grateful to Chalearn for donating prizes and Google for providing computing resources. This work was supported by ChaLearn, the ANR Chair of Artificial Intelligence HUMANIA ANR-19-CHIA-0022 and TAILOR EU Horizon 2020 grant 952215.

\section*{Software and Data}

All software (including starting kit and winning solutions) is open-sourced on our website (\url{https://metalearning.chalearn.org/}). The meta-datasets will remained private on the challenge platform ({\textit{Codalab}}) to serve as a long-lasting benchmark for research in meta-learning.

\bibliographystyle{icml2022}
\newpage

redpajama_set_name: RedPajamaArXiv


text:
Lew Rockwell Books
Ron Paul Books
Murray N. Rothbard Library & Resources
LRC Bestselling Books
Celebrating 20 Years LewRockwell.com ANTI-STATE•ANTI-WAR•PRO-MARKET
Let the Government Do It
By Laurence M. Vance
Following the lead of other states, my state of Florida now has a "heartbeat bill" in the legislature that, if passed and signed into law by the governor, would effectively ban abortions in Florida once a fetal heartbeat is detected. Abortions up to 24th week of pregnancy are currently allowed in the state of Florida.
HB 235 requires:
a physician to perform an examination for, and inform a woman obtaining an abortion of the presence of, a detectable fetal heartbeat;
the physician to review the results of such examination with the patient before the woman gives informed consent for the abortion procedure;
that a woman who declines to review the results certify in writing that she did so of her own free will and without undue influence.
The bill was introduced by Representative Mike Hill, a Republican from Pensacola. It would make it a third-degree felony to perform abortions "when a fetal heartbeat is detected." The bill also redefines a fetus as an "unborn human being" categorized as such "from fertilization until live birth."
"I feel it is my obligation, it is my duty, to protect life liberty and the pursuit of happiness, including those who are unborn," said Representative Hill. Another legislator, Representative Anna V. Eskamani, a Democrat from Pensacola, who is the former senior director of public affairs at Planned Parenthood Southwest Central Florida, responded that "decisions about a woman's pregnancy should be 'left between a woman, her family, her doctor, and her faith—not politicians.'" The Free Society Laurence M. Vance Best Price: $12.00 Buy New $18.00 (as of 10:25 EST - Details)
A similar bill, SB 792, has been introduced in the Florida Senate.
"Heartbeat bills" have been proposed for the last few years, although every time one is enacted it is struck down by the federal courts. The plan seems to be to get a case involving a "heartbeat" bill before the Supreme Court so that the infamous case of Roe v. Wade (1973) can be overturned by the Court's conservative majority. In Roe, the Court held that a woman's right to an abortion fell within the right to privacy protected by the Fourteenth Amendment. States may not ban abortions before the fetus is determined to be "viable." The case of Planned Parenthood v. Casey (1992) reaffirmed Roe, and further prohibited states from imposing an "undue burden" on a woman seeking an abortion.
The problem with these "heartbeat bills," aside from the hundreds of thousands of dollars in legal fees that some states have spent defending them, is that the person least likely to want to discover a baby's heartbeat—the abortionist—is the one responsible to check for a heartbeat.
The recent flood of "heartbeat bills" in state legislatures is a response to laws like that recently passed in New York that effectively allow a woman to terminate her pregnancy (kill her baby) up to the point of birth.
How sad that "heartbeat bills" are challenged in courts, but abortions in the last trimester of pregnancy are not. How tragic that there is waiting period to buy a gun, but not to have an abortion.
Conservative Christians, of which I am one, are cheering these "heartbeat bills," and I am cheering, in principle, with them. However, there is an unintended consequence of these bills that I should point out. They serve to reinforce the attitude that is all to prevalent among conservative Christians: let the government do it.
No need to persuade women to not get an abortion—let the government do it by imposing restrictions on abortion.
And it's not just abortion where many conservative Christians have abdicated their responsibility.
No need to persuade people not to gamble—let the government do it by prohibiting gambling or imposing restrictions on it.
No need to persuade people not to abuse drugs—let the government do it by declaring war on drugs.
No need to persuade women not to sell their bodies—let the government do it by outlawing prostitution.
No need to persuade teenagers not to listen to music with raunchy lyrics—let the government do it by mandating warning labels on records.
No need to have prayer and Bible reading in the home—let the government do it by requiring these things in public schools.
No need to persuade people to spend time in church or with their families—let the government do it by forbidding car dealers from opening on Sunday.
No need to persuade teenagers not to smoke—let the government do it by raising the tobacco purchase age to 21. Free Trade or Protecti... Laurence M. Vance Buy New $5.95 (as of 10:20 EST - Details)
No need to feed the hungry—let the government do it with food stamps.
No need to care for the aged, widows, orphans, and disabled—let the government do it with Social Security.
No need to persuade people not to abuse alcohol—let the government do it by instituting Prohibition, raising the drinking age to 21, or preventing businesses from selling alcohol on Sunday.
No need to persuade unmarried people not to commit fornication—let the government do it by criminalizing fornication.
On this last point I note that the Utah legislature recently passed a bill to decriminalize fornication.
In Title 76, Chapter 7, Part 1, Section 104 of the Utah Criminal Code it says this about fornication:
76-7-104. Fornication.
(1) Any unmarried person who shall voluntarily engage in sexual intercourse with another is guilty of fornication.
(2) Fornication is a class B misdemeanor.
Violation of this law, which is no longer enforced, can result in up to six months in jail and a fine of $1,000.
The Utah state legislature recently passed legislation to repeal this law. Yet, some conservatives in the Utah House disapproved of the repeal.
Conservative Christians, of all people, should not look to the government to do anything.
The Best of Laurence M. Vance
Laurence M. Vance [send him mail] writes from central Florida. He is the author of The War on Drugs Is a War on Freedom; War, Christianity, and the State: Essays on the Follies of Christian Militarism; War, Empire, and the Military: Essays on the Follies of War and U.S. Foreign Policy; King James, His Bible, and Its Translators, and many other books. His newest books are Free Trade or Protectionism? and The Free Society.
Previous article by Laurence M. Vance: Can US Soldiers Be Heroes?
Chilling Bipartisan Attacks
The Best Reason Not To Buy
Christopher Tolkien, RIP
Court Refuses To Use Personal Pronouns
Jimmy Dore on UFO Claims
Catholic prof fired by bishop after posting Viganò's criticisms of Bergoglio online
'Mendacity, Hubris & Lies'
The Climate Change of the 1840s
Where are the transitioning to male athletes?
Murray and Me
Iowa's Bernie Team Locks Down Top Twitter Accounts Following James O'Keefe Exposé
German Government Escalates Its War on Gold
LRC Blog
The Endless Night: A Valentine to Film Noir
Rand Paul on Impeachment: Foreign Aid is the Real Problem, Not Delaying It
Tucker Carlson on the Angry Exchange Between Elizabeth Warren and Bernie Sanders
Chris Plante on CNN's New War on Bernie Sanders
Trump Trial Begins – Guilty As Charged?
"Recovering" What Wasn't Yours to Begin With
A Horrible Day in History
First Man?
Lapdog Europeans Join Trump To Kill Iran Deal
17 Years After 'Liberation,' Iraq Again Designated US 'Enemy'
LRC Podcasts


Patterdale Hall is a Residential Adventure Learning Centre, based in Glenridding on the shores of Lake Ullswater, Cumbria, United Kingdom, which provides a variety of land and water based activities for school children and commercial clients. The Hall is part of Bolton School in Lancashire.
The Centre is looking to recruit two people to join their team, one person to work in a combined role as part-time Cook and part-time General Assistant (making full time hours), and the second to work as a full-time General Assistant. These roles will be for a fixed-term commencing late February 2019 until the end of November 2019.
The combined role of Cook and General Assistant will be responsible for assisting in the provision of catering services, as well as working within the areas of Housekeeping and evening duty. For the Cook part of this role, applicants should ideally have previous experience in food preparation and basic food hygiene. The salary for the Cook part of this role up to £5,640.32 per annum, and the salary for the General Assistant part will be up to £11,491.22 per annum, both salaries will pro-rata for the period of employment.
The General Assistant is a multi-functional role working in three main areas; Catering, Housekeeping and evening duties. Previous experience in a similar role would be an advantage but is not essential.
The salary for this role will be up to £17,007 per annum, pro-rata for the period of employment.
The above positions could be ideal for a couple as live-in accommodation is available, or two individual applicants.
Please note that the General Assistant role has been advetised separately.
To be responsible for the preparation, cooking and service of food, under the direction of the Patterdale Hall Residential Manager and/or in his absence the Cook in Charge, for the provision of catering services, in line with the requirements of the customer.
The hours of work for the Cook part of the role will be 12 hours per week, and for the General Assistant part they will be on average 25 hours per week, all to be worked between Monday and Sunday, for the period of the fixed-term.
Where appropriate fee remission arrangements will be offered for children to attend the Senior School.
In line with school policy employees will have the opportunity to arrange to educate their children from Nursery age to 6th form; all on the Bolton School campus.
Convenient, affordable, quality childcare provision will be available before and after the school day for children who are also Bolton School pupils.
Sporting and leisure facilities will be available including gym, outdoor pursuits centre, swimming pool and tennis courts.
The opportunity to take part in the Childcare Voucher Scheme will be available.
Lunches will be provided free of charge on campus.
Training and Development Opportunities will be offered.
28 days annual holiday. Pro-rata for the period of the fixed-term.
The Governors of Bolton School will offer new employees the opportunity to join the Bolton School Group Personal Pension Scheme. The employee will be given the option to choose their contribution but as a minimum all members will be required to contribute 6% of their gross earnings and the School will contribute 9%. If you choose not to join the Group Personal Pension scheme and you fall into the category of "eligible worker" you will be automatically enrolled into the School's alternative pension scheme, arranged via the Peoples Pension. The minimum employee contribution is 3% of "qualifying earnings" and the School will contribute 2%. Further details are available upon appointment.
All offers of employment will be made subject to the successful applicant providing the original qualification certificates, relevant to the position and as outlined in their application form. Identification documents will also be required including passport or picture driving licence, birth certificate, marriage certificate where applicable, current pay slip and evidence of address e.g. utilities bill, bank statement. Eligibility to work in the UK will be checked. After the appointment is made the successful candidate will be required to complete a medical questionnaire, a check of the Children's Barred List will be done and an Enhanced Criminal Records Check will be carried out through the Disclosure and Barring Service. The School will seek two references prior to the interview.
The successful applicant will be required to complete a six month probationary period.
Bolton School is committed to safeguarding and promoting the welfare of children and to equality and diversity in all aspects of employment. The successful candidate will be required to undertake an enhanced criminal records check.


Conversion of an existing 19th century lakefront ice storage building on Squam Lake in NH. The project was an exercise in simplicity and grace within the strict confines of the building size (15' x 19') and the local zoning regulations. From an Boston Globe article by Architecture critic Robert Campbell: "The architects had to turn the magic trick of pulling a big rabbit out of a small hat. They had to find room for a spacious house inside a tiny shack. Room for a kitchen, a bathroom, a bedroom, a living room, an entry and storage...They pulled off their magic trick. When you enter you can't believe the sense of space and light inside the shell."
Further quote in Boston Globe by Robert Campbell "Its hard to forget this little house. It seems to crystallize a kind of peace. To get to it you must drive down a long, twisting road through a dark wood. You feel you've left the world far behind. You arrive at the house, you step up on the rough natural boulder that serves as a threshold, and you enter. You turn towards the lake, where perhaps the sun is setting, or you watch the glow of the fire playing across the natural ceder walls. You listen to the water lapping and the wind rustling among the oaks and maples and hemlocks. You hear three loons call, and you hear Coolidge's little schnauzer dog, Kate, answer with accurate mimickry. You feel both calm and alert, very much as you do in a ceremonial Japanese teahouse."


Churchtown Primary School - Year 4 Gymnasts going for gold!
Year 4 Gymnasts going for gold!
On Wednesday 6th February, Miss Armour and the Year 4 Gymnastics team attended the North Sefton Gymnastics competition at Stanley High School. After their win last year the team were eager to defend their title and had been training hard in the weeks leading up to the event.
On the day they performed exceptionally well and achieved the highest individual scores of the competition, as well as the overall title. Well done to Rebecca, Isobel, Elisabeth, Jamie and Jacey - we are all very proud!


Calling all princes and princesses. This is a magical event on our ice rink not to be missed. Featuring tunes from all your favourite films and a live DJ for requests, this is a fun session for you and your friends. A snow machine will add the finishing touches.
After your skating session, your ticket includes a burger/hot dog, chips and drink in our Apres Skate Barn where you will meet a few special guests!