text:
Actor, Director, Screenwriter
Family & Companions
Jacques Tatischeff
Le Pecq, Yvelines, FR
Jacques Tati is a chess master of modern film comedy, a creator of complex comic structures in which gag constructions and audience expectations become pawns on his cinematic board. The recurring figure in these games is Monsieur Hulot (played by the director), a blank-faced comic cipher garbed in a crumbled raincoat and ill-fitting trousers, an ever-present pipe muffling any words he ma...
Parade (1974)
Trafic (1970)
Mon Oncle (1958)
Mr. Hulot's Holiday (1953)
Jour de Fete (1949)
Mr. Hulot's Holiday - Movie Poster
Jour de Fete (1949) -- (Movie Clip) Nature's Been Generous With Him
Mon Oncle (1958) -- (Movie Clip) Un Film De Jacques Tati
Mon Oncle (1958) -- (Movie Clip) He Simply Won't Obey
Jour de Fete (1949) -- (Movie Clip) The Americans Have You Beat
Micheline Winter
Married on May 25, 1944.
Jacques Tati is a chess master of modern film comedy, a creator of complex comic structures in which gag constructions and audience expectations become pawns on his cinematic board. The recurring figure in these games is Monsieur Hulot (played by the director), a blank-faced comic cipher garbed in a crumbled raincoat and ill-fitting trousers, an ever-present pipe muffling any words he may say, an umbrella clutched in indecisive hands. His determinedly irresolute stride across Tati's expansive canvases is the unlikely spark that sets the comic machinery afire. On the basis of a mere four features ("Mr. Hulot's Holiday" 1953; "Mon Oncle" 1958; "Playtime" 1967; and "Traffic" 1971) over a 20-year period, Tati managed to reshape slapstick comedy, turning it into an intellectual parlor game.
Tati began performing in French music halls and cafes as a pantomimist and impersonator. In 1931, he filmed a comedy short, "Oscar, Champion de Tennis," but it was never completed. Following were a number of short films which anticipated his later features in their use of natural and mechanical sounds--"On Demande une brute" (1934), "Gai dimanche" (1935), and "Soigne ton gauche" (1936). After WWII, Tati appeared in the features "Sylvie et le Fantome" (1945) and "Le Diable au corps" (1946). In his short film, "L'Ecole de facteurs" (1947), Tati created the character of Francois the postman, a character he would play himself in his first self-directed feature, "Jour de Fete" (1948). "Jour" used the riffing gag structure Tati would explore more fully in his later features, plus creative sound as a source for gags.
Unhappy with the Francois character, Tati sought a persona with a more universal appeal. With Monsieur Hulot, Tati found his cosmic archetype: a zero who creates comic anarchy in his wake. In "Mr. Hulot's Holiday," Tati applies Hulot to the gag structures of "Jour de Fete." "Mon Oncle" deals with the tension between Hulot's old world sensibilities and the new world of modern mechanization and consumerism. "Playtime," Tati's masterpiece, released in 70mm and stereophonic sound, examines the disappearance of humanity within the maze-like confines of post-industrial society. "Trafic" portrays the anthropomorphism of automobiles and the mechanization of human beings.
Tati's cold, crisp examinations are a result of his re-inventing film comedy structures. Hulot has no purpose except to ignite the gag machinery. He is never the center of a gag sequence and frequently disappears from the gag situation once the perpetual motion machine takes hold. (In one sequence in "Playtime," Hulot appears merely as a reflection in a glass window.) Once the gag machinery begins, Tati subverts the punchline by either delaying it or ignoring it altogether. The result creates a tension for audience expectations: will the punchline continue to be prolonged or simply demolished? Tati does not allow his audience to identify with the main character in the scene; as a result, the subject of the shot becomes everything that appears within the frame. A Tati film is characterized by a tangled texture (especially on his densely packed soundtracks) that requires many viewings to unravel.
This complexity was Tati's commercial undoing; because of the prolonged preparations required to plan his films, Tati lost his audience. The nine-year gap between "Mon Oncle" and "Playtime" crippled the momentum of his career, and after the extravagances of "Playtime," Tati never recovered financially. When "Trafic" was released, it seemed a throwback to his films before "Playtime" and was a financial failure. In 1974, Tati released his final film, "Parade," a low-budget celebration of pantomime recalling his shorts from the thirties.
Although Tati influenced filmmakers as diverse as Jerry Lewis and Robert Altman, his career seems in a way to be both the beginning and the end of a comic tradition. Nevertheless, Tati's structural experiments did breathe life for a time into a moribund form.
Hulot
Monsieur Hulot
Francois--The Postman
Soigne ton Gauche (1936)
Writer (Feature Film)
From Story
Splendor (1988)
Cast (Special)
Fanfare (1954)
Director (Short)
Forza Bastia (2002)
The School for Postmen (1947)
Cours du Soir (1967)
Gai dimanche! (1935)
On demande une brute (1934)
Writer (Short)
Apprenticed to Spiller's, a British picture-framer, in London (date approximate)
Worked as amateur entertainer, performing sports-related pantomimes in London
Returned to France and became professional cabaret and music hall entertainer
Experiments as filmmaker--writing, directing and starring in short, "Oscar, champion de tennis"
Major recognition on stage at the Ritz on bill with Maurice Chevalier
Debut as producer, "Retour a la terre" (also starred)
Directorial debut, short "L'Ecole des facteurs"
Directed and starred in first feature film, "Jour de fete/The Big Day"
Created play, "Jour de fete a Olympia", based on "Jour de fete"
After failure of "Traffic," creditors seized Tati's assets, impounding all his previous features
Directed first program for Swedish TV, "Parade"
Paris distributor paid off Tati's $1.6 million debt and re-released impounded features
Here is the original release French movie poster for Mr. Hulot's Holiday (1953), starring and directed by Jacques Tati.
Jour de Fete (1949) -- (Movie Clip) Nature's Been Generous With Him Director, star and co-writer Jacques Tati, in his first feature, has just been introduced as the mailman Francois in the tiny French village (the real Sainte-Sévère-sur-Indre), where Guy Decomble and Paul Frankeur have arrived with their carnival, and the famous bit with the flagpole begins, in Jour de Fete, 1949.
Jour de Fete (1949) -- (Movie Clip) The Americans Have You Beat Tipsy and annoyed if not despondent, rural French mail carrier Francois (first-time director, co-writer and star Jacques Tati) has just seen an American newsreel about the dazzling advances in mail delivery in America, resulting in some inspired bumbling, in Jour de Fete, 1949.
Jour de Fete (1949) -- (Movie Clip) Speed! Speed! Motivated now by the newsreel he saw about the American methods of delivery, rural French postman Francois (director Jacques Tati) sets about his rounds with new determination, with some of the best gags, in Tati's first feature, Jour de Fete, 1949.
Mon Oncle (1958) -- (Movie Clip) Un Film De Jacques Tati A not inconsiderable portion of invention and tone setting just in the title sequence, from Jacques Tati's second film starring himself as "Mr. Hulot," Mon Oncle, 1958, winner of the Academy Award for Best Foreign Language Film.
Mon Oncle (1958) -- (Movie Clip) He Simply Won't Obey The modern house now famous as writer, director and star Jacques Tati's "Villa Arpel" is central, as Gerard (Alain Becourt) is delivered by his uncle (Tati), while mother (Adrienne Servantie) entertains before father (Jean-Pierre Zola) gets home, in Mon Oncle, 1958.
Mon Oncle (1958) -- (Movie Clip) Now That His Uncle Is Free Writer, director, and title character Jacques Tati is home with the family, having been sacked from the job Arpel (Jean-Pierre Zola) got him, now nominated by his sister, Mrs. Arpel (Adrienne Servantie) to watch over her son, his nephew, Gerard (Alain Becourt), with famous bumbling, in Mon Oncle, 1958.
Mon Oncle (1958) -- (Movie Clip) Thank You, Mr. Hulot Having been visible but not identifiable in a couple of early shots, writer, director and star Jacques Tati here introduces himself as Mr. Hulot, the title character, and his house, which is also a character, and Betty (Betty Schneider), his landlord's niece, early in Mon Oncle, 1958.
Playtime -- (Movie Clip) Arriving From Frankfurt The actor Jacques Tati not in evidence but the director very much, in the rather still opening sequence from the acclaimed Playtime, 1967, the third in his "M. Hulot" series.
Playtime -- (Movie Clip) Boston For Breakfast M. Hulot (star and director Jacques Tati) in and out of things, as his adventure along the Champs Elysee takes him to a travel agency, Americans everywhere, in Playtime (1967),Telluride Film Festival honoree 2006.
Mr. Hulot's Holiday -- (Movie Clip) Visiting Martine Mr. Hulot (director & star Jacques Tati) makes a hash of waiting for his date, the fetching Martine (Nathalie Pascaud) in Mr. Hulot's Holiday, 1953.
Mr. Hulot's Holiday -- (Movie Clip) Tennis Mr. Hulot (director & star Jacques Tati) fashions a hat from a newspaper and undertakes to play tennis, to the amusement of Martine (Nathalie Pascaud) in Mr. Hulot's Holiday, 1953.
Mr. Hulot's Holiday -- (Movie Clip) Beach Mr. Hulot (director & star Jacques Tati) attempts to defend the honor of Martine (Nathalie Pascaud) then causes a panic at the beach in Mr. Hulot's Holiday, 1953.
TCM Video Podcast -- (Movie Promo) October 2011, Part Two Tati, Fellini, De Sica. On the latest TCM Podcast, TCM's Scott McGee and Guest Blogger Brandon Kyle Goco discuss why these pillars of foreign cinema should be household names. Share your thoughts on why classic foreign cinema does, or doesn't appeal to you. Post your answers at facebook.com/tcmtv
Count Dimitri Tatischeff
Attache at Russian embassy in Paris.
Sophie Tatischeff
Film editor, director. Died on October 27, 2001 at age 55 after a long illness.
Pierre Tatischeff


Find out about Online Billing, our secure, user-friendly, web-based system for paying monthly health premiums.
Employers that follow Generally Accepted Accounting Principles (GAAP) must report under GASB. The Trust helps you with a GASB 74 –Other Postemployement Benefits Compliance Report.
Sample language for Trust members for your financial statement note, to be included with your annual financial report.
View rates for all available Trust benefits, including medical rates for WellCity Award recipients.
The Trust provides tax-favored account administration for Health Reimbursement Arrangements (HRA), Flexible Spending Accounts (FSA), and Health Savings Accounts (HSA). Find sample templates and information about this optional service.


text:
L'Aeroporto di Voghera-Rivanazzano  è un aeroporto italiano situato a 5 km dalla città di Voghera, nel territorio del comune di Rivanazzano Terme.

Storia
Il primo campo di aviazione del Vogherese risale al periodo della prima guerra mondiale, mentre è nel 1926 la costituzione della sezione locale dell'aeroclub provinciale. Il 16 giugno 1951 si costituisce il nuovo Aeroclub di Voghera ed i fondatori iniziano a prodigarsi per trovare un'area per poter operare, individuando il luogo dove oggi sorge l'infrastruttura aeroportuale.
Dopo aver svolto i lavori di realizzazione della pista in erba anche con mezzi di fortuna ed in prestito, l'aeroporto fu autorizzato ed aperto il 25 settembre 1956. La dimensione iniziale della pista era di 600x50 metri.

Un potenziamento della struttura aeroportuale fu pensato fin dal 1980 con un progetto, ma divenne fattivo nel 1991. Nel 1993 venne realizzata la pista in asfalto con i raccordi ed il piazzale di sosta; successivamente vennero acquisiti i terreni necessari all'ampliamento che l'ha resa come si presenta attualmente, terminato nel 2006.

L'aeroporto è stato chiuso al traffico commerciale per quasi totale assenza di traffico e situazione debitoria eccessiva nel 2011.
Ad oggi viene utilizzato da un locale aeroclub che ne concede l'utilizzo anche per altre manifestazioni come concerti e gare di dragster. Viene impiegato saltuariamente dal personale dell'Aeronautica Militare per test go-around e dal soccorso sanitario su ala rotante qualora la struttura risultasse in posizione comoda al soccorso territoriale.

Note

Voci correlate
Elenco degli aeroporti italiani

Collegamenti esterni

Voghera
Infrastrutture della provincia di Pavia
Voghera

redpajama_set_name: RedPajamaWikipedia


Brice Marden, The Muses, 1991–93, oil on linen, 108 × 180 inches. Private collection. Courtesy of Thomas Ammann Fine Art Zürich.
Brice Marden's The Muses is a marvelous painting, horizontal but consisting of vertical linear movements which will reassure those who find relief in the potentially figurative. But I think what's exciting about it lies beyond fears of the formless, returning one to the body by a route at once more circuitous and direct. Lines are painted in and then painted or washed out, beginning and ending and beginning again as other lines are painted in other colors, in a constant play between the tangible and the spectral, addition and erasure (a kind of addition), performed by a constantly considered yet improvisational brush-stroke, the colors naturalistic and combining with white to produce a shimmering optical grayness. What Marden does here with thin paint recalls his thick waxy paintings of the earlier 1970s, which like this one needed the colors of landscape to maintain the possibility of an atmosphere which can undermine the surface and save it from being merely the place of inscription. This seems important because if the visual is always about what it's not (e.g. time), in Marden it may be about touch, a kind of inscribing, and The Muses suggests to me that the reason I was not convinced by (for instance) intervening works like the Cold Mountain series—where I thought the drawings good but the paintings not—is that his sense of touch and (or as) inscription only comes into its own when it can get lost in a space of its own making.
The artist-run curatorial project HKJB favors art over concept. Their first show "Personal Abstraction" takes art away from a gallery setting and puts it back into a studio, in an effort to change the way we think about viewing and talking about art.


text:
Händelser

1 juli
 Kroatien går som 28:e medlemsland med i Europeiska unionen.

3 juli
 Egyptens militär avsätter president Muhammad Mursi.
 Belgiens kung Albert II av Belgien meddelar att han tänker abdikera på Belgiens nationaldag, den 21 juli.

6 juli
 Asiana Airlines Flight 214 havererar utanför San Francisco International Airport.

8 juli
 Sökmotorn Altavista stängs.

13 juli
 Det blir offentligt att professor Stefan Svallfors nominerat Edward Snowden, som läckte information om USA:s National Security Agency, till Nobels fredspris.

18 juli
 Amerikanska staden Detroit begärs i konkurs.

21 juli
 Philippe av Belgien tillträder som Belgiens kung.

22 juli
 Prins Williams och Catherines son föds och blir nummer tre i samväldesrikenas tronföljd. Två dagar senare meddelas att prinsens ska heta George Alexander Louis.

24 juli
 78 personer omkommer i en järnvägsolycka i Santiago de Compostela, nordvästra Spanien.

28 juli
 Finalen i Europamästerskapet i fotboll för damer 2013 äger rum i Solna, Sverige. Tyskland slår Norge med 1-0.

Källor

Externa länkar


text:
\section{Acknowledgments}
\label{sec:acks}

This work was made possible in part thanks to Dell-EMC, NetApp, and
IBM support; and NSF awards CCF-1918225, CNS-1900706, CNS-1729939, and
CNS-1730726.

\section{Background and Motivation}
\label{sec:background}

We now explain how ML can help storage systems and address potential
concerns regarding integrating ML into storage systems and OSs.

\paragraph{Our vision}
KML\xspace is designed to replace OS-level storage system heuristics and
system parameter tuning.  To this end, a KML\xspace application first
observes the target OS component by collecting data from probes that
were placed in the kernel.  The data collected from these probes is
used to train an ML model using the functionality provided by KML\xspace.
The KML\xspace application then makes predictions using this trained model
and tunes the system.  We implemented our use cases with an
\emph{observe-and-tune} paradigm to reduce overhead introduced by the
ML models.  Therefore, we do not impose extra overheads on storage
components that require low latency and predictable performance.  The
two KML\xspace use-cases that we detail in this paper are ML models
developed (1) to tune readahead sizes on a per-disk and per-file basis
and (2) to tune NFS \stt{rsize} value.  We chose these two examples
because:
\begin{inparaenum}[(i)]
\item their storage components can significantly benefit from
  fine-tuned parameters,
\item they require adaptation to a variety of different workloads, which is crucial in providing
  optimal performance, and
\item it proves that adding asynchronous ML computation (see
  Section~\ref{sec:eval:kml}) will \emph{not} negatively impact
  critical I/O paths.
\end{inparaenum}

\paragraph{Unexpected behavior \& explainability of ML}
Concerns regarding the potentially unexpected behavior and
explainability of ML applications are justified when it comes to
tuning OS parameters, where traditional heuristics have well-defined
behaviors often expressed as closed-form formulas.
An ML algorithm may behave erratically when used in new, unforeseen settings,
which may hamper the performance of the system where it is deployed.
This type of issue is difficult to troubleshoot due to the
long-standing explainability problems that plague ML models.
KML\xspace currently supports two ML models: neural networks and
decision trees.
Predictions made by decision trees are more easily explainable because they can be represented
and visualized as successive \textsc{if-then} statements, bisecting the
range of the features considered.
Meanwhile, deep neural networks are considerably more challenging to explain and
verify.
However, recent research is dedicated to improving these
aspects of ML~\cite{jeyakumar2020can, samek2020toward}.
While we plan to address possible unstable behavior using
feedback-based control algorithms in future work,
we currently focus on demonstrating that ML can tune
storage system parameters \emph{better} than existing heuristics.
In Section~\ref{sec:eval:readahead}, we compare the performance and
accuracy of the neural network vs.\ decision tree models that we built.

\section{Conclusion}
\label{sec:conclusion}

Operating systems and storage systems have begun to support many
ever-changing workloads and devices.  To provide the best performance,
we have to configure storage system knobs based on workloads' needs and
device characteristics.  Unfortunately, current heuristics may not adapt
to workload changes quickly enough
and require constant development efforts to support
new devices.  We propose KML\xspace to solve these problems---an ML framework
inside the OS that adapts quickly to optimize storage performance.  KML\xspace
enables finer granularity optimizations for individual files in
mixed workloads---a challenging problem.  Our preliminary results show
that, for a readahead problem, we can boost I/O throughput by up to
2.3$\times$ without imposing significant CPU/memory overheads.
For the NFS \stt{rsize} problem, the improvement was up to 15$\times$.

\paragraph{Future work}
We plan on using KML\xspace to tune knobs for other OS subsystems:
\eg packet and I/O schedulers, networking, and the page cache.
We are enriching KML\xspace's capabilities by supporting other ML techniques
such as reinforcement learning~\cite{kaelbling1996reinforcement}, which
can be a better fit for solving OS problems.
To support more advanced ML approaches (\eg Recurrent
Neural Networks (RNNs)~\cite{wiki-rnn}) and Long Short-Term Memory
(LSTM)~\cite{hochreiter1997long}), we are extending KML\xspace to support
arbitrary computation DAGs.

\subsection{Evaluation Goals}
\label{sec:eval:goals}

Our primary evaluation goal is to show that using ML techniques inside
the OS can unlock the hidden potential of tuning parameters such as
readahead for files and block devices, and \stt{rsize} for NFS.

To this end, we evaluate both readahead ML models
and the NFS neural network
to show how these models improve the I/O performance and quickly adapt
the system in the presence of changing workloads.
To show that
our neural network models are
capable of learning abstract workload patterns, we first present
the generalization power of our models by testing it on workloads
\emph{not} included in the training dataset.
Next, we present benchmarks on a device type that was not used in the data
collection phase nor training.
Moreover, we evaluate KML\xspace's versatility
by applying the readahead neural network model on a per-file basis.
This demonstrates KML\xspace's ability to optimize individual workloads when
they are mixed together.

One of the main concerns with ML approaches is unpredictable behavior
due to mispredictions.
We evaluate what happens when the readahead ML models
mispredict and how quickly they recover.
Finally, we evaluate KML\xspace's own system overheads, in terms of (i)
training cost, (ii) inference cost, and (iii) memory usage.

\subsection{KML\xspace's Own Overheads}
\label{sec:eval:kml}

An ML model's overhead depends on its architecture.  Generally, deeper
or higher-dimensional neural networks consume more memory and CPU than,
say, decision-tree models.  It is vital that an ML component,
especially one that may run inside the kernel, consume as little CPU
and memory as possible.  Next, we evaluate the readahead neural network overheads.

\paragraph{CPU overheads}
On average, data processing and normalization transactions took
22~nanoseconds for the readahead use-case and 36~nanoseconds for the
NFS use-case.  This means that data processing, which is the only
inline operation in our readahead neural network, adds a negligible
latency on the running system compared to typical I/O latencies.
Such low overhead is crucial in enabling high sampling rates.

The readahead neural network performs inference (prediction) and
changes the block-layer readahead value in 21$\mu$s on average.  This
is a small number and executes in a separate, asynchronous kernel
thread.
Hence, it has negligible impact on the overall performance of the OS.
When the readahead neural network runs in per-file mode, KML\xspace runs
inferences an average 135 times (\ie the number of open files) every
second; running inference for all open files consumes 1.7ms on
average.
We measured that the readahead decision tree inference takes only
8$\mu$s.  The readahead neural network and decision tree have the same
data pre-processing and normalization implementation---the only
difference between them is in the inference part.

As discussed in Section~\ref{sec:usecase-readahead}, our readahead
neural network prototype offloads training to the user level.  We
measured the time to perform one training iteration in user level
at 51$\mu$s on average; this training iteration includes the forward
pass, back-propagation, and weight update stages.

\paragraph{Memory overheads}
The readahead neural network allocates 3,916 bytes of dynamic memory
during the model's initialization phase.  While inferencing, KML\xspace
temporarily allocates 676 bytes before returning the inference
results.  This overall memory footprint is negligible in today's
multi-GB systems.
The readahead decision tree occupies only 2,432 bytes of dynamic
memory during the initialization.  No dynamic memory allocation
happens during the inference for the readahead decision tree.
Lastly, the kernel module \stt{readhead.ko} has a memory footprint of
432KB and the kernel module \stt{nfs.ko} is 636KB, while KML\xspace entire
framework (\stt{k-Mlib.ko}) has a
memory footprint of 5.5MB.

\subsection{NFS Evaluation}
\label{sec:eval:nfs}

\input{fig-nfs-overall-performance}

Figure~\ref{fig:nfs-overall-performance} shows the NFS \stt{rsize}
neural network performance improvements using the same evaluation
techniques of readahead.  Throughout these experiments, we ran
multiple iterations of the same workloads.
Since \stt{rsize} is a mount point parameter for NFS, our NFS neural
network can tune \stt{rsize} values only in the beginning of the
iteration.  (We plan to fix the Linux kernel to permit \stt{rsize} to
change dynamically.)  Hence, in sequential workloads, if the NFS
neural network makes even one misprediction, it will affect the entire
iteration, leading to performance degradation.  Nevertheless, in
random workload cases, we still measured around 15$\times$ performance
improvement; in separate experiments (not shown), performance
improvements for random workloads reached up to 20$\times$.  This
demonstrates the significant potential of KML\xspace.

\subsection{Readahead Evaluation}
\label{sec:eval:readahead}

\paragraph{Readahead background}
There are two places in the Linux kernel where readahead is defined:
the block layer and the file system level.  When a file is opened, the
VFS initializes an open \stt{struct file} and copies the readahead
value for that file from the corresponding block layer.  Upon a page
fault for that file, the page-cache layer uses the value stored in the
file to initiate reading-ahead the desired number of sectors of that
file.
However, the readahead value in the file structure is initialized only
once when the file is opened.  So when KML\xspace changes the block layer readahead
value, the Linux kernel does not copy the new value
to any file already opened.
This means that open files may continue to use
a sub-optimal readahead value, even if better values are available
(\eg due to workload changes).
That is why we implemented a mechanism that changes the readahead size
for \emph{open files} when KML\xspace changes the disk-level readahead
value.  This propagates newer readahead values to each
open file, improving our adaptability.
Conversely, if KML\xspace mispredicts the workload type and changes the
readahead size to a suboptimal value, short-term performance
degradation can happen, which might hurt overall performance.

\input{fig-overall-performance}

\input{fig-ultimate}

\input{fig-per-disk-vs-per-file-nvme}

\paragraph{Back-to-back workloads on NVMe}
Figure~\ref{fig:back-to-back} shows four workloads running back to
back with each subfigure comparing a vanilla run (colored orange) to
our KML\xspace-enabled readahead run (colored blue).
The readahead value was
set to the default value (\ie 256) at the start of both vanilla and
KML\xspace-enabled runs, but when the next workload started, it used the last
readahead value from the previous workload's run (\eg the readahead
value at the end of the leftmost subfigure is the same at the start of
the subfigure immediately to its right).
This experiment evaluates KML\xspace's ability to optimize the readahead
values on complex systems where the I/O workload may change every few
minutes.
The X axes indicate the run time in minutes.  The Y axes indicate
throughput in thousands of ops/sec (higher is better), and have
different scales for each experiment.
The Y2 axes show the readahead values used or predicted by
KML\xspace over time in terms of number of sectors (denoted with a green line
and using the same scale).
Each workload ran 15--50 times in a row, so it ran long enough to
observe mis/predictions patterns.
As seen in Figure \ref{fig:back-to-back}, KML\xspace adapts quickly to
changing workloads by tuning the readahead value in about one second.

Although we observe some mis/prediction patterns, overall
throughput improved across all four runs, averaging 63.25\% improvement:
140\% improvement for readrandom, 2\% for readsequential, 109\% for
\stt{mixgraph}, and 12\% for readreverse.
The potential benefits are clear: even a small percentage improvement
in throughput can yield significant cumulative energy and economic cost
savings for long-running servers~\cite{li-hotstorage2014-cost}.

\paragraph{Read-sequential workloads}
Out of the six workloads we ran, Figure~\ref{fig:overall-performance}
shows the one where KML\xspace performed the worst: read-sequential.
Reading data sequentially directly from the raw SATA-SSD is nearly
1,000$\times$ faster than the \stt{mixgraph} workload, and nearly
400$\times$ faster with the NVMe-SSD.
There is little opportunity for KML\xspace to improve throughput for a
sequential workload that reads at speeds near the maximum throughput of
the physical device.

\paragraph{Read-reverse workloads}
As we can see from the fluctuating green line (readahead values in
Figure~\ref{fig:back-to-back}) KML\xspace mispredicts readreverse as
readseq
and changes the readahead value to something suboptimal.  These two workloads
both access files sequentially---one reading forward and one backward.
Interestingly, readseq and readreverse are quite close
from a feature representation perspective, which explains the
mispredictions.  But since both of these workloads access files
sequentially, their optimal readahead values are also quite close to
each other.  Thus, even if KML\xspace mispredicts readreverse as
readseq or vice versa, it had a small overall impact on
performance.

\paragraph{Summary of readahead neural network results}
For brevity, we summarize all readahead neural network results in
Figure~\ref{fig:overall-performance}.  We observe that the average
throughput improvement for NVMe-SSD is ranging from 0\% to 65\%.  We saw
greater improvements in the SATA-SSD case, ranging from 2\% to 130\%
(2.3$\times$).
Lastly, we ran the complex \stt{mixgraph} workload on NVMe-SSD with the
system memory set to the maximum (\ie 24GB) and the database size set to a
relatively large 65GB.  This experiment ran for nearly an hour (48.5
minutes) and resulted in an average throughput improvement of 38\%.

\paragraph{Mixed workloads}
Mixed workloads are considered a challenging optimization problem.
In Figure~\ref{fig:per-disk-vs-per-file}, we present a timeline
performance comparison using the readahead neural network model
running on a per-disk vs.\ per-file basis.
Per-file mode performs better overall because readahead values are set
for each open file independently.
Conversely, in the per-disk mode, a single readahead value is set at
the disk level and hence to all open files: a readahead value good for
one workload is likely to be bad for a second one running
concurrently.
One reason why the per-disk mode cannot predict workload types correctly
is that when different workloads are mixed---even sequential ones or
ones with regular
patterns---the mix looks more like a purely random workload
at the disk level.
Figure~\ref{fig:ultimate} shows overall mixed workloads performance
comparisons.
Per-file mode performed overall better in every combination of mixed
workloads.
If we compare only the sequential parts of the mixed workload
combination (orange bars in Figure~\ref{fig:ultimate}), in per-disk
mode, we observe significant performance degradation.
However, in per-file mode, we can observe performance improvements for
both the sequential and random (blue bars in
Figure~\ref{fig:ultimate}) parts of the mixed workload combination.
The reason why per-disk mode performs better for the random parts of
the mixed workload combinations is for the same aforementioned reason
(mixing workloads looks more random-like at the disk level); KML\xspace
predicts these as readrandom or readrandomwriterandom which
coincidentally fits this part of the workload, but significantly hurts
non-random workloads.

\paragraph{Decision tree evaluation}
In addition to the neural network model, we implemented a decision
tree model for the readahead problem to compare the two ML approaches
on the same problem.  We tested the readahead decision tree the same
way.  Figure~\ref{fig:dt-overall-performance} shows that there is a
performance improvement for workloads with a random component.
However, performance degrades for sequential workloads.
We investigated this performance degradation.
Figure~\ref{fig:dt-readseq-performance} shows the readseq
workload running on a RocksDB instance stored on an NVMe-SSD.  Here,
the readahead decision tree predicts the workload correctly in the
first three minutes, despite some fluctuations.  Afterwards, the
decision tree model's predictions fluctuate wildly, and at around
minute 10 it consistently makes completely wrong predictions.
For the readahead decision tree, we measure average throughput
improvement for random workloads on NVMe-SSD as ranging from 48\% to
59\%; and in the SATA-SSD case, ranging from 99\% to 119\%
(2.19$\times$).  However, the readahead decision tree model degrades
performance for sequential workloads on NVMe-SSD, ranging from 15\% to
40\%; and in the SATA-SSD case, ranging from 36\% to 73\%.
Overall, this was somewhat expected: it is known that neural network
models, while more complex to build and use, are generally more accurate
and adaptable than decision-tree models~\cite{hall2003neural}.
\input{fig-dt-overall-performance}
\input{fig-dt-readseq-performance}

\paragraph{TPC-H benchmarks}
\input{fig-tpch-overall-performance}
As we mentioned in Section~\ref{sec:eval:testbed}, we evaluated our
readahead neural network model---trained on \stt{db\_bench} RocksDB
workloads that run on NVMe-SSD---on TPC-H queries (also NVMe-SSD).
This was to show our model's accuracy limitations when presented with
vastly different workload and application combinations.
See Figure~\ref{fig:tpch-overall-performance}.
We measured performance improvements as much as 39\% for most query types.
For query 11, however, the readahead neural network failed to
characterize the workload correctly and resulted in a 53\% performance
reduction.
Nevertheless, overall TPC-H performance still improved by 6\%.

\subsection{Testbed}
\label{sec:eval:testbed}

We ran the benchmarks on two identical Dell R-710 servers, each with
two Intel Xeon
quad-core CPUs (2.4GHz, 8 hyper-threads), 24GB of RAM
and an Intel 10GbE NIC.
In some experiments, we intentionally configured the system with only
1GB of memory to force more memory pressure on the I/O system; but we
also show experiments with the full 24GB of system RAM.  We used the
CentOS 7.6 Linux distribution.
We developed KML\xspace for Linux kernel version 4.19.51, the long-term
support stable kernel released on June 15, 2019; we added our readahead
ML models
to this kernel and used it in all experiments.
Because HDDs are becoming less popular in servers, especially when I/O
performance is a concern, we focused all of our experiments on SATA
and NVMe SSDs.
We used Intel SSDSC2BA200G3 200GB as our SATA-SSD disk and a Samsung
MZ1LV960HCJH-000MU 960GB as our NVMe-SSD disk, both formatted with
Ext4.
These two devices were used exclusively for RocksDB databases.
To
avoid interference with the installed CentOS,
the two servers had a dedicated Seagate ST9146852SS 148GB SAS boot
drive for CentOS, utilities, and RocksDB benchmark software.
We used 10GbE switches to connect our machines.  We observed an
average RTT time of 0.2 milliseconds.

\input{fig-back-to-back-1gb-4gb-db-trained-workloads}

\paragraph{Benchmarks}
We chose RocksDB's \stt{db\_bench} tool to generate diverse workloads
for evaluating the readahead
and NFS \stt{rsize}
neural networks.
RocksDB~\cite{rocksdb} is a popular key-value store and covers an
important segment
of realistic storage systems; \stt{db\_bench} is a versatile
benchmarking tool that includes a diverse set of realistic workloads.
Workloads can be run individually or in series, and the working set
size can be easily configured (\eg increased to generate more I/O
pressure on a system).

To demonstrate that our ML models
can learn from
and optimize for different types of workloads, we chose the
following six popular yet different \stt{db\_bench} workloads to run:
\begin{inparaenum}[(1)]
\item readrandom,
\item readseq,
\item readrandomwriterandom (alternating random reads and writes),
\item readreverse,
\item updaterandom (read-modify-write in random offsets),
and
\item \stt{mixgraph} (a complex mix of sequential and random accesses, based
  on Facebook's realistic data that follow certain Pareto and
  power-law distributions~\cite{cao2020characterizing}).
\end{inparaenum}
This diverse set of workloads tests reads and writes, sequential and
random patterns, and even complex mixed patterns; this provides a good
coverage of many possible real-world workloads users could face.

We trained our readahead neural network on traces that contain only
four of these workloads: readrandom, readseq, readreverse,
readrandomwriterandom---all running on the NVMe-SSD.
These four tend to be the simpler workloads, because we wanted to see
whether KML\xspace can train on simpler workloads yet accurately predict on
more complex workloads not trained on.
This also ensures a balanced representation of randomness and
sequentiality in the training dataset.

After the training phase was done, we tested our models on all six
workloads as well as different devices.  This was done to show that our
models not only perform accurate predictions on
the training set samples, but they also generalize
to three new and complex workloads (updaterandom and two variants of
\stt{mixgraph} as well as a different device (SATA-SSD))---which were
excluded from the training data.
We evaluated mixed workloads by running two concurrent \stt{db\_bench}
instances, each on a separate RocksDB database, both stored on the
same device.
We kept the hardware configuration the same as before to increase
system and page-cache pressure.

We also experimented with our readahead network model using
TPC-H~\cite{TPC-H} queries running on MySQL~\cite{mysql20} to evaluate
how generalizable the readahead neural network is.  TPC-H is a
well-known database benchmarking framework.  We tested our readahead
neural network on this setup to show the limits of our model.
Because \stt{db\_bench} benchmarks and TPC-H queries generate
different I/O patterns, we anticipate that models built with the
former may mispredict for the latter.
In this paper we do \emph{not} claim that our readahead neural network
model will work universally to optimize readahead values for all
possible workloads.  Rather, these use cases are meant to demonstrate
the KML\xspace framework's versatility.  Developing universal solutions for
these problems requires more comprehensive datasets and potentially
more powerful or complex models, which is beyond the scope of this paper.
Figure~\ref{fig:overall-performance} demonstrates the performance
improvements as a result of predictions made by the readahead neural
network---detailed further below.

\section{Evaluation}
\label{sec:evaluation}

Our evaluation proceeds as follows:
First, we explain our evaluation goals in Section~\ref{sec:eval:goals}.
We then describe the testbed design and benchmarks that we used to
evaluate the readahead neural network in
Section~\ref{sec:eval:testbed}.
Section~\ref{sec:eval:readahead} shows how the readahead ML models
improve performance.
In Section~\ref{sec:eval:nfs}, we present our evaluation of the
\stt{rsize} neural network model for NFS.
Finally, in Section~\ref{sec:eval:kml} we provide performance details
regarding KML\xspace's training and inferencing.

\input{eval-goals}
\input{eval-testbed}
\input{eval-readahead}
\input{eval-nfs}
\input{eval-kml-details}

\section{Introduction}
\label{sec:introduction}

Computer hardware, software, storage, and workloads have been changing
for decades---an accelerating trend in recent years.  Storage
performance heavily depends on workloads and the precise system
configuration~\cite{fast17instability, fast10fsgreen}.  Storage
systems and OSs include many parameters that can affect overall
performance~\cite{atc18auto-tuning,
  fast20carver}.  Yet, users often do not have the time or expertise
to tune these parameters.  Worse, the storage and OS communities are very conservative:
they resist making significant changes to algorithms and software to
prevent instability or data loss.  Thus, many techniques
currently used were historically developed based on human intuition
after studying a few workloads; but such techniques cannot easily adapt to
ever-changing workloads and system diversities.

For example, readahead values, while tunable, are often fixed and left
at their defaults.  Yet correctly setting them is important and difficult
when workloads change: too little readahead wastes potential
throughput and too much readahead pollutes caches---both hurting
performance.
Some OSs let users pass hints via \stt{fadvise} and \stt{madvise} to
help the OS recognize files that will be used purely sequentially or
randomly, but these often fail to find optimal values for complex
and changing workloads.
In this paper, we experimented with a variety of modern workloads and
many different values of readahead: we found that no single readahead
value is optimal for all workloads.
Another example of tunable parameters in the network settings is the
default read-size (\stt{rsize}) parameter in NFS: if it is set too
small or too large, performance suffers.

Machine Learning (ML) techniques can address this complex relationship
between workloads and tunable parameters by observing actual
behavior and adapting on-the-fly, and hence may be more promising than
fixed heuristics.
ML techniques were recently used to predict index
structures in KV stores~\cite{kraska2018case, dai2020wisckey}, for
database query optimization~\cite{sagedbkraska}, improved
caching~\cite{subedi2018stacker}, cache eviction
policies~\cite{vietri-hotstorage18-lecar}, I/O
scheduling~\cite{hao2020linnos}, and more.

In this paper we describe our ML approach to improve storage
performance by dynamically adapting to changing workloads.
We designed and developed a versatile, low-overhead, light-weight
system called \emph{KML\xspace}, for conducting ML training and prediction
in  storage systems.  KML\xspace defines generic ML APIs that can be
used for a variety of subsystems; we currently support several deep
neural networks and decision tree models.
We designed KML\xspace to be embeddable inside an OS or the critical path of
the storage system: KML\xspace imposes low CPU and memory overheads.
KML\xspace can run synchronously or asynchronously, giving users the ability
to trade-off prediction accuracy vs.\ overhead.

Developing and tuning ML-based applications can be its own challenge.
Therefore, we designed KML\xspace to run identically in user- or
kernel-level.  Users can develop and debug ML solutions in the user
level, then upload the same model to run identically in the kernel.

In this paper we demonstrate KML\xspace's usefulness with two case studies:
(i) adapting readahead values dynamically and (ii) setting NFS
\stt{rsize} values automatically.  In both cases we aim to adapt these
values within seconds under changing and even mixed workloads.
This paper makes several contributions:
\begin{enumerate}
\smallitemsep
\item We show that lightweight ML can indeed become a first-class
  citizen inside storage systems and OSs;
\item We offer flexibility through synchronous or asynchronous
  training and the ability to offload training to the user-level;
\item We introduce the idea of generic ML APIs that can be expanded to
  support many additional and future ML techniques;
\item We apply KML\xspace to two important optimization problems (readahead
  and \stt{rsize} values);
and
\item We evaluate our solutions using multiple, complex, and even
  mixed workloads, as well as two different storage devices.  We
  demonstrate throughput improvements as high as 2.3$\times$ for
  readhead and up to 15$\times$ for \stt{rsize}.  We show that ML
  models trained on a few workloads can generalize and optimize
  throughput for never-seen-before workloads or devices.  And finally,
  we show that KML\xspace has negligible overheads or memory footprint,
  making it suitable for embedding into storage systems.
\end{enumerate}

The rest of this paper is organized as follows.
Section~\ref{sec:background} provides brief ML background.
Section~\ref{sec:kml-arch} describes KML\xspace's main design.
Section~\ref{sec:use-cases} describes our two use cases (readahead and
NFS \stt{rsize}).
Detailed evaluation of KML\xspace and two use cases are in
Section~\ref{sec:evaluation}.
We survey related work in Section~\ref{sec:related} and conclude in
Section~\ref{sec:conclusion}.

\section{KML\xspace's Architecture}
\label{sec:kml-arch}

We begin by discussing high-level KML\xspace design choices and why we
chose to implement a new machine learning framework instead of
porting an existing one.

\paragraph{KML\xspace high-level design choices}
\input{fig-kml-arch-comparison}
Figure~\ref{fig:arch-comparison} demonstrates two different operating
modes that we built.  KML\xspace supports (a) in-kernel training and
inference and (b) user space offline training and in-kernel inference.
KML\xspace has a highly modular design: the core ML code base is shared by
both user and kernel space.  However, operation mode (a) is designed
for performance and accuracy whereas operation mode (b) is built to
support ease of ML development.  We recommend using mode (a) when the
problem at hand requires processing data with a high collection rate.  Mode
(a) also allows training and running inference in kernel space
without any prior data collection and processing.  Mode (a) also
enables more advanced ML techniques (\eg
Reinforcement Learning).

Conversely, mode (b) helps OS developers iterate over their ML model
design in user space with different features, architectures, and
hyper-parameters until a stable and accurate model is reached.
Experimenting with and debugging a model is much easier in user space.
Once a model is built in user space, the same model can be loaded into
the kernel without modification.

\paragraph{Yet another ML framework}
Developing new solutions and optimizations for storage and operating
systems requires a highly efficient design which carefully considers
the needs of the OS.  Modern ML libraries are designed for building
general-purpose
ML approaches, and tend to rely on many third-party
libraries (\eg in C++ or Python) to handle core ML components.  This
is why porting an existing ML framework to run in the kernel requires
redesigning the entire ML core.  Instead of porting a relatively large
and complicated existing ML framework, we designed and implemented
KML\xspace from scratch, enabling a low-overhead, light-weight, and highly
tailored experience for OSs and storage systems.

\input{kml-design-overview}
\input{kml-ml-lib-design}
\input{kml-modular-design}
\input{kml-overheads}
\input{kml-safety}
\input{kml-implementation}

\subsection{Design Overview}
\label{sec:kml-design-overview}

\paragraph{Easy to develop and extend}
In Figure~\ref{fig:arch-comparison}(b), KML\xspace is compiled and linked
with an application for both kernel and user space.  \stt{u-MLib.a}
and \stt{k-MLib.ko} are compiled from the same KML\xspace source code.  We
developed a wrapper layer for the KML\xspace development API; thus, KML\xspace's core
code base is uniform across both user and kernel APIs.  Having
this abstraction speeds up development, eases debugging, and
facilitates extensibility (see Section~\ref{sec:kml-modular-design}).
While these abstractions make it easy to build ML solutions for
storage systems, we recognize that developing KML\xspace-based solutions
requires a good understanding of OS and storage system
internals.

\paragraph{Low overhead}
Among the most critical challenges in making ML
approaches practical for storage systems is reducing computational and
memory overheads.  ML solutions have 3 phases that consume much
memory/CPU resources: (i) inference, (ii) training and (iii) data
processing \& normalization.  We detail our design choices to reduce
these overheads in Section~\ref{sec:kml-overheads}.

\paragraph{Safe programming model}
Even a single new line of code integrated into an OS kernel can introduce
safety and security concerns.
For that reason, we explain how KML\xspace avoids instability or security
breaches in Section~\ref{sec:kml-safety}.

\subsection{Implementation}
\label{sec:kml-implementation}

KML\xspace contains 10,703 lines of C/C++ code (LoC).
KML\xspace's core ML part has 5,539 LoC, which can be compiled in both user
and kernel space.
Our readahead neural network model code is nearly 1K LoC long: 486 LoC
for collecting data, initializing the model, creating an inference
thread, and changing block-level readahead sizes; and another 351 LoC
for model definition, data processing, and normalization.
Our NFS neural network model also includes nearly 1K LoC: 435
LoC for data collection, model initialization, and running
inference to predict workload type; and 338 LoC for creating the model
and manipulating data.

\subsection{Fundamentals of Core ML library}
\label{sec:kml-core-ml}

KML\xspace provides primitives for building and extending ML models.
This involves building algorithms for training ML models (\eg
back-propagation, decision-tree induction) and building the mathematical
functions needed to implement them.  The library design allows for
seamless extensibility of library functionality.  Additionally, the
library is easily debuggable in user space by ensuring identical behavior
of all ML functionality also in kernel space.

\paragraph{Mathematical and matrix operations}
Most machine learning algorithms heavily rely on basic
mathematical functions and matrix algebra.  For example, a neural
network classifier uses functions such as matrix
multiplication/addition, \stt{softmax}, and exponentiation.  Hence, we
implemented kernel versions of such common ML functions.

\paragraph{Layer and loss-function implementations}
One can think of a neural network as a composition of layers and one
or more loss functions.  Many of these building blocks are used across
many different neural network architectures.  Layers like a fully
connected layer, ReLU~\cite{nair2010rectified}, or sigmoid are
essential components of many neural networks regardless of the
end goal; loss functions are also fairly standard across many
applications.  In addition, both layers and loss functions implement
two main functionalities, one during the inference (forward) phase and
another during the back-propagation phase.  Our initial implementation
is focused on these common components and modes of operation,
implementing the forward and back-propagation functionality for the
layers and loss functions included in KML\xspace.

Implementing new layers and loss functions in KML\xspace requires
implementing the aforementioned functionality for the layer/loss
function to be added, as well as a function for initializing the
data structures related to the layer/loss (\eg a weight matrix in the
fully-connected layer).

\paragraph{Inference and training}
When stacked together, the elements of a conventional neural network
may be represented as a DAG.  From this perspective, neural network
inference means traversing the DAG starting from the initial node(s)
(where the inputs are provided), toward the resulting nodes (where the
neural network output is produced).  KML\xspace implements a standard training
method used in neural
networks---back-propagation~\cite{rumelhart1986learning}---where
the chain rule allows for efficient computation of the gradients.  KML\xspace
includes Stochastic Gradient Descent (SGD) which uses the gradients
computed using back-propagation to optimize the neural network
weights.

\subsection{KML\xspace's Modular Design}
\label{sec:kml-modular-design}

We now elaborate on KML\xspace's operation modes: (i) in-kernel training and
inference (see Figure~\ref{fig:arch-comparison}(a)), and (ii) user
space training and in-kernel inference (see
Figure~\ref{fig:arch-comparison}(b)).

\paragraph{Training in kernel space}
\input{fig-kml-online-kernel-arch}
We use the readahead use case to describe how KML\xspace works in kernel
training and inference mode.  Figure~\ref{fig:arch-online} shows KML\xspace's
framework (\stt{k-MLib.ko}), a KML\xspace application (\stt{readahead.ko}), and
target storage components (Block device and Memory Management
subsystems).  The yellow background denotes KML\xspace related components.  The
blue background depicts the target storage components, which are
readahead problem specific.  The green line represents execution and
data flow.  Numbered boxes refer to transitions happening between the
components.

As we mentioned in Section~\ref{sec:background}, we designed our use
cases based on the \emph{observe-and-tune} principle.  In its first
stage, the readahead module needs to observe and collect data.  Since
our target component is the memory management system, the readahead
module starts collecting data from this component
(Figure~\ref{fig:arch-online}(1)).  The readahead module then extracts
features and transfers them to the KML\xspace framework to be normalized
(Figure~\ref{fig:arch-online}(2)).  After the data processing and
normalization stage is done, if the readahead module is configured for
training, it trains on the normalized data, and the execution flow
ends here for this processing cycle.  However, if the readahead module
is configured for inference, it feeds the normalized data to the
readahead neural network model and tunes the target components based
on the prediction made by the model.

The way the KML\xspace application optimizes target components is highly
dependent on the problem and its solution.  In this use case, the
readahead module updates readahead sizes using block layer
\stt{ioctl} calls (Figure~\ref{fig:arch-online}(5)).  In addition, we
developed another approach for tuning readahead which changes readahead sizes
on a per-file basis (Figure~\ref{fig:arch-online}(4)).  When the readahead
module is running in inference mode, execution flow forms a
\emph{closed-circuit}.
After the readahead module changes readahead sizes, memory management
state changes.  When the memory management state changes, the input for
the readahead neural network model changes and then
predictions change.  If users want to train and inference in the
kernel, they have to control the switching between training and
inference carefully.  We have implemented the readahead use case using
both in-kernel training and offline training in user space.  In this
section, we focus on how execution and data flow happens in KML\xspace's
in-kernel training and inference mode.  We provide features and the ML
model details for the readahead use case in
Section~\ref{sec:usecase-readahead}.

In the ML ecosystem, data collection is a crucial part.  One reason we
offer kernel training is to train on data collected with a high sampling
rate.  Tracing OS's and storage systems with high accuracy and sampling
rates is challenging~\cite{systor20reanimator}.  Nevertheless, research
shows that tracing tools like LTTng~\cite{lttng-web} can bring
overhead down to as little as 5\%.
Additionally, traces may still be inaccurate due to data loss.  LTTng
collects trace data in user/kernel circular buffers.
This means that under heavy
sampling loads, trace events could be overwritten and lost before they
are processed by user space threads.
While this problem may still occur in our modes of operation,
operating in kernel space gives us more control over thread scheduling
to reduce loss of sampled events.  Considering the high data sampling
rate needed in our use cases, placing data processing and
normalization in user space results in more data loss than in the
kernel.  Still, we believe a user-kernel co-operated design may be
beneficial in some cases.  That is why we are exploring implementing a
user/kernel co-operated design for use cases that do not require a high
data sampling rate.  Thanks to KML\xspace's modular architecture, supporting
a user/kernel co-operated design can be achieved without fundamental
design changes.

In the long term, we believe that reinforcement learning can be used
to achieve truly \emph{self-adaptive and learned} storage
systems~\cite{kaelbling1996reinforcement}.  Hence, kernel training is
critical to reaching this goal.
One of KML\xspace's current limitations in the inability to adapt to
file-access patterns not present in the training data set;
reinforcement learning would make this possible.

\paragraph{Training in user space}
Building ML solutions is an iterative development life cycle.  Hence,
to find essential features and build accurate neural network models,
we need to run multiple data analyses and train/test our ML model with
different architectures and hyper-parameters.  Recognizing the
advantages of kernel training, building this iterative development
life cycle in the kernel is challenging.  To facilitate faster model
development and debugging, KML\xspace offers offline user space training and
kernel inference operation mode (see
Figure~\ref{fig:arch-comparison}(b)).  Since KML\xspace's user and kernel
space libraries are accessed by the same APIs and compiled from the
same uniform code base, porting trained ML models to kernel
space for inference is effortless.
\input{fig-kml-offline-arch}
Figure~\ref{fig:arch-offline} shows how the readahead model works with
this mode of operation.
Components highlighted in yellow represent KML\xspace-specific
implementations.  The red arrows denote the offline data collection
and training paths.

First, we built our training data set by tracing the memory management
subsystem.  We used Re-Animator~\cite{systor20reanimator} to collect
data from the target storage component.  Next, we moved on to the
feature-extraction phase.  This is where user-space training was
useful, because we could run various analyses, test different features,
and implement numerous data normalization techniques without
re-running experiments.  After we finalized the feature selection,
we trained and tested the readahead ML model in user space.  The
training phase often requires iterating over different ML
architectures and testing with multiple hyper-parameters.  When the
readahead ML model is ready for real-time testing, the only step
the user needs to take to deploy the trained model is to save the
readahead ML model to a KML\xspace-specific deployment file and load it
into the readahead kernel module.  KML\xspace APIs provide all the
functionality necessary for building, training, and
deploying ML models in-kernel (see examples in
Table~\ref{tab:api-table}).  The rest of the in-kernel execution is
identical to how KML\xspace runs when it is configured for inference in
kernel training/inference mode (Figure~\ref{fig:arch-online}).
\input{tab-api}
We implemented a wrapper layer to abstract external functionality
away from KML\xspace.  This way KML\xspace can be compiled and used in both
user and kernel space with the same API.
KML\xspace's development API falls into five categories:
\begin{inparaenum}[(i)]
\item memory management,
\item threading,
\item logging,
\item atomic operations,
and
\item file operations.
\end{inparaenum}
KML\xspace's development API provides a total of 30 functions to support KML\xspace's
functionality.

\subsection{Computational \& Memory Overheads}
\label{sec:kml-overheads}

OSs and storage systems are susceptible to performance degradation and
increased latency if computational and memory resources are not
carefully managed.  Therefore, we designed KML\xspace with efficient CPU and
memory usage in mind.  There is often a positive correlation
between the computational and memory footprint of an ML model and its
training and inference accuracy.  Hence, KML\xspace is highly configurable,
letting users trade-off overheads vs.\ prediction accuracy to best
suit the problem at hand.

\paragraph{Reducing computational overheads}
Matrix manipulation is a computationally intensive ML building block that
relies on floating-point (FP) operations.  However, OSs often disable
the floating-point unit (FPU) in the kernel because of the
overhead incurred by context-switching.  To address this, we considered three
approaches:
\begin{inparaenum}[(1)]
\item quantization,
\item fixed-point representations, and
\item enabling the floating-point unit in kernel space.
\end{inparaenum}
Quantization provides compact representation, allows developers to
compute matrix manipulation operations, and does not require an
FPU~\cite{choi2019accurate, gupta2015deep, desa2018understanding,
  de2018high, hao2020linnos}.  Quantization can help reduce
computational and memory overheads, but it reduces
accuracy~\cite{hubara2017quantized}.
Fixed-point representation computes FP operations using integer
registers.  Since all FP operations are emulated, integration of
fixed-point representation is fairly easy and even faster in
certain cases~\cite{chen2020mlloadbalancing, lin2016fixed}.  However,
fixed-point representation works within fixed ranges which can result in
numerical instability~\cite{lai2017deep}.
For these reasons, KML\xspace enables the FP register usage in the Linux
kernel using the \stt{kernel\_fpu\_begin} and \stt{kernel\_fpu\_end}
functions.  To avoid context-switch overheads, we minimize the number
of code blocks that use FPs.

\paragraph{Reducing memory overheads}
There are three factors affecting KML\xspace's memory consumption:
\begin{inparaenum}[(1)]
\item ML model-specific data (\eg metadata for model structure,
  weights and biases),
\item KML\xspace's internal memory allocations during training and
  inference, and
\item data collection for both training and inference.
\end{inparaenum}
ML model-specific data and KML\xspace's internal memory usage depend highly
on the number of layers, layer sizes, and layer types in the ML model.
KML\xspace gathers input data in a lock-free circular buffer; then, an
\emph{asynchronous training thread} trains on gathered data.  When
collecting data with a high sampling rate, the size of the lock-free circular
buffer plays a significant role in the ML model's performance and
accuracy.  Users need to configure the size of the circular buffer in
consideration of the data sampling rate such that the asynchronous
training thread can catch up with processing.  If the size of the
circular buffer is misconfigured, KML\xspace may lose useful training data,
which can reduce the resulting ML model's accuracy.

\paragraph{Operating under resource-constrained conditions}
KML\xspace exposes a memory allocation and \emph{reservation} API for ML
internals.  The primary motivation behind KML\xspace's memory reservation
capabilities is to extract predictable performance and accuracy, even
under memory pressure.  This allows KML\xspace to operate without fear of
memory allocation lagging or failing, which would otherwise hurt
performance and accuracy.

\paragraph{Data processing \& asynchronous training}
To make ML solutions generalizable, Data normalization is often used
as a powerful tool.  To this end, KML\xspace supports data normalization
functionalities such as moving average, standard deviation, and
Z-score calculation.  However, data normalization often requires heavy
FP computation.  One of our main principles while building ML
solutions for storage systems is avoiding inline computation on I/O
and data paths.  Thus, KML\xspace supports offloading training, inference,
and data normalization to a separate \emph{asynchronous thread}.  This
thread communicates with other KML\xspace components (\eg data collection)
over a lock-free circular buffer.  KML\xspace also provides development APIs
to make this communication seamless.

We recommend allocating one CPU core to the training thread.
This ensures it
gets scheduled frequently
enough to keep up with processing and training---otherwise,
crucial training data may be lost.  Currently, KML\xspace
supports only one asynchronous training thread.  This is because our
prototype supports only chain computation graphs that must be serially
processed (\ie the previous layer's output is processed and handed to
the next layer as input).

\subsection{Safety \& Security}
\label{sec:kml-safety}

Both the training and inference phases for machine learning solutions are
computationally intensive.  Except for model initialization and ML
model-saving operations, there is no I/O involved in KML\xspace APIs.
KML\xspace's impact on the safety and stability of storage components is limited to
memory allocation and concurrency.
Memory allocations in both user and kernel space can use locking
mechanisms.  Therefore, there might be unexpected latencies or
deadlocks.  To avoid these problems, KML\xspace allocates memory only in the
asynchronous training thread.  KML\xspace also uses a lock-free circular
buffer for data communication and reserves additional memory for extra
safety.

Obtaining predictable accuracy and performance from ML solutions
depends on how the ML model is architected.  We applied standard (\eg
N-fold) validation techniques to ensure the stability of our ML solutions
(see Section~\ref{sec:use-cases}).
From a security perspective, we recognize that loading an unverified
ML model into a running kernel opens up new attack surfaces.  We are
exploring solutions to digitally sign and certify loadable
models~\cite{linux-code-signing, kim2018broken}.

\section{Related Work}
\label{sec:related}

\paragraph{Machine learning in systems and storage}
In follow-up work to Mittos~\cite{hao2017supporting}, a custom neural
network was built that makes inferences inside the OS's I/O scheduler
queue.  The neural network decides synchronously whether to submit
requests to the device using binary
classification~\cite{hao2020linnos}.  There are notable
differences between that system and our KML\xspace.  That system was trained
offline using TensorFlow and exclusively trained in user
space.  Additionally, inferences were tested only on NVMe-SSDs,
and the model could not be easily re-trained.  Finally, each of the
two layers in their neural network were custom built.
Conversely, KML\xspace provides a more flexible architecture.  Training,
retraining, normalization, repeated inference---all are possible and
accomplished with ease in any combination of online, offline,
synchronous, or asynchronous settings.  Additionally, KML\xspace easily
supports an arbitrary number of generalizable neural network layers,
and experiments demonstrate more expressive classification abilities
on a more diverse set of devices.

Laga \etal~\cite{laga2016lynx} improved readahead performance in the
Linux Kernel with Markov chain models, netting a 50\% I/O performance
improvement in TPC-H~\cite{TPC-H} queries on SATA-SSDs.
In contrast, our experimentation was conducted on a wider selection of
storage media (NVMe-SSD and SATA-SSD) and workloads.  In TPC-H, we
show improvements up to 39\% despite TPC-H being a completely new
workload for our readahead model.  Moreover, our results illustrate
that our readahead model can improve I/O throughput by as much as
2.4$\times$---all while keeping memory consumption under 4KB, in
comparison to Laga \etal's much larger 94MB Markov chain model.

Some research has attempted to apply ML techniques to OS task
scheduling~\cite{negi2005applying, chen2020mlloadbalancing}, with
negligible reported performance improvements (0.1--6\%).
Nevertheless, it is becoming increasingly popular to apply ML
techniques to storage and OS problems including, but not limited to:
memory allocation~\cite{maas2020learning}, TCP
congestion~\cite{dong2018pcc}, predicting index structures in
key-value stores~\cite{kraska2018case, dai2020wisckey}, offline
black-box storage parameter optimization~\cite{zhen2018towards}, local
and distributed caching~\cite{vietri-hotstorage18-lecar,
  subedi2018stacker}, database query optimization~\cite{sagedbkraska},
cloud resource management~\cite{cortez2017resource,
  delimitrou2013paragon, delimitrou2014quasar}, and more.

\paragraph{Machine learning libraries for resource-constraint systems}
A myriad of ML libraries exist---some general purpose and others
specialized or designed with constraints in mind.  Popular
general-purpose ML libraries include
Tensorflow~\cite{abadi2016tensorflow},
PyTorch~\cite{paszke2019pytorch}, and CNTK~\cite{cntk}.
Conversely, libraries like ELL~\cite{microsoft-ell}, Tensorflow
Lite~\cite{tensorflow-lite}, SOD~\cite{SOD}, and Dlib~\cite{Dlib}
specialize in constrained or on-device environments,
KML\xspace differentiates itself by targeting OS-level applications.
Inside the OS, resources are highly constrained, prediction accuracy
is vital, and even small data-path overheads are unacceptable.

\paragraph{Adapting readahead and prefetching}
Readahead and prefetching methods are both well-studied
problems~\cite{Shriver1998AnAB, shriver1999does, ding2007diskseen,
  kroeger2001design} and see use in distributed
systems~\cite{lee2018aps, tran2004automatic, chen2019rnn,
  dong2010correlation, liao2015performing, nalajala2019improving,
  liang2007step, cherubini-icdm17-prefetching}.
Many have attempted to build statistical models to optimize and tune
systems~\cite{Shriver1998AnAB, shriver1999does, fox2008quantifying}.
However, the main limitation of statistical models is their inability
to adapt to novel new workloads and devices.  We have shown that our
model can adapt to \emph{never-before-seen} workloads and devices.
Another way to improve a readahead system is to predict individual I/O
requests and file accesses by observing workload
patterns~\cite{ding2007diskseen, kroeger2001design, Wu2008OnTD,
  uppal2012flashy, hu2010feature, amer2002file, whittle2003using,
  xu2020frequent}.
Predicting file accesses using hand-crafted algorithms is a reasonable
first approach.  However, such manual labor simply cannot keep up with
the breadth of workload complexity possible on modern systems.
Conversely, as long as we have training data, ML models can adapt and
optimize much faster.
Simulations are also viable solutions for readahead and
prefetching problems~\cite{ganfure2020deepprefetcher,
  chakraborttii2020learning, ravichandran2005making, xu2011evaluation,
  zheng2017adaptive}.
However, simulations are computationally expensive and are limited to
the datasets that the models are trained and tested with.
Additionally, the models produced in simulations are not designed for
resource-constrained environments, making it non-trivial to migrate
such models to the kernel.  It is possible to use a user-space library
to intercept file accesses~\cite{won2018ifetcher} or to require
application-level changes~\cite{yang2002atc}.  In contrast, KML\xspace
requires no application changes and is capable of intercepting
\stt{mmap}-based file accesses.

Finally, while techniques exist to improve NFS performance, to the
best of our knowledge, none are performed automatically using
ML~\cite{improvenfswrite}.

\subsection{Use Case: NFS rsize}
\label{sec:usecase-nfs}

\paragraph{Problem definition}
Networked storage systems such as NFS are popular and heavily used.
NFS is used for storing virtual machine disks~\cite{nfs-vmdk}, hosting
NoSQL databases~\cite{linkedin-nfs}, and more.  A misconfiguration of NFS
can hurt performance.  We experimented with different applications
using NFS and found out that one critical NFS configuration parameter
is the \stt{rsize}---default network read-block size.  Hence, we focus
on the NFS \stt{rsize} problem: predicting an optimal \stt{rsize} value
based on workload characteristics.

\paragraph{Studying the problem}
We tested NFS using the same methodology we used for readahead.  The
only difference here is tuning \stt{rsize} instead of readahead.  We
used NFSv4 for all of our tests.  The NFSv4 implementation we used
supports only seven different \stt{rsize} values (4K--256K).
However, in the NFS use case, there are additional external factors
not present in the readahead problem that can affect I/O performance
(\eg NFS server configuration, network speed, and number of clients
connected to the same server).
We experimented with four different RocksDB
benchmarks under different NFS server configurations and network
conditions.  We configured our server with two different NFS mount
point options---one backed by NVMe-SSD and one backed by SATA-SSD.  We injected
artificial network delays to simulate slower networks.  Our
experiments revealed that random and sequential workloads require
different \stt{rsize} values to achieve optimal performance.

\paragraph{Data collection}
We enabled NFS and page cache-related kernel tracepoints to collect
training data (\eg{} \stt{nfs4\_read}, \stt{nfs4\_readpage\_done},
\stt{vmscan\_lru\_shrink\_inactive}, and \stt{add\_to\_page\_cache}).
Unlike the readahead neural network model, we collected data from
tracepoints not only to model page cache behavior, but also network
conditions.  After studying these traces, we chose our feature set and
placed our hook functions.  Our feature set includes eight features
(described below) which are calculated using the following five data
points:
\begin{inparaenum}[(i)]
\item time difference from the beginning of execution for each
  tracepoint transaction,
\item NFS file handles,
\item file offsets in NFS requests,
\item page offsets of the files that were accessed,
and
\item number of reclaimed pages during LRU scans.
\end{inparaenum}

\paragraph{Data preprocessing \& normalization}
We applied the same data preprocessing and normalization techniques
that we used for the readahead neural network.  The NFS neural network
model consists of eight features which are computed every second:
\begin{inparaenum}[(1)]
\item number of tracepoint transactions,
\item average time difference between each \stt{nfs4\_read} and
  \stt{nfs\_readpage\_done} matching pair,
\item average time difference between each consecutive \stt{nfs4\_read}
  request,
\item average time difference between each consecutive
  \stt{nfs4\_readpage\_done} request,
\item mean absolute requested offset difference between each consecutive
  \stt{nfs4\_read} request,
\item mean absolute page offset difference between each consecutive
  \stt{add\_to\_page\_cache},
\item average number of reclaimed pages
and
\item current \stt{rsize}.
\end{inparaenum}

\paragraph{Neural network model}
We trained and tested our NFS neural network model using exactly the
same methodology as the readahead problem; for brevity, we detail only
the differences between the neural network models.
We approached the NFS problem as a workload characterization problem
and constructed our NFS neural network model with four linear layers
(with hidden layer sizes of 25, 10, and 5) with sigmoid activation
functions in between.  Similar to the readahead neural network, we
used cross entropy as the loss function and SGD as the optimizer.  We
evaluated the NFS neural network model and found out that it results
in a prediction accuracy of 98.6\% (using 10-fold cross-validation).

\subsection{Use Case: Readahead}
\label{sec:usecase-readahead}

\paragraph{Problem definition}
Readahead is a technique to prefetch an additional amount of storage
data into the OS caches in anticipation of its use in the near term.
Determining how much to read ahead has always been challenging: too
little readahead necessitates more disk reads later and too much readahead
pollutes caches with useless data---both hurt performance.
The readahead value is a typical example of a storage system
parameter: while tunable, it is often fixed and left at its default.
Some OSs let users pass hints via \stt{fadvise} and \stt{madvise} to
help the OS recognize files that will be used purely sequentially or
randomly, but these often fail to find optimal values for varied,
mixed, or changing workloads.
Next, we detail our readahead neural network design.  See
Figure~\ref{fig:arch-offline}.  After implementing KML\xspace's base ML
functionality, we focused on the readahead problem: predicting
optimal readahead values in the face of dynamic workloads.

\paragraph{Studying the problem}
We experimented with running 4 different RocksDB~\cite{rocksdb}
benchmarks, each with 20 different readahead sizes (8--1024), and
attempted to determine the readahead sizes that yield the best performance (in
ops/sec) for each workload.
This became our training data,
which can help predict readahead values for \emph{other} workloads and
environments.  This investigation revealed that each workload has a
unique behavior and requires a different readahead size to
reach optimal performance.
We further investigated the correlations between file access patterns
and performance.  This allowed us to zero in on the exact nature of
the information that was needed to model the readahead problem.

\paragraph{Data collection}
We used LTTng~\cite{lttng-web} to collect trace data, which we then
used for finding useful features for the readahead problem.  We
captured most page cache tracepoints~\cite{linux-tracepoints} (\eg{}
\stt{add\_to\_page\_cache}, \stt{writeback\_dirty\_page}).
We collected and processed over 20GB of traces by running multiple
10-minute RocksDB~\cite{rocksdb} benchmarks on an NVMe-SSD device.
Ten minutes was sufficient for RocksDB to reach a steady state.
After examining these traces, we selected a set of candidate features
based on our expertise.
We then picked the features of interest
and decided where to call hook functions which are responsible for
gathering necessary information (\eg{} \stt{struct page}) for
inference.
Our hook functions provide three important raw values:
\begin{inparaenum}[(1)]
\item time difference from the beginning of execution,
\item \stt{inode} number,
and
\item page offsets of the files that were accessed at the places where
the hook functions were called.
\end{inparaenum}

\paragraph{Data preprocessing \& normalization}
Data preprocessing and normalization are essential to ensure that our
readahead neural network is generalizable.  We summarize the input
data at one-second intervals.
The first input feature we selected for our model was the number of
transactions taking place each second.
The next two features were the calculated cumulative moving mean and
the cumulative moving standard deviation of page offsets.
Another important feature was the mean absolute page offset differences
for successive transactions.
We used the \stt{inode} number to filter only RocksDB file accesses.
Before we fed these features to our readahead neural network, we
applied Z-score normalization to each feature.

\paragraph{Neural network model}
We modeled the readahead problem as a classification problem and
designed a neural network with three linear layers (with hidden layer
sizes of 5 and 15), using sigmoid non-linearities in between layers,
and with a cross-entropy loss method as the loss function.  We used an
SGD optimizer~\cite{robbins1951stochastic, kiefer1952stochastic}, and
set a learning rate of 0.01 and a momentum of 0.99 after trying
different values; these values are common in the
literature~\cite{bengio2012practical}.
Our readahead neural network trains on the aforementioned input data
and predicts the workload type.  We trained on the following four
types of RocksDB workloads on NVMe-SSD because they provide a
diverse combination of random and sequential operations:
\begin{inparaenum}[(i)]
\item readrandom,
\item readseq,
\item readrandomwriterandom,
and
\item readreverse.
\end{inparaenum}
Class frequencies were close, suggesting that classification accuracy
is a good metric to evaluate the performance, with the least frequent
class being 21.4\% and most frequent class being 28.8\%.

We tested the neural network's performance with the data described
above via k-fold cross validation with $k=10$, and found out that it
achieved an average accuracy of 95.5\%.
We also performed an analysis to understand the contribution of each
feature to the classification performance.  One technique is to
randomize the order of the feature of interest across samples in the
validation dataset, and then calculate the 10-fold validation
performance~\cite{breiman2001random}.
Using Pearson correlation analysis~\cite{pearson1895vii}, we found that
two features were highly correlated: the cumulative moving
standard deviation and the cumulative moving mean of page offsets.
Including both would have hampered this analysis, so we excluded the
cumulative standard deviation of page offsets.
Cross validation results were 69.6\%, 76.4\%, 42.6\%, and 89.1\% for
number of transactions, cumulative moving mean of page offsets, mean
absolute page offset differences, and current readahead value,
respectively.  This shows that mean absolute page offset differences
is the most important feature, because randomizing its order reduced
the validation results the most (down to 42.6\%)---followed by number
of transactions, cumulative moving mean of page offsets, and finally
the currently used readahead value.

After obtaining classification predictions, we set the empirically
determined optimal readahead sizes according to the predicted workload
type.
Next, in Section~\ref{sec:eval:readahead}, we evaluate the readahead
neural network not only on workloads we trained on but also
on workloads that were \emph{not} included in the training data and
workloads running on different devices (NVMe vs.\ SATA SSDs).
We also experimented with the readahead neural network using
TPC-H~\cite{TPC-H} queries running on MySQL~\cite{mysql20} to show how
our readahead neural network behaves on completely different types of
workloads and how generalizable the models are.

\paragraph{Decision-tree models}
We also built a decision-tree model for workload type classification
based on the same features and training data.  The readahead decision
tree contains 59 nodes with a depth of 9.  We tested the prediction
accuracy of the decision tree using the same procedure with the
readahead neural network (10-fold cross-validation), and observed that
it results in an average prediction accuracy of 75.4\%.  As mentioned
in Section~\ref{sec:background}, we included decision trees in KML\xspace
because decision trees are relatively more explainable than neural
networks and run considerably faster.
We evaluated the readahead decision tree using the same procedure as
the neural networks (Section~\ref{sec:eval:readahead}).

\paragraph{Readahead in per-file basis}
So far, we have shown how we approach the readahead problem when a
single workload is running on a device.  Storage system developers
recognize the challenge of handling mixed storage workloads running on
the same system---a common occurrence.
In that case, the readahead problem needs to be addressed at a finer
granularity than setting one readahead value for the entire block
device.
Instead, readahead values have to be set at a higher logical
layer---on a per-file basis.
To show the usefulness of our neural network model, we use the same
neural network model to tune readahead size not only on a per-disk
basis but also on a per-file basis.
Whereas before we ran inference every second and set one readahead
value for an entire device, here we ran inference every second on
\emph{each} open file and set a readahead value in Linux's \stt{struct
file}.
We evaluated the per-file basis approach and found that it could
predict and improve I/O throughput for \emph{mixed workloads} better
than both the vanilla and per-disk basis approaches
(Section~\ref{sec:eval:readahead}).

\section{Use Cases}
\label{sec:use-cases}

We now detail our two use cases: (1) readahead neural network and
decision tree model and (2) NFS neural network model.  We describe the
following for each:
\begin{inparaenum}[(i)]
\item problem definition,
\item data collection for training,
\item data preprocessing and feature extraction,
and
\item building the ML model.
\end{inparaenum}

\input{use-case-readahead}
\input{use-case-nfs.tex}

redpajama_set_name: RedPajamaArXiv


It was nearly sunup, still sultry and humid.
It was another beautiful morning, with the hint of summer—Roscoe had always loved the summer and hated the cold, and he wondered if he would get back in time to enjoy the sultry days of July and August, when it was so hot even the river hardly seemed to move.
He wrapped his blanket around him tightly to prevent Ed from slipping in, but it was a hot sultry night and he was soon sweating so profusely that he couldn't sleep anyway.
There are no more uses of "sultry" in Lonesome Dove.


Temporary interventions, mobile artworks and project organizations or even flexible exhibition spaces without a fixed location are a growth market in contemporary art practice.
Perhaps, in the present time, a nomadic existence should be considered a necessity of life and the interest in mobility a productive response to mounting institutional uncertainty. Crucial manifestations such as Parasite Paradise (Leidsche Rijn 2003) and 11ha (Stork Terrain Amsterdam 2001) explore the immense diversity of temporary constructions by artists, architects and urban planners, and show that these practices are often inextricably linked with the ambition to reach alternative target groups, to create new sites for art and to vanquish the restrictions of a rigidly regulated society. Do these 'unregulated mobile longings' 2 lead to a resurgence of artistic refuges and social involvement? Or is there cause for questioning this contemporary phenomenon?
Holger Nickisch is developing the mobile unit ME/folly based on an inventory of mobile structures already created, personal experiences with art projects in public space and a consecutive series of interviews with artists involved. This concrete trial structure is offered to possible users in the cultural field as a facility, thereby forming the base of operations and knowledge centre for practical research.
Specific user needs, the boundaries of public space and the strong or weak position of art part in urban renewal processes will be analyzed.
2 Tom van Gestel, Director of SKOR, Parasite Paradise catalogue, Amsterdam 2003.
Holger Nikisch is a visual artist living and working in Amsterdam. In his work, he focuses on public space as a site for his work and interventions, and as inspiration source. Besides his practice as visual artist, Nikisch also curates projects nationally and internationally. He also founded ARTWALK and in 2009 was appointed director of the artspace Kunstfort Vijfhuizen.


text:
© 2020 Clemens Wilhelm All rights reserved
During the opening the visitors of the show individually entered a white room which was closed off with white hospital curtains. This room only contained a chair and a camera. The visitors were told to take a seat and speak the sentence "Je crois que je suis malade (I believe I am sick)" into the camera. For all other visitors and also for Clemens Wilhelm it was invisible what was happening behind the curtains. The performance video is presented as a lifesize video projection. It is silent which raises the viewer's attention to body language and facial expressions of the "sick", and enhances the theatricality and articificiality of the individual performances.
Installation views at Votivkirche Vienna in the exhibition "Leiblichkeit & Sexualität", 2014.
(photo: Christof Bartylla)
JE CROIS QUE JE SUIS MALADE
This performance was developed for the exhibition "wasistdas" at Espace Artcore, Paris in 2009. This show was held in the apartment where French playwright Molière lived. He died there in 1673 after a performance of his last play "The Imaginary Invalid", a comedy about hypochondria, health mania, belief in doctors, lack of attention and the public display of illness and ego. These very contemporary themes in combination with the special location were the background for Clemens Wilhelm's performance piece.
TitleJE CROIS QUE JE SUIS MALADEMediumVideo Performance | Video InstallationLength25 min loop (silent)CountryParis, FranceYear2009
MACHT NICHTS

redpajama_set_name: RedPajamaCommonCrawl


text:
Tracing The Global Rise Of China's Tech Giants
Chicago Tech Information
Business Finance Fundamentals
Business Finance Information & Matters
Main Health And Fitness Organizations Precision Diet And The American Council On ..
Kementerian Komunikasi Dan Informatika
Difference Between Computer Science & Info Expertise
2020 Business & Finance Awards
AT&T Is Launching Another New Streaming Service This Fall Called AT&T TV
A Cultural Evolution Approach To Digital Media
Nintendo Game Store
We cannot go to precise bars anymore, so drinking cheap beer at your go-to dive and betting your pals you may sink the eight ball is off the desk. But there's digital version that permits you to compete one-on-one or in eight-player tournaments. You can join via Facebook and challenge […]
We cannot go to precise bars anymore, so drinking cheap beer at your go-to dive and betting your pals you may sink the eight ball is off the desk. But there's digital version that permits you to compete one-on-one or in eight-player tournaments. You can join via Facebook and challenge your friends from the comforts of your couch.
Booba pillow in ffxiv, I'm still catching up on game stuff even when sick pic.twitter.com/sRbDLDCoi7
— Sukai•Amai🍨スカイ•甘い🍨 (@lilsuckai) May 17, 2021
This horrifically attractive game is a direct sequel to the wonderful Resident Evil 7, happening a number of years later in a Romanian village. Not only that, but using an excellent gaming keyboard and mouse can provide the aggressive edge over anyone utilizing a gamepad, particularly in first-person-shooter titles. Pair all that with a suitably snug gaming chair and you will not must settle into the sofa for an incredible gaming experience. The program, offered by the Starfall Education Foundation, a nonprofit group, was conceived by Dr. Stephen Schutz.
Search Jobs And Discover Your Dream Job Right Now
Games For Core Players
Who is the best game in the world 2020?
Begin Your Engines Games
Still, these games do not require a big area in which to play them, large quantities of energy or stamina, or specialised gear aside from what is obtainable in a box. Popular sports may have spectators who're entertained just by watching games. A community will typically align itself with a local sports activities staff that supposedly represents it ; they usually align themselves towards their opponents or have conventional rivalries.
When the ball sticks and one participant tries to do an excessive quantity of, the staff suffers. They should play as a staff and belief their teammates and maintain the ball shifting. Also, if the skin Games shots aren't falling for them, as they sometimes aren't on the road, they need to attempt to get to the basket and both get the score or draw the foul.
The expertise is normally fairly different from traditional role-playing games. Single-player games embody Final Fantasy, Fable, The Elder Scrolls, and Mass Effect.
Developed by Epic Games and released in 2017, Fortnite remains one of the most popular video games of 2020. (It won the best multiplayer/competitive game at the 2019 Webby awards.) It has three different modes: Battle Royale, Save the World, and Creative.
Play Tony Hawk's Pro Skater & Tony Hawk's Pro Skater 2 in a single epic assortment, rebuilt from the bottom up in unimaginable HD. All the pro skaters, ranges and methods are back and fully-remastered, plus more. Get immediate entry to over one hundred Marketplace items with new additions every month. Have fun with friends and family with Board and Card Games. The games above provided by Ducksters are relatively simple games which would possibly be usually puzzle or instructional games. We are at all times wanting into including new games, so examine again often.
With photoreal real-time rendering, free entry to all supply code, and the skilled support you need, Unreal Engine means critical business. How do you entice, have interaction, and retain at present's audiences? With the world's most open and superior real-time 3D creation platform, you can create jaw-dropping MR experiences for reside broadcast and stay occasions that may hold them on the edge of their seats. From design evaluate to digital showroom, the world's most open and advanced real-time 3D creation platform is transforming workflows. With Unreal Engine's high-fidelity real-time setting, stakeholders can consider concepts and configure options with confidence.
particularly "deathmatch" games performed towards hundreds of thousands of peers around the world for games corresponding to Call of Duty Modern Warfare). Nowadays, many games have a web-based element that vastly improves the gameplay expertise and interactivity, typically superseding the significance of the player's offline game objectives. grew extra in style with the release of Marathon on the Macintosh in 1994 and particularly after first-person multiplayer shooter Quake hit shops in 1996. By this point, the discharge of Windows 95 and inexpensive Ethernet playing cards introduced networking to the Windows PC, further expanding the recognition of multiplayer LAN games. Zynga is a number one developer of the world's most popular social games which might be played by hundreds of thousands of individuals all over the world each day.
Travel & Leisure Names Winter Park Considered One Of America's Most Romantic Cities
Mon May 24 , 2021
Cover price is $6.99 an issue, current renewal price is 12 points for $40.00. Travel + Leisure, printed by Meredith, at present publishes 12 instances yearly. Cover price is $6.ninety nine an issue, current renewal fee is 12 points for $37.00. Food & Wine, printed by Meredith, currently publishes 12 […]


text:
Pataliputra (en sánscrito) o Pataliputta (en palī), fue una antigua ciudad del noreste de la India localizada a orillas del río  Ganges. Actualmente está en ruinas y prácticamente englobada en su sucesora, Patna.

Pataliputra fue fundada en el siglo V a.C. en tiempos de Buda Gautama, quien según las crónicas residió en ella.

El griego Megástenes la menciona en su libro Indika con el nombre de Palibothra.

Pataliputra fue una de las capitales del reino de Magadha. Se considera que esta ciudad fue sede del tercer gran concilio budista, realizado en el 341 a. C.; concretamente fue la capital del Imperio Maurya en tiempos de Aśoka (siglo III a. C.) y también fue alcanzada por las fuerzas del Reino indogriego en tiempos de Menandro I.

Luego fue saqueada entre los siglos V y VIII por invasores kushán, saka y principalmente por los heftalitas. Parte de su antiguo esplendor quedó restaurado en el siglo XVI con la instauración del Gran Mogol, aunque a partir de entonces su historia se confunde con la de Patna.

Localidades de Bihar
Antiguas capitales nacionales de India
Ciudades santas del budismo
Localidades establecidas en el siglo V a. C.
Localidades a orillas del río Ganges


text:
Catholics in the Archdiocese of Philadelphia not wishing to attend Sunday Mass for fear of spreading or contracting the coronavirus are no longer obligated to do so. The following options for viewing the Mass from home are available both on television and streaming online:
Live feed of Sunday Mass from Cathedral Basilica of Saints Peter and Paul and the Shrine of Saint Katharine Drexel, Philadelphia, PA - Every Sunday ay 11:00 am
EWTN (Eternal Word Television Network): See the schedule for broadcast times here, and see this link for the Sunday Mass streamed live online.
WPVI TV, Channel 6: Sunday Masses at St. Malachy Parish, Philadelphia are aired every Sunday morning from 5:30 to 6 a.m.
CatholicTV: Daily and Sunday Masses are broadcast live and available for later viewing online.
Also click below to select a televised Mass streaming live from parishes and shrines in the Archdiocese of Philadelphia:
St. Raymond of Penafort Parish, Philadelphia - Sundays at 10:00 am
St. John Neumann Parish, Bryn Mawr - Sundays at 10:30 am
St. Cornelius Parish, Chadds Ford - Sundays at 11:00 am
St. Mary Magdalen Parish, Media – All Masses streamed
Miraculous Medal Shrine, Philadelphia - Monday 9:00 am Novena and Mass


Watch Jake on the rise.
Jake may be a bit plain but he's certainly no lostie in the modern men's style space. Overnight Jake Gyllenhaal rocked up to the 'Clash De Cartier' launch in Paris looking rather….Parisian in an ascot a less formal version of a cravat that's tied like a scarf.
Gyllenhaal's look was all about the black tuxedo jacket played down without a bow tie. The ascot is a rather simple menswear accessory to pull off and much of its success in a man's outfit is derived from confidence. In this particular case Gyllenhaal played it safe by going monotone. It's only when bold colours and patterns are involved that the look can get harder to pull off. See some of the men below to see what we mean.
Nonetheless Gyllenhaal made it work in his favour especially for a more formal event like Cartier.
He paired the look with what appears to be black polished Cuban heeled shoes (not that he needed it) and a Cartier Santos timepiece which paired perfectly with the whole ensemble.
Here's a closer look at the immaculate look which Gyllenhaal gave a seriously masculine edge. With the 38-year-old actor set to play a villainous role in the upcoming Spiderman movie, we look forward to seeing what else the man comes up with on the red carpet.


SeaWeb is the only international, nonprofit organization exclusively dedicated to using the science of communications to fundamentally shift the way people interact with the ocean. We transform knowledge into action by shining a spotlight on workable, science-based solutions to the most serious threats facing the ocean, such as climate change, pollution and depletion of marine life.
GRACE Communications Foundation重视食品、水和能量的相互关系,通过一些在线项目为消费者、拥护者以及政客传递相关知识,包括Sustainable Table, Eat Well Guide, The Meatrix, Network for New Energy Choices, H2O Conserve 和 the Ecocentric blog.
Andean Collection was founded to bring sustainable change to impoverished communities in South America. We offer artisans the opportunity to participate in the global market while inspiring our customers with access to the elusive world of the rural Andes through clothing and jewelry. We have a non-profit arm, Andean Project, to ensure that this change is productive and healthy and to address other poverty related social issues, all funded through our sales and private donations.
Product Policy Institute (PPI) is working to transform our "throw-away" society into a vibrant one, where we meet the needs of all people with sustainable, energy-efficient goods and services. Our goal is to develop cradle-to-cradle producer responsibility – which makes manufacturers responsible for collecting and recycling their products and packaging – into the central approach to designing and managing products so that market forces drive the use and reuse of safer chemicals and materials.
Justmeans is the world's leading source of information and connections for the sustainable business industry.