\section{Introduction}

The COVID-19 pandemic has brought critically neglected areas of infection control onto the global stage \cite{dancer2020}.
Most notably this includes the risk of airborne transmission
and the strategies required to mitigate it \cite{morawska2020}.
The prevailing view presupposes a binary classification between \emph{large} and \emph{small} droplets (also called aerosols), which respectively transmit disease via the \emph{droplet} and \emph{airborne} routes.
This dichotomy originates in research conducted in the 1930s \cite{wells1934}, and underlies current measures for infection control.
Arbitrary threshold diameters (typically \SIrange{5}{10}{\um}) are widely used to distinguish aerosols from large droplets, which have been critiqued by various authors \eg in \Refscite{tellier2009,asadi2019,vuorinen2020,bourouiba2020}.
In any case, the size distribution of respiratory particles (and their resulting dynamical behaviour) is wide and continuous.

Many outbreaks are suggestive of airborne transmission \cite{morawska2020,anderson2020},
aerosolised SARS-CoV-2 virus remains viable under laboratory conditions \cite{vandoremalen2020,fears2020},
viable virus has been cultured from the air surrounding sick patients \cite{lednicky2020},
and its predecessor SARS-CoV-1 is considered to have been transmitted by the airborne route \cite{morawska2020,li2005,xiao2017,yu2005,olsen2003}.
Taken together these present a plausible case that the airborne route is significant in transmission of SARS-CoV-2 \cite{morawska2020,jayaweera2020,kampf2020,anderson2020,azimuddin2020,asadi2020}, although the transmission mechanism by air is complex and many questions remain including the typical viral load of airborne exhaled particles and the required infectious dose.
However, absence of evidence is \emph{not} evidence of absence, and so a precautionary approach is justified to improve public safety \cite{dancer2020,morawska2020,prather2020,greenhalgh2020} similar to that employed to suppress transmission by large droplets or surface contamination where the evidence of transmission is also incomplete \cite{morawskaAirborneTime2020,kampf2020}.

Use of face coverings in public spaces is mandatory or strongly encouraged around the world \cite{masks4allCountries,yougov2020}.
Agencies like the Centers For Disease Control and Prevention recommend members of the public wear reusable fabric coverings \cite{cdc}, whereas disposable surgical masks are more common in East Asian countries such as China \cite{landi2020}.
Widespread use of surgical masks will inevitably send enormous volumes of plastic waste into the environment, which can take decades (or even centuries) to degrade \cite{weinstein2016}; a central question is thus how do fabric coverings perform relative to surgical masks?
The efficacy of both types of face coverings is controversial \cite{vincent2016,greenhalgh2020,delve2020}, largely because the gold standard of medical evidence is not available for their use; that being said, the current evidence is indicative that they reduce airborne spread \cite{greenhalgh2020,delve2020,howard2020}.

It would be unethical to conduct a randomised-placebo controlled trial with high-risk activities such as the spread of SARS-CoV-2, and so we must rely on alternative forms of evidence for the effectiveness of face coverings such as plausibility arguments \cite{greenhalgh2020,smith2003,potts2006}.
Here we lay out such a plausibility argument, by combining experimental literature and theoretical calculations.
Our goal is to suggest more precisely in which scenarios face coverings are expected to be effective in reducing transmission of SARS-CoV-2.
We argue that face coverings do not eliminate the risk of secondary transmissions, especially in high-risk settings such as in hospitals or amongst large crowds, but should significantly reduce the risk in most settings.
Moreover, we will suggest criteria where washable cloth masks are expected to be comparable to disposable surgical masks; these may be preferred because they generate less plastic waste.

\begin{figure}
  \includegraphics[width=\linewidth]{collection-efficiency-panic-complete}
  \caption{
    Variation in mask filtration efficiency with incoming particle size.
    (a) Diagram of capture of viral particles by a fibre within a mask.
    Larger particles are more easily captured because they are less mobile; smaller aerosols by contrast are transported around the fibre by the gas flow.
    Larger particles can also carry more virions, and submicron aerosols are unlikely to contain even a single virion (\cf text).
    The filtration efficiency of perfectly fitting (b) surgical masks and (c) cloth masks formed from 4 identical plain-woven cotton layers are shown as a function of particle size.
    We show experimental measurements from \Refscite{konda2020,zangmeister2020} (points) and predictions from our model (lines) which is described elsewhere \cite{robinson2020} and the Supplementary Material.
    The shaded envelopes around the lines in (c) show the uncertainty in the model predictions, obtained by propagating uncertainties in the geometric parameters given in \Refcite{zangmeister2020}.
    We set the velocity of the gas through the mask to \SI{6.3}{\centi\metre\per\second} in our calculations for comparison with data from \Refcite{zangmeister2020}.
  }
  \label{fig:fabric-filter-efficiency}
\end{figure}

\section{Masks are personal air filters}

A mask or face covering is nothing more or less than an air filter worn on the face.
How air filters work and how efficient they are at filtering out particles of differing sizes is reasonably well understood \cite{robinson2020,wang2013}.
Droplets and coarser aerosols (with diameter $\gtrsim \SI{1}{\micro\metre}$), which are more capable of containing significant viral doses (\cf next section), are more easily filtered because they are less mobile; fine aerosols (\SIrange{0.1}{1}{\um}) by contrast are transported around the fibre by the gas flow.
We illustrate this schematically in \Figref{fig:fabric-filter-efficiency}(a).

In \Figref{fig:fabric-filter-efficiency}(b) we have plotted both measurements of surgical mask filtering efficiency (symbols) and theoretical calculations (curves).
The theoretical calculations involved following the trajectories of particles inserted into the gas flow around fibres, described elsewhere \cite{robinson2020} and in the Supplementary Material (SM).
The efficiency is plotted as a function of the particle diameter, because the particle size ultimately determines how hard or easy it is to filter out.
The measurements and model agree on the same basic facts:
\begin{itemize}
  \item Filtering efficiency is essentially \SI{100}{\percent} for particles $\gtrsim \SI{5}{\micro\metre}$ in diameter or larger.
  \item However, filtering efficiency is low (\SIrange{30}{60}{\percent}) in the range \SIrange{0.1}{1}{\micro\metre}.
\end{itemize}
These predictions make quantitative the picture we laid out in the preceding paragraph and \Figref{fig:fabric-filter-efficiency}(a).
Both surgical and cotton masks are thus only partially effective at filtering out sub-micrometre aerosols.
However, their efficiency rapidly increases as the size increases beyond a micrometre, so masks are generally highly effective in this regime.
For reference, SARS-CoV-2 is approximately $\SI{0.1}{\micro\metre}$ in diameter \cite{bar-on2020}, so any particle larger than this \emph{can} potentially carry a virus.
However, in the next section we will argue that only particles larger than $\gtrsim \SI{1}{\micro\metre}$ are \emph{likely} to contain any virus in the majority of cases; thus filtration efficiency in this regime is sufficient to significantly reduce transmission.

Surgical masks are disordered homogeneous arrangements of fibres (\cf \Figref{fig:fabrics}(b)), whereas cloth face coverings are typically more ordered (\eg woven fabric in \Figref{fig:fabrics}(a)).
More work needs to be done to understand the effect of heterogeneities in cloth masks, but we can obtain some understanding by approximating them as a homogeneous medium \cite{robinson2020} and making a small correction for the effect of inter-yarn pores on collection of submicrometer particles (SM).
In \Figref{fig:fabric-filter-efficiency}(c) we perform such a calculation for two masks formed from 4 layers of plain-woven cotton fabrics sampled in \Refcite{zangmeister2020}.
Despite considerable variation between different fabrics, the behaviour is more-or-less identical to surgical masks; this is broadly in agreement with the findings of \Refscite{zangmeister2020,konda2020,lustig2020,wangData2020} and
demonstrates that reusable cloth masks can be suitable replacements for disposable surgical masks.

Returning to the mechanism, we briefly discuss the physics underlying this behaviour (more details can be found in \Refcite{robinson2020}).
Masks are fundamentally arrays of fibres, see \Figref{fig:fabrics}, so air must flow around and between these fibres.
Particles a fraction of a micrometre in size have very little inertia and so tend to follow the air flow through the mask faithfully and are not filtered.
However, the particle inertia varies with its mass \ie the cube of its diameter (its volume).
This means it rapidly increases with diameter.
Beyond around $\SI{1}{\micro\metre}$ in diameter the particles have too much inertia to follow the air as it twists in between the fibres, and so they impact onto the fibres.
On microscopic lengthscales most surfaces are attractive, so colliding particles will stick and remain on the fibres \cite{robinson2020}.
Because of this basic physics, the filtering efficiency of particles larger than $\sim\SI{5}{\micro\metre}$ is likely to be limited only by the leakage of air around the mask.
For the intermediate range \SIrange{1}{5}{\micro\metre} the exact behaviour will depend on the details of the material, but the rapid rise in filtration efficiency with particle size is a robust feature \cite{robinson2020}.
Finally, we note that the filtration efficiency increases for capture of the smallest aerosols ($\lesssim \SI{0.3}{\um}$) in \Figref{fig:fabric-filter-efficiency}(b-c) where capture is enhanced by Brownian motion.
We have not focused on this mechanism because such small aerosols are highly unlikely to carry significant doses of virus, and our theory is thus less accurate in this region.

\begin{figure}
  \includegraphics[width=\linewidth]{fabrics_circles-panic2}
  \caption{
    Fabrics are broadly categorised as \emph{knitted} (not shown), \emph{woven} or \emph{non-woven}.
    (a) Woven fabrics formed by intersecting perpendicular yarns (the ``warp'' and ``weft'').
    (b) Nonwoven fabrics are formed by entangling fibres through other means, resulting in less ordered arrangements.
    Scanning electron microscope images of example fabrics show scalebars of (a) \SI{100}{\um} and (b) \SI{50}{\um}.
  }
  \label{fig:fabrics}
\end{figure}

It is useful to compare the effect of wearing a mask with social distancing \SIrange{1}{2}{\metre} apart.
Droplets $\sim\SI{20}{\micro\metre}$ (and smaller) can easily travel several metres \cite{xie2007,liuCoughJets2017,bourouiba2014,bazant2020}, so a perfectly fitting mask would be superior to social distancing for droplets of this size.
However, in practice masks are inevitably far from airtight and so many droplets will bypass the mask around the leakage sites (estimated as around $\SI{20}{\percent}$ in \Refcite{hossain2020}), and so there is still a case for social distancing even when all parties are masked.

Neither social distancing over \SIrange{1}{2}{\metre} nor masks are expected to completely mitigate any risk of transmission from sub-micrometre aerosols, because of their long persistence times \cite{vuorinen2020} and the poor filtration efficiency in this range.
The capturing efficiency for particles of diameter \SIrange{0.1}{1}{\um} is only \SIrange{30}{60}{\percent}, which is a possible cause for concern if these fine aerosols can transmit the virus.
Note that respirators (\eg N95/KN95/FFP2) make use of so-called \emph{electret} fibres which can sustain considerable electrostatic charge \cite{chen1993,kravtsov2000}; this increases the efficiency in the \SIrange{0.1}{1}{\um} regime.
In the next section we will argue that such high-performance filters are unlikely to be needed for most members of the public, and the expected efficacy of face coverings is sufficient for preventing community transmission.

\begin{figure}
  \centering
  \includegraphics[width=\linewidth]{droplet-distribution-virus-laden}
  \caption{
    Histograms of key properties of expiratory particles, \ie normalised so that the area under the curves gives the relative frequency.
    (a) Distributions of SARS-CoV-2 viral loads in testing from RT-PCR in two studies \cite{jones2020,jacot2020} (points) and our bimodal fits for calculating percentiles (lines).
    Note the large distribution, and the presence of a tail of patients with extraordinarily large viral loads (shaded purple) corresponding to so-called ``superreplicators''.
    (b-c) Aerosol distributions for virus-laden particles exhaled during speech and voluntary coughing under viral loads typical of the top 25th percentile in (a).
    We show the distributions of aerosols that contain at least one virus (solid lines) and those that bypass a surgical mass worn by an inhaler (dashed lines); the latter are unnormalised to show the effect of filtering.
    We calculate the former distributions using data from \Refscite{gregson2020,johnson2011} to characterise the exhaled aerosols in healthy patients.
  }
  \label{fig:bioaerosol-distribution}
\end{figure}

\section{Which respiratory droplets and aerosols are likely to contain significant doses of virus?}
\label{sec:viral-droplets}

To assess the effect of mask wearing we need to know which respiratory droplets and aerosols are likely to carry significant doses of virus.
We thus need to know the size distribution of exhaled droplets, and how concentrated the (viable) virus is in droplets produced by shedding in an infected individual.

Instead of an arbitrary size cutoff, expiratory particles can be more meaningfully categorised by their site of origin in the respiratory system.
Larger droplets are more likely to be deposited in the respiratory tract \cite{zhang2012,cheng2014,chalvatzaki2020,guo2020}, so a large droplet originating deep in the respiratory tract would immediately deposit and be unlikely to escape; as a rule, smaller droplets emerge from lower in the respiratory tract.
The majority of droplets produced in the oral cavity vary in size from $\sim$\SIrange{10}{1000}{\um} \cite{johnson2011}, whereas droplets produced in the Larynx and the lower respiratory tract are seen in the range $\sim$\SIrange{0.1}{10}{\um} \cite{morawska2009,johnson2011,asadi2019,gregson2020}.
The former presumably contain the majority of virus because volume scales as the diameter cubed \footnote{The concentration of viable virus may be lower in droplets: in influenza it is a factor of $\sim10$ smaller \cite{milton2013}, however even accounting for this a factor of 10 increase in diameter increases the expected number of virions by $\sim 100$.},
but the latter are expected to be responsible for transmission by the airborne route so they will be our focus.

Fluid particles immediately begin to evaporate upon exhalation \cite{xie2007,liuCoughJets2017} and the aerosols will reach their dessicated steady states (which we will refer to as ``droplet nuclei'') in less than \SI{1}{\second} \cite{nicas2005}.
We use parameterisations of the measured size distributions (\ie uncorrected for evaporation) reported in \Refscite{johnson2011,gregson2020} in our calculations.
Their hydrated state would be around a factor of $\sim3$ larger on exhalation, allowing for coverings to filter aerosols more effectively at the source \cite{liuCoughJets2017,robinson2020}.

Testing for SARS-CoV-2 is primarily performed by detecting the presence of viral RNA in respiratory fluid using reverse-transcriptase polymerase chain reaction (RT-PCR).
The distribution of concentrations of viral RNA, the \emph{viral load}, using this technique has been reported in \Refscite{jones2020,jacot2020} which we show in \Figref{fig:bioaerosol-distribution}(a).
The distribution is extremely broad, spanning around 10--12 orders of magnitude: patients in the upper tail of the distribution (the ``superreplicators'') are responsible for hosting most of the virus in circulation, which may be a factor in superspreading events.
Each study sampled the viral loads from thousands of patients in all stages of disease progression.
These studies primarily involved testing naso- and oropharyngeal samples, which we assume to be representative of \emph{all} respiratory fluids where data is lacking.
Finally, we note that there is ample time for evaporation between sample collection and testing in these studies so we assume these concentrations to refer to the dessicated states (the nuclei).

Viral load typically peaks around the onset of symptoms \cite{zou2020} which is also the most contagious stage of disease progression \cite{anderson2020}; we thus expect viral loads from the upper half of the distributions in \Figref{fig:bioaerosol-distribution}(a) to be most relevant to disease transmission.
The actual number of \emph{viable} virus, as measured from viral plaque assays, in aerosol vs the RT-PCR result is typically only one part in \numrange{e2}{e4} \cite{milton2013,fears2020}.
We thus take the upper limit of viral loads as \SIrange{e8}{e10}{\per\milli\litre} instead of \SIrange{e10}{e12}{\per\milli\litre}.

The probability that an aerosol contains at least one virus depends on its volume and the viral load.
By combining this information we can estimate the fraction of particles at any specific size that contains the virus, and thus the variation in the relative number of particles that contain the virus across the size distribution (details of calculation in SM).
In \Figref{fig:bioaerosol-distribution}(b-c) we show the resulting distributions produced in speech and coughing.
For the moderately large viral load of \SI{e8}{\per\milli\litre} the majority of viral aerosols extend into the micron regime $\gtrsim \textrm{\SI{1}{\um}}$.
Only for extremely large viral loads of \SI{e10}{\per\milli\litre} do the submicron droplets begin to contain significant numbers of virus.
Thus submicron droplets should be unimportant vectors in the majority of people infected with SARS-CoV-2.

\begin{table}
  \begin{ruledtabular}
  \begin{tabular}{lccc}
    & \multicolumn{3}{c}{Fraction filtered} \\
    \cline{2-4}
    Expiratory particles & Inhaler & Exhaler & Both \\
    \hline
    Viral speech aerosols & \LineSIrange{47}{85}{\percent} & \LineSIrange{57}{99}{\percent} & \LineSIrange{77}{100}{\percent} \\
    Viral cough aerosols & \LineSIrange{49}{79}{\percent} & \LineSIrange{58}{99}{\percent} & \LineSIrange{79}{100}{\percent} \\
    Droplets $\ge \SI{5}{\um}$ & \LineSIrange{80}{100}{\percent} & \LineSIrange{60}{100}{\percent} & \LineSIrange{92}{100}{\percent} \\
  \end{tabular}
  \end{ruledtabular}
  \caption{Expected efficacy of surgical masks at filtering virus-laden expiratory particles involved in different transmission modes.
    These numbers are expected to be similar for multi-layered cloth masks (\cf \Figref{fig:fabric-filter-efficiency}).
    The lower bounds were obtained assuming large viral loads of \SI{e10}{\per\milli\litre} and 20\% mask leakage on inhalation (consistent with \Refcite{hossain2020}) and an estimated 40\% leakage for exhalation.
    The upper bounds were obtained assuming the median viral load of \SI{e6}{\per\milli\litre} and perfect fit (no mask leakage).}
  \label{table:droplets-removed}
\end{table}

\section{Are face coverings likely to prevent spread in community settings?}

To understand how masks reduce transmission we need to compare the filtration efficiency of masks in \Figref{fig:fabric-filter-efficiency}(b) with the droplets that need to be filtered.
We do this by showing the distribution of viral droplets that pass through a surgical mask in this with dashed lines in \Figref{fig:bioaerosol-distribution}(b-c).
We see that masks remove the majority of viral aerosols, leaving primarily those in the submicron regime as expected.
Moreover the masks are more effective with decreasing viral loads, so with the exception of ``superreplicators'' we anticipate masks to be highly effective at reducing transmission rates from the vast majority of people infected with SARS-CoV-2.
Where transmission by submicron aerosols is possible, masks can only partially reduce the risk.

While our focus has been on the airborne route, we note here that given face coverings are essentially perfect filters of particles larger than \SI{5}{\um} in diameter (assuming perfect compliance and appropriate usage) they should provide excellent protection for transmission by the large droplet route and prevent surface contamination.
As such, we expect them to provide a net-benefit regardless of whether transmission is dominated by the aerosol or droplet routes.

A key limiting factor for mask efficiency is how much leakage there is around the sides of the mask.
For inhalation we expect this to be around \SI{20}{\percent} \cite{hossain2020}, though we are not aware of any data on exhalation so we make a conservative (\ie pessimistic) estimate of \SI{40}{\percent} leakage.
Note that personal masks can be customised to the individual to improve the fit, so these are upper bounds.
Accounting for mask leakage we estimate the total fraction of droplets and aerosols filtered in table \ref{table:droplets-removed} when masks are worn by either the (susceptible) inhaler, the (infected) exhaler or both people.
Even with greater leakage masking the exhaler is generally more effective at preventing airborne transmission than masking the inhaler, because aerosols are larger (and thus filtration is more effective) before the aerosols evaporate into their dessicated states.
We have underestimated the protection from large droplets whilst masking the exhaler because the mask would deflect and remove momentum from the expiratory jets \cite{tang2009,tang2011}; taking this into account, masking the exhaler (``source control'') is expected to offer more protection than masking the inhaler.
In all cases masking both people is more effective as this partially makes up for deficiencies in leakage and the worse filtration in the submicron regime.

\section{Discussion}

The SARS-CoV-2 pandemic has exposed critical knowledge gaps in infection control and prevention \cite{dancer2020,poon2020}.
We believe these gaps necessitate cross-disciplinary collaborations on matters of public policy, and in this spirit we have presented a physical scientists' perspective on the effectiveness of face coverings.

Both masks made from simple cotton fabrics, and surgical masks, are predicted to reduce transmission of respiratory viruses.
Cloth masks can be washed and reused making them preferable to disposable surgical masks because they generate less plastic waste which is better for the environment.
As masks are cheap, and wearing a mask is a relatively minor inconvenience compared to contracting SARS-CoV-2, recommending mask use is a simple way to reduce transmission.
The extent of the reduction depends to some degree on whether transmission is via large droplets or aerosols, however we have argued that in all cases masks are likely to significantly reduce the risk from viral particulates.
Due to the inevitable poor fit of these masks, filtration is never \SI{100}{\percent} effective for any transmission mode, but they filter larger droplets much more efficiently than smaller ones.

Transmission of respiratory viruses is complex and poorly understood, so more data is needed.
We need either direct data on transmission rates as a function of conditions, with and without masks, and a much better idea of the infectivity of aerosolised virus including the required dose for infection
Both of these will be challenging but both are possible.
However, the basic physics of filtration tells us about how capture varies with droplet size.

Both masks and social distancing reduce the exposed dose by some percentage, dependent on how big the droplets are that are involved in transmission.
It now seems well established that with SARS-CoV-2 some infected people have viral loads thousands or millions of times higher than others \cite{jones2020,jacot2020}.
Thus a say \SI{50}{\percent} reduction in dose due to mask wearing corresponds to very different absolute reductions in dose from infected people with high and low viral loads.
As typically the viral load of an infectious person will not be known, other forms of interventions may be warranted in addition to masking.

\begin{acknowledgements}
  The authors wish to thank Kate Oliver for helpful discussions on textiles, Patrick Warren for guidance on LB simulations, Mahesh Bandi for making us aware of his ingenious use of a candyfloss maker, and Mike Allen, Jens Eggers, and Daan Frenkel for helpful discussions
  We gratefully acknowledge Daniel Bonn, Patrick Charbonneau, K.\ K.\ Cheng, Rosie Dalzell, Tanniemola Liverpool, John Russo and Hajime Tanaka for providing valuable comments on this work.

  JFR, JPR and CPR wish to thank the Bristol Aerosol COVID-19 group for valuable discussions and feedback on this work.
  JFR would like to thank Kirsty Wynne for assistance in debugging the code used in the theoretical calculations.
  The authors would like to thank Judith Mantell and Jean-Charles Eloi of the Wolfson Bioimaging Facility and the Chemical Imaging Facility (EPSRC Grant ``Atoms to Applications'', EP/K035746/1), respectively, for the SEM images and assistance in this work.
\end{acknowledgements}

\section{Theoretical model for filtration}

Here, we briefly summarise the theoretical model used to obtain the theoretical filtration curves in the main text.
We have provided a full account of this model in \Refcite{robinson2020}, which goes through this theory in more detail for an audience of fluid mechanics specialists.
The theory is a development in filtration theory, which is widely used in the treatment of air filters.
Our developments have extended its applicability to fabrics, which enabled the agreement between theory and experiments shown in \Figref{fig:fabric-filter-efficiency} in the main text.

\subsection{Particle motion}

We consider particles of radius $a_p$ moving at velocity $\vec{v}$ while being transported by a gas flow field $\vec{u}$.
To model the gas flow, we treat the mask as a system of infinitely long cylinders of radius $a_f$ occupying a volume fraction $\alpha$ inside the mask.
Assuming the effective neighbourhood around each fibre can be modelled by the co-axial region with distance from its centre $r \in [a_f, a_f/\sqrt{\alpha}]$ and Stokes flow yields the Kuwabara flow field for $\vec{u}$ \cite{kuwabara1959,robinson2020}.
We write the characteristic flow speed through the mask as $U_0$, which typically takes on values of $U_0 \sim \mathcal{O}(\SI{1}{\centi\metre\per\second})$ for flows through a face covering \cite{robinson2020}.

Next we insert the particles into the gas flow and follow their trajectories.
Assuming Stokes flow, the particle equation of motion reads
\begin{equation}\label{eq:particle-newton2}
  m_p \frac{d \vec{v}}{d t} = - \frac{\vec{v} - \vec{u}}{B}
\end{equation}
where $m_p$ is the particle mass.
The term on the right hand side is the Stokes' drag.
In this term $B = C/6\pi \mu a_p$ is the particle mobility, with $\mu$ the dynamic viscosity of air and $C$ the Cunningham slip correction factor.
We have assumed that the particle interacts with the flow field as a point particle so that: (a) the flow field $\vec{u}$ is unperturbed by the presence of the particle and (b) the Stokes drag couples only to the particle's centre of mass.

\subsection{Filtration efficiency}

To understand the filtering capacity of a single fibre, we consider the flow around an infinitely long cylinder aligned perpendicular to the direction of flow.
We define the single-fibre efficiency $\eta$ as the fraction of particle trajectories that terminate on the fibre surface.
We further assume that far from the fibre the particles are:
\begin{enumerate}
\item Aligned with the gas flow field, which is uniform before being disturbed by the fibre.
\item The particles are randomly positioned in the gas flow.
\end{enumerate}
The single-fibre efficiency can then be expressed as $\eta = \lambda/W$ where $W$ is the width of the mask (approximately the width of the face), which introduces $\lambda$ as the width of the \emph{collection window}.
If the particle is initially placed inside this window then it will ultimately terminate on the surface of the fibre, otherwise it will bypass the fibre.

In the main text we argued that the main collection mechanism for masks arises from particle inertia (also called ``impaction''), and we indicated that Brownian motion (diffusion) acts as a secondary mechanism for the smallest aerosols.
We respectively write the single-fibre efficiency for these two mechanisms as $\lambda_I$ and $\lambda_D$.
These mechanisms act in distinct size regimes, so their combined efficiency is well-approximated as.
\begin{equation}
  \lambda \simeq \lambda_I + \lambda_D.
\end{equation}
We numerically determined the inertial term $\lambda_I$ by repeatedly inserting particles into the gas flow field at varying initial conditions and integrated \eqref{eq:particle-newton2}.
As diffusion is the less important mechanism in this work, we have simply made use of the standard relation \cite{stechkina1966}
\begin{equation}\label{eq:diffusion-efficiency}
  \frac{\lambda_D}{2 a_f}
  =
  \frac{2.9}{(K \Pe^2)^{1/3}}
  + \frac{0.624}{\Pe}
  + \frac{1.24 R^{2/3}}{\sqrt{K\Pe}},
\end{equation}
where $R = a_p / a_f$, $\Pe$ is the P\'eclet number and $K$ is the hydrodynamic factor emerging from the Kuwabara flow field \cite{kuwabara1959}.

\subsection{Filtration efficiency of masks}

We consider the mask as an assembly of perfectly cylindrical fibres aligned perpendicular to the flow and randomly distributed with uniform probability.
We find that the \emph{penetration}, the fraction of particles that pass through the mask without being filtered, is \cite{robinson2020}
\begin{subequations}\label{eq:total-penetration}
  \begin{align}
  P
  =&
  \exp{\left( - \frac{L}{\xi} \right)},
  \intertext{with penetration length}
  \label{eq:penetration-length}
  \xi
  =&
  \frac{(1 - \alpha) \pi}{4\alpha \overline{\lambda}}
  \int_0^\infty d_f^2 \, p(d_f) \, dd_f,
  \intertext{and effective collection window}
  \overline{\lambda}
  =&
  \int_0^\infty \lambda(d_f) \, p(d_f) \, dd_f.
  \end{align}
\end{subequations}
where $L$ is the thickness of the mask, $\alpha$ is the volume fraction occupied by the fibres and $p(d_f)$ gives the distribution of diameters in the sample which are well-described by a log-normal distribution \ie $\ln{(d_f / \si{\um})} \sim \mathcal{N}(\mu, \sigma^2)$.
We have previously determined the parameters characterising fibres in surgical masks and cotton fabrics from scanning electron microscope images \cite{robinson2020}.

Cotton fibres have $\mu = 2.68$ (corresponding to a modal fibre diameter of \SI{14.6}{\um}) and $\sigma = 0.34$ \cite{robinson2020}.
Woven fabrics are heterogeneous: the fibres are twisted into yarns which are then woven into their final structure as shown in \Figref{fig:fabrics}(a).
To treat this heterogeneity we start by calculating $\lambda$ for an effective homogeneous system, and then we correct for the fact that most of the flow occurs through the inter-yarn pores where the fibres are less dense.
We calculate the average thickness and volume fraction for the plain-woven cotton fabrics considered in \Refcite{zangmeister2020} from their stated values of areal density, yarn thicknesses and yarn densities.
To take this average we assume the yarns are cylindrical so that the maximum thickness is thus the sum of the warp and weft yarn diameters; the average thickness is thus less than this maximum thickness because of the pores and the cylindrical shape of the yarns.
The average volume fraction can then be estimated by combining this quantity with the areal mass and the bulk density of cotton.
For the two fabrics in \Figref{fig:fabric-filter-efficiency}(c) we find average thicknesses of $L=\SI{0.16}{\milli\metre}$ and \SI{0.24}{\milli\metre} and average volume fractions of $\alpha=0.48$ and 0.26.

The flow speed is essentially zero inside the yarn, so most of the flow goes through the inter-yarn pores.
Consequently, compared to flow through a homogeneous material: (i) the effective fibre density will be reduced and (ii) the typical flow speed will be increased.
Effect (i) generically lowers the collection efficiency as there are fewer fibres to collect particles, whereas the effect of (ii) depends on the collection mechanism.
Collection by inertia (impaction) is enhanced by increasing the flow speed, opposing the effect from an effectively reduced fibre density.
After cancellation we thus expect the resulting change in efficiency to be small, and so we do not correct this collection mechanism.
However, the efficiency of collection by diffusion decreases with increasing flow speed, reinforcing effect (i), which is potentially significant.
We correct  $\lambda_D$ by inserting a locally increased flow speed into the P\'eclet number entering \eqref{eq:diffusion-efficiency}.
We estimate the fabric permeability $k$ (the area fraction of pores) from the yarn parameters given in \Refcite{zangmeister2020}, and from this we obtain the flow speed inside the pores as $U_0 \to U_0 / k$.
No fitting was required to match the experiments of \Refcite{zangmeister2020} because all parameters are known.

For surgical masks we found the inner side of the mask was composed of fine fibres with $0.75 \lesssim \mu \lesssim 1$ and $0.35 \lesssim \sigma \lesssim 0.45$, and on the outer side they were coarser with $3 \lesssim \mu \lesssim 3.2$ and $0.07 \lesssim \sigma \lesssim 0.1$ \cite{robinson2020}.
We model this as two layers of thickness $L_1$ and $L_2$ with total thickness $L_1 + L_2 = L \simeq \SI{0.8}{\milli\metre}$ determined from optical microscopy \cite{robinson2020}.
We set $\mu_1 = 0.9$ (mode \SI{2.5}{\um}), $\mu_2 = 3$ (mode \SI{20}{\um}), $\sigma_1 = 0.4$, $\sigma_2 = 0.1$ and the volume fraction as $\alpha = 0.075$ in both layers (determined previously in \Refcite{robinson2020} by weighing samples).
We fit the total penetration to the experimental data to find the relative thicknesses of each layer $L_1$ and $L_2$, with the fit performed while their combined thickness was kept fixed.
Surgical masks are nonwoven (and thus homogeneous) so we did not need to correct for changes in collection efficiency due to large inter-yarn pores.

\section{Exhaled aerosol distributions containing at least one virus}

The average number of virions in a spherical homogeneous particle of diameter $d_p = 2a_p$ with viral concentration $v$ is
\begin{equation*}
  \langle n_v \rangle = v \frac{\pi d_p^3}{6}.
\end{equation*}
Assuming the Poisson distribution, the probability that a particle of this size contains at least one virion is
\begin{equation*}
  p(v|d_p) = 1 - e^{-\langle n_v \rangle}.
\end{equation*}
This probability is negligible for $d_p \lesssim \SI{1}{\um}$ for $v \ll \SI{e10}{\per\milli\litre}$, as we stressed throughout the main text.
Application of Bayes' theorem gives the probability distribution for viral aerosols as
\begin{equation*}
  p(d_p|v) \propto p(v|d_p) p(d_p).
\end{equation*}
with the proportionality constant ensuring that the final distribution is normalised.
The quantity $p(d_p)$ is the distribution of \emph{all} exhaled aerosols (\ie viral or otherwise), which we take directly from the fits to experimental data in \Refscite{gregson2020,johnson2011}.
Multiplying these distributions by $d_p$ gives the probabilities normalised with $\ln{d_p}$ instead of $d_p$; we did this in \Figref{fig:bioaerosol-distribution} so that the area under the curve corresponds to the frequency.


\section{Introduction}
In the past, the usage of robots has been confined to heavy industry (e.g. automotive) where the manipulation tasks could be performed in constrained environments with limited human presence. Although this technology was very successful in this setting, it is inadequate for some of the growing areas, such as the light manufacturing industries (e.\,g., food, consumer electronics) and domestic robotics (e.\,g., home assistants and companions).

In order to create robots capable of living and operating in the same space with humans, robots have to be able to interact and co-operate with them.
Manipulating objects of daily living scenarios is an easy cognitive task for humans, who are capable to generalise their grasping and manipulation skills to a wide spectra of materials and shapes. This creates the necessity of developing robots that can cope with such object diversity while performing handover and manipulation tasks. This becomes even more challenging when dealing with previously unseen objects as their dimensions, mass and material are unknown.
While it is relatively simple for humans to estimate the physical properties (such as mass, stiffness) of the object using vision and other sensing modalities (e.g. tactile, force feedback), it still is an open problem in robotics.

In recent decades, the capabilities of computer vision algorithms have increased significantly due to increased availability in labeled vision data and more performing vision algorithms, based on learnt visual features (such as \cite{he2016deep}).
Computer Vision researchers have put efforts into developing solutions that support robotic manipulation. However, they rely mostly on pre-selected sets of object models (e.g. \cite{peng2019pvnet}, \cite{wang2019densefusion}, \cite{xiang2017posecnn}). For real robotic scenarios, it is important to maintain a certain level of flexibility, reducing the prior knowledge required to deal with the wide spectra of situations required for the collaboration between a human and a robot.
Furthermore, while vision is very effective in localising and estimating the physical properties of solid objects (\cite{phillips2016seeing}, \cite{Wang_2019_CVPR}, \cite{wang2019fast} \cite{hampali2020honnotate}, \cite{sajjan2020clear}, \cite{kokic2019learning}), it is much harder to estimate the filling properties of container objects with different possible types of filling, unless the container is transparent \cite{mottaghi2017see}.
Previous works, such as \cite{liang2019making}, \cite{liang2020robust} and \cite{griffith2012object}, have proved that sound and haptics can be used to estimate the quantity and quality of the filling in a container.
Thus, combining different types of modalities can improve to estimate relevant object features that can support safe and accurate handovers.

In this work, we introduce an approach to estimate the filling mass of a container object using audio and multi-view RGB-D sensing modalities. In particular, we split the problem of estimating the filling mass of an object (container) into 3 tasks. We rely on the CORSMAL Container Manipulation Dataset \cite{CORSMAL} which is designed for this problem.
\begin{enumerate}[label=$T_{\arabic*}$]
    \item \label{task:filling_type}\textbf{Filling type}.
    In this task, we predict the type of content present in the container object, if it is not empty. Thus, the possible types of content are: empty, pasta, rice, or water (for drinking glasses and cups). The classification of the filling level is performed combining 2 audio-based classifiers: the first one uses ``classical'' audio features (MFCCs, chromagram, energy, spread, etc.) \cite{giannakopoulos2015pyaudioanalysis} in a random forest classifier, while the second one uses VGGish features \cite{Hershey2017} in a GRU \cite{chung2014empirical} model.
    \item \label{task:filling_level}\textbf{Filling level}.
    In this task, we estimate the percentage of the container that is filled with the filling items. The percentage values are discretised into 3 classes: 0, 50, and 90 percent. The classification of this task combines three models: an RGB-based and two audio-based classifiers. The audio-based models are similar to the \ref{task:filling_type} while the RGB-based classifier uses the R(2+1)d features and a GRU model.
    \item \label{task:container_capacity}\textbf{Container capacity}. In this task, we estimate the volume of the container object. Different from the previous tasks, the target value is a real number.
    For this task, RGB-D + IR data is used to localise the object and estimate its dimensions \cite{2019Xomperomultiview}. The volume is then computed using a cylindrical approximation.
\end{enumerate}
Finally, once estimated the filling type ($T_1$), filling level ($T_2$), and container capacity ($T_3$), we can calculate the mass of the object easily, assuming a pre-estimated average mass density for each filling type.
The solution that we present resulted the top-ranking one for overall performance among the participants of the CORSMAL Challenge 2020 \cite{2020matilla}.

The main contributions of this article are the following:
\begin{enumerate}
    \item We present an approach to estimate the container's filling mass based on several different modalities (audio, RGB, IR, and depth) which is formulated as the combination of three sub-tasks: estimation of the filling level of an object, the filling type, and its capacity.
    \item An experiment is performed on a public dataset provided in the context of the CORSMAL Challenge 2020 \cite{2020matilla}. Our model achieves state-of-the-art performance on both public and private test subsets proving generalization capabilities on previously unseen objects.
\end{enumerate}

Please note that this work is conducted in the context of a competition and some decisions are taken under time limitation.

The paper is organized as follows. Section~2 describes our approach to the estimation of the container filling mass and each of the sub-tasks individually as well as the implementation details. The experimentation is presented in Section~3, which includes dataset description, metrics, and results. Section~4 concludes.
\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{main_fig.pdf}
    \caption{The design of an individual model for each sub-task: container filling level (left), filling type (middle) classification, and container capacity estimation (right). For the \textbf{filling level} estimation we rely on both modalities: RGB (R(2+1)d features) and audio (``classical'' and VGGish features). In particular, we encode streams from each camera placed at different view points with the individual GRU models. Next, we sum up the logits and apply softmax at the output. Next, we process VGGish featured extracted from the audio stream and process it with another GRU model which outputs the class probabilities. Then, the ``classical'' features are classified with the random forest algorithm. Finally, the probabilities from all three models are averaged to produce the final prediction for the filling level. The procedure for \textbf{filling type} classification resembles the one for the filling level except for the absence of RGB stream. The \textbf{capacity estimation} pipeline starts with the extraction of two frames from two camera views (C1 and C2): the 1$^{\text{st}}$ and the 20$^{\text{th}}$ frames to the end. The frames are passed through the Mask R-CNN to detect a container. The detected bounding boxes are used to crop both frames. The crops are sent to the LoDE model which outputs the estimation of the container capacity. If an object was not detected on either of the frames, we use the training prior.}
    \label{fig:main}
\end{figure}

\section{Our Approach}
The task of container mass estimation requires solving three individual sub-tasks: container's filling type and filling level as well as container's capacity estimation. The first two are classification tasks and the latter one is 3D localization. We solve each of the three sub-tasks individually and using the obtained predictions to calculate the mass of a container. We summarize the approach in Figure~\ref{fig:main}.

\subsection{Filling Level and Filling Type Estimation}
The first two sub-tasks require to estimate the filling level and filling type of the container. We formulate both sub-tasks as classification problems. We combine the details for both sub-tasks in one section as we are using a similar approach which differs only in the number of output classes. The approach to these two sub-tasks could be roughly divided into three parts: feature extraction, a classification model, and how the predictions are combined if several classifiers were used.

\subsubsection{Feature Extraction}
Being some of the containers not transparent, it would be difficult to rely on vision to recognise the filling type.
Thus, it can be assumed that a model designed for \textit{filling type} classification should mostly benefit from audio input as a human, if blinded, could distinguish what is being poured by relying on the sound (e.\,g. rice vs. water).
Similarly, for the \textit{filling level}, the audio might provide information if the container was empty or almost full. However, it might also be useful to let a model utilize some information from the vision as some of the containers are transparent and the current \textit{filling level} could be guessed from the visual information. However, for the opaque containers, vision is not informative. Therefore, we rely on audio in both and additionally employ visual modality for the \textit{filling level} task.

To extract audio features for the filling type classification, we rely on both ``classical'' audio features and features from a pre-trained deep learning model (VGGish \cite{Hershey2017}).The ``classical'' features are extracted in two phases. In the first phase, we extract short-term features for each $50$ ms window of the audio. These features include the MFCCs, energy, spectral characteristics, zero crossing rate, chroma vector and deviation \cite{giannakopoulos2015pyaudioanalysis}. Then, these short-term features are used to derive long-term features that are fed to the classifier. Long-term features summarize a larger portion of the audio. These contain the means and standard deviations of multiple short-term features. There are 136-dimensions of features in the final model.

The  features are extracted from every $0.96$ seconds of the original one-channel audio and represented as a 128-d vector obtained from the output of the pre-classification layer. The  features form a tensor of size $ \mathbf{R}^{T_{\text{vgg}} \times 128} $ for each event where $ T_{\text{vgg}} = \text{floor}(T_{\text{sec}} / 0.96) $, $T_{\text{sec}}$ is the duration of the audio in seconds.

For the filling level estimation sub-task we also utilize visual information, which was encoded with a pre-trained model for action recognition (R(2+1)d RGB-only \cite{tran2018closer}). The R(2+1)d features are a 512-d vector extracted from the pre-classification layer which spans about 0.5 second of the original video (16 RGB frames @ 30 fps). Therefore, for each RGB stream obtained from a camera, we extract $ \mathbf{R}^{T_{\text{r21d}} \times 512} $ tensor where $ T_{\text{r21d}} = \text{floor}(T_f / 16) $ with $T_f$ being the total number of frames in a video.
These specific feature extractors (VGGish and R(2+1)d) as they show promising results not only on the tasks they are trained to solve (sound and action classification) but also when applied for other tasks (\cite{MDVC_2020}, \cite{BMT_2020}, \cite{Liu19a}). Adding these extra features are also motivated with the fact that it is useful to have more diverse individual ``opinions'' (predictions) for a stronger combined prediction \cite{breiman1996bagging}, \cite{king2000better}.

\subsubsection{Classification Models}
We applied the ``classical'' audio features to both SVM and random forest classifiers \cite{Statistics01randomforests}. We selected the random forest as it performed better on the validation set. The random forest fits a set of decision tree classifiers on random subsets of observations from the dataset as well as the random subset of features.

Considering that the VGGish and R(2+1)d features are sequence and the expected output of our model is the class label for filling type and filling level this problem can be treated as ``many-to-one'' type. For these types of problems, an RNN is usually considered as a natural candidate. In this work, we tried both LSTM \cite{hochreiter1997long} and GRU \cite{chung2014empirical} but the latter one was preferred due to both efficiency and accuracy.

The last hidden state at the top layer of GRU is used for classification. Specifically, we pass it through one fully-connected layer which maps the hidden state space into the space of labels (for instance, 3 logits for the filling level and 4 for the filling type tasks). Since we may have several RGB sequences (an event is recorded from several cameras) we may aggregate predictions by summing the individual logits obtained from each RGB sequence, e.\,g. R(2+1)d features. Note that we do not take into consideration the fact that if the container is a food box, the filling type could not be liquid.

\subsubsection{Post-Processing}
We aggregate the predictions from the models: random forest on ``classical`` audio features, GRU on VGGish features, and, for the filling level, also the GRU output aggregated from multiple cameras on R(2+1)d features. In our work, we use the simple average of probabilities (``opinions'') from each model to form the final prediction for the sub-tasks.

\subsection{Container capacity Estimation}
This task requires the estimation of the capacity of the container. Since the shape and size of a container can vary significantly and the model could be tested in different scenarios, it is a challenging task to tackle.
Our approach builds on the Localisation and Object Dimensions Estimator (LoDE) algorithm \cite{2019Xomperomultiview}. LoDE is a method based on Mask R-CNN \cite{he2017mask} to simultaneously localise objects used as a container and estimate their dimensions. It makes use of RGB frames extracted from the two calibrated RGB cameras positioned on the left and right corners of the room.
The depth and infrared data are also used to improve robustness.

A mask of the object is created using a Mask R-CNN ResNet-50-Feature Pyramid Network (FPN) \cite{huang2017speed} pre-trained on Common Objects in Context (COCO) dataset~\cite{lin2014microsoft}.
The two 2D centroids ($x^{1}$ and $x^{2}$) are estimated in the segmented images from the left and right camera.
To estimate the 3D centroid ($X$) of an object, a triangulation with the two 2D centroids is performed.
The shape of the object is then predicted by initialising a cylindrical model around the estimated 3D object centroid which iteratively fits the object shape.
The complete algorithm is given in \cite{2019Xomperomultiview}.
Figure \ref{fig:LoDEAlgorithm} \cite{2019Xomperomultiview} shows the above-mentioned method. If the algorithm fails to detect the container, we employ the prior from the training dataset as our prediction.
\begin{figure}[t]
    \centering
    \includegraphics[width=\textwidth]{LoDEAlgorithm.png}
    \caption{Two frames $m^{1}$ and $m^{2}$ extracted from cameras $C^{1}$ and $C^{2}$ from two different viewpoints (left and right angulation). The 2D centroids, $x^{1}$ and $x^{2}$, are used to triangulate the 3D centroid $X$ of the object. Image published in \cite{2019Xomperomultiview}.}
    \label{fig:LoDEAlgorithm}
\end{figure}

After the algorithm predicts the width and height of the object, we determine the capacity with the following equation which we designed for the task:
\begin{equation}
    C = \overline{r}^{2} \cdot h \cdot \pi
\end{equation}
where $C$ is the estimated capacity, $\overline{r}$ the average of the radius of the calculated cylindrical shape and $h$ the estimated height of the object.

\subsection{Implementation Details}
We rely on \texttt{pyAudioAnalysis} library\footnote{\href{https://github.com/tyiannak/pyAudioAnalysis}{\UrlFont{github.com/tyiannak/pyAudioAnalysis}}} to extract the ``classical'' features from the audio and tune the hyper-parameters of the random forest model \cite{giannakopoulos2015pyaudioanalysis}.
VGGish and R(2+1)d features are extracted using \texttt{video\_features}\footnote{\href{https://github.com/v-iashin/video_features}{\UrlFont{github.com/v-iashin/video\_features}}} script. To estimate the capacity of the containers, a modified version of the official LoDE codebase\footnote{\href{https://github.com/CORSMAL/LoDE}{\UrlFont{github.com/CORSMAL/LoDE}}} was used.

We tune only the \emph{number of trees} parameter of the random forest classifier. The other parameters are kept as the default values. We retrain the model with different values of this parameter and choose the value that achieves the highest accuracy in the validation set.
To train the GRU models we employ batching of size 64 while padding the shorter sequences in a batch to the length of the longest sequence. The sizes of the hidden states are 512. The GRU has 5 layers for VGGish features and 3 layers for R(2+1)d features. We train the GRU models for up to 30 epochs with Adam optimizer \cite{kingma2014adam} with learning rate $3\cdot 10^{-4}$ and cross-entropy loss. These hyper-parameters are selected on the 3-fold cross-validation which is described next.

Since there is uncertainty about the objects which will occur during the inference, a model requires to be capable to generalize its predictions to unexpected forms and sound feedback of containers. To prevent our model from overfitting, we employ 3-fold cross-validation.

The splits are constructed following the distribution of the hold-out test set. Specifically, we split ``train'' objects according to their type 2:1 such that the validation dataset will have one object and the other two are used for training. Therefore, we had 3 train:validation splits with 6:3 objects (if 9 objects are in the training set) in each such that each object appears in a validation set once.

Running the training and evaluation scripts are executed on one 1080Ti and takes about 8 hours with R(2+1)d feature extraction allocates more than half of the execution time. This time can be significantly reduced by increasing the number of GPUs. We provide the \texttt{Docker}\footnote{\href{https://hub.docker.com/r/iashin/corsmal}{\UrlFont{hub.docker.com/r/iashin/corsmal}}} image to reproduce the results and the source code\footnote{\href{https://github.com/v-iashin/CORSMAL}{\UrlFont{github.com/v-iashin/CORSMAL}}}.
\begin{table}[t]
\centering
\setlength{\tabcolsep}{5pt}
\begin{tabular}{lrrrrrr}
\toprule
  & \multicolumn{3}{c}{\textbf{Validation Set}} & ~ & \multicolumn{2}{c}{\textbf{Test Set}} \\ \cmidrule{2-7}
\textbf{Sub-task} & \textbf{``class.'' feats.} & \textbf{VGGish} & \textbf{R(2+1)d} & ~ & \textbf{Public} & \textbf{Private} \\ \midrule
Filling Level & 69.9 & 75.5 & 74.7 & ~ & 78.14 & 81.16 \\
Filling Type & 93.3 & 91.3 & -- & ~ & 93.83 & 94.70 \\\bottomrule
\end{tabular}
\vspace{1em}
\caption{Performance of individual classification models for the container's filling level and filling type sub-tasks. The results are shown using the average F1-measure obtained by averaging the scores obtained with 3-fold cross-validation and the values obtained from the CORSMAL Challenge 2020 leader board. \label{tab:val}}
\end{table}

\section{Experiments}

\subsection{Dataset} We train our model on the official training set of the challenging audio-visual-inertial CORSMAL dataset \cite{CORSMAL}. The CORSMAL dataset is a collection of human interaction with different containers recorded with multi-sensor devices: 8-element circular microphone and 4 cameras from different viewpoints recording RGB, IR, depth, IMU streams.

The dataset consists of three parts: public train, public test, and private test. The public train holds 9 containers: 3 plastic drinking cups, 3 drinking glasses, and 3 cardboard food boxes (e.\,g. cereal); 1 cup, 1 glass, and 1 food box is reserved for the public test set. The private test is hidden and known to have 3 more containers. The difference between the public and private test sets is in the absence of information about the input.

A container can be filled with rice, pasta, or water (only glasses and cups) and at 3 different levels: 0, 50, and 90\,\% with respect to the capacity of the container. All combinations of containers are executed by 12 different subjects for 2 different backgrounds and 2 different illumination conditions. In total, there are 1140 different combinations.

\subsection{Metrics} To track the performance of our classification algorithms, we employ the F1-score which favors a balance between precision and recall. The metric is weighted to account for the class imbalance. While the modified absolute error was used in evaluating container capacity. For a more detailed explanation of how the metrics are calculated on the public and private subsets of CORSMAL, we refer a reader to the technical document provided by CORSMAL Challenge 2020.

\subsection{Results}
\begin{table}[t]
\centering
\setlength{\tabcolsep}{5pt}
\begin{tabular}{l r r}
\toprule
                             & \multicolumn{2}{c}{\textbf{Test set}} \\ \cmidrule{2-3}
\textbf{Sub-task}                & \textbf{Public}   & \textbf{Private}  \\ \midrule
Filling Level                & 78.14             & 81.16             \\
Filling Type                 & 93.83             & 94.70             \\
Container Capacity           & 60.56             & 60.58             \\ \midrule
Overall Performance          & 64.98             & 65.15               \\\bottomrule
\end{tabular}
\vspace{1em}
\caption{The results from the CORSMAL Challenge 2020 leader board by November 23rd 2020. The performance is shown using the modified absolute error between the estimated filling mass and the ground truth. \label{tab:test}}
\end{table}

\subsubsection{Container Filling Level Classification}
We compare the performance of the container's filling level between each classification model individually. Table~\ref{tab:val} (first row) shows the results of the classification models averaged among three validation folds on the F1 metric. According to the results, the performance of the VGGish-based GRU model (audio) performs on par with the GRU model which is based on R(2+1)d (visual) features (75.5 and 74.7) while the random forest-based on ``classical'' features performs the worst (69.9), yet adding it to the final combination improves the overall performance.

The results of the combination of three classification models show the importance of aggregation of the individual predictions as the results on both public and private sets are higher than of the individual models. We also highlight the fact that according to the difference between private and public scores our model has not shown any sign of over-fitting but rather a strong generalization capability.

\subsubsection{Container Filling Type Classification}
Similar to the filling level sub-task, the classification results of filling type models are compared individually (see Table~\ref{tab:val}, the second row). The results show the importance of audio modality and ``classical'' features, in particular. The random forest classifier achieves 93.3 F1 score which slightly higher than VGGish-based GRU with 91.3 F1 score.\footnote{We also trained the GRU model on R(2+1)d features, yet it reached only F1=67.3.}

For this sub-task, we also observe the improvement from combining the predictions. Yet, fewer gains are obtained in performance possibly because we rely on only one modality (audio). Nevertheless, the model also preserves the generalization capabilities and absence of over-fitting.

\subsubsection{Container Capacity Estimation}
For the prediction of the container capacity, a modified version of the LoDE method is implemented.
The original LoDE algorithm was developed for calculating the capacity of transparent objects and was tested on a supervised environment with no occluded objects.
For the CORSMAL challenge, the objects were manipulated by humans with multiple occlusions and additional items in the camera view.

The three target objects for the public set of the challenge are a beer cup, a cocktail glass, and a pasta box.
Since the original Mask R-CNNs were pre-trained on the COCO dataset, there was no label to recognise a pasta box or other similar boxes. To alleviate this, an additional label was introduced.
The label is equal to the book (present in the COCO dataset), considering the similarity of its shape to a box.
For the future, transfer learning will be considered to better adjust the final weight of the fully connected layer of the Mask R-CNN and better segment the objects.

In the dataset, some videos start with an empty room (no object or person in it).
To avoid having a frame with no object to detect, two frames were extracted from the videos: the first and the $20^{th}$ to last frame.
An average of the capacities detected in the two frames was then used for final validation.

Figure \ref{fig:task3_results} shows two examples of the resulting capacity estimation through LoDE estimation. For the beer cup (object 10), occlusion by the human hand breaks the mask into two.
Thus, predicting a smaller capacity for the container than the real one.
For the pasta box (object 12), the model correctly identifies the container object, although not trained for it.
The calculated capacity, similarly to the beer cup, is smaller than the ground-truth, because of the occlusion by the manipulator's hands.
Nevertheless, the model achieves an accuracy of 60.56\% when tested on the public dataset and 60.58\% on the private dataset, for an overall of 60.58\%, which is higher than the average (see Table~\ref{tab:test}).
\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{task3_results}
    \caption{Examples showing the limitations of the capacity estimation of two of the objects (10 and 12) of the CORSMAL Challenge. The object 10 is occluded by the hand and the object 12 is predicted to be the wrong object, because the training set did not contain boxes. }
    \label{fig:task3_results}
\end{figure}

\subsubsection{Container Filling Mass Estimation (the overall task)}
Table~\ref{tab:test} (bottom row) shows the results of the container filling mass estimation which takes into consideration the performance of all three sub-tasks together. Since the overall performance metric favours the least ``performing'' sub-task in the final score we believe it could be further improved by exploring the container capacity estimation sub-task. Our approach outperforms all other submissions to CORSMAL Challenge 2020 leaving us on the top of the leader board (see Figure~\ref{fig:lb}).
\begin{figure}[t]
    \centering
    \includegraphics[width=\textwidth]{lb.png}
    \caption{The state of the leader board on 22$^\text{th}$ of November 2020 showing our team (``Because It's Tactile'') placed first in the overall task (filling mass estimation) \cite{corsmalwebsite}.}
    \label{fig:lb}
\end{figure}

\section{Conclusion}
Complex real-world sensing tasks as those posed in the CORSMAL 2020 Challenge require analysis of multiple data modalities. The amount of information gained by a modality is dependent on the nature of the task and it may be predicted by human intuition. However, a modality may still improve the combined performance, despite having poor performance alone. We observed this in our experiments. We predicted audio modality to be informative in filling level and type classification. Indeed, it achieves decent performance. However, the addition of visual features improves overall accuracy. Besides different modalities, using multiple models that follow different paradigms on the same modality increased the accuracy of our predictions. Achieving the highest score in this challenge was possible thanks to the variety in both the modalities and the algorithms.

In future works, other modalities such as IMU and tactile sensing can be explored. Also, the capacity estimation algorithm will be further improved by training the model for the Mask-RCNN which will permit to recognize objects more reliably.
In addition, the LoDE model will be ulterior improved by developing optimal capacity evaluation calculations based on the label of the recognised object.

\bibliographystyle{splncs04}