text:
\section{Security Analysis and Limitations}
\label{sec:analysis}

In this section, we conduct an informal security analysis of {\scshape BYOTee}\xspace,
in which we discuss the attacks {\scshape BYOTee}\xspace can and cannot defend from.

\subsection{Malicious and Compromised Hardcore System Software}
\textbf{Read and Write to Enclaves.}
The hardcore system software, including the operating system and UA, is not a part of the TCB in {\scshape BYOTee}\xspace.
Even if the hardcore system software is compromised at run-time, the attacker cannot access the data on/from enclave hardware resources, including BRAM and peripherals, because they are in the isolated address space of the target enclave.
The attacker cannot breach the confidentiality of SSA code and data as well, because they are encrypted at build time.
The enclave-hardcore system 2-way communication is based on interrupts and the shared DRAM.
Malicious hardcore system software can raise the interrupt to the enclave to carry out a DoS attack.
Utilizing priority interrupts in sophisticated softcore processors, {\scshape BYOTee}\xspace can prevent these attacks from the hardcore system side.

\textbf{Reconfiguring FPGA.} The secure or trusted boot, i.e., static root of trust, process of {\scshape BYOTee}\xspace ensures the secure loading of the bitstream to configure the FPGA at boot time.
An alternative dynamic root of trust approach ensures a compromised hardcore system software cannot load an attacker-controlled bitstream to reconfigure the FPGA if the hardware supports run-time reconfiguration.

\subsection{Compromised {\scshape Firmware}\xspace and SSAs}
If the software layer of an enclave, i.e., {\scshape Firmware}\xspace and SSA, is compromised at run-time by untrusted input sent by the hardcore system software,
it can disclose information in the compromised enclave address space, including data on the BRAM and data from the connected peripherals.
But it cannot read data from the BRAM or peripherals of other enclaves, since they are in different address spaces and cannot be seen by the compromised software.
Therefore, the attack is confined within the compromised enclave.

\subsection{Hardware IPs and Malicious Peripherals}
Malicious hardware IPs cannot be loaded since a bitstream is signed by a trusted developer and verified before loading.
Even if peripherals are malicious and send out rogue DMA requests to access sensitive memory regions,
they are confined in the enclave they are assigned to.
Therefore, a malicious peripheral can only cause limited damages.

\subsection{Cold-boot Attack}
\label{subsec:analysisColdBoot}
Cold-boot attacks rely on the observation that the contents in memory are not immediately erased after power is lost.
While cold-boot attacks on DRAM even at
room temperature are proven very effective~\cite{halderman2009lest}, attacks on SRAM without external power sources are less feasible~\cite{rahmati2012tardis}.
Most data {\scshape BYOTee}\xspace stores on the DRAM is either encrypted or does not need to be protected.
For instance, even if the SEB is located on the DRAM and subject to cold-boot attacks,
the SSA*, which includes developer keys, is encrypted.
Obviously, \emph{Chal}, \emph{PreExecAtt}, \emph{PostExecAtt} do not need to be protected.
It is, however, possible to dump the input and output fields of the SEB using cold-boot attacks on DRAM.
Other sensitive data, such as developer keys, plaintext SSA, program states, are placed on an enclave's BRAM.
Cold-boot attacks on BRAM are difficult because:
i) the BRAM cells are hardware initialized during FPGA configuration in many SoC FPGA systems~\cite{bramInit};
ii) even without initialization, the contents in BRAM decays faster~\cite{rahmati2012tardis};
iii) BRAM is embedded on-chip and cannot be physically taken out, so attackers have to bypass software protections to dump its content.

\subsection{Side-channel Attacks}
\textbf{Cache side-channel.} Because the CPU is time-shared between the REE and TEE in SGX and TrustZone, cache side-channel attacks are effective~\cite{brasser2017software, zhang2016return, cho2018prime, zhang2016truspy}.
In {\scshape BYOTee}\xspace, the REE on the hardcore system side and enclaves do not time-share any CPU resources; hence, there is no cache side-channel between the REE and TEEs.

\textbf{Power side-channel.}
In FPGA-based remote power side-channel attacks, the attacker builds an on-chip ring oscillators-based power monitor to conduct power analysis attacks on other modules on the same FPGA or a CPU on the same SoC~\cite{zhao2018fpga}.
{\scshape BYOTee}\xspace cannot directly mitigate these attacks but can them by only loading authenticated and trusted enclave bitstreams that do not have a power monitor.

\textbf{Other side-channels.}
When multiple enclaves reside on the same SoC FPGA, they share FPGA hardware resources.
Therefore, it is possible to conduct other sharing-based side-channel attacks, such as FPGA long wire-based attacks~\cite{giechaskiel2018leaky, ramesh2018fpga}.
Similar to power side-channel attacks, {\scshape BYOTee}\xspace cannot prevent these attacks directly.

\section{Experiment Device}

Figure~\ref{fig:z7} shows the top and bottom view of the Cora Z7-07S development board with a single-core 667MHz Arm Cortex-A9 processor, a Xilinx Zynq-7000 FPGA, and the connected Pmod I2S2 audio device.

\section{Automatically Generated Hardware Design of the Example SSAs}

Figure~\ref{fig:corr-res}(a) shows the hardware for SSA-1 with a MicroBlaze Debugging Module (MDM)~\cite{mdmTRM} to help the developers debug SSAs.
A master interface of MDM is also connected to the ps7\_axi\_periph to enable debugging from the hardcore system side.
Figure~\ref{fig:corr-res}(b) shows the same hardware configuration without the debugging module.
As shown in Figure~\ref{fig:corr-res}(c), the hardware design of SSA-2 includes two peripherals that are only connected to the FPGA.
A pulse width modulation IP~\cite{pwm} connects the RGB\_LED ports, and an AXI GPIO IP~\cite{axiGpio} connects the button ports to the mb\_axi\_interconnect\_0 interconnect.
As shown in Figure~\ref{fig:corr-res}(d),
the hardware for SSA-3 includes several more IPs for different functionality.
An I2S transmitter RTL (SPI) is added to interact with the Pmod I2S2 module Codec module.
Additionally, an AXI direct memory access IP~\cite{axiDMA} with BRAM and an AXI stream data FIFO IP~\cite{axififo} to offload audio data streaming computation from Microblaze are inserted by the {\scshape HardwareBuilder}\xspace.
An ADC module for reading voltages, which is used for performance monitoring~\cite{xadc}, is also connected.
Figure~\ref{fig:corr-res}(e) represents the hardware of SSA-4 with two enclaves.
Both the enclaves have their own BRAM, 128KB, and 32KB respectively.
Enclave-1 have a shared DRAM with Zynq processor as SEB for communication, while Enclave-2 does not have any SEB with Zynq.
Instead of inter enclave communication, one 8KB BRAM is shared between Enclave-1 and Enclave-2 without the Zynq processing system access.

\begin{comment}
\section*{To-do list}
1. Need to verify when a peripheral is connected to PL, is it true that the PS can't access it anymore. We need to confirm this. I am not confident about this. If not, we need to design a mechanism to stop the PS from accessing the peripheral.

2. Implementation: add authentication to SSA.

3. Implementation: develop a simple verifier that runs on a server (PC).

4. Finish app-2 and app-4.

5. Performance measurement. Need add timer to count CPU cycles.

5. Related work needs to be reorganized and rewritten.

6. Think about how to automated the hardwarebuilder.
7. Evaluation plan: Give me a plan by Sep 28th. What tables will be there. What figures will be there. How many Figures will be there.

7. Acknowledgment the discussion with Ni Kai, Zhenhua Wu, and RIT startup fund, Air force summer visit, NSF-CRII in the final version.

8. A new addition could be SSA sealing. Santury also has this feature.

9. Also add attestation for hardware configurations.

10. ShEF~\cite{zhao2021shef}

\end{comment}

\begin{figure*}[!t]
    \centering
    \subfloat[Hardware for SSA-1 with a Debugging Module]{\includegraphics[width = 0.49\linewidth]{images/dsgn_aes_with_mdm.pdf}} \hfill
    \subfloat[Hardware for SSA-1 without a Debugging Module]{\includegraphics[width = 0.49\linewidth]{images/dsgn_aes_without_mdm.pdf}}\\
    \subfloat[Hardware for SSA-2 without a Debugging Module]{\includegraphics[width = 0.49\linewidth]{images/dsgn_led_without_mdm.pdf}} \hfill
    \subfloat[Hardware for SSA-3 without a Debugging Module]{\includegraphics[width = 0.49\linewidth]{images/dsgn_secure_drm_without_mdm.pdf}}
    \subfloat[Hardware for SSA-4 without a Debugging Module]{\includegraphics[width = 0.49\linewidth]{images/dsgn_app4_without_mdm.pdf}}
    \caption{Block diagrams of the hardware designs of the four enclaves for the four example SSAs. The ZYNQ processing system represents the hardcore Cortex-A processor. The Enclave-1, Enclave-2, and Enclave-3 have one MicroBlaze softcore processor, whereas Enclave-4 has two MicroBlaze softcore processors.
The MicroBlaze interrupt interface is connected to an AXI interrupt controller and configured with an AXI GPIO.
All output GPIO to the softcore is connected to the hardcore system for triggering interrupts.
Two AXI Interconnects, mb\_axi\_mem\_interconnect\_0 for Microblaze and ps7\_axi\_periph for Cortex-A processor are used to connect external IPs.
The ZYNQ, Microblaze, and other IPs get primary clock input from the clk\_in1 port.
The reset port is connected to the reset interfaces of the IPs.}
    \label{fig:corr-res}
\end{figure*}

\begin{comment}
\begin{table*}[t]

	\centering
	\footnotesize
	\begin{tabular}{l|c|c|c|c|c|c|c|c}
		\hline
		 & Pipeline &  Max clock &  ICache  &  DCache  & FPU& Memory  & Debug   & Open-sourced? \\
		 & depth &  frequency &    &     &  Protection &  Module &  Resource &  \\ \hline
		MicroBlaze$^{1}$~\cite{mbsoft} & 3, 5& 200 MHz & * &*& \cmark&\cmark & \cmark&\xmark\\ \hline

		NIOS$^{1}$~\cite{nios}   & 5, 6& 210 MHz& *&*&  \cmark &\cmark & \cmark&\xmark\\ \hline

		VexRisc$^{1}$~\cite{VexRiscv}   & 2 - 5+& 243 MHz& *&*& \cmark &\cmark & \cmark&\cmark\\ \hline

		NEO430$^{1}$~\cite{Neo430}  & 5& 122 MHz&*&*&\xmark&\xmark&\xmark &\cmark\\ \hline
		Microwatt$^{2}$~\cite{microwatt}  & 5& 100 MHz &16KB&32KB&\cmark& \cmark&\cmark&\cmark\\ \hline
		A2I$^{2}$~\cite{a2i}    &7 &2.3 GHz&32KB&32KB&\cmark&\cmark&\cmark&\cmark\\ \hline
		A2O$^{2}$~\cite{a2o}   & 4&2.3 GHz&32KB&32KB&\cmark&\cmark &\cmark&\cmark\\ \hline
		OpenSPARC T1$^{2}$~\cite{OpenSPARC}   &6 &1.2 GHz&*&*&\cmark&\cmark& \cmark&\cmark\\ \hline
		Libre-SOC$^{2}$~\cite{libreSoc}    & 4&300 MHz&*&*&\cmark& \cmark&\cmark&\cmark\\ \hline

	\end{tabular}

	\caption{Example softcore CPUs. * : configurable depend on available FPGA hardware; \cmark: Available; \xmark: Not available; 1 : 32 bit processor; 2: 64 bit processor}
	\label{t:softcores}
\end{table*}

\section{Example Application}

As an example application of multiple protected SSA sessions, we develop a factorial calculator computational SSA for a CTEE with MicroBlaze softcore.
Interrupt handlers in MicroBlaze can access execution sensitive information such as interrupt return address, stack pointer through registers which are stored in \verb|r14| and \verb|r1| respectively.
While the SSA is running, UA triggers an interrupt handled by {\scshape Firmware}\xspace which generates a secure state blob using AES encryption on the necessary data and adds a SHA512-HMAC authentication tag.
The secure state blob can be re-invoked later from the previous execution point, and in the meantime, other SSAs can execute on the same CTEE.

\section{Deployment Model}
{\scshape BYOTee}\xspace, supports two deployment models: local deployment and cloud infrastructure deployment model.
There are three stakeholders: cloud provider, {\scshape Firmware}\xspace developer, and SSA developer.
Apart from software development, the stakeholders are responsible for key management in the system.
The device keys, e.g., AES ($k_{d}$), RSA ($sk_{d}, pk_{d}$) are stored in nonvolatile eFUSE memory or BBRAM along with configuration bits.
For local deployment, the {\scshape Firmware}\xspace developers program the device with the keys through the JTAG interface, which is used by the FPGA hardware AES module for decryption at boot time\cite{xilinxSecure}.
For cloud deployment, the provides program their infrastructure with devices keys, and developer/tenant sends their hardware configuration JSON file, {\scshape Firmware}\xspace, and UA to the trusted cloud provided.
In this case, multiple tenants can be present in the same infrastructures, the provider merges all tenants configuration JSON to generate the bitstream, combine with their {\scshape Firmware}\xspace, and lastly, generate the protected FPGA image.
The UAs for each tenant run individually in the untrusted  environment.
Using BBRAM allows the feature to update the device's keys at some later stage if the previous gets compromised, whereas the eFUSE memory is one-time programmable.

On the other hand, the developer's keys, e.g., RSA ($sk_{u}, pk_{u}$) are used for SSA decryption and verification.
For both deployment models, the developer embeds the keys on the {\scshape Firmware}\xspace.
The SSA developers need to register their keys on the {\scshape Firmware}\xspace with corresponding SSA keys.
They use RSA ($ pk_{u}$) to compose a special message in a predefined format including a special header, symmetric SSA key, and lastly the digest of the SSA to be executed.
The {\scshape Firmware}\xspace, after receiving the message whitelists the SSA key and records the SSA digest.
Upon receiving SSA execution instruction, the {\scshape Firmware}\xspace decrypts the SSA and verifies the SSA digest.
If the developer's key need to be updated, the {\scshape Firmware}\xspace needs to be replaced, and the newly generated protected FPGA image should be loaded on the device.

\section {TODO:}
\begin{enumerate}
	\item Threat Model: Needs to clear. Not contradictory with security analysis or evaluation.
	\item Security analysis and security evaluation can be merged.
	\item Motivation part needs to rewritten: Deployment use of BYOT, benefit of customized hardware TCB, Cloud or only local.
	\item Evaluation comparison with OP-TEE is not fair.
	\item Trusted I/O path: who establishes the trusted paths?
	\item Keys: how those keys are secured? how often they are updated.
	\item How can BYOT provide partial reconfiguration security?
	\item Design Change: After load there can be a attestation of the system software as a proof of secure load.
	\item Proof of physical isolation.
\end{enumerate}
\begin{figure}[t]
	\begin{centering}
		\centering
		\includegraphics[width=0.47\textwidth]{images/BYOT-state.pdf}

		\caption{Multiple SSA sessions}
		\label{fig:SSA state}
	\end{centering}
\end{figure}
\end{comment}

\begin{comment}
We consider both local and cloud deployment models for {\scshape BYOTee}\xspace.
In the local deployment model, the manufacturers or service provides embed the device key onto the SoC FPGA.
The device key are burned into the eFUSE or BBRAM, which are used for the secure loading of protected bitstream images.
The application providers, on the other hand, embeds the developer keys on {\scshape Firmware}\xspace which are used for secure loading and attestation mechanism of SSAs in {\scshape BYOTee}\xspace.
To facilitate different SSAs from different application providers {\scshape Firmware}\xspace is populated with a white list of developer keys.

For the cloud deployment model, {\scshape BYOTee}\xspace enables multiple tenants to run on top of the same SoC FPGA in different CTEEs.

\section{Evaluation of multiple SSA sessions}

\begin{table}[t]
	\centering
	\footnotesize
	\caption{Protected SSA Sessions Performance (Time in milliseconds; size in bytes)}
	\label{t:FwLocRes}
	\begin{tabular}{c|c|c|c}
		\hline
		SSA & Binary & Export & Import \\
		\hline
		SSA-5 & 2452 & 703.4 & 687.3\\
		\hline
	\end{tabular}
\end{table}

\end{comment}

\section{{\scshape BYOTee}\xspace Applications}

\subsection{Example Application: Secure Audio DRM}

Designing a secure audio DRM was a challenge in 2020 MITRE Collegiate Embedded Capture-The-Flag competition (eCTF).
\section{{\scshape BYOTee}\xspace Applications}
\label{s:apps}
In this section, we use multiple security-sensitive application examples to demonstrate how {\scshape BYOTee}\xspace can secure real-world applications in several classes, namely computational application, peripheral interacting application, and peripheral and PS interacting application.
We present the design of an example application in each class, which we use to evaluate {\scshape BYOTee}\xspace.

\subsection{Computational Applications}

Computational applications take input from the PS or other CTEEs, perform the intended computational operations, and send the outputs back.
They represent computational tasks, such as encryption, decryption, machine learning-based classification, etc., that do not need peripherals.
\emph{Example application:}
We implement an AES accelerator SSA (SSA-1).
To use SSA-1, a UA places the plaintext or ciphertext in the SEB and notifies the SSA.
When the encryption or decryption is finished, SSA-1 places the outputs on the SEB.

\subsection{Peripheral Interacting Applications}

Simple peripheral interacting applications control one or more peripherals, but they do not communicate with other SSAs or applications on the PS.
An example usage is cyber-physical applications that read from sensors, make local decisions, and control an actuator.
\emph{Example application:}
We develop an LED toggler SSA (SSA-2) that uses a button and an LED.
When the button is pressed, SSA-2 toggles the color of the LED.
Both the button and LED are directly connected to the CTEE of SSA-2 and cannot be accessed by the PS or other CTEEs.

\begin{comment}
\begin{lstlisting}[label={ssa2},caption={SSA-2 Code Snippet}, language=c] .
void GpioHandler(void *CallbackRef)
{
	XGpio *GpioPtr = (XGpio *)CallbackRef;
	IntrFlag = 1;
	XGpio_InterruptClear(GpioPtr, GlobalIntrMask);
}

int main()
{
	u32 *led = (u32 *)XPAR_RGB_PWM_0_PWM_AXI_BASEADDR;
	xil_printf("Wait for Button GPIO interrupt: \r\n");

	while(true)
	{
		if(IntrFlag)
		{
			toggleLED(led);
		}
	}

	return 0;
}
\end{lstlisting}
Listing~\ref{ssa2} represents the code snippet of the LED Switch SSA, where it configures GpioHandler function as the button interrupt handler.
\end{comment}

\subsection{Peripheral and PS Interacting Applications}
\label{sec:DRM}

[Need to clearly explain why this app needs BYOT. Why TrustZone can;t provide the protection (code boot attack).]

Peripheral and PS interacting applications do not only control some peripherals but also continuously interacts with the UA.

\emph{Example application:}
We develop a music player SSA with digital rights management (SSA-3).
The untrusted module running on the PS takes user's commands, such as login (using username and Pin), logout, query (i.e., display a song's owner information), share (i.e., share a song to another user), play, pause, resume, and stop.
Depending on the user's request, it communicates with the SSA and forwards necessary data through SEB.

The trusted module is the SSA running on the CTEE, which implements the DRM properties of the music player.
Allow only authorized users to play songs, regional provisioning of songs, prevent unprovisioned media files from being played are some of the DRM properties that the SSA needs to ensure.
To validate those properties at runtime, the required DRM meta-information is embedded in the SSA or put onto the media files themselves.
For example, user passwords, encryption keys are embedded into the SSA, however, region lock, owner and shared user information, etc. are put onto the media files.
Embedded information is only available to the SSA as it is in BRAM of the CTEE, and the media files are authenticated and can only be decrypted by the SSA on BRAM.
So the untrusted module at runtime does not have access to sensitive data to temper with.

\subsection{Intercommunicating SSA}
Intercommunicating SSAs represents a set of applications where multiple SSA runs on different physically separated CTEEs and needs secure intercommunication between them to carry out the interdependent sensitive task.

\emph{Example application:}
As an example, we develop two computational applications deployed in two physically separated CTEEs.
When the untrusted module wants to authenticate and decrypt some data, it sends the signature and ciphertext through the SEB to the CTEE-1's SSA.
The SSA of CTEE-1 performs the AES decryption task and internally utilizes the SSA on CTEE-2 to generate the signature (SHA512-HMAC).
The communication between the SSAs are through a shared BRAM without untrusted PS software access.
This example also illustrates the dynamic and static loading of SSAs onto the CTEEs.
While the {\scshape Firmware}\xspace of the CTEE-1 dynamically loads the AES accelerator SSA into its BRAM, the SSA on CTEE-2 is loaded statically onto the BRAM with the bitstream.

\section{{\scshape BYOTee}\xspace Architecture}
\label{s:arch}
In this section, we first present the security and functional design goals of {\scshape BYOTee}\xspace followed by an overview of the {\scshape BYOTee}\xspace architecture and workflow.
We then illustrate hardware TCB customization, bootstrapping trust in enclaves, secure execution of SSA, and remote attestation mechanisms.

\subsection{Design Goals}
\label{subsec:designgoal}

{\scshape BYOTee}\xspace provides physically isolated execution environments on-demand, which even hardware debuggers and DMA-enabled devices cannot access.
With the help of {\scshape BYOTee}\xspace, users can use the exact and even formally verified hardware and software needed for their applications.
{\scshape BYOTee}\xspace has the following security and functional design goals:

\goal{General-purpose execution environments.} {\scshape BYOTee}\xspace should provide general-purpose execution environments similar to SGX and TrustZone, not application-specific accelerators.
SSAs can be implemented in any language as long as they can be linked against the {\scshape Firmware}\xspace.

\goal{Multiple isolated execution environments.} {\scshape BYOTee}\xspace should provide multiple execution environments at the same time and guarantee the secrecy and integrity of the SSA running inside each.

\goal{Physical execution isolation.} {\scshape BYOTee}\xspace should provide dedicated CPUs for execution environments, and all hardware resources for enclaves are physically isolated from the REE and each other.
With physical isolation, {\scshape BYOTee}\xspace mitigates the side-channel attacks that are prevalent in CPU-sharing TEEs, such as SGX and TrustZone.

\goal{Customizable hardware TCB.} The hardware TCB of each enclave should be customizable, allowing for a minimal TCB that only includes the hardware, e.g., peripherals, necessary for the SSA running inside and excludes other hardware on the system.
Note that formally verifying the customized hardware~\cite{sieh1997verify, breuer1997refinement} is beyond the scope of {\scshape BYOTee}\xspace.

\goal{Isolated path between SSA and peripherals.} An enclave should isolate the communication path between the SSA inside and peripherals from the hardcore system and other enclaves, preventing software-based eavesdropping and tampering.

\goal{Enclave-to-Hardcore System and Inter-enclave communication.} The SSA in an enclave should be able to communicate with software modules on the hardcore system and other enclaves.
The inter-enclave communication should be isolated from the hardcore system and other non-participating enclaves.

\goal{Minimal software TCB.} The firmware serving an SSA should only include the housekeeping libraries and drivers that are necessary for the SSA execution and exclude other software.

\goal{Remote attestation mechanisms.} {\scshape BYOTee}\xspace should provide mechanisms to support sophisticated protocols to attest the integrity of an enclave's hardware and software stacks, including boot loaders, bitstream, {\scshape Firmware}\xspace, SSAs and their inputs and outputs to remote verifiers.

\goal{Easy to use.} {\scshape BYOTee}\xspace should be easy to use, especially for the software developers who do not have hardware programming experience. As a rule of thumb, developing a {\scshape BYOTee}\xspace SSA should not take significantly more time and effort than developing a Linux application with the same functionality.

\subsection{{\scshape BYOTee}\xspace Overview}
\label{subsec:overview}

Figure~\ref{fig:BYOTee} presents an overview of the architecture and workflow of the {\scshape BYOTee}\xspace framework.
The {\scshape BYOTee}\xspace tools and codebase mainly include the {\scshape HardwareBuilder}\xspace, {\scshape Firmware}\xspace, and {\scshape SSAPacker}\xspace.
During the development stage, the {\scshape HardwareBuilder}\xspace generates synthesizer commands, e.g., Tcl, based on the SSA's needs specified in the developer's hardware description JSON input.
Then, the SoC FPGA vendor-provided synthesizer, e.g., Xilinx Vivado~\cite{vivvado}, Intel Quartus Prime~\cite{quartus}, generates the bitstream file using the synthesizer commands.
The developer can customize the {\scshape Firmware}\xspace by only including the needed source code and writing the SSA source codes, which are compiled with the corresponding compiler for the softcore CPU, e.g., \verb|mb-gcc| for MicroBlaze.
The bitstream and {\scshape Firmware}\xspace binary are encrypted, signed, and packed by the vendor-provided merger, e.g., \verb|UpdateMEM| from Xilinx, into a protected FPGA image.
The SSA binary is encrypted, signed, and packed by the {\scshape SSAPacker}\xspace into a protected SSA.
At boot-time, the bitstream is loaded onto the FPGA.
As a result, multiple enclaves are created, and the corresponding {\scshape Firmware}\xspace starts running.
Then, an untrusted application can trigger the loading of a protected SSA into an enclave.

{\scshape BYOTee}\xspace meets \emph{G1}, \emph{G2}, \emph{G3}, \emph{G4}, and \emph{G5} by configuring the FPGA to build enclaves.
{\scshape BYOTee}\xspace constructs enclaves with softcore CPUs, which provide general-purpose computing environments (\emph{G1}).
The {\scshape Firmware}\xspace includes the standard C libraries and connected peripheral HAL libraries for SSAs to invoke at run-time.
Each enclave has its own set of hardware (\emph{G2}), including softcore CPU, e.g., MicroBlaze, UltraSPARC, etc., Block RAM, and peripherals.
With FPGA routing, these hardware resources within an enclave are connected together but isolated from the hardcore system and other enclaves (\emph{G3}).
The softcore CPU in an enclave is not time-shared with the hardcore system and other enclaves, mitigating the cache side-channel attacks (\emph{G3}).
No additional hardware modules, such as debuggers, can be connected to an enclave unless it is explicitly specified by the developer (\emph{G4}).
The connections among these resources are also physically isolated from the hardcore system and other enclaves, preventing eavesdropping and tampering (\emph{G5}).

Enclaves use interrupts on softcore CPUs and shared physical memory regions on the DRAM to communicate with the hardcore system, whereas enclaves use interrupts and shared regions on the BRAM to communicate with each other.
Since a shared BRAM region is only mapped in the address spaces of the participating enclaves, it is isolated from the hardcore system and other enclaves (\emph{G6}).
{\scshape BYOTee}\xspace includes {\scshape Firmware}\xspace, which can be customized and only consists of libraries, a HAL for the needed peripherals, and a loader for the SSA (\emph{G7}).
The {\scshape Firmware}\xspace also provides softcore CPU interrupt-based remote attestation mechanisms for proving the integrity of enclaves (\emph{G8}).
{\scshape BYOTee}\xspace provides an easy-to-use toolchain for developers to focus on SSA development, increasing the usability of {\scshape BYOTee}\xspace and decreasing the chances of developer-induced misconfigurations (\emph{G9}).

\subsection{Customizing Hardware TCB for Enclaves}
\label{subsec:customizeHWTCB}

To customize the hardware TCB, the developer designs one or more enclaves using a hardware description language, e.g., Verilog or VHDL.
The output is a bitstream file that configures the FPGA.
To facilitate this step, {\scshape BYOTee}\xspace has a component, named {\scshape HardwareBuilder}\xspace, which takes developer-specified hardware description in JSON format as input (See an example in \S\ref{subsec:dev}), allocates hardware resources, and outputs a script, e.g., in Tcl format, that can be processed by a synthesis tool to generate the bitstream.
Each enclave's hardware description includes but is not limited to:
i) a softcore CPU and its configurations, e.g., clock frequency, cache size, etc;
ii) a corresponding debug IP to enable software debugging on the softcore CPU;
iii) its main BRAM memory address and size;
iv) the address and size of the shared DRAM with the hardcore system;
v) the address and size of the shared BRAM with other enclaves;
and vi) connected peripherals.
The {\scshape HardwareBuilder}\xspace assigns a contiguous address space of the BRAM to each enclave and connects the hardware components automatically.

\subsection{Bootstrapping Trust in Enclaves}
\label{subsec:bootstrapTrust}

Because the details of trust bootstrap are related to the specific deployment model and policies, {\scshape BYOTee}\xspace provides the mechanisms to build trust bootstrap paths.
{\scshape BYOTee}\xspace supports FPGAs that can be configured at boot time and/or reconfigured at run-time by providing trust bootstrap in enclaves both statically and dynamically.
The static trust bootstrap builds on secure or trusted boot, whereas the dynamic trust bootstrap relies on some dynamic root-of-trust modules at run-time.

\begin{figure}[t]
	\begin{centering}
		\centering
		\includegraphics[width=.47\textwidth]{images/bootstraptrust.pdf}

		\caption{Example static and dynamic trust bootstrap paths on Zynq SoC. \Circled{\textbf{1}} static trust bootstrap. \Circled{\textbf{2}} dynamic trust bootstrap without TrustZone. \Circled{\textbf{3}} dynamic trust bootstrap with TrustZone.}
		\label{fig:bootstrap}
	\end{centering}

\end{figure}

Without loss of generality, Figure~\ref{fig:bootstrap} shows three example paths for static and dynamic trust bootstrap on the Xilinx Zynq SoC with Arm hardcore CPUs.
In the static bootstrap case \Circled{\textbf{1}}, {\scshape BYOTee}\xspace relies on secure or trusted boot to launch enclaves.
BootROM, which is the root of trust for measurement, first decrypts, measures, i.e., $m_1=H(FSBL)$, and executes the FSBL using the device keys.
The FSBL initializes the hardcore system by decrypting, measuring, and executing the SSBL using the device keys.
A measurement $m_2=H(m_1||SSBL)$ is generated in this step.
The FSBL also initializes the FPGA by decrypting, measuring, and configuring the FPGA using the device keys, after which bitstream is programmed on the FPGA, and the {\scshape Firmware}\xspace starts execution.
A measurement $m_3=H(m_2||BS||FW)$ is generated by the FSBL and placed on the shared DRAM region with the enclave for the future use of attestation report generation, which we discuss in~\S\ref{subsec:attest}.
The {\scshape Firmware}\xspace uses the developer keys, e.g., $k_{u}$ and/or $pk_{u}$, to decrypt and measure the SSA.
Allowed developer keys are embedded in the {\scshape Firmware}\xspace at the development stage.
Because the {\scshape Firmware}\xspace is encrypted at rest and only decrypted on the BRAM, the developer keys are secure.

In the dynamic case, software running on the hardcore system of a Xilinx Zynq device can use the device configuration (DevC) or Processor Configuration Access Port (PCAP) interfaces to reconfigure the FPGA.
Therefore, the software module that reconfigures the FPGA needs to access the device key and must be trusted.
In the case that TrustZone is not available or used \Circled{\textbf{2}}, the trusted software module can be part of the REE kernel, which uses the device keys to decrypt and measure the bitstream and {\scshape Firmware}\xspace.
If TrustZone is available \Circled{\textbf{3}}, the trust software module can be a kernel module inside the TrustZone secure world operating system.

\subsection{Executing SSAs in Enclaves}
\label{subsec:execintee}

To create an SSA, the developer links the source code against the {\scshape Firmware}\xspace.
After the launching of an enclave, the {\scshape Firmware}\xspace initializes the softcore CPU and other components, then it waits for requests from the hardcore system.
Both the {\scshape Firmware}\xspace and SSA use a shared DRAM region in the SSA Execution Block (SEB) format as shown in Figure~\ref{fig:sebctee}, to have two-way data transmissions with UA on the hardcore system.
To initiate the transmission from the hardcore system to an enclave, UA on the hardcore system raises interrupts on the enclave's softcore CPU, which are handled by the {\scshape Firmware}\xspace.
{\scshape BYOTee}\xspace defines three service primitives through the \textsf{LdExec*} interrupts: i) load and execute an SSA (\textsf{LdExec});
ii) load and execute an SSA with pre-execution attestation (\textsf{LdExecPreAtt});
iii) load and execute an SSA with post-execution attestation (\textsf{LdExecPostAtt}).
The softcore CPU interrupts can be implemented as GPIO interrupts in the enclave and memory-mapped to a DRAM address for the UA to access.
In this subsection, we focus on \textsf{LdExec}, and the other two primitives are discussed in \S\ref{subsec:attest}.

\begin{figure}[t]
	\begin{centering}
		\centering
		\includegraphics[width=0.47\textwidth]{images/BYOT-FM-Layout.pdf}
		\caption{SSA Execution Block (SEB) layout, simplified enclave address space layout, and steps in executing an SSA.}
		\label{fig:sebctee}
	\end{centering}
\end{figure}

To execute an SSA on an enclave, the untrusted application first fills data into the SEB and raises a \textsf{LdExec*} interrupt.
As shown in Figure~\ref{fig:sebctee}, a SEB has regions for the encrypted and signed SSA (SSA*), input data for the SSA, output data from the SSA, a challenge $Chal$ from a remote verifier, a pre-execution ($PreExecAtt$), a post-execution attestation measurement ($PostExecAtt$) and other data.
When the {\scshape Firmware}\xspace receives a \textsf{LdExec*} interrupt, it copies SSA* and optionally $Chal$ and input data in the SEB from DRAM to its own BRAM \Circled{\textbf{1}}.
The {\scshape Firmware}\xspace can also disable \textsf{LdExec*} interrupts after the data is copied.
Note that it is critical for the {\scshape Firmware}\xspace to perform measurement on the BRAM since the DRAM can be changed asynchronously by the hardcore system.
The {\scshape Firmware}\xspace then decrypts and verifies the encrypted SSA* using the corresponding developer's keys \Circled{\textbf{2}}.
Upon the successful verification of the SSA's integrity, the {\scshape Firmware}\xspace loads sections of the decrypted SSA to the right locations and gives the control to the SSA \Circled{\textbf{5}}.
If there is an output, the SSA writes it in the output region on the BRAM, and yields the control of the softcore CPU back to the {\scshape Firmware}\xspace.
The {\scshape Firmware}\xspace copies the output from the BRAM to the DRAM \Circled{\textbf{6}}.
Finally, the {\scshape Firmware}\xspace cleans up all the input, output, and SSA-related regions on the BRAM and awaits new requests from the hardcore system \Circled{\textbf{7}}.
While SSAs can execute concurrently on their own enclaves respectively,
the {\scshape Firmware}\xspace also supports executing multiple SSAs sequentially or the same SSA multiple times on the same enclave without reconfiguring the FPGA but just re-initializing the enclave, e.g., flush the cache,
clean up the BRAM \Circled{\textbf{7}}, etc.

\begin{comment}
\begin{figure}[t]
	\begin{centering}
		\centering
		\includegraphics[width=0.47\textwidth]{images/SSC_execution_time_line.pdf}

		\caption{Executing two SSAs concurrently on two enclaves}
		\label{fig:ssc_time_line}
	\end{centering}
\end{figure}

\begin{figure}[t]
	\begin{centering}
		\centering
		\includegraphics[width=0.47\textwidth]{images/Temoral_SSC_execution_time_line.pdf}

		\caption{Executing two SSAs sequentially on one enclave}
		\label{fig:temporal_ssc_time_line}
	\end{centering}
\end{figure}
\end{comment}

\subsection{Pre-execution and Post-execution Attestation Mechanisms}
\label{subsec:attest}

{\scshape BYOTee}\xspace provides two attestation mechanisms, namely pre-execution and post-execution attestations, as shown in Figure~\ref{fig:sebctee}.
The former extends the traditional remote attestation of code and input integrity with bitstream, whereas the latter extends the output data integrity attestation~\cite{abera2019diat}.
Note that {\scshape BYOTee}\xspace only provides the mechanism for attestation, which can support sophisticated attestation protocols.
With the help of trust bootstrapping discussed in \S\ref{subsec:bootstrapTrust}, the measurement mechanism not only captures the identity of the loaded SSA but also the bitstream, {\scshape Firmware}\xspace and programs, e.g., FSBL and SSBL, before it.

In pre-execution attestation, a verifier sends a cryptographic nonce as $Chal$, which is copied to the BRAM by the {\scshape Firmware}\xspace \Circled{\textbf{1}}.
After loading the SSA sections to the right addresses, the {\scshape Firmware}\xspace computes a measurement $PreExecAtt$ on the vector table, {\scshape Firmware}\xspace code and data, $m_3$, $Chal$, input data and SSA sections \Circled{\textbf{3}}, and copies the measurement to the DRAM.
Depending on scenarios and attestation protocol details, the {\scshape Firmware}\xspace can use a device key, developer key, or other shared keys to compute the measurement.
In post-execution attestation, the $Chal$ from $\mathtt{Vrf}$ is also copied to the BRAM by the {\scshape Firmware}\xspace \Circled{\textbf{1}}.
After the SSA finishes execution \Circled{\textbf{4}}, the {\scshape Firmware}\xspace computes a measurement $PostExecAtt$ on the vector table, {\scshape Firmware}\xspace code and read-only data, $m_3$, $Chal$, input data, output data generated by the SSA, SSA's read-only sections and $PreExecAtt$, and copies the measurement to the DRAM \Circled{\textbf{5}}.

\subsection{Multiple Inputs to SSA}
\label{subsec:coninput}

In the case that the UA on the hardcore system needs to continuously send input data to the SSA, e.g., not all input data is available at the beginning, the size of input in SEB is not big enough, etc., the UA writes the newly available input data in the input region inside the SEB, and it can use two mechanisms to notify the {\scshape Firmware}\xspace and SSA that new data is available.
The first mechanism works for softcore CPUs that support priority interrupts.
On such systems, {\scshape BYOTee}\xspace defines a \textsf{NewData} interrupt, which UA can raise.
The \textsf{NewData} interrupt has a low priority so that it cannot interrupt the execution of the SSA.
Only after the SSA finishes execution and yields the control back to the {\scshape Firmware}\xspace, the {\scshape Firmware}\xspace can copy and measure the input data from the DRAM to the BRAM, and gives the control to the SSA again.
On softcore CPUs without priority interrupts, the {\scshape Firmware}\xspace uses global variables to indicate whether new data is available in the input region to synchronize with the SSAs on the enclaves.

\subsection{Optional Multiple Protected SSA Sessions}
As an option, an enclave can also interleave the execution of multiple eligible SSAs with proper hardware re-initialization, e.g., flush cache, reset all memory, etc.
To this end, {\scshape BYOTee}\xspace defines two interrupt-based service primitives: i) suspend and export the SSA state (\textsf{SusExp});
ii) restore and execute a saved and encrypted SSA state (\textsf{ReExec}).
When a \textsf{SusExp} interrupt is raised, the {\scshape Firmware}\xspace copies the SSA context, e.g., general and system registers, onto the BRAM.
Then, the {\scshape Firmware}\xspace uses the developer key to encrypt and sign the saved SSA context and all of the SSA's writable memory regions, e.g., stack, .data, .bss, etc.
The encrypted blob is placed in the SEB output region for the UA to retrieve,
after which the {\scshape Firmware}\xspace cleans up BRAM and awaits new requests.
When a \textsf{ReExec} interrupt is raised, the {\scshape Firmware}\xspace retrieves an encrypted blob from the SEB input region.
Upon a successful signature verification, the {\scshape Firmware}\xspace loads the decrypted memory contents to the right locations, restores the registers, and resumes the SSA execution.

\section{Background: SoC FPGA}
\label{s:back}

\begin{comment}
\begin{table*}[!t]
\center
\scriptsize
\begin{tabular}{lp{3.5cm}p{11cm}}
\hline
Acronym           & Full name        & Description                                                                                \\ \hline\hline
PS & Processing System & PS encompasses `hard' processors, cache memory, memory interfaces, interconnect, clock generation circuitry and other components that can be found on `hard' CPUs.
PS components exist as dedicated silicon element on the device. \\ \hdashline
FPGA & Programmable Logic & FPGA is the FPGA part of an SoC. The FPGA can implement `soft' processors, and other high-speed logic, arithmetic and data flow subsystems. \\ \hdashline
BRAM & Block RAM & BRAMs are used for storing large amounts of data inside of the FPGA. \\ \hline
CTEE & Customized Trusted Execution Environment & A CTEE has its own TSC, BRAM, and peripherals. \\ \hdashline
FM & Firmware running on `soft' processor. & A software component provided by {\scshape BYOTee}\xspace that provides hardware abstraction layer. \\ \hdashline
SSA & Security-Sensitive Application & An SSA runs on isolated software cores. SSC could be PAL or TA. \\ \hdashline
UA & Untrusted Application & UA runs on hardware core, usually on top of an operating system. \\ \hline
$K_{D}, SK_{D}, PK_{D}$ & Device keys & Device keys, e.g., AES, RSA, are used to encrypt and sign system modules, such as FSBL, SSBL. They are unique for each device and are programmed in eFUSE/BBRAM using hardware interfaces. \\ \hdashline
$K_{U}, SK_{U}, PK_{U}$ & Developer keys & Developer keys are used to encrypt and sign SSCs. Valid developer keys are embedded into RT at development stage. \\ \hline \\
\end{tabular}
\caption{Acronyms and descriptions}
\label{tab:acronym}
\end{table*}
\end{comment}

In this section,
we discuss the hardware components of SoC FPGA, followed by its design, development, and run-time workflow.

\subsection{Hardware Components}

\begin{comment}
System-on-Chip implements the functionality of an entire system on a single silicon chip.
Compared with system-on-printed-circuit-board, SoC is lower cost, enables more secure data transfers,
and has higher speed and lower power consumption.
However, traditional application-specific integrated circuit SoCs lack flexibility, making them suitable only for products with a limited lifetime.
SoC FPGA is a type of flexible system-on-programmable-chip, where FPGAs can be reconfigured as desired.
The market of SoC FPGA is expected to grow significantly with the increase in the global adoption of artificial intelligence and internet-of-things solutions~\cite{FPGA2019}.
\end{comment}

A SoC FPGA comprises:
i) hardcore CPU system, which is formed around hard processors, such as the Cortex-A processor,
to run the traditional operating systems and applications;
ii) FPGA, which can implement arbitrary systems, including softcore CPUs, high-speed logic, arithmetic, and data flow subsystems.
In addition to the general fabric, the FPGA has Block RAMs (BRAM) to store data.
Note that BRAM is made of Static RAM (SRAM) on existing SoC FPGA platforms.
Compared to DRAM whose cells are made of capacitors and is vulnerable to cold-boot attacks due to the slow decay~\cite{halderman2009lest}, SRAM decays faster~\cite{rahmati2012tardis}.
FPGA is configured with a \emph{bitstream}, which is programmed in hardware design description languages, such as Verilog, VHDL, etc.;
iii) other integrated on-chip memory and high-speed communications interfaces.

\begin{figure}[t]
	\begin{centering}
		\centering
		\includegraphics[width=0.48\textwidth]{images/Zynq_PL_PS.pdf}
		\vspace{-0.2cm}
		\caption{An overview of SoC FPGA hardware}
		\label{fig:zynq_device}
	\end{centering}
			\vspace{-0.4cm}
\end{figure}

Figure~\ref{fig:zynq_device} shows the architecture of SoC FPGAs.
The main building blocks of the hardcore CPU system include the hardcore CPU, its Memory Management Unit (MMU), On-Chip Memory (OCM), caches, etc.
Additionally, the hardcore system connects external I/O interfaces, such as SPI, I2C, UART, etc.
Bridges and interconnects connect the hardcore system and FPGA interfaces.
The FPGA side is mainly composed of configurable logic blocks, logic cells, Adaptive Logic Modules (ALMs), Lookup Tables (LUTs), flip-flops, switch matrix, carry logic, BRAM, and Input/Output Blocks (IOB) for interfacing.
The FPGA also includes modules for Analogue to Digital Conversion (ADC) and a set of  Joint Test Action Group (JTAG) ports for configuration and debugging.
Usually several security-relevant modules, including non-volatile Battery-backed RAM (BBRAM), one-time programmable electronic fuse (eFUSE),
and cryptographic accelerators, are connected to both the hardcore system and FPGA.

\subsection{Development and Run-time Workflow}
A typical design and development flow of general-purpose computing platforms on SoC FPGA involves:
i) the development of the hardware system on the FPGA, including designing the peripheral blocks and creating the connections between these blocks and the hardcore CPU system.
In this step, a developer can use and customize open-sourced and proprietary hardware IPs;
ii) the development of the software system on the hardcore and softcore CPUs.
To be compatible with SoCs without FPGAs, when a SoC FPGA device is powered on, the hardcore system boots first before going on to configure the FPGA.

To enable secure and trusted boot, unique AES and RSA keys can be generated off the device and programmed to the persistent secure storage, e.g., eFUSE array, on the device.
The keys are used to encrypt and sign the bitstream and the firmware (e.g., in ELF format) that runs on the softcore CPU in the development stage.
The device first starts with the hard-wired boot ROM, which verifies, decrypts, and loads the First Stage Boot Loader (FSBL) from supported interfaces, such as SD card, JTAG, etc.
The FSBL, in turn verifies and sets up the FPGA with bitstream using the device configuration (DevC)~\cite{ZynqTrm} interface and firmware, after which the firmware on FPGA starts execution.
The FSBL also verifies the Second Stage Boot Loader (SSBL), such as U-Boot, and gives control of the hardcore system to it.
The SSBL verifies and boots the operating system on hardcore CPU.
In addition to boot-time configuration, some FPGAs can also be programmed at run-time.
For instance, software running on the hardcore of some Xilinx Zynq devices can use the DevC and Processor Configuration Access Port (PCAP) interfaces to reconfigure the whole or part of the FPGA.

\begin{comment}
In Intel FPGAs, on the other hand, the early boot is handled by Secure Device Manager (SDM) on a flash controller and this in turn boots the hardcore system and FPGA.
During the first stage, the SDM boots and takes control.
From the bitstream, it determines the configuration source of FPGA, handles device security, initializes the hardcore system, and loads the FSBL in the on-chip RAM.
The hard processor now executes the FSBL, loads SSBL on DRAM, loads the OS, and lastly the applications.
SDM provides two modes of booting: FPGA configuration first and hardcore system boot first (Early I/O).
FPGA is fully configured including core and peripheral I/O before booting the hardcore system, in FPGA configuration first mode.
In hardcore system boot first, only FPGA peripheral I/O is configured before booting the hardcore system.
After that, the FSBL, SSBL, or OS boot stages can be configured to perform FPGA core configuration.
Thus, SDM here ensures the secure and trusted boot of Intel FPGA SoCs.
\end{comment}

\section{Conclusion}
\label{s:conclusion}

Even though existing hardware-assisted TEEs on commodity computing devices have been widely adopted in commercial systems, they have many issues that make them either ineffective, inefficient, or untrustworthy.
In this paper, we present {\scshape BYOTee}\xspace, which is a framework for building multiple equally secure TEEs on-demand with configurable hardware and software TCBs utilizing commodity SoC FPGA devices.
In {\scshape BYOTee}\xspace, enclaves, which include softcore CPUs, memory, and peripherals, are created on the FPGA, and the {\scshape BYOTee}\xspace {\scshape Firmware}\xspace provides necessary software libraries for the SSAs to use.
Additionally, the {\scshape Firmware}\xspace offers two attestation mechanisms, namely pre-execution and post-execution attestations, to verify the hardware and software stacks.
We implemented a proof-of-concept {\scshape BYOTee}\xspace for Xilinx SoC FPGA.
The evaluation results on the low-end Zynq-7000 system for benchmark applications and example SSAs show the effectiveness and performance of {\scshape BYOTee}\xspace.
\section{{\scshape BYOTee}\xspace Applications and Developer's Perspective}
\label{s:dev}

\begin{comment}
\begin{figure*}[ht]
	\begin{centering}
		\centering
		\includegraphics[width=\linewidth]{images/dev_work_flow.pdf}
		\vspace{-0.3cm}
		\caption{Development and run-time process}
		\label{fig:}
	\end{centering}
\end{figure*}
\end{comment}

In this section, we first use multiple SSA examples to demonstrate how {\scshape BYOTee}\xspace can secure real-world applications in several classes.
Then, we discuss how developers can easily develop and deploy enclave hardware, SSAs, and UAs using the {\scshape BYOTee}\xspace toolchain.

\subsection{{\scshape BYOTee}\xspace Applications}
\label{sub:apps}

\textbf{Computational Applications.}
Computational applications take input from the hardcore system or other enclaves, perform the intended computational operations, and send the outputs back.
They represent computational tasks, such as encryption, decryption, machine learning-based classification, etc., that do not need peripherals.
{\scshape BYOTee}\xspace protects such applications from code and data disclosure, memory corruption, and cache side-channel attacks from the hardcore system and other enclaves at run-time.
We implement an AES accelerator SSA (SSA-1) as an example for computational applications.
To use SSA-1, a UA places the plaintext or ciphertext in the SEB and notifies the SSA.
When the encryption or decryption is finished, SSA-1 places the outputs on the SEB.

\textbf{Peripheral Interacting Applications.}
Peripheral interacting applications use some peripherals, but they do not communicate with other SSAs or the hardcore system.
An example usage is cyber-physical applications that read from sensors, make local decisions, and control an actuator.
Besides the attacks {\scshape BYOTee}\xspace protects the computational applications from, {\scshape BYOTee}\xspace also protects the paths between the SSA and peripherals from attacks for these applications.
We develop an LED toggler SSA (SSA-2) that uses a button and an LED.
When the button is pressed, SSA-2 toggles the color of the LED.
Both the button and LED are connected to the enclave of SSA-2 only and cannot be accessed by the hardcore system or other enclaves.

\begin{comment}
\begin{lstlisting}[label={ssa2},caption={SSA-2 Code Snippet}, language=c] .
void GpioHandler(void *CallbackRef)
{
	XGpio *GpioPtr = (XGpio *)CallbackRef;
	IntrFlag = 1;
	XGpio_InterruptClear(GpioPtr, GlobalIntrMask);
}

int main()
{
	u32 *led = (u32 *)XPAR_RGB_PWM_0_PWM_AXI_BASEADDR;
	xil_printf("Wait for Button GPIO interrupt: \r\n");

	while(true)
	{
		if(IntrFlag)
		{
			toggleLED(led);
		}
	}

	return 0;
}
\end{lstlisting}
Listing~\ref{ssa2} represents the code snippet of the LED Switch SSA, where it configures GpioHandler function as the button interrupt handler.
\end{comment}

\textbf{Peripheral and Hardcore System Interacting Applications.}
Peripheral and hardcore system interacting applications do not only control some peripherals but also continuously interacts with a UA.
For demonstration, we develop a music player system with digital rights management that guarantees the confidentiality, integrity, and authenticity of songs for artists and recording studios.
This means i) songs cannot be digitally disclosed, ii) songs cannot be modified,
and iii) only songs that were protected can be played.

To this end, the music player system has three components:
i) a trusted song protector (in Python with 160 SLOC), which is an offline component to encrypt and sign a song file (WAV format);
ii) a UA (in C with 695 SLOC) running on the hardcore system, which provides a user interface to play, pause, resume, and stop a protected song.
The UA awaits for the user's commands, reads protected songs from storage, e.g., SD card, and sends them to the song playing SSA.
Because the protected song file is big (e.g., an original  77 seconds, 48KHz, and a single channel WAV file is around 33MB.
The protected song file is several hundred bytes bigger.), the UA needs to continuously read the protected song file data from the storage and send it to the SSA;
iii) a song playing SSA (SSA-3) that authenticates, decrypts, and plays a song by sending the plaintext data of it to a hardware audio module.
The hardware audio module is only connected to the enclave running SSA-3, so the hardcore system and other enclaves cannot access it.

We emphasize that any software solution that solely trusts SGX or TrustZone cannot meet the security requirements of this music player system because:
i) there is no isolated or trusted I/O path between an SGX enclave and the hardware audio module; hence a malicious REE OS can breach the confidentiality, integrity, and authenticity of a song.
Some solutions, such as SGXIO~\cite{weiser2017sgxio}, attempt to address this issue but they need to add additional hardware, e.g., the hypervisor, into the TCB;
ii) a TrustZone application must decrypt the song in DRAM before sending it to play; hence, vulnerable to cold-boot attacks.

\textbf{Distributed Applications.}
A distributed application consists of multiple inter-communicating SSAs running on different enclaves at the same time.
The SSAs communicate through a shared BRAM region.
{\scshape BYOTee}\xspace not only protects each of the SSAs but also their communications from the hardcore system and other enclaves.
For demonstration, we develop an application that processes data in sequence with two SSAs.
SSA-1 first receives data from a UA, decrypts the data, outputs to the shared BRAM instead of DRAM, and the second SSA (SSA-4) takes the output of SSA-1 and performs a SHA512-HMAC signature verification.

\subsection{Developer's Perspective}
\label{subsec:dev}

\begin{figure}[!t]
\begin{lstlisting}[label={l:CTEEHS},caption={An example hardware description defining three enclaves in JSON format. Each enclave has its own customized softcore CPU (processor model, cache, FPU, etc.), peripheral connections, memory, etc.}]
{"Enclaves": [
    {"Name": "Enclave-1",
     "Processor": {"Type": "MicroBlaze 32bit",
      "Debugging": "Enabled"},
     "Memory Size": "512KB",
     "Shared DRAM SEB": {
     	"Base": "0x20000000", "Size": "2MB"}},
    {"Name": "Enclave-2",
     "Processor": {"Type": "VexRisc 32-bit",
      "Data Cache": "16KB",
      "Instruction Cache": "16KB", "FPU": "F32",
      "Debugging": "Disabled"},
     "Memory Size": "32MB",
     "Shared DRAM SEB": {
      "Base": "0x20000800", "Size": "128MB"}},
    {"Name": "Enclave-3",
     "Processor": {"Type": "A2I 64bit",
      "Data Cache": "64KB",
      "Instruction Cache": "64KB",
      "MMU": "Enabled", "MMU Page Size": "4KB",
      "FPU": "AXU", "Debugging": "Disabled"},
     "Memory Size": "64MB",
     "Shared DRAM SEB": {
      "Base": "0x20020800", "Size": "256MB"}}],
 "Peripherals": [
    {"Type": "AXI Gpio",
     "Board Interface": "Btns 2bits",
     "Access": ["Hardcore system", "Enclave-2"]},
    {"Type": "Uart Lite 8bit",
     "Baud Rate": "115200",
     "Access": ["Enclave-1"]},
    {"Type": "Dual Port BRAM Generator",
     "Base Address": "0x1F0000", "Size": "2MB",
     "Access": ["Enclave-1", "Enclave-3"]}]}
\end{lstlisting}
\end{figure}

\textbf{Creating Enclave Hardware.}
A developer can use the {\scshape HardwareBuilder}\xspace to design and create the hardware comprising of one or multiple enclaves.
Listing~\ref{l:CTEEHS} shows an example hardware description in JSON format of three enclaves.
Enclave-1 has a 32-bit MicroBlaze softcore CPU~\cite{mbsoft}. Enclave-2 has a 32-bit VexRisc softcore CPU~\cite{VexRiscv}, Enclave-3 uses a 64-bit A2I softcore CPU~\cite{a2i}.
The softcore CPUs of Enclave-2 and Enclave-3 have FPUs, instruction, and data caches.
A2I softcore of Enclave-3 has MMU enabled with a page size of 4KB.
Debugging is enabled on the softcore CPU of Enclave-1 only, for which {\scshape HardwareBuilder}\xspace inserts a debugging module to help the developer debug and trace the SSA on this enclave.
A DRAM region is reserved for the SEB of each enclave, respectively.
A UART peripheral is only connected to the Enclave-1 and cannot be accessed by the hardcore system or the other enclave.
Additionally, a GPIO peripheral is connected to both the hardcore system and Enclave-2 but cannot be accessed by Enclave-1 or Enclave-3.
Each enclave shares a DRAM region with the hardcore system for 2-way enclave-to-hardcore System communication.
Enclave-1 and Enclave-3 can also use the shared BRAM region to communicate.

The developer uses {\scshape HardwareBuilder}\xspace to generate hardware configurations, which outputs the Tcl scripts containing the synthesizer commands.
The \verb|-d| parameter specifies the JSON configuration file, and \verb|-o| defines the output Tcl path:

\begin{lstlisting}[label={header}, language=bash, numbers=none]
hardwareBuilder.py -d <CONFIG_JSON> -o <TCL>
\end{lstlisting}

Then, the developer invokes the synthesizer tool with the Tcl script as input.
The \verb|-n| parameter specifies the name of the hardware project,
and the \verb|bf| parameter specifies the mode of operation, which includes generating bitstream, combining bitstream with {\scshape Firmware}\xspace, etc.
The final output of the {\scshape HardwareBuilder}\xspace is a bitstream file specified by \verb|-o|:

\begin{lstlisting}[label={header}, language=bash, numbers=none]
createFPGAImage -d <TCL> -n <PROJ_NAME> -bf <BUILD_FLAG> -o <FPGA_IMAGE>
\end{lstlisting}

\textbf{Creating Boot Images.}
After the {\scshape HardwareBuilder}\xspace, the developer uses the boot loader creation tool with the developer-defined boot image format, e.g., \verb|.bif|, and the protected FPGA image to create a deployable binary file.
\begin{lstlisting}[label={header}, language=bash, numbers=none]
createBootImage <SYSTEM_BIF> <FPGA_IMAGE> -o <BYOTee_BIN>
\end{lstlisting}

\textbf{Creating SSAs and UAs.}
{\scshape BYOTee}\xspace\ {\scshape Firmware}\xspace is primarily developed in C, but software modules developed in any language that can be linked against the {\scshape Firmware}\xspace can be included in an SSA.
The SSAs, which have their own \verb|main| functions with a declaration of \verb|int main()| \verb|__attribute__| \verb|((section (".text.ssa_entry")))|, are developed as separate applications from the {\scshape Firmware}\xspace.
In our prototype implementation, SSAs are statically linked against the {\scshape Firmware}\xspace with the linker script {\scshape BYOTee}\xspace provides,
which reserves physical memory regions for the SSAs' code and data sections.

\begin{comment}
\begin{lstlisting}[label={header},caption={Header file defining SSA entry}, language=c]
#ifndef BYOT_HEADER_H_
#define BYOT_HEADER_H_

int main() __attribute__ ((section (".text.ssa_entry")));

#endif
\end{lstlisting}
\end{comment}

Not all libc functions are available for the SSAs to use.
For example, \verb|printf| may is not available since \verb|stdout| is not defined in the {\scshape Firmware}\xspace for a specific FPGA system.
To move data among DRAM, BRAM, and peripheral memories, system-specific underlying mechanisms will be used.
The {\scshape Firmware}\xspace provides a HAL with interfaces like \verb|BYOT_MemCpy| to replace the libc \verb|memcpy|.
The UAs execute as unprivileged applications on the operating system, e.g., PetaLinux for Xilinx devices, of the hardcore system.
The UAs use a UIO interface to communicate with the {\scshape Firmware}\xspace and SSA running on the FPGA.

\begin{comment}
\subsection{Usage}
After specifying the hardware description in the JSON file, a developer can generate hardware configurations with {\scshape HardwareBuilder}\xspace in two simple steps.
First, the developer invokes our developed python tool to generate the Tcl scripts containing the synthesizer commands:

\begin{lstlisting}[label={header}, language=bash, numbers=none]
parseConfig.py -d <CONFIG_JSON_path>
\end{lstlisting}
In the second step of the {\scshape HardwareBuilder}\xspace, the developer invokes the synthesizer tool with the Tcl script as input.
\begin{lstlisting}[label={header}, language=bash, numbers=none]
buildDevice -p <TCL_SRC_PATH> -n <PROJ_NAME> -bf <BUILD_FLAG>
\end{lstlisting}
The first two arguments specify the source Tcl file path, and the -n argument with the PROJ\_NAME specifies the name of the hardware project.
The last two option specifies the mode of operation, which includes creating project (cp), generating bitstream (gb), combining bitstream with {\scshape Firmware}\xspace (cb), and running all functions (all).
The final output of the {\scshape HardwareBuilder}\xspace is the bitstream file.

After the hardware generation and software development, the developer needs to invoke three {\scshape BYOTee}\xspace tools for deployment in a SoC device.
First, a FPGA boot loader creation tool with two arguments: the developer-defined boot image format and the path to protected FPGA image (bitstream) to create the deployable bin file.
\begin{lstlisting}[label={header}, language=bash, numbers=none]
createBootImage <SYSTEM.bif> <FPGA_IMAGE_PATH> -o <BYOT.BIN_PATH>
\end{lstlisting}
\end{comment}

The developer uses the {\scshape SSAPacker}\xspace to generate protected SSA binaries:

\begin{lstlisting}[label={header}, language=bash, numbers=none]
SSAPACKER -d <SSA_BIN> -o <PROTECTED_SSA>
\end{lstlisting}

{\scshape BYOTee}\xspace also includes a tool to put the boot loader, PetaLinux image, UA, protected FPGA image, and protected SSA binaries on the SD card for the deployment and debugging on Xilinx SoC FPGA systems:

\begin{lstlisting}[label={header}, language=bash, numbers=none]
deploySoC <SD_DEVICE> <BYOTee_BIN> <FPGA_IMAGE> <PROTECTED_SSA> <UA> <IMAGE.ub>
\end{lstlisting}

\subsection{Security Evaluation}

To demonstrate the security of {\scshape BYOTee}\xspace, we report the software TCB size of the four example SSAs and evaluate the cold-boot attacks on DRAM and BRAM of the same board.

\subsubsection{Software TCB Size}
\label{sub:tcbAnalysis}

Table~\ref{t:FwLocRes} presents the size of the software TCB for the four example SSAs and their corresponding {\scshape Firmware}\xspace.
As the table shows, the size of {\scshape Firmware}\xspace increases as the SSA gets more complicated and needs more services.
Nevertheless, the run-time software TCB (SSA and {\scshape Firmware}\xspace combined) of SSA-3, which is a functional digital right management music player, has only 10,727 SLOC, representing a significant software TCB reduction from its counterpart implemented as a TrustZone, e.g., TF-M~\cite{TFMsrc} has over 117K SLOC, or SGX application, e.g., the Gramine library OS~\cite{gramine} has 83K SLOC.

\subsubsection{Cold-boot Attacks on DRAM and BRAM}

\begin{figure}[!t]
    \vspace{-0.2cm}
	\centering
	\subfloat[0 s]{\includegraphics[width = 0.19\linewidth]{images/0.png}}\hfill
	\subfloat[5 s]{\includegraphics[width = 0.19\linewidth]{images/5.png}}\hfill
	\subfloat[15 s]{\includegraphics[width = 0.19\linewidth]{images/10.png}}\hfill
	\subfloat[20 s]{\includegraphics[width = 0.19\linewidth]{images/15.jpg}}\hfill
	\subfloat[30 s]{\includegraphics[width = 0.19\linewidth]{images/20.jpg}}  \\
	\subfloat[0 s]{\includegraphics[width = 0.19\linewidth]{images/0.png}}\hfill
	\subfloat[1 m]{\includegraphics[width = 0.19\linewidth]{images/f1m.png}}\hfill

	\subfloat[10 m]{\includegraphics[width = 0.19\linewidth]{images/f10m.jpg}}\hfill
	\subfloat[13 m]{\includegraphics[width = 0.19\linewidth]{images/20.jpg}}    \hfill
	\subfloat[BRAM initialization]{\includegraphics[width = 0.19\linewidth]{images/sram.png}}\hfill

	\caption{Visualizing DRAM decay after power down and BRAM hardware initialization after power up on the same Z7-07S board. We loaded a bitmap image (150$\times$150 pixel; 90.1kB) on the DRAM and BRAM.
	We then measure 1) DRAM decay at room temperature (20$^{\circ}$C/68$^{\circ}$F) after (a) power reset (0 second) and (b) - (e) losing power for different intervals;
	2) DRAM decay at -18$^{\circ}$C/0$^{\circ}$F after (f) power reset and (g) - (i) losing power for different intervals; and 3) (j) BRAM hardware initialization after power up. The reconstructed image from the fully decayed DRAM is red because half of the cells are measured as 1s and the other half as 0s. The reconstructed image from the BRAM is transparent because all the bits are initialized to 0s.}
	\label{fig:cold-boot}
	\vspace{-.3cm}
\end{figure}

We evaluated the feasibility of cold-boot attacks on DRAM and BRAM on the same Cora Z7-07S board.
In these experiments, we loaded a bitmap image onto the DRAM and BRAM and measured the DRAM decay at room temperature (20$^{\circ}$C/68$^{\circ}$F) and -18$^{\circ}$C/0$^{\circ}$F after power reset (0 seconds) and losing power for different intervals.
We dumped the content of BRAM, for which the Xilinx Zynq-7000 FPGA has a non-bypassable hardware initialization mechanism after power up to clear all the bits to 0s.
As we discussed in~\ref{subsec:analysisColdBoot}, even if the BRAM is not initialized, cold-boot attacks on it are much more difficult than on DRAM.
Figure~\ref{fig:cold-boot} visualizes the cold-boot attack results, which clearly show cold-boot attacks on DRAM are feasible but not on BRAM.

\subsection{Performance on the Low-end MicroBlaze Softcore CPU}

We evaluate the performance of the low-end MicroBlaze-powered enclaves, which provides a lower-bound performance estimation of available softcore CPUs.
We first evaluate the performance using 12 Embench-IoT benchmark applications~\cite{embenchIOT}.
We then evaluate the {\scshape BYOTee}\xspace software run-time performance by measuring the time cost of different {\scshape Firmware}\xspace operations.

\subsubsection{Benchmark Performance Evaluation}
To show the performance of the MicroBlaze softcore compared to the Cortex-M4 hardcore, we use 12 applications from the Embench-IoT benchmarks~\cite{embenchIOT}.
As Table~\ref{t:embench} shows, the applications run comparatively faster on the low-end MicroBlaze softcore CPU than the hardcore Cortex-M4.
For better performance, users can choose more advanced softcore CPUs.

\subsubsection{{\scshape Firmware}\xspace Performance}
We evaluate the time {\scshape Firmware}\xspace spends on the dynamic loading, decrypting, integrity and authenticity verification, attestation, cleaning up, suspending, and restoring operations of three SSAs,
As Table~\ref{t:all} shows,
the time spent by the {\scshape Firmware}\xspace on these operations is linear to the size of the SSA and its data combined.
To copy the protected SSA and its input from DRAM to BRAM, {\scshape Firmware}\xspace running on the Z7-07S spends around 1.07 ms for every 10,000 bytes.
To decrypt the protected SSA and its data using 256-bit CBC mode AES, {\scshape Firmware}\xspace spends around 2182 ms for every 10,000 bytes.
The integrity and authenticity verification costs around 93.43 ms for every 10,000 bytes.
The BLAKE-based pre-execution and post-execution cost around 124.77 ms for every 10,000 bytes.
Cleaning up the BRAM takes around 0.45 ms for every 10,000 bytes.
The SHA512-HMAC and AES 256-bit with CBC mode based suspending and restoring cost roughly 2834 ms for every 10,000 bytes.

\begin{comment}
\emph{SSA Dynamic Load and Clean Up}

Table~\ref{t:all} includes the performance of {\scshape Firmware}\xspace with respect to dynamic load and clean-up of BRAM for different sizes of SSAs.
Dynamic load of SSA performance is directly related to the size, as before loading of SSAs {\scshape Firmware}\xspace will perform integrity and authenticity check operations on SSA modules.
For clean-up operation, the performance depends on reserved memory size for SSAs by the {\scshape Firmware}\xspace

\textbf{SSA Decryption}
\label{s:decrypt-res}
Table~\ref{t:all} represents the ciphertext size for each SSA and their decryption execution time inside the {\scshape Firmware}\xspace.
The ciphertext data for the SSAs is made up of the extracted code section, data sections, and metadata.
We are using symmetric encryption of SSAs, namely AES with developer-defined key.
For multiple invocation of the same SSA, the {\scshape Firmware}\xspace performs this decryption once.

\textbf{SSA Authentication}
SSA authentication data include the SSA binary, input data, any metadata from SEB of CTEEs.
From the table~\ref{t:all} we observe that the authentication performance of {\scshape BYOTee}\xspace depends on the size of data to be authenticated.
For SSA-3 application user inputs are put to SEB at runtime instead of load time by UA.
Authentication is not performed with the runtime data, but for files loaded from SD card such as encrypted song files are authenticated.
\textbf{Attesation on SEB}
\label{sub:attst}

Table~\ref{t:all} represents the {\scshape Firmware}\xspace attestation performance on the three SSAs.
For \emph{preExeAtt}, the attestation data is the sum of the extracted SSA modules, metadata, and inputs.
For \emph{postExeAtt}, we include the SSA outputs along with the \emph{preExeAtt} attestation data.
The \emph{preExeAtt} and \emph{postExeAtt} are performed on each invocation of the SSAs.
Although based on the result, it may seem the attestation is a slow process, this can be mitigated with a more powerful FPGA unit and more sophisticated processors.

\end{comment}

\begin{comment}
\textbf{CTEE communication performance}
{\scshape BYOTee}\xspace supports communication of the SSAs in different CTEEs through shared memory.
Thus, the communication speed is analogous to the access of local memory and no additional constraints are present except the bus speed.
While in traditional TEEs such as Intel SGX, two secure enclaves even in the same platform need to go through local attestation and requires the OS to perform additional key exchange, encryption operations to ensure secure communication~\cite{costan2016intel}.
In {\scshape BYOTee}\xspace as the BRAM is physically isolated from the PS (REE), the communication medium is trusted and can be used directly to communicate with other SSAs on different CTEEs.
\end{comment}

\begin{comment}
\subsection{Attacks and Mitigations}
\label{sub:atkDef}
{\scshape BYOTee}\xspace can be subject to different attack scenarios.
We demonstrate some attacks on {\scshape BYOTee}\xspace, which are within the scope in the following:

\subsubsection{SSA Integrity} As described in \S~\ref{subsec:attest}, the SEB between the PS and the CTEEs are not secure, and PS software has full access.
Before the {\scshape Firmware}\xspace copies the SSA modules onto BRAM, the PS software can inject or temper code and data sections of the SSA modules.
We demonstrate such an attack from the PS software, which disrupts the integrity of SSAs.
Firstly, the SSA modules are encrypted while it's in the SEB.
Thus, code or data manipulation to leak sensitive information is infeasible from the PS software.
Even if the malicious PS software manipulates the encrypted SSA modules, the \emph{preExeAtt} measurement makes those attacks detectable.
Besides, the \emph{postExeAtt} after each SSA execution provides PoX for the SSAs.

\subsubsection{Input and Output Data Integrity} Even though the SSA modules are encrypted in the SEB, the input and output data of the SSAs are not encrypted.
We construct an attack where the UA disrupts the integrity of the input or output data.
Our attestation mechanism detects the tempering of the I/O data similarly to the tempering of SSA modules.
Additionally, we include a \textit{chal} number for each measurement, which detects replay attacks from the UA.

\end{comment}

\section{Implementation and Evaluation}
\label{s:impleval}

In this section, we present an implementation of the {\scshape BYOTee}\xspace framework for the Xilinx SoC FPGA and evaluate it on a low-end Digilent Cora Z7-07S development board ($\approx$\$130) with a Zynq-7000 FPGA.

\subsection{Experiment Environment}

\begin{figure*}[!t]
    \centering
    \subfloat[Enclave-1 w/ debugger]{\includegraphics[width = 0.19\linewidth]{images/Impl_aes_with_mdm.png}}\hfill
    \subfloat[Enclave-1 wo/ debugger]{\includegraphics[width = 0.19\linewidth]{images/Iml_aes_without_mdm.png}}\hfill
    \subfloat[Enclave-2 wo/ debugger]{\includegraphics[width = 0.19\linewidth]{images/Impl_led_without_mdm.png}}\hfill
    \subfloat[Enclave-3 wo/ debugger]{\includegraphics[width = 0.19\linewidth]{images/Impl_secure_DRM_without_mdm.png}}\hfill
    \subfloat[Enclave-1 and Enclave-4 wo/ debugger]{\includegraphics[width = 0.19\linewidth]{images/Impl_aapp4_with_mdm.png}}
    \vspace{-0.2cm}
    \caption{Resource footprints of the enclaves with MicroBlaze softcore CPUs for the example SSAs on the Cora Z7-07S.
The yellow and red portions represent the MicroBlaze softcore CPU cells.
The purple portion represents the cells of BRAMs.
The pink rectangle represents the I/O ports. In (d), the rectangle on top of the I/O ports represents an analog to digital conversion module. The blue portions represent all other IPs, such as different peripherals, debugging modules, interconnects.}
    \label{fig:corr-res-iml}

\end{figure*}

\begin{table*}[t]
	\centering
	\footnotesize
	\caption{Hardware TCB size and resource utilization of enclaves for the example SSAs on Cora Z7-07S}
	\label{t:resApps}
	\begin{tabular}{c|c|c|c|c|c|c|c|c}
		\hline
		 & \multicolumn{2}{c|}{Enclave-1}   & \multicolumn{2}{c|}{Enclave-2}  & \multicolumn{2}{c|}{Enclave-3}& \multicolumn{2}{c}{Enclave-4}  \\ \hline
		Resource & w/ debugger   & w/o debugger  & w/ debugger   & w/o debugger & w/ debugger   & w/o debugger & w/ debugger   & w/o debugger \\ \hline
		LUT     & 5,255 (36.5\%)      &   2,232 (15.5\%)    & 6,385 (44.3\%) &  3,291 (22.9\%) & 10,781 (74.9\%) & 6,778 (47.1\%) & 5,302 (36.8\%) & 3,130 (21.7\%) \\ \hline
		LUTRAM   &  419 (7.0\%)        &    211 (3.5\%)    &   507 (8.5\%) & 282 (4.7\%)  & 725 (12.1\%) & 427 (7.1\%) & 319 (5.3\%) & 145 (2.4\%)\\ \hline
		Flip-flop    & 5,245 (18.2\%)       &  2,259 (7.8\%)    &     6,759 (23.5\%)  & 37,64 (13.1\%) & 11,363 (39.5\%)&   7,721 (26.8\%) & 5,497 (19.1\%) & 3,014 (10.5\%)\\ \hline
		BRAM  & 18 (36.0\%)  & 16 (32.0\%)    &     34 (68.0\%) & 32 (64.0\%)   &  48 (95.0\%)   & 45.50 (91.0\%)   & 28 (56.0\%)  & 26 (52.0\%)\\ \hline
		DSP    & 3 (4.5\%) & 0 (0.0\%)&  3 (4.5\%)     &   0 (0.0\%) & 3 (4.5\%)  & 0 (0.0\%) & 3 (4.5\%) & 0 (0.0\%) \\ \hline
		IOB     & 0 (0.0\%) & 0 (0.0\%)   &  6 (6.0\%)   & 6 (6.0\%)   &   28 (28.0\%) & 28 (28.0\%) & 0 (0.0\%) & 0 (0.0\%) \\ \hline

	\end{tabular}
	\vspace{0.25cm}
\end{table*}

The Cora Z7-07S board has a single-core 667MHz Arm Cortex-A9 processor with 512MB DDR3 memory, 32KB L1 cache, 512KB L2 cache and a Xilinx Zynq-7000 FPGA.
As shown in Table~\ref{t:fpgas}, the Zynq-7000 FPGA has 3,600 logic cells, 14,400 LUTs, 6,000 LUTRAM, 28,800 flip-flops, a 225KB BRAM, 66 Digital Signal Processing (DSP) slices, and 100 IOBs.
The development board also has an SPI header, two push-buttons, two RGB LEDs, a microSD card slot, two Pmod connectors, etc.
We connect a Pmod I2S2 stereo audio input and output device~\cite{pmod} to the board for SSA-3 evaluation.
Figure~\ref{fig:z7} (in Appendix) shows the top and bottom view of the board with the connected audio device.
We create two partitions, namely \verb|boot| and \verb|root|, on an SD card and use the Xilinx \verb|bootgen| tool to generate device boot images~\cite{bootgen}.
\verb|Bootgen| stitches a stock FSBL for the hardcore system and the protected FPGA image together to create a binary boot file.
The boot file, U-Boot as the SSBL for the hardcore system, and a PetaLinux image for the hardcore system are stored in the \verb|boot| partition, whereas the protected SSAs, UAs, and other application files are stored in the \verb|root| partition.

\subsection{{\scshape BYOTee}\xspace Implementation}
We implemented the {\scshape BYOTee}\xspace infrastructure and toolchain, which include {\scshape HardwareBuilder}\xspace, {\scshape SSAPacker}\xspace, and {\scshape Firmware}\xspace, for the Xilinx SoC FPGA.
The {\scshape HardwareBuilder}\xspace was developed in Python (2.5K SLOC).
The {\scshape SSAPacker}\xspace include Python (63 SLOC) and C code (420 SLOC).
The {\scshape Firmware}\xspace was developed in C and has an SSA loader and cleaner (1.1K SLOC), an attestation module (333 SLOC), an interrupt initialization and handling module (101 SLOC), and a linker script (212 lines).
The {\scshape Firmware}\xspace is linked against the vendor-provided HAL (7.9K SLOC) and libraries, e.g., libc (1.2MB), etc.
The {\scshape Firmware}\xspace, especially the HAL, can be customized to reflect an SSA's needs.
Our implementation of the SSA loader uses AES-256 for SSA encryption and SHA512-HMAC to protect the integrity and authenticity of SSAs.
We use the BLAKE2~\cite{Blk2} hash algorithm to implement the pre-execution- and post-execution-attestations of SSA applications in the attestation module.
On the hardcore system side, a userspace I/O interface is used for the UAs to access the shared DRAM regions between the hardcore system and FPGA.

\begin{table}[t]
	\centering
	\footnotesize
	\caption{Size of the example SSAs' software TCB}
	\label{t:FwLocRes}
	\begin{tabular}{c|c|c|c|c|c|c|c}
		\hline
		 & \multicolumn{2}{c|}{SSA} & \multicolumn{5}{c}{Corresponding {\scshape Firmware}\xspace} \\
			\hline
		& SLOC & Bytes & SLOC & .text & .data & .bss & Total \\
		\hline
		SSA-1 & 717 & 12,892& 3,143 &   27,296    &  3,236    &  448    &      30,532 \\
			\hline
		SSA-2 & 346 & 2,868 & 3,532&  30,748     &  2,800  &   440   &  33,988   \\
		\hline
		SSA-3 & 1,029 & 20,380 & 9,698 &     57,142  &    4,308  &    635  &  62,085   \\
		\hline
		SSA-4 & 622 & 31,088 & 3,235 & 28,377   &   3,608   &   528  &  35,748  \\
		\hline
	\end{tabular}
    \vspace{0.1cm}
\end{table}

\begin{table*}[t]
	\centering
	\footnotesize
	\caption{{\scshape Firmware}\xspace performance evaluation on the low-end softcore MicroBlaze CPU (Time in milliseconds; size in bytes). The experiments show SSA-3 can verify, decrypt, and play 48KHz WAV music smoothly on the very low-end softcore CPU.}
	\label{t:all}
	\begin{tabular}{c|c|c|c||c|c|c|c|c|c||c|c}
		\hline
		&  Binary & Input & Output & Loading & Decryption & Integrity and & pre-execution & post-execution  & Cleaning & Suspend & Restore \\
		&  size & size & size &  & & authenticity & attestation & attestation & up & and & and\\
		&  &  & &  & & verification & & & & export & execute\\
		 \hline
		SSA-1 & 12,892 & 64 & 64 & 1.39 & 2784.56 & 118.54 & 153.94  & 154.79 & 0.54 & 3694.55 & 3609.65\\ \hline
		SSA-2 &  2,596 & N/A & N/A & 0.30 & 579.11 & 29.15 & 30.90 & 30.90 & 0.11 & 741.71
		 & 729.69 \\ \hline
		SSA-3 & 20,152 & 128
		& 132 & 2.17 & 4414.32 & 185.30 & 258.30 & 260.50 & 0.90 & 5787.97 & 5638.76 \\ \hline

	\end{tabular}
    \vspace{-0.2cm}
\end{table*}

\begin{table}[t]
	\centering
	\footnotesize
	\caption{Embench-IoT performance evaluation in millisecond on  softcore MicroBlaze CPU (Zynq-7000 FPGA implemented, version 10.0, 100MHz, no cache, no FPU) and hardcore Cortex-M4 CPU (16MHz, no cache, no FPU, officially reported performance from ~\cite{embencm4}).}
	\label{t:embench}
	\begin{tabular}{l|p{3cm}|c|c}
	\hline
	 Application & Description & M4~\cite{embencm4} & MicroBlaze \\ \hline
	 aha-mont64 & Modulo generator& 4,004 &  501 \\ \hline
	 crc32 & 32 bit error detector & 4,010 & 193\\ \hline
	 huffbench & Data compressor & 4,120 &  111\\ \hline
	 minver&Floating point matrix inversion&3,998 & 327\\ \hline
	 nettle-aes & Low level AES library & 4,026 & 245\\ \hline
	 nsichneu & Computes permutation&4,001 & 237\\ \hline
	 primecount &Prime counter & n/a & 193\\ \hline
	 sglib-combined& Sort, search, and query on array, list, and tree &3,981 &  189\\ \hline
	 slre&Regex matching&4,010&113\\ \hline
	 statemate&Car window lift control & 4,001 & 139\\ \hline
	 tarfind&Archive file finder& n/a &163
	  \\ \hline
	 ud & Matrix factorization & 3,999 & 343\\ \hline
	 \multicolumn{2}{c|}{Geometric Mean}&4,015&208\\ \hline
	\end{tabular}
\end{table}

\subsection{Enclaves for the Example SSAs}
We specified the hardware description for each example SSA in~\S\ref{sub:apps} and used the {\scshape HardwareBuilder}\xspace and synthesizer to generate its enclave bitstream, e.g., Enclave-1 for SSA-1.
All the enclaves are configured with a 32-bit Microblaze CPU (version 10.0, 100MHz, no instruction/data cache, no FPU).
The Enclave-1, Enclave-2, Enclave-3 have a 128KB BRAM, whereas Enclave-4 has a 32KB BRAM as their main memory.
The peripherals that belong to an enclave are connected through a dedicated AXI Interconnect IP.
Figure~\ref{fig:corr-res-iml} shows the footprints of the four hardware designs on the Z7-07S device.
These figures demonstrate the configurable nature of the {\scshape BYOTee}\xspace hardware TCB and the physical isolation of the enclaves from each other and from the hardcore processor.
Figure~\ref{fig:corr-res-iml}(a) shows the hardware design with a debugger, which developers can use to debug the SSA and {\scshape Firmware}\xspace on the softcore,
whereas all other designs do not have a debugger for the minimum hardware TCB.
The orange portion of the footprint represents cells of the ZYNQ7 processing system, whereas yellow represents softcore cells.
The purple straight lines represent the cells of BRAMs.
The pink rectangle represents the I/O ports, and in Figure~\ref{fig:corr-res-iml}(d) the rectangle on top of the I/O ports represents an analog to digital conversion module.
Figure~\ref{fig:corr-res-iml}(e) represents hardware implementation of two enclaves, namely Enclave-1 (yellow) and Enclave-4 (red), to run SSA-1 and SSA-4, respectively.
There are three types of BRAM in this design.
The purple straight lines represent the contiguous memory of Enclave-1 (128KB), and the green straight lines represent the contiguous memory of Enclave-4 (32KB).
The ash straight lines represent the shared BRAM (8KB) between Enclave-1 and Enclave-4.
The blue portions in Figure~\ref{fig:corr-res-iml} represent the rest of the IPs used in the hardware design.
These include interconnects, interrupt controllers, etc.

Table~\ref{t:resApps} presents each enclave's hardware TCB and resource utilization on the Cora Z7-07S board with and without a debugger IP.
As the table shows, the debugger IP significantly increases the resource utilization of an enclave as it uses three DSP slices, two BRAMs, and many other resources.
Since SSA-1 and SSA-4 do not use peripherals, Enclave-1 and Enclave-4
do not have any IOB.
Figure~\ref{fig:corr-res} (in Appendix) shows the block diagrams of the enclaves generated by the {\scshape HardwareBuilder}\xspace for the four example SSAs.

\section{Introduction}

Existing hardware-assisted Trusted Execution Environments (TEE) on commodity computing devices make use of hardware security primitives offered by the CPU, such as Intel SGX~\cite{costan2016intel} and Arm TrustZone~\cite{pinto2019demystifying}, to guarantee code and data loaded inside to be protected, with respect to confidentiality and integrity, from the Rich Execution Environment (REE).
In recent years, we have witnessed unprecedented growth in using SGX and TrustZone in real-world products and academic projects,
which include real-time kernel protections~\cite{azab2014hypervision,ge2014sprobes}, securing containers and runtime libraries~\cite{arnautov2016scone,tsai2017graphene, santos2014using}, shielding applications from attacks~\cite{baumann2015shielding,lind2017glamdring,schuster2015vc3}, etc.

\textbf{Limitations of Existing TEEs.} The hardware and software of existing TEEs nevertheless have the following issues, making them either ineffective, inefficient, or untrustworthy:
i) existing TEE architectures, such as Intel SGX and Arm TrustZone, are proprietary and only work for specific computing architectures, which users have to trust blindly.
It is impossible for users to verify the correctness of the TEE designs or attest the hardware states at run-time;
ii) they only provide a static and fixed hardware Trusted Computing Base (TCB), which cannot be customized for different applications.
For TrustZone, it means the TEE has the highest privilege to control the REE and communicate with all peripherals.
The design violates the principle of least privilege by including unnecessary peripherals and buggy peripheral drivers in the software TCB~\cite{cerdeira2020sok} and exposing the TEE to malicious peripheral inputs~\cite{gross2019breaking}.
For SGX, it means applications in enclaves have to go through and trust the REE OS to communicate with peripherals~\cite{schneider2020pie}, bloating the size of software TCB by including a usually monolithic REE OS kernel, e.g., 27.8M Source Lines of Code (SLOC) of the Linux kernel~\cite{LinuxSLOC};
iii) the TEE shares a processor core with the REE in a time-sliced fashion, not only costing many CPU cycles~\cite{song2016enforcing, zhang2019icore} for the expensive context switches between the TEE and REE but also making it vulnerable to cache side-channel attacks~\cite{zhang2016truspy, cho2018prime, brasser2017software, zhang2016return, gutierrez2018cachelight}, which directly undermine TEE's security promises;
iv) the software TCBs in TEEs are large, creating big attack surfaces for run-time attacks that hijack the control or data flow~\cite{almakhdhubmurai2020, zhou2020silhouette}.
For example, Haven~\cite{haven} places a whole Windows 8 library OS inside the enclave.
OP-TEE, a Cortex-A secure world OS, has 277K SLOC~\cite{OP-TEE},
and TF-M~\cite{TFMsrc}, a trusted firmware for Cortex-M TrustZone, also over 117K SLOC.
Partly due to these reasons, in 2022 Intel has removed support for SGX in some future CPUs~\cite{sgxdepre}.
In addition,
TrustZone has many other limitations:
i) TrustZone only provides \emph{one} isolated execution environment;
ii) TrustZone does not encrypt the contents of Dynamic RAM (DRAM).
Therefore, it is subject to code and data disclosure from cold-boot attacks~\cite{halderman2009lest};
and iii) TrustZone does not have native hardware supports for remote attestations of code and data integrity~\cite{nunes2020apex, abera2019diat}
due to the lack of platform registers to accumulate measurements.

\begin{table}[t]
	\centering
	\footnotesize

	\begin{tabular}{c|c|c|c}
		\hline
		Vendor & {Intel}   & \multicolumn{2}{c}{Xilinx}   \\ \hline
		 &  Agilex F  & Zynq-7000 &  UltraScale+ EG  \\
		SoC FPGA  & R25A~\cite{Agilex}  & XC7Z007S~\cite{CoraZ7z}& ZU19EG~\cite{evUltra}  \\ \hline
		LE/LC & 2,692,760 & 3,600& 1,143,000 \\ \hline
		ALMs/LUT  & 912,800&14,400& 523,000\\ \hline
		PLLs & 28& 2 & 12\\ \hline
		DSP  & 8,528&66 & 1,968 \\ \hline
		BRAM (MB) & 259 &0.225 & 80.4\\ \hline
 		I/O & 624&49&347 \\ \hline

		Hardcore CPU &Cortex-A53 (Q)&Cortex-A9 (S) & Cortex-A53 (Q) \\ \hline
		DRAM (GB)& 16 & 0.512 & 8 \\ \hline

		Price in 2022 & $\approx$\$10K& $\approx$\$130& $\approx$\$4.3K\\ \hline
	\end{tabular}

	\caption{Example SoC FPGA products. LE: Logic Elements (Intel); LC: Logic Cells (Xilinx); ALM: Adaptive Logic Modules (Intel: Multiple LUTs, registers, adders, and multiplexers make up an ALM); LUT: Look Up Table (Xilinx); DSP: Digital Signal Processing blocks; PLLs: Fabric and I/O Phase-Locked Loops; I/O: Maximum User I/O Pin; S: Single-core; Q: Quad-core. We use a Zynq-7000 system in our evaluations.}
	\label{t:fpgas}
\end{table}
\begin{comment}

\begin{table*}[t]
	\centering
	\footnotesize

	\begin{tabular}{c|c|c|c|c|c|c}
		\hline
		Vendor & \multicolumn{3}{c|}{Intel}   & \multicolumn{3}{c}{Xilinx}   \\ \hline
		& Arria V GZ & Stratix V GX  & Agilex F  & Zynq-7000 &  UltraScale+ RFSoC&  UltraScale+ EG  \\
		SoC FPGA & 5AGZE5~\cite{arria} & 5SGXA7~\cite{stratix}  & R25A~\cite{Agilex}   &  XC7Z007S~\cite{CoraZ7z}  & ZU27DR~\cite{rfsoc} & ZY19EG~\cite{evUltra}  \\ \hline
		LE/LC  &400,000 & 622,000 & 2,692,760 & 3,600&  930,300& 1,143,000 \\ \hline
		ALMs/LUT  &150,960& 234,720 & 912,800&14,400&  425,280&523,000\\ \hline
		PLLs &24 & 28 & 28& 2 & 8 & 12\\ \hline
		DSP  &71,092& 256& 8,528&66&4,272 & 1,968 \\ \hline
		BRAM (MB) & 33.5& 50& 259 &0.225&  73.5&80.4\\ \hline
		I/O &674& 696& 624&49&120&347 \\ \hline

		Hardcore CPU &Cortex-A9 (D) &Cortex-A53 (D) &Cortex-A53 (Q)&Cortex-A9 (S) & Cortex-A53 (Q)&Cortex-A53 (Q) \\ \hline

		Price in 2022 (\$) & $\approx$5K& $\approx$8.5K& $\approx$10K& $\approx$130& $\approx$4.5K& $\approx$4.3K\\ \hline
	\end{tabular}

	\caption{Example SoC FPGA products in the market. LE: Logic Elements (Intel); LC: Logic Cells (Xilinx); ALM: Adaptive Logic Modules (Intel: Multiple LUTs, registers, adders, and multiplexers make up an ALM); LUT: Loop Up Table (Xilinx); DSP: Digital Signal Processing Blocks; PLLs: Fabric and I/O Phase-Locked Loops; I/O: Maximum User I/O Pin; S: Single-core; D: Dual-core; Q: Quad-core.}
	\label{t:fpgas}
\end{table*}
\end{comment}
While many software-based projects attempted to address one or more of the aforementioned issues by building software layers on SGX, TrustZone, or RISC-V, some issues, e.g., non-verifiable design, non-customizable hardware TCB, time-sharing of the processor, vulnerable to cold-boot attacks due to the slow DRAM decay, etc., are rooted in the design of their respective hardware system, business model and ecosystem, etc.; therefore they cannot be addressed by software alone (refer to Table~\ref{tab:comparison} for comparisons).
For example, {\scshape Sanctuary}~\cite{brasser2019sanctuary} uses the memory access controller to provide multi-domain isolation for sensitive applications on Cortex-A TrustZone,
and {\scshape Cure}~\cite{bahmani2021cure} enables the exclusive assignment of system resources, e.g., peripherals, CPU cores, or cache resources, to each enclave on RISC-V.
Even though they address some of the aforementioned issues, both solutions still suffer from cold-boot attacks and a large software TCB.
Other hardware-based solutions, such as HECTOR-V~\cite{nasahl2020hector}, Graviton~\cite{volos2018graviton}, etc., do not address all the issues either, and they cannot be deployed on commodity devices because hardware changes are required.

\textbf{Benefits of FPGA.}
Field-Programmable Gate Array (FPGA) is designed to be configured by users using a hardware description language after manufacturing.
Besides reconfigurability, it has advantages of high performance, fast development round, etc.
As shown in Table~\ref{t:fpgas}, a variety of FPGA products ranging from embedded systems, i.e., Xilinx Zynq-7000 with only 3,600 logic elements ($\approx$\$130), to data center devices, i.e., Agilex F R25A with 2.6 millions of logic elements ($\approx$\$10k), are available in the market.
Various FPGA-based application-specific accelerators, such as for deep convolutional neural networks~\cite{zhang2015optimizing, suda2016throughput}, recurrent neural networks~\cite{li2019rnn}, classic and post-quantum cryptographic algorithms~\cite{chelton2008fast, elkhatib2021high},  Memcached~\cite{lavasani2013fpga}, etc., have been proposed and deployed on such devices~\cite{qiu2016going, huang2016programming}.

More interestingly, FPGA can also be used to build general-purpose computing platforms, in which users can design and implement their own softcore CPUs or customize existing open-sourced~\cite{OpenCores, VexRiscv, Neo430, microwatt, a2i, a2o, OpenSPARC, libreSoc} or proprietary ones~\cite{lysecky2009design, nios}.
The available softcore CPUs range from the partially configurable (i.e., cache size, pipeline depth, memory management unit, etc.) and proprietary low-end 32-bit MicroBlaze~\cite{lysecky2009design}, to the fully customizable and open-sourced mid-end 32-bit RISC-V~\cite{VexRiscv, matthews2017taiga} and high-end 64-bit A2I POWER processor~\cite{a2i}.
While the low-end softcore CPUs are comparable to micro-controllers, their mid-end and high-end counterparts have performances comparable to hardcore micro-processors~\cite{heinz2019catalog}.
Because it is possible to formally verify the FPGA implementations of system-on-chip resources~\cite{sieh1997verify, breuer1997refinement}, including the softcore CPUs,
users do not have to blindly trust a whole proprietary CPU but just the FPGA configuration modules and verifiers.

\textbf{Customizing Enclaves Using FPGA.}
We present a hardware and software co-design framework to \underline{B}uild \underline{Y}our \underline{O}wn \underline{T}rusted \underline{E}xecution \underline{E}nvironments ({\scshape BYOTee}\xspace) on commodity FPGA devices.
Using the {\scshape BYOTee}\xspace toolchain, users can easily build multiple equally secure and customized enclaves on-demand to execute their Security-Sensitive Applications (SSA).
An enclave can include only the hardware and software functionality necessary for the SSA and exclude other hardware and software components on the system, minimizing the sizes of hardware and software TCB.

At the hardware layer, {\scshape BYOTee}\xspace utilizes commodity System-on-Chip (SoC) FPGA devices,
which integrate both hardcore CPU system, e.g., Cortex-A, RISC-V, etc., and FPGA programmable logic architectures.
The nature of FPGA enables on-demand configurations and reconfigurations of enclaves' hardware TCBs, which may include softcore CPUs, Block RAM based (BRAM; same as Static RAM on known SoC FPGA devices) main memory, and peripherals.
For example, an enclave may have a softcore A2I POWER CPU and a hardware GPU as its dedicated peripheral for acceleration.
Each enclave has its own isolated physical address space, which maps its own main memory, system configuration registers, and peripherals.
The software on the hardcore CPU and other enclaves cannot see an enclave's address space unless it is explicitly specified in the design.
By assigning the hardware resources to co-resident enclaves, {\scshape BYOTee}\xspace builds a multiprogramming environment, isolates software faults, and provides memory protection on FPGAs.
Because enclaves do not time-share any processor with each other and the hardcore system, the cache side-channel attack vector is removed.
Cold-boot attacks on these enclaves are also difficult due to the characteristics of BRAM.

FPGA alone, however, does not address all the issues of existing TEEs, and the SSAs need external libraries to execute and drivers to control the peripherals.
On the software front, the configurable {\scshape Firmware}\xspace component of {\scshape BYOTee}\xspace provides necessary software libraries, e.g., libc, and a Hardware Abstraction Layer (HAL), for a minimum software TCB.
Even though formally verifying the correctness and security of hardware designs, e.g., softcore CPU, is beyond our scope,
{\scshape BYOTee}\xspace offers mechanisms to bootstrap the trust and attest the integrity of the hardware implementations.
In particular, to bootstrap trust in enclaves statically and dynamically, {\scshape BYOTee}\xspace offers example paths building on the secure boot or a run-time trusted module, respectively.
Additionally, the {\scshape Firmware}\xspace offers two attestation mechanisms, namely pre-execution and post-execution attestations.
The former extends the traditional remote attestation of code and input to include hardware integrity, whereas the latter extends output data integrity attestation~\cite{abera2019diat}.

We implement a proof-of-concept {\scshape BYOTee}\xspace infrastructure and toolchain for the Xilinx SoC FPGA.
The toolchain includes {\scshape HardwareBuilder}\xspace, which takes developers' hardware resource need as input and automatically generates hardware modules and interconnections.
{\scshape HardwareBuilder}\xspace enables developers to focus on SSA development, increases the usability of {\scshape BYOTee}\xspace, and decreases the chances of developer-induced misconfigurations.
The system and toolchain also include {\scshape Firmware}\xspace, which provides a run-time environment, and {\scshape SSAPacker}\xspace, which encrypts and signs an SSA binary.
We demonstrate the usage of {\scshape BYOTee}\xspace with four example SSAs.
We emphasize none of the existing TEE solutions can meet all the security needs of the SSA-3, which is a secure music player with digital rights management.
We evaluate the security and performance of {\scshape BYOTee}\xspace with 12 benchmark applications and the four SSAs.
The evaluation results on the low-end Zynq-7000 system show {\scshape BYOTee}\xspace is effective in defeating attacks and efficient in executing SSAs.
The contributions of this paper are summarized as follows:

\begin{itemize}
\item We present {\scshape BYOTee}\xspace, a framework for building multiple general-purpose enclaves with configurable hardware and software TCBs utilizing commodity FPGA devices.
The idea of {\scshape BYOTee}\xspace can be implemented on FPGA systems from any vendor;

\item We present example paths to bootstrap trust in an enclave's hardware and software statically or dynamically. We design and implement two attestation mechanisms that capture the identifies of boot loaders, bitstream (hardware implementation), {\scshape Firmware}\xspace, and SSA;

\item We implement the {\scshape BYOTee}\xspace system and toolchain for the Xilinx SoC FPGA, which include the {\scshape HardwareBuilder}\xspace, {\scshape Firmware}\xspace, {\scshape SSAPacker}\xspace, and other helper tools, e.g., linker scripts, to facilitate developers to design, develop, and debug their SSAs.
\ifdefined
    We open-source the {\scshape BYOTee}\xspace system and toolchain\footnote{https://github.com/CactiLab/BYOTee-Build-Your-Own-TEEs};
\else
   To show the good faith of open-source, we put an anonymized copy of the source code online for artifact evaluations\footnote{https://anonymous.4open.science/r/byot-8011};
\fi

\item We choose the very low-end MicroBlaze softcore CPU and Zynq-7000 system to demonstrate {\scshape BYOTee}\xspace's usage, security, effectiveness, and performance with the Embench-IoT benchmark suite and four SSAs.
\end{itemize}

\begin{comment}
The rest of this paper is organized as follows.
In~\S\ref{s:back}, we introduce the background knowledge of SoC FPGA.
\S\ref{s:thread} presents the system and threat model of {\scshape BYOTee}\xspace.
\S\ref{s:arch} presents the architecture and workflow of the {\scshape BYOTee}\xspace framework and illustrates its components in detail.
\S\ref{sec:analysis} presents the security analysis.
In \S\ref{s:dev}, we discuss how to develop and build SSAs from developer's perspective and illustrate several examples of real-world applications that can be protected by {\scshape BYOTee}\xspace.
\S\ref{s:impleval} presents the implementation and evaluation results on the security and performance of {\scshape BYOTee}\xspace.
We discuss and compare {\scshape BYOTee}\xspace with related work in \S\ref{s:rel}.
\S\ref{s:conclusion} concludes this paper.
\end{comment}
\section{Future Research Directions}
\label{s:future}
After our current implementation of {\scshape BYOTee}\xspace, we realize there are several future research opportunities.
Some of the possible research directions are listed below:

\subsection{Compartmentalize Applications} Given a bare-metal  application, one future direction could be to compartmentalize them into different secure and non-secure modules.
The secure modules will be loaded onto the CTEEs dynamically, and the non-secure modules will execute on the PS side.
This automated process will make the system easier to use, and developers will develop one single application.
\subsection{Decouple {\scshape Firmware}\xspace from CTEEs} To reduce the software TCB further {\scshape Firmware}\xspace can be decoupled from the CTEE hardware.
Instead of individual {\scshape Firmware}\xspace for each CTEE, single {\scshape Firmware}\xspace to manage all the CTEEs will reduce the software TCB further.
\subsection{Dynamic Reconfiguration of CTEEs}
Using the PS interfaces such as DevC and PCAP, partial or full reconfiguration of the CTEEs will benefit from dynamic hardware TCB.
This dynamic hardware configurable feature for individual sensitive applications can further reduce the hardware TCB.
\end{comment}

\section{Acknowledgment}

We thank Paul Ratazzi, Zhenhua Wu, Kai Ni, and Arnob Paul for the discussions while this work was in progress.
We thank Sandro Pinto for bringing TEEOD to our attention.
We also thank the organizers of MITRE eCTF 2020 for inspiring this research.
This material is based upon work supported in part by the National Science Foundation under Grant No. 2037798 and by the Air Force Visiting Faculty Research Program.
Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of United States Government or any agency
thereof.

\bibliographystyle{abbrv}

\section{Security Analysis and Limitations}
\label{sec:analysis}

In this section, we conduct an informal security analysis of {\scshape BYOTee}\xspace,
in which we discuss the attacks {\scshape BYOTee}\xspace can and cannot defend from.

\subsection{Malicious and Compromised Hardcore System Software}
\textbf{Read and Write to Enclaves.}
The hardcore system software, including the operating system and UA, is not a part of the TCB in {\scshape BYOTee}\xspace.
Even if the hardcore system software is compromised at run-time, the attacker cannot access the data on/from enclave hardware resources, including BRAM and peripherals, because they are in the isolated address space of the target enclave.
The attacker cannot breach the confidentiality of SSA code and data as well, because they are encrypted at build time.
The enclave-hardcore system 2-way communication is based on interrupts and the shared DRAM.
Malicious hardcore system software can raise the interrupt to the enclave to carry out a DoS attack.
Utilizing priority interrupts in sophisticated softcore processors, {\scshape BYOTee}\xspace can prevent these attacks from the hardcore system side.

\textbf{Reconfiguring FPGA.} The secure or trusted boot, i.e., static root of trust, process of {\scshape BYOTee}\xspace ensures the secure loading of the bitstream to configure the FPGA at boot time.
An alternative dynamic root of trust approach ensures a compromised hardcore system software cannot load an attacker-controlled bitstream to reconfigure the FPGA if the hardware supports run-time reconfiguration.

\subsection{Compromised {\scshape Firmware}\xspace and SSAs}
If the software layer of an enclave, i.e., {\scshape Firmware}\xspace and SSA, is compromised at run-time by untrusted input sent by the hardcore system software,
it can disclose information in the compromised enclave address space, including data on the BRAM and data from the connected peripherals.
But it cannot read data from the BRAM or peripherals of other enclaves, since they are in different address spaces and cannot be seen by the compromised software.
Therefore, the attack is confined within the compromised enclave.

\subsection{Hardware IPs and Malicious Peripherals}
Malicious hardware IPs cannot be loaded since a bitstream is signed by a trusted developer and verified before loading.
Even if peripherals are malicious and send out rogue DMA requests to access sensitive memory regions,
they are confined in the enclave they are assigned to.
Therefore, a malicious peripheral can only cause limited damages.

\subsection{Cold-boot Attack}
\label{subsec:analysisColdBoot}
Cold-boot attacks rely on the observation that the contents in memory are not immediately erased after power is lost.
While cold-boot attacks on DRAM even at
room temperature are proven very effective~\cite{halderman2009lest}, attacks on SRAM without external power sources are less feasible~\cite{rahmati2012tardis}.
Most data {\scshape BYOTee}\xspace stores on the DRAM is either encrypted or does not need to be protected.
For instance, even if the SEB is located on the DRAM and subject to cold-boot attacks,
the SSA*, which includes developer keys, is encrypted.
Obviously, \emph{Chal}, \emph{PreExecAtt}, \emph{PostExecAtt} do not need to be protected.
It is, however, possible to dump the input and output fields of the SEB using cold-boot attacks on DRAM.
Other sensitive data, such as developer keys, plaintext SSA, program states, are placed on an enclave's BRAM.
Cold-boot attacks on BRAM are difficult because:
i) the BRAM cells are hardware initialized during FPGA configuration in many SoC FPGA systems~\cite{bramInit};
ii) even without initialization, the contents in BRAM decays faster~\cite{rahmati2012tardis};
iii) BRAM is embedded on-chip and cannot be physically taken out, so attackers have to bypass software protections to dump its content.

\subsection{Side-channel Attacks}
\textbf{Cache side-channel.} Because the CPU is time-shared between the REE and TEE in SGX and TrustZone, cache side-channel attacks are effective~\cite{brasser2017software, zhang2016return, cho2018prime, zhang2016truspy}.
In {\scshape BYOTee}\xspace, the REE on the hardcore system side and enclaves do not time-share any CPU resources; hence, there is no cache side-channel between the REE and TEEs.

\textbf{Power side-channel.}
In FPGA-based remote power side-channel attacks, the attacker builds an on-chip ring oscillators-based power monitor to conduct power analysis attacks on other modules on the same FPGA or a CPU on the same SoC~\cite{zhao2018fpga}.
{\scshape BYOTee}\xspace cannot directly mitigate these attacks but can them by only loading authenticated and trusted enclave bitstreams that do not have a power monitor.

\textbf{Other side-channels.}
When multiple enclaves reside on the same SoC FPGA, they share FPGA hardware resources.
Therefore, it is possible to conduct other sharing-based side-channel attacks, such as FPGA long wire-based attacks~\cite{giechaskiel2018leaky, ramesh2018fpga}.
Similar to power side-channel attacks, {\scshape BYOTee}\xspace cannot prevent these attacks directly.

\section{Experiment Device}

Figure~\ref{fig:z7} shows the top and bottom view of the Cora Z7-07S development board with a single-core 667MHz Arm Cortex-A9 processor, a Xilinx Zynq-7000 FPGA, and the connected Pmod I2S2 audio device.

\section{Automatically Generated Hardware Design of the Example SSAs}

Figure~\ref{fig:corr-res}(a) shows the hardware for SSA-1 with a MicroBlaze Debugging Module (MDM)~\cite{mdmTRM} to help the developers debug SSAs.
A master interface of MDM is also connected to the ps7\_axi\_periph to enable debugging from the hardcore system side.
Figure~\ref{fig:corr-res}(b) shows the same hardware configuration without the debugging module.
As shown in Figure~\ref{fig:corr-res}(c), the hardware design of SSA-2 includes two peripherals that are only connected to the FPGA.
A pulse width modulation IP~\cite{pwm} connects the RGB\_LED ports, and an AXI GPIO IP~\cite{axiGpio} connects the button ports to the mb\_axi\_interconnect\_0 interconnect.
As shown in Figure~\ref{fig:corr-res}(d),
the hardware for SSA-3 includes several more IPs for different functionality.
An I2S transmitter RTL (SPI) is added to interact with the Pmod I2S2 module Codec module.
Additionally, an AXI direct memory access IP~\cite{axiDMA} with BRAM and an AXI stream data FIFO IP~\cite{axififo} to offload audio data streaming computation from Microblaze are inserted by the {\scshape HardwareBuilder}\xspace.
An ADC module for reading voltages, which is used for performance monitoring~\cite{xadc}, is also connected.
Figure~\ref{fig:corr-res}(e) represents the hardware of SSA-4 with two enclaves.
Both the enclaves have their own BRAM, 128KB, and 32KB respectively.
Enclave-1 have a shared DRAM with Zynq processor as SEB for communication, while Enclave-2 does not have any SEB with Zynq.
Instead of inter enclave communication, one 8KB BRAM is shared between Enclave-1 and Enclave-2 without the Zynq processing system access.

\begin{comment}
\section*{To-do list}
1. Need to verify when a peripheral is connected to PL, is it true that the PS can't access it anymore. We need to confirm this. I am not confident about this. If not, we need to design a mechanism to stop the PS from accessing the peripheral.

2. Implementation: add authentication to SSA.

3. Implementation: develop a simple verifier that runs on a server (PC).

4. Finish app-2 and app-4.

5. Performance measurement. Need add timer to count CPU cycles.

5. Related work needs to be reorganized and rewritten.

6. Think about how to automated the hardwarebuilder.
7. Evaluation plan: Give me a plan by Sep 28th. What tables will be there. What figures will be there. How many Figures will be there.

7. Acknowledgment the discussion with Ni Kai, Zhenhua Wu, and RIT startup fund, Air force summer visit, NSF-CRII in the final version.

8. A new addition could be SSA sealing. Santury also has this feature.

9. Also add attestation for hardware configurations.

10. ShEF~\cite{zhao2021shef}

\end{comment}

\begin{figure*}[!t]
    \centering
    \subfloat[Hardware for SSA-1 with a Debugging Module]{\includegraphics[width = 0.49\linewidth]{images/dsgn_aes_with_mdm.pdf}} \hfill
    \subfloat[Hardware for SSA-1 without a Debugging Module]{\includegraphics[width = 0.49\linewidth]{images/dsgn_aes_without_mdm.pdf}}\\
    \subfloat[Hardware for SSA-2 without a Debugging Module]{\includegraphics[width = 0.49\linewidth]{images/dsgn_led_without_mdm.pdf}} \hfill
    \subfloat[Hardware for SSA-3 without a Debugging Module]{\includegraphics[width = 0.49\linewidth]{images/dsgn_secure_drm_without_mdm.pdf}}
    \subfloat[Hardware for SSA-4 without a Debugging Module]{\includegraphics[width = 0.49\linewidth]{images/dsgn_app4_without_mdm.pdf}}
    \caption{Block diagrams of the hardware designs of the four enclaves for the four example SSAs. The ZYNQ processing system represents the hardcore Cortex-A processor. The Enclave-1, Enclave-2, and Enclave-3 have one MicroBlaze softcore processor, whereas Enclave-4 has two MicroBlaze softcore processors.
The MicroBlaze interrupt interface is connected to an AXI interrupt controller and configured with an AXI GPIO.
All output GPIO to the softcore is connected to the hardcore system for triggering interrupts.
Two AXI Interconnects, mb\_axi\_mem\_interconnect\_0 for Microblaze and ps7\_axi\_periph for Cortex-A processor are used to connect external IPs.
The ZYNQ, Microblaze, and other IPs get primary clock input from the clk\_in1 port.
The reset port is connected to the reset interfaces of the IPs.}
    \label{fig:corr-res}
\end{figure*}

\begin{comment}
\begin{table*}[t]

	\centering
	\footnotesize
	\begin{tabular}{l|c|c|c|c|c|c|c|c}
		\hline
		 & Pipeline &  Max clock &  ICache  &  DCache  & FPU& Memory  & Debug   & Open-sourced? \\
		 & depth &  frequency &    &     &  Protection &  Module &  Resource &  \\ \hline
		MicroBlaze$^{1}$~\cite{mbsoft} & 3, 5& 200 MHz & * &*& \cmark&\cmark & \cmark&\xmark\\ \hline

		NIOS$^{1}$~\cite{nios}   & 5, 6& 210 MHz& *&*&  \cmark &\cmark & \cmark&\xmark\\ \hline

		VexRisc$^{1}$~\cite{VexRiscv}   & 2 - 5+& 243 MHz& *&*& \cmark &\cmark & \cmark&\cmark\\ \hline

		NEO430$^{1}$~\cite{Neo430}  & 5& 122 MHz&*&*&\xmark&\xmark&\xmark &\cmark\\ \hline
		Microwatt$^{2}$~\cite{microwatt}  & 5& 100 MHz &16KB&32KB&\cmark& \cmark&\cmark&\cmark\\ \hline
		A2I$^{2}$~\cite{a2i}    &7 &2.3 GHz&32KB&32KB&\cmark&\cmark&\cmark&\cmark\\ \hline
		A2O$^{2}$~\cite{a2o}   & 4&2.3 GHz&32KB&32KB&\cmark&\cmark &\cmark&\cmark\\ \hline
		OpenSPARC T1$^{2}$~\cite{OpenSPARC}   &6 &1.2 GHz&*&*&\cmark&\cmark& \cmark&\cmark\\ \hline
		Libre-SOC$^{2}$~\cite{libreSoc}    & 4&300 MHz&*&*&\cmark& \cmark&\cmark&\cmark\\ \hline

	\end{tabular}

	\caption{Example softcore CPUs. * : configurable depend on available FPGA hardware; \cmark: Available; \xmark: Not available; 1 : 32 bit processor; 2: 64 bit processor}
	\label{t:softcores}
\end{table*}

\section{Example Application}

As an example application of multiple protected SSA sessions, we develop a factorial calculator computational SSA for a CTEE with MicroBlaze softcore.
Interrupt handlers in MicroBlaze can access execution sensitive information such as interrupt return address, stack pointer through registers which are stored in \verb|r14| and \verb|r1| respectively.
While the SSA is running, UA triggers an interrupt handled by {\scshape Firmware}\xspace which generates a secure state blob using AES encryption on the necessary data and adds a SHA512-HMAC authentication tag.
The secure state blob can be re-invoked later from the previous execution point, and in the meantime, other SSAs can execute on the same CTEE.

\section{Deployment Model}
{\scshape BYOTee}\xspace, supports two deployment models: local deployment and cloud infrastructure deployment model.
There are three stakeholders: cloud provider, {\scshape Firmware}\xspace developer, and SSA developer.
Apart from software development, the stakeholders are responsible for key management in the system.
The device keys, e.g., AES ($k_{d}$), RSA ($sk_{d}, pk_{d}$) are stored in nonvolatile eFUSE memory or BBRAM along with configuration bits.
For local deployment, the {\scshape Firmware}\xspace developers program the device with the keys through the JTAG interface, which is used by the FPGA hardware AES module for decryption at boot time\cite{xilinxSecure}.
For cloud deployment, the provides program their infrastructure with devices keys, and developer/tenant sends their hardware configuration JSON file, {\scshape Firmware}\xspace, and UA to the trusted cloud provided.
In this case, multiple tenants can be present in the same infrastructures, the provider merges all tenants configuration JSON to generate the bitstream, combine with their {\scshape Firmware}\xspace, and lastly, generate the protected FPGA image.
The UAs for each tenant run individually in the untrusted  environment.
Using BBRAM allows the feature to update the device's keys at some later stage if the previous gets compromised, whereas the eFUSE memory is one-time programmable.

On the other hand, the developer's keys, e.g., RSA ($sk_{u}, pk_{u}$) are used for SSA decryption and verification.
For both deployment models, the developer embeds the keys on the {\scshape Firmware}\xspace.
The SSA developers need to register their keys on the {\scshape Firmware}\xspace with corresponding SSA keys.
They use RSA ($ pk_{u}$) to compose a special message in a predefined format including a special header, symmetric SSA key, and lastly the digest of the SSA to be executed.
The {\scshape Firmware}\xspace, after receiving the message whitelists the SSA key and records the SSA digest.
Upon receiving SSA execution instruction, the {\scshape Firmware}\xspace decrypts the SSA and verifies the SSA digest.
If the developer's key need to be updated, the {\scshape Firmware}\xspace needs to be replaced, and the newly generated protected FPGA image should be loaded on the device.

\section {TODO:}
\begin{enumerate}
	\item Threat Model: Needs to clear. Not contradictory with security analysis or evaluation.
	\item Security analysis and security evaluation can be merged.
	\item Motivation part needs to rewritten: Deployment use of BYOT, benefit of customized hardware TCB, Cloud or only local.
	\item Evaluation comparison with OP-TEE is not fair.
	\item Trusted I/O path: who establishes the trusted paths?
	\item Keys: how those keys are secured? how often they are updated.
	\item How can BYOT provide partial reconfiguration security?
	\item Design Change: After load there can be a attestation of the system software as a proof of secure load.
	\item Proof of physical isolation.
\end{enumerate}
\begin{figure}[t]
	\begin{centering}
		\centering
		\includegraphics[width=0.47\textwidth]{images/BYOT-state.pdf}

		\caption{Multiple SSA sessions}
		\label{fig:SSA state}
	\end{centering}
\end{figure}
\end{comment}

\begin{comment}
We consider both local and cloud deployment models for {\scshape BYOTee}\xspace.
In the local deployment model, the manufacturers or service provides embed the device key onto the SoC FPGA.
The device key are burned into the eFUSE or BBRAM, which are used for the secure loading of protected bitstream images.
The application providers, on the other hand, embeds the developer keys on {\scshape Firmware}\xspace which are used for secure loading and attestation mechanism of SSAs in {\scshape BYOTee}\xspace.
To facilitate different SSAs from different application providers {\scshape Firmware}\xspace is populated with a white list of developer keys.

For the cloud deployment model, {\scshape BYOTee}\xspace enables multiple tenants to run on top of the same SoC FPGA in different CTEEs.

\section{Evaluation of multiple SSA sessions}

\begin{table}[t]
	\centering
	\footnotesize
	\caption{Protected SSA Sessions Performance (Time in milliseconds; size in bytes)}
	\label{t:FwLocRes}
	\begin{tabular}{c|c|c|c}
		\hline
		SSA & Binary & Export & Import \\
		\hline
		SSA-5 & 2452 & 703.4 & 687.3\\
		\hline
	\end{tabular}
\end{table}

\end{comment}

\section{{\scshape BYOTee}\xspace Applications}

\subsection{Example Application: Secure Audio DRM}

Designing a secure audio DRM was a challenge in 2020 MITRE Collegiate Embedded Capture-The-Flag competition (eCTF).
\section{{\scshape BYOTee}\xspace Applications}
\label{s:apps}
In this section, we use multiple security-sensitive application examples to demonstrate how {\scshape BYOTee}\xspace can secure real-world applications in several classes, namely computational application, peripheral interacting application, and peripheral and PS interacting application.
We present the design of an example application in each class, which we use to evaluate {\scshape BYOTee}\xspace.

\subsection{Computational Applications}

Computational applications take input from the PS or other CTEEs, perform the intended computational operations, and send the outputs back.
They represent computational tasks, such as encryption, decryption, machine learning-based classification, etc., that do not need peripherals.
\emph{Example application:}
We implement an AES accelerator SSA (SSA-1).
To use SSA-1, a UA places the plaintext or ciphertext in the SEB and notifies the SSA.
When the encryption or decryption is finished, SSA-1 places the outputs on the SEB.

\subsection{Peripheral Interacting Applications}

Simple peripheral interacting applications control one or more peripherals, but they do not communicate with other SSAs or applications on the PS.
An example usage is cyber-physical applications that read from sensors, make local decisions, and control an actuator.
\emph{Example application:}
We develop an LED toggler SSA (SSA-2) that uses a button and an LED.
When the button is pressed, SSA-2 toggles the color of the LED.
Both the button and LED are directly connected to the CTEE of SSA-2 and cannot be accessed by the PS or other CTEEs.

\begin{comment}
\begin{lstlisting}[label={ssa2},caption={SSA-2 Code Snippet}, language=c] .
void GpioHandler(void *CallbackRef)
{
	XGpio *GpioPtr = (XGpio *)CallbackRef;
	IntrFlag = 1;
	XGpio_InterruptClear(GpioPtr, GlobalIntrMask);
}

int main()
{
	u32 *led = (u32 *)XPAR_RGB_PWM_0_PWM_AXI_BASEADDR;
	xil_printf("Wait for Button GPIO interrupt: \r\n");

	while(true)
	{
		if(IntrFlag)
		{
			toggleLED(led);
		}
	}

	return 0;
}
\end{lstlisting}
Listing~\ref{ssa2} represents the code snippet of the LED Switch SSA, where it configures GpioHandler function as the button interrupt handler.
\end{comment}

\subsection{Peripheral and PS Interacting Applications}
\label{sec:DRM}

[Need to clearly explain why this app needs BYOT. Why TrustZone can;t provide the protection (code boot attack).]

Peripheral and PS interacting applications do not only control some peripherals but also continuously interacts with the UA.

\emph{Example application:}
We develop a music player SSA with digital rights management (SSA-3).
The untrusted module running on the PS takes user's commands, such as login (using username and Pin), logout, query (i.e., display a song's owner information), share (i.e., share a song to another user), play, pause, resume, and stop.
Depending on the user's request, it communicates with the SSA and forwards necessary data through SEB.

The trusted module is the SSA running on the CTEE, which implements the DRM properties of the music player.
Allow only authorized users to play songs, regional provisioning of songs, prevent unprovisioned media files from being played are some of the DRM properties that the SSA needs to ensure.
To validate those properties at runtime, the required DRM meta-information is embedded in the SSA or put onto the media files themselves.
For example, user passwords, encryption keys are embedded into the SSA, however, region lock, owner and shared user information, etc. are put onto the media files.
Embedded information is only available to the SSA as it is in BRAM of the CTEE, and the media files are authenticated and can only be decrypted by the SSA on BRAM.
So the untrusted module at runtime does not have access to sensitive data to temper with.

\subsection{Intercommunicating SSA}
Intercommunicating SSAs represents a set of applications where multiple SSA runs on different physically separated CTEEs and needs secure intercommunication between them to carry out the interdependent sensitive task.

\emph{Example application:}
As an example, we develop two computational applications deployed in two physically separated CTEEs.
When the untrusted module wants to authenticate and decrypt some data, it sends the signature and ciphertext through the SEB to the CTEE-1's SSA.
The SSA of CTEE-1 performs the AES decryption task and internally utilizes the SSA on CTEE-2 to generate the signature (SHA512-HMAC).
The communication between the SSAs are through a shared BRAM without untrusted PS software access.
This example also illustrates the dynamic and static loading of SSAs onto the CTEEs.
While the {\scshape Firmware}\xspace of the CTEE-1 dynamically loads the AES accelerator SSA into its BRAM, the SSA on CTEE-2 is loaded statically onto the BRAM with the bitstream.

\section{{\scshape BYOTee}\xspace Architecture}
\label{s:arch}
In this section, we first present the security and functional design goals of {\scshape BYOTee}\xspace followed by an overview of the {\scshape BYOTee}\xspace architecture and workflow.
We then illustrate hardware TCB customization, bootstrapping trust in enclaves, secure execution of SSA, and remote attestation mechanisms.

\subsection{Design Goals}
\label{subsec:designgoal}

{\scshape BYOTee}\xspace provides physically isolated execution environments on-demand, which even hardware debuggers and DMA-enabled devices cannot access.
With the help of {\scshape BYOTee}\xspace, users can use the exact and even formally verified hardware and software needed for their applications.
{\scshape BYOTee}\xspace has the following security and functional design goals:

\goal{General-purpose execution environments.} {\scshape BYOTee}\xspace should provide general-purpose execution environments similar to SGX and TrustZone, not application-specific accelerators.
SSAs can be implemented in any language as long as they can be linked against the {\scshape Firmware}\xspace.

\goal{Multiple isolated execution environments.} {\scshape BYOTee}\xspace should provide multiple execution environments at the same time and guarantee the secrecy and integrity of the SSA running inside each.

\goal{Physical execution isolation.} {\scshape BYOTee}\xspace should provide dedicated CPUs for execution environments, and all hardware resources for enclaves are physically isolated from the REE and each other.
With physical isolation, {\scshape BYOTee}\xspace mitigates the side-channel attacks that are prevalent in CPU-sharing TEEs, such as SGX and TrustZone.

\goal{Customizable hardware TCB.} The hardware TCB of each enclave should be customizable, allowing for a minimal TCB that only includes the hardware, e.g., peripherals, necessary for the SSA running inside and excludes other hardware on the system.
Note that formally verifying the customized hardware~\cite{sieh1997verify, breuer1997refinement} is beyond the scope of {\scshape BYOTee}\xspace.

\goal{Isolated path between SSA and peripherals.} An enclave should isolate the communication path between the SSA inside and peripherals from the hardcore system and other enclaves, preventing software-based eavesdropping and tampering.

\goal{Enclave-to-Hardcore System and Inter-enclave communication.} The SSA in an enclave should be able to communicate with software modules on the hardcore system and other enclaves.
The inter-enclave communication should be isolated from the hardcore system and other non-participating enclaves.

\goal{Minimal software TCB.} The firmware serving an SSA should only include the housekeeping libraries and drivers that are necessary for the SSA execution and exclude other software.

\goal{Remote attestation mechanisms.} {\scshape BYOTee}\xspace should provide mechanisms to support sophisticated protocols to attest the integrity of an enclave's hardware and software stacks, including boot loaders, bitstream, {\scshape Firmware}\xspace, SSAs and their inputs and outputs to remote verifiers.

\goal{Easy to use.} {\scshape BYOTee}\xspace should be easy to use, especially for the software developers who do not have hardware programming experience. As a rule of thumb, developing a {\scshape BYOTee}\xspace SSA should not take significantly more time and effort than developing a Linux application with the same functionality.

\subsection{{\scshape BYOTee}\xspace Overview}
\label{subsec:overview}

Figure~\ref{fig:BYOTee} presents an overview of the architecture and workflow of the {\scshape BYOTee}\xspace framework.
The {\scshape BYOTee}\xspace tools and codebase mainly include the {\scshape HardwareBuilder}\xspace, {\scshape Firmware}\xspace, and {\scshape SSAPacker}\xspace.
During the development stage, the {\scshape HardwareBuilder}\xspace generates synthesizer commands, e.g., Tcl, based on the SSA's needs specified in the developer's hardware description JSON input.
Then, the SoC FPGA vendor-provided synthesizer, e.g., Xilinx Vivado~\cite{vivvado}, Intel Quartus Prime~\cite{quartus}, generates the bitstream file using the synthesizer commands.
The developer can customize the {\scshape Firmware}\xspace by only including the needed source code and writing the SSA source codes, which are compiled with the corresponding compiler for the softcore CPU, e.g., \verb|mb-gcc| for MicroBlaze.
The bitstream and {\scshape Firmware}\xspace binary are encrypted, signed, and packed by the vendor-provided merger, e.g., \verb|UpdateMEM| from Xilinx, into a protected FPGA image.
The SSA binary is encrypted, signed, and packed by the {\scshape SSAPacker}\xspace into a protected SSA.
At boot-time, the bitstream is loaded onto the FPGA.
As a result, multiple enclaves are created, and the corresponding {\scshape Firmware}\xspace starts running.
Then, an untrusted application can trigger the loading of a protected SSA into an enclave.

{\scshape BYOTee}\xspace meets \emph{G1}, \emph{G2}, \emph{G3}, \emph{G4}, and \emph{G5} by configuring the FPGA to build enclaves.
{\scshape BYOTee}\xspace constructs enclaves with softcore CPUs, which provide general-purpose computing environments (\emph{G1}).
The {\scshape Firmware}\xspace includes the standard C libraries and connected peripheral HAL libraries for SSAs to invoke at run-time.
Each enclave has its own set of hardware (\emph{G2}), including softcore CPU, e.g., MicroBlaze, UltraSPARC, etc., Block RAM, and peripherals.
With FPGA routing, these hardware resources within an enclave are connected together but isolated from the hardcore system and other enclaves (\emph{G3}).
The softcore CPU in an enclave is not time-shared with the hardcore system and other enclaves, mitigating the cache side-channel attacks (\emph{G3}).
No additional hardware modules, such as debuggers, can be connected to an enclave unless it is explicitly specified by the developer (\emph{G4}).
The connections among these resources are also physically isolated from the hardcore system and other enclaves, preventing eavesdropping and tampering (\emph{G5}).

Enclaves use interrupts on softcore CPUs and shared physical memory regions on the DRAM to communicate with the hardcore system, whereas enclaves use interrupts and shared regions on the BRAM to communicate with each other.
Since a shared BRAM region is only mapped in the address spaces of the participating enclaves, it is isolated from the hardcore system and other enclaves (\emph{G6}).
{\scshape BYOTee}\xspace includes {\scshape Firmware}\xspace, which can be customized and only consists of libraries, a HAL for the needed peripherals, and a loader for the SSA (\emph{G7}).
The {\scshape Firmware}\xspace also provides softcore CPU interrupt-based remote attestation mechanisms for proving the integrity of enclaves (\emph{G8}).
{\scshape BYOTee}\xspace provides an easy-to-use toolchain for developers to focus on SSA development, increasing the usability of {\scshape BYOTee}\xspace and decreasing the chances of developer-induced misconfigurations (\emph{G9}).

\subsection{Customizing Hardware TCB for Enclaves}
\label{subsec:customizeHWTCB}

To customize the hardware TCB, the developer designs one or more enclaves using a hardware description language, e.g., Verilog or VHDL.
The output is a bitstream file that configures the FPGA.
To facilitate this step, {\scshape BYOTee}\xspace has a component, named {\scshape HardwareBuilder}\xspace, which takes developer-specified hardware description in JSON format as input (See an example in \S\ref{subsec:dev}), allocates hardware resources, and outputs a script, e.g., in Tcl format, that can be processed by a synthesis tool to generate the bitstream.
Each enclave's hardware description includes but is not limited to:
i) a softcore CPU and its configurations, e.g., clock frequency, cache size, etc;
ii) a corresponding debug IP to enable software debugging on the softcore CPU;
iii) its main BRAM memory address and size;
iv) the address and size of the shared DRAM with the hardcore system;
v) the address and size of the shared BRAM with other enclaves;
and vi) connected peripherals.
The {\scshape HardwareBuilder}\xspace assigns a contiguous address space of the BRAM to each enclave and connects the hardware components automatically.

\subsection{Bootstrapping Trust in Enclaves}
\label{subsec:bootstrapTrust}

Because the details of trust bootstrap are related to the specific deployment model and policies, {\scshape BYOTee}\xspace provides the mechanisms to build trust bootstrap paths.
{\scshape BYOTee}\xspace supports FPGAs that can be configured at boot time and/or reconfigured at run-time by providing trust bootstrap in enclaves both statically and dynamically.
The static trust bootstrap builds on secure or trusted boot, whereas the dynamic trust bootstrap relies on some dynamic root-of-trust modules at run-time.

\begin{figure}[t]
	\begin{centering}
		\centering
		\includegraphics[width=.47\textwidth]{images/bootstraptrust.pdf}

		\caption{Example static and dynamic trust bootstrap paths on Zynq SoC. \Circled{\textbf{1}} static trust bootstrap. \Circled{\textbf{2}} dynamic trust bootstrap without TrustZone. \Circled{\textbf{3}} dynamic trust bootstrap with TrustZone.}
		\label{fig:bootstrap}
	\end{centering}

\end{figure}

Without loss of generality, Figure~\ref{fig:bootstrap} shows three example paths for static and dynamic trust bootstrap on the Xilinx Zynq SoC with Arm hardcore CPUs.
In the static bootstrap case \Circled{\textbf{1}}, {\scshape BYOTee}\xspace relies on secure or trusted boot to launch enclaves.
BootROM, which is the root of trust for measurement, first decrypts, measures, i.e., $m_1=H(FSBL)$, and executes the FSBL using the device keys.
The FSBL initializes the hardcore system by decrypting, measuring, and executing the SSBL using the device keys.
A measurement $m_2=H(m_1||SSBL)$ is generated in this step.
The FSBL also initializes the FPGA by decrypting, measuring, and configuring the FPGA using the device keys, after which bitstream is programmed on the FPGA, and the {\scshape Firmware}\xspace starts execution.
A measurement $m_3=H(m_2||BS||FW)$ is generated by the FSBL and placed on the shared DRAM region with the enclave for the future use of attestation report generation, which we discuss in~\S\ref{subsec:attest}.
The {\scshape Firmware}\xspace uses the developer keys, e.g., $k_{u}$ and/or $pk_{u}$, to decrypt and measure the SSA.
Allowed developer keys are embedded in the {\scshape Firmware}\xspace at the development stage.
Because the {\scshape Firmware}\xspace is encrypted at rest and only decrypted on the BRAM, the developer keys are secure.

In the dynamic case, software running on the hardcore system of a Xilinx Zynq device can use the device configuration (DevC) or Processor Configuration Access Port (PCAP) interfaces to reconfigure the FPGA.
Therefore, the software module that reconfigures the FPGA needs to access the device key and must be trusted.
In the case that TrustZone is not available or used \Circled{\textbf{2}}, the trusted software module can be part of the REE kernel, which uses the device keys to decrypt and measure the bitstream and {\scshape Firmware}\xspace.
If TrustZone is available \Circled{\textbf{3}}, the trust software module can be a kernel module inside the TrustZone secure world operating system.

\subsection{Executing SSAs in Enclaves}
\label{subsec:execintee}

To create an SSA, the developer links the source code against the {\scshape Firmware}\xspace.
After the launching of an enclave, the {\scshape Firmware}\xspace initializes the softcore CPU and other components, then it waits for requests from the hardcore system.
Both the {\scshape Firmware}\xspace and SSA use a shared DRAM region in the SSA Execution Block (SEB) format as shown in Figure~\ref{fig:sebctee}, to have two-way data transmissions with UA on the hardcore system.
To initiate the transmission from the hardcore system to an enclave, UA on the hardcore system raises interrupts on the enclave's softcore CPU, which are handled by the {\scshape Firmware}\xspace.
{\scshape BYOTee}\xspace defines three service primitives through the \textsf{LdExec*} interrupts: i) load and execute an SSA (\textsf{LdExec});
ii) load and execute an SSA with pre-execution attestation (\textsf{LdExecPreAtt});
iii) load and execute an SSA with post-execution attestation (\textsf{LdExecPostAtt}).
The softcore CPU interrupts can be implemented as GPIO interrupts in the enclave and memory-mapped to a DRAM address for the UA to access.
In this subsection, we focus on \textsf{LdExec}, and the other two primitives are discussed in \S\ref{subsec:attest}.

\begin{figure}[t]
	\begin{centering}
		\centering
		\includegraphics[width=0.47\textwidth]{images/BYOT-FM-Layout.pdf}
		\caption{SSA Execution Block (SEB) layout, simplified enclave address space layout, and steps in executing an SSA.}
		\label{fig:sebctee}
	\end{centering}
\end{figure}

To execute an SSA on an enclave, the untrusted application first fills data into the SEB and raises a \textsf{LdExec*} interrupt.
As shown in Figure~\ref{fig:sebctee}, a SEB has regions for the encrypted and signed SSA (SSA*), input data for the SSA, output data from the SSA, a challenge $Chal$ from a remote verifier, a pre-execution ($PreExecAtt$), a post-execution attestation measurement ($PostExecAtt$) and other data.
When the {\scshape Firmware}\xspace receives a \textsf{LdExec*} interrupt, it copies SSA* and optionally $Chal$ and input data in the SEB from DRAM to its own BRAM \Circled{\textbf{1}}.
The {\scshape Firmware}\xspace can also disable \textsf{LdExec*} interrupts after the data is copied.
Note that it is critical for the {\scshape Firmware}\xspace to perform measurement on the BRAM since the DRAM can be changed asynchronously by the hardcore system.
The {\scshape Firmware}\xspace then decrypts and verifies the encrypted SSA* using the corresponding developer's keys \Circled{\textbf{2}}.
Upon the successful verification of the SSA's integrity, the {\scshape Firmware}\xspace loads sections of the decrypted SSA to the right locations and gives the control to the SSA \Circled{\textbf{5}}.
If there is an output, the SSA writes it in the output region on the BRAM, and yields the control of the softcore CPU back to the {\scshape Firmware}\xspace.
The {\scshape Firmware}\xspace copies the output from the BRAM to the DRAM \Circled{\textbf{6}}.
Finally, the {\scshape Firmware}\xspace cleans up all the input, output, and SSA-related regions on the BRAM and awaits new requests from the hardcore system \Circled{\textbf{7}}.
While SSAs can execute concurrently on their own enclaves respectively,
the {\scshape Firmware}\xspace also supports executing multiple SSAs sequentially or the same SSA multiple times on the same enclave without reconfiguring the FPGA but just re-initializing the enclave, e.g., flush the cache,
clean up the BRAM \Circled{\textbf{7}}, etc.

\begin{comment}
\begin{figure}[t]
	\begin{centering}
		\centering
		\includegraphics[width=0.47\textwidth]{images/SSC_execution_time_line.pdf}

		\caption{Executing two SSAs concurrently on two enclaves}
		\label{fig:ssc_time_line}
	\end{centering}
\end{figure}

\begin{figure}[t]
	\begin{centering}
		\centering
		\includegraphics[width=0.47\textwidth]{images/Temoral_SSC_execution_time_line.pdf}

		\caption{Executing two SSAs sequentially on one enclave}
		\label{fig:temporal_ssc_time_line}
	\end{centering}
\end{figure}
\end{comment}

\subsection{Pre-execution and Post-execution Attestation Mechanisms}
\label{subsec:attest}

{\scshape BYOTee}\xspace provides two attestation mechanisms, namely pre-execution and post-execution attestations, as shown in Figure~\ref{fig:sebctee}.
The former extends the traditional remote attestation of code and input integrity with bitstream, whereas the latter extends the output data integrity attestation~\cite{abera2019diat}.
Note that {\scshape BYOTee}\xspace only provides the mechanism for attestation, which can support sophisticated attestation protocols.
With the help of trust bootstrapping discussed in \S\ref{subsec:bootstrapTrust}, the measurement mechanism not only captures the identity of the loaded SSA but also the bitstream, {\scshape Firmware}\xspace and programs, e.g., FSBL and SSBL, before it.

In pre-execution attestation, a verifier sends a cryptographic nonce as $Chal$, which is copied to the BRAM by the {\scshape Firmware}\xspace \Circled{\textbf{1}}.
After loading the SSA sections to the right addresses, the {\scshape Firmware}\xspace computes a measurement $PreExecAtt$ on the vector table, {\scshape Firmware}\xspace code and data, $m_3$, $Chal$, input data and SSA sections \Circled{\textbf{3}}, and copies the measurement to the DRAM.
Depending on scenarios and attestation protocol details, the {\scshape Firmware}\xspace can use a device key, developer key, or other shared keys to compute the measurement.
In post-execution attestation, the $Chal$ from $\mathtt{Vrf}$ is also copied to the BRAM by the {\scshape Firmware}\xspace \Circled{\textbf{1}}.
After the SSA finishes execution \Circled{\textbf{4}}, the {\scshape Firmware}\xspace computes a measurement $PostExecAtt$ on the vector table, {\scshape Firmware}\xspace code and read-only data, $m_3$, $Chal$, input data, output data generated by the SSA, SSA's read-only sections and $PreExecAtt$, and copies the measurement to the DRAM \Circled{\textbf{5}}.

\subsection{Multiple Inputs to SSA}
\label{subsec:coninput}

In the case that the UA on the hardcore system needs to continuously send input data to the SSA, e.g., not all input data is available at the beginning, the size of input in SEB is not big enough, etc., the UA writes the newly available input data in the input region inside the SEB, and it can use two mechanisms to notify the {\scshape Firmware}\xspace and SSA that new data is available.
The first mechanism works for softcore CPUs that support priority interrupts.
On such systems, {\scshape BYOTee}\xspace defines a \textsf{NewData} interrupt, which UA can raise.
The \textsf{NewData} interrupt has a low priority so that it cannot interrupt the execution of the SSA.
Only after the SSA finishes execution and yields the control back to the {\scshape Firmware}\xspace, the {\scshape Firmware}\xspace can copy and measure the input data from the DRAM to the BRAM, and gives the control to the SSA again.
On softcore CPUs without priority interrupts, the {\scshape Firmware}\xspace uses global variables to indicate whether new data is available in the input region to synchronize with the SSAs on the enclaves.

\subsection{Optional Multiple Protected SSA Sessions}
As an option, an enclave can also interleave the execution of multiple eligible SSAs with proper hardware re-initialization, e.g., flush cache, reset all memory, etc.
To this end, {\scshape BYOTee}\xspace defines two interrupt-based service primitives: i) suspend and export the SSA state (\textsf{SusExp});
ii) restore and execute a saved and encrypted SSA state (\textsf{ReExec}).
When a \textsf{SusExp} interrupt is raised, the {\scshape Firmware}\xspace copies the SSA context, e.g., general and system registers, onto the BRAM.
Then, the {\scshape Firmware}\xspace uses the developer key to encrypt and sign the saved SSA context and all of the SSA's writable memory regions, e.g., stack, .data, .bss, etc.
The encrypted blob is placed in the SEB output region for the UA to retrieve,
after which the {\scshape Firmware}\xspace cleans up BRAM and awaits new requests.
When a \textsf{ReExec} interrupt is raised, the {\scshape Firmware}\xspace retrieves an encrypted blob from the SEB input region.
Upon a successful signature verification, the {\scshape Firmware}\xspace loads the decrypted memory contents to the right locations, restores the registers, and resumes the SSA execution.

\section{Background: SoC FPGA}
\label{s:back}

\begin{comment}
\begin{table*}[!t]
\center
\scriptsize
\begin{tabular}{lp{3.5cm}p{11cm}}
\hline
Acronym           & Full name        & Description                                                                                \\ \hline\hline
PS & Processing System & PS encompasses `hard' processors, cache memory, memory interfaces, interconnect, clock generation circuitry and other components that can be found on `hard' CPUs.
PS components exist as dedicated silicon element on the device. \\ \hdashline
FPGA & Programmable Logic & FPGA is the FPGA part of an SoC. The FPGA can implement `soft' processors, and other high-speed logic, arithmetic and data flow subsystems. \\ \hdashline
BRAM & Block RAM & BRAMs are used for storing large amounts of data inside of the FPGA. \\ \hline
CTEE & Customized Trusted Execution Environment & A CTEE has its own TSC, BRAM, and peripherals. \\ \hdashline
FM & Firmware running on `soft' processor. & A software component provided by {\scshape BYOTee}\xspace that provides hardware abstraction layer. \\ \hdashline
SSA & Security-Sensitive Application & An SSA runs on isolated software cores. SSC could be PAL or TA. \\ \hdashline
UA & Untrusted Application & UA runs on hardware core, usually on top of an operating system. \\ \hline
$K_{D}, SK_{D}, PK_{D}$ & Device keys & Device keys, e.g., AES, RSA, are used to encrypt and sign system modules, such as FSBL, SSBL. They are unique for each device and are programmed in eFUSE/BBRAM using hardware interfaces. \\ \hdashline
$K_{U}, SK_{U}, PK_{U}$ & Developer keys & Developer keys are used to encrypt and sign SSCs. Valid developer keys are embedded into RT at development stage. \\ \hline \\
\end{tabular}
\caption{Acronyms and descriptions}
\label{tab:acronym}
\end{table*}
\end{comment}

In this section,
we discuss the hardware components of SoC FPGA, followed by its design, development, and run-time workflow.

\subsection{Hardware Components}

\begin{comment}
System-on-Chip implements the functionality of an entire system on a single silicon chip.
Compared with system-on-printed-circuit-board, SoC is lower cost, enables more secure data transfers,
and has higher speed and lower power consumption.
However, traditional application-specific integrated circuit SoCs lack flexibility, making them suitable only for products with a limited lifetime.
SoC FPGA is a type of flexible system-on-programmable-chip, where FPGAs can be reconfigured as desired.
The market of SoC FPGA is expected to grow significantly with the increase in the global adoption of artificial intelligence and internet-of-things solutions~\cite{FPGA2019}.
\end{comment}

A SoC FPGA comprises:
i) hardcore CPU system, which is formed around hard processors, such as the Cortex-A processor,
to run the traditional operating systems and applications;
ii) FPGA, which can implement arbitrary systems, including softcore CPUs, high-speed logic, arithmetic, and data flow subsystems.
In addition to the general fabric, the FPGA has Block RAMs (BRAM) to store data.
Note that BRAM is made of Static RAM (SRAM) on existing SoC FPGA platforms.
Compared to DRAM whose cells are made of capacitors and is vulnerable to cold-boot attacks due to the slow decay~\cite{halderman2009lest}, SRAM decays faster~\cite{rahmati2012tardis}.
FPGA is configured with a \emph{bitstream}, which is programmed in hardware design description languages, such as Verilog, VHDL, etc.;
iii) other integrated on-chip memory and high-speed communications interfaces.

\begin{figure}[t]
	\begin{centering}
		\centering
		\includegraphics[width=0.48\textwidth]{images/Zynq_PL_PS.pdf}
		\vspace{-0.2cm}
		\caption{An overview of SoC FPGA hardware}
		\label{fig:zynq_device}
	\end{centering}
			\vspace{-0.4cm}
\end{figure}

Figure~\ref{fig:zynq_device} shows the architecture of SoC FPGAs.
The main building blocks of the hardcore CPU system include the hardcore CPU, its Memory Management Unit (MMU), On-Chip Memory (OCM), caches, etc.
Additionally, the hardcore system connects external I/O interfaces, such as SPI, I2C, UART, etc.
Bridges and interconnects connect the hardcore system and FPGA interfaces.
The FPGA side is mainly composed of configurable logic blocks, logic cells, Adaptive Logic Modules (ALMs), Lookup Tables (LUTs), flip-flops, switch matrix, carry logic, BRAM, and Input/Output Blocks (IOB) for interfacing.
The FPGA also includes modules for Analogue to Digital Conversion (ADC) and a set of  Joint Test Action Group (JTAG) ports for configuration and debugging.
Usually several security-relevant modules, including non-volatile Battery-backed RAM (BBRAM), one-time programmable electronic fuse (eFUSE),
and cryptographic accelerators, are connected to both the hardcore system and FPGA.

\subsection{Development and Run-time Workflow}
A typical design and development flow of general-purpose computing platforms on SoC FPGA involves:
i) the development of the hardware system on the FPGA, including designing the peripheral blocks and creating the connections between these blocks and the hardcore CPU system.
In this step, a developer can use and customize open-sourced and proprietary hardware IPs;
ii) the development of the software system on the hardcore and softcore CPUs.
To be compatible with SoCs without FPGAs, when a SoC FPGA device is powered on, the hardcore system boots first before going on to configure the FPGA.

To enable secure and trusted boot, unique AES and RSA keys can be generated off the device and programmed to the persistent secure storage, e.g., eFUSE array, on the device.
The keys are used to encrypt and sign the bitstream and the firmware (e.g., in ELF format) that runs on the softcore CPU in the development stage.
The device first starts with the hard-wired boot ROM, which verifies, decrypts, and loads the First Stage Boot Loader (FSBL) from supported interfaces, such as SD card, JTAG, etc.
The FSBL, in turn verifies and sets up the FPGA with bitstream using the device configuration (DevC)~\cite{ZynqTrm} interface and firmware, after which the firmware on FPGA starts execution.
The FSBL also verifies the Second Stage Boot Loader (SSBL), such as U-Boot, and gives control of the hardcore system to it.
The SSBL verifies and boots the operating system on hardcore CPU.
In addition to boot-time configuration, some FPGAs can also be programmed at run-time.
For instance, software running on the hardcore of some Xilinx Zynq devices can use the DevC and Processor Configuration Access Port (PCAP) interfaces to reconfigure the whole or part of the FPGA.

\begin{comment}
In Intel FPGAs, on the other hand, the early boot is handled by Secure Device Manager (SDM) on a flash controller and this in turn boots the hardcore system and FPGA.
During the first stage, the SDM boots and takes control.
From the bitstream, it determines the configuration source of FPGA, handles device security, initializes the hardcore system, and loads the FSBL in the on-chip RAM.
The hard processor now executes the FSBL, loads SSBL on DRAM, loads the OS, and lastly the applications.
SDM provides two modes of booting: FPGA configuration first and hardcore system boot first (Early I/O).
FPGA is fully configured including core and peripheral I/O before booting the hardcore system, in FPGA configuration first mode.
In hardcore system boot first, only FPGA peripheral I/O is configured before booting the hardcore system.
After that, the FSBL, SSBL, or OS boot stages can be configured to perform FPGA core configuration.
Thus, SDM here ensures the secure and trusted boot of Intel FPGA SoCs.
\end{comment}

\section{Conclusion}
\label{s:conclusion}

Even though existing hardware-assisted TEEs on commodity computing devices have been widely adopted in commercial systems, they have many issues that make them either ineffective, inefficient, or untrustworthy.
In this paper, we present {\scshape BYOTee}\xspace, which is a framework for building multiple equally secure TEEs on-demand with configurable hardware and software TCBs utilizing commodity SoC FPGA devices.
In {\scshape BYOTee}\xspace, enclaves, which include softcore CPUs, memory, and peripherals, are created on the FPGA, and the {\scshape BYOTee}\xspace {\scshape Firmware}\xspace provides necessary software libraries for the SSAs to use.
Additionally, the {\scshape Firmware}\xspace offers two attestation mechanisms, namely pre-execution and post-execution attestations, to verify the hardware and software stacks.
We implemented a proof-of-concept {\scshape BYOTee}\xspace for Xilinx SoC FPGA.
The evaluation results on the low-end Zynq-7000 system for benchmark applications and example SSAs show the effectiveness and performance of {\scshape BYOTee}\xspace.
\section{{\scshape BYOTee}\xspace Applications and Developer's Perspective}
\label{s:dev}

\begin{comment}
\begin{figure*}[ht]
	\begin{centering}
		\centering
		\includegraphics[width=\linewidth]{images/dev_work_flow.pdf}
		\vspace{-0.3cm}
		\caption{Development and run-time process}
		\label{fig:}
	\end{centering}
\end{figure*}
\end{comment}

In this section, we first use multiple SSA examples to demonstrate how {\scshape BYOTee}\xspace can secure real-world applications in several classes.
Then, we discuss how developers can easily develop and deploy enclave hardware, SSAs, and UAs using the {\scshape BYOTee}\xspace toolchain.

\subsection{{\scshape BYOTee}\xspace Applications}
\label{sub:apps}

\textbf{Computational Applications.}
Computational applications take input from the hardcore system or other enclaves, perform the intended computational operations, and send the outputs back.
They represent computational tasks, such as encryption, decryption, machine learning-based classification, etc., that do not need peripherals.
{\scshape BYOTee}\xspace protects such applications from code and data disclosure, memory corruption, and cache side-channel attacks from the hardcore system and other enclaves at run-time.
We implement an AES accelerator SSA (SSA-1) as an example for computational applications.
To use SSA-1, a UA places the plaintext or ciphertext in the SEB and notifies the SSA.
When the encryption or decryption is finished, SSA-1 places the outputs on the SEB.

\textbf{Peripheral Interacting Applications.}
Peripheral interacting applications use some peripherals, but they do not communicate with other SSAs or the hardcore system.
An example usage is cyber-physical applications that read from sensors, make local decisions, and control an actuator.
Besides the attacks {\scshape BYOTee}\xspace protects the computational applications from, {\scshape BYOTee}\xspace also protects the paths between the SSA and peripherals from attacks for these applications.
We develop an LED toggler SSA (SSA-2) that uses a button and an LED.
When the button is pressed, SSA-2 toggles the color of the LED.
Both the button and LED are connected to the enclave of SSA-2 only and cannot be accessed by the hardcore system or other enclaves.

\begin{comment}
\begin{lstlisting}[label={ssa2},caption={SSA-2 Code Snippet}, language=c] .
void GpioHandler(void *CallbackRef)
{
	XGpio *GpioPtr = (XGpio *)CallbackRef;
	IntrFlag = 1;
	XGpio_InterruptClear(GpioPtr, GlobalIntrMask);
}

int main()
{
	u32 *led = (u32 *)XPAR_RGB_PWM_0_PWM_AXI_BASEADDR;
	xil_printf("Wait for Button GPIO interrupt: \r\n");

	while(true)
	{
		if(IntrFlag)
		{
			toggleLED(led);
		}
	}

	return 0;
}
\end{lstlisting}
Listing~\ref{ssa2} represents the code snippet of the LED Switch SSA, where it configures GpioHandler function as the button interrupt handler.
\end{comment}

\textbf{Peripheral and Hardcore System Interacting Applications.}
Peripheral and hardcore system interacting applications do not only control some peripherals but also continuously interacts with a UA.
For demonstration, we develop a music player system with digital rights management that guarantees the confidentiality, integrity, and authenticity of songs for artists and recording studios.
This means i) songs cannot be digitally disclosed, ii) songs cannot be modified,
and iii) only songs that were protected can be played.

To this end, the music player system has three components:
i) a trusted song protector (in Python with 160 SLOC), which is an offline component to encrypt and sign a song file (WAV format);
ii) a UA (in C with 695 SLOC) running on the hardcore system, which provides a user interface to play, pause, resume, and stop a protected song.
The UA awaits for the user's commands, reads protected songs from storage, e.g., SD card, and sends them to the song playing SSA.
Because the protected song file is big (e.g., an original  77 seconds, 48KHz, and a single channel WAV file is around 33MB.
The protected song file is several hundred bytes bigger.), the UA needs to continuously read the protected song file data from the storage and send it to the SSA;
iii) a song playing SSA (SSA-3) that authenticates, decrypts, and plays a song by sending the plaintext data of it to a hardware audio module.
The hardware audio module is only connected to the enclave running SSA-3, so the hardcore system and other enclaves cannot access it.

We emphasize that any software solution that solely trusts SGX or TrustZone cannot meet the security requirements of this music player system because:
i) there is no isolated or trusted I/O path between an SGX enclave and the hardware audio module; hence a malicious REE OS can breach the confidentiality, integrity, and authenticity of a song.
Some solutions, such as SGXIO~\cite{weiser2017sgxio}, attempt to address this issue but they need to add additional hardware, e.g., the hypervisor, into the TCB;
ii) a TrustZone application must decrypt the song in DRAM before sending it to play; hence, vulnerable to cold-boot attacks.

\textbf{Distributed Applications.}
A distributed application consists of multiple inter-communicating SSAs running on different enclaves at the same time.
The SSAs communicate through a shared BRAM region.
{\scshape BYOTee}\xspace not only protects each of the SSAs but also their communications from the hardcore system and other enclaves.
For demonstration, we develop an application that processes data in sequence with two SSAs.
SSA-1 first receives data from a UA, decrypts the data, outputs to the shared BRAM instead of DRAM, and the second SSA (SSA-4) takes the output of SSA-1 and performs a SHA512-HMAC signature verification.

\subsection{Developer's Perspective}
\label{subsec:dev}

\begin{figure}[!t]
\begin{lstlisting}[label={l:CTEEHS},caption={An example hardware description defining three enclaves in JSON format. Each enclave has its own customized softcore CPU (processor model, cache, FPU, etc.), peripheral connections, memory, etc.}]
{"Enclaves": [
    {"Name": "Enclave-1",
     "Processor": {"Type": "MicroBlaze 32bit",
      "Debugging": "Enabled"},
     "Memory Size": "512KB",
     "Shared DRAM SEB": {
     	"Base": "0x20000000", "Size": "2MB"}},
    {"Name": "Enclave-2",
     "Processor": {"Type": "VexRisc 32-bit",
      "Data Cache": "16KB",
      "Instruction Cache": "16KB", "FPU": "F32",
      "Debugging": "Disabled"},
     "Memory Size": "32MB",
     "Shared DRAM SEB": {
      "Base": "0x20000800", "Size": "128MB"}},
    {"Name": "Enclave-3",
     "Processor": {"Type": "A2I 64bit",
      "Data Cache": "64KB",
      "Instruction Cache": "64KB",
      "MMU": "Enabled", "MMU Page Size": "4KB",
      "FPU": "AXU", "Debugging": "Disabled"},
     "Memory Size": "64MB",
     "Shared DRAM SEB": {
      "Base": "0x20020800", "Size": "256MB"}}],
 "Peripherals": [
    {"Type": "AXI Gpio",
     "Board Interface": "Btns 2bits",
     "Access": ["Hardcore system", "Enclave-2"]},
    {"Type": "Uart Lite 8bit",
     "Baud Rate": "115200",
     "Access": ["Enclave-1"]},
    {"Type": "Dual Port BRAM Generator",
     "Base Address": "0x1F0000", "Size": "2MB",
     "Access": ["Enclave-1", "Enclave-3"]}]}
\end{lstlisting}
\end{figure}

\textbf{Creating Enclave Hardware.}
A developer can use the {\scshape HardwareBuilder}\xspace to design and create the hardware comprising of one or multiple enclaves.
Listing~\ref{l:CTEEHS} shows an example hardware description in JSON format of three enclaves.
Enclave-1 has a 32-bit MicroBlaze softcore CPU~\cite{mbsoft}. Enclave-2 has a 32-bit VexRisc softcore CPU~\cite{VexRiscv}, Enclave-3 uses a 64-bit A2I softcore CPU~\cite{a2i}.
The softcore CPUs of Enclave-2 and Enclave-3 have FPUs, instruction, and data caches.
A2I softcore of Enclave-3 has MMU enabled with a page size of 4KB.
Debugging is enabled on the softcore CPU of Enclave-1 only, for which {\scshape HardwareBuilder}\xspace inserts a debugging module to help the developer debug and trace the SSA on this enclave.
A DRAM region is reserved for the SEB of each enclave, respectively.
A UART peripheral is only connected to the Enclave-1 and cannot be accessed by the hardcore system or the other enclave.
Additionally, a GPIO peripheral is connected to both the hardcore system and Enclave-2 but cannot be accessed by Enclave-1 or Enclave-3.
Each enclave shares a DRAM region with the hardcore system for 2-way enclave-to-hardcore System communication.
Enclave-1 and Enclave-3 can also use the shared BRAM region to communicate.

The developer uses {\scshape HardwareBuilder}\xspace to generate hardware configurations, which outputs the Tcl scripts containing the synthesizer commands.
The \verb|-d| parameter specifies the JSON configuration file, and \verb|-o| defines the output Tcl path:

\begin{lstlisting}[label={header}, language=bash, numbers=none]
hardwareBuilder.py -d <CONFIG_JSON> -o <TCL>
\end{lstlisting}

Then, the developer invokes the synthesizer tool with the Tcl script as input.
The \verb|-n| parameter specifies the name of the hardware project,
and the \verb|bf| parameter specifies the mode of operation, which includes generating bitstream, combining bitstream with {\scshape Firmware}\xspace, etc.
The final output of the {\scshape HardwareBuilder}\xspace is a bitstream file specified by \verb|-o|:

\begin{lstlisting}[label={header}, language=bash, numbers=none]
createFPGAImage -d <TCL> -n <PROJ_NAME> -bf <BUILD_FLAG> -o <FPGA_IMAGE>
\end{lstlisting}

\textbf{Creating Boot Images.}
After the {\scshape HardwareBuilder}\xspace, the developer uses the boot loader creation tool with the developer-defined boot image format, e.g., \verb|.bif|, and the protected FPGA image to create a deployable binary file.
\begin{lstlisting}[label={header}, language=bash, numbers=none]
createBootImage <SYSTEM_BIF> <FPGA_IMAGE> -o <BYOTee_BIN>
\end{lstlisting}

\textbf{Creating SSAs and UAs.}
{\scshape BYOTee}\xspace\ {\scshape Firmware}\xspace is primarily developed in C, but software modules developed in any language that can be linked against the {\scshape Firmware}\xspace can be included in an SSA.
The SSAs, which have their own \verb|main| functions with a declaration of \verb|int main()| \verb|__attribute__| \verb|((section (".text.ssa_entry")))|, are developed as separate applications from the {\scshape Firmware}\xspace.
In our prototype implementation, SSAs are statically linked against the {\scshape Firmware}\xspace with the linker script {\scshape BYOTee}\xspace provides,
which reserves physical memory regions for the SSAs' code and data sections.

\begin{comment}
\begin{lstlisting}[label={header},caption={Header file defining SSA entry}, language=c]
#ifndef BYOT_HEADER_H_
#define BYOT_HEADER_H_

int main() __attribute__ ((section (".text.ssa_entry")));

#endif
\end{lstlisting}
\end{comment}

Not all libc functions are available for the SSAs to use.
For example, \verb|printf| may is not available since \verb|stdout| is not defined in the {\scshape Firmware}\xspace for a specific FPGA system.
To move data among DRAM, BRAM, and peripheral memories, system-specific underlying mechanisms will be used.
The {\scshape Firmware}\xspace provides a HAL with interfaces like \verb|BYOT_MemCpy| to replace the libc \verb|memcpy|.
The UAs execute as unprivileged applications on the operating system, e.g., PetaLinux for Xilinx devices, of the hardcore system.
The UAs use a UIO interface to communicate with the {\scshape Firmware}\xspace and SSA running on the FPGA.

\begin{comment}
\subsection{Usage}
After specifying the hardware description in the JSON file, a developer can generate hardware configurations with {\scshape HardwareBuilder}\xspace in two simple steps.
First, the developer invokes our developed python tool to generate the Tcl scripts containing the synthesizer commands:

\begin{lstlisting}[label={header}, language=bash, numbers=none]
parseConfig.py -d <CONFIG_JSON_path>
\end{lstlisting}
In the second step of the {\scshape HardwareBuilder}\xspace, the developer invokes the synthesizer tool with the Tcl script as input.
\begin{lstlisting}[label={header}, language=bash, numbers=none]
buildDevice -p <TCL_SRC_PATH> -n <PROJ_NAME> -bf <BUILD_FLAG>
\end{lstlisting}
The first two arguments specify the source Tcl file path, and the -n argument with the PROJ\_NAME specifies the name of the hardware project.
The last two option specifies the mode of operation, which includes creating project (cp), generating bitstream (gb), combining bitstream with {\scshape Firmware}\xspace (cb), and running all functions (all).
The final output of the {\scshape HardwareBuilder}\xspace is the bitstream file.

After the hardware generation and software development, the developer needs to invoke three {\scshape BYOTee}\xspace tools for deployment in a SoC device.
First, a FPGA boot loader creation tool with two arguments: the developer-defined boot image format and the path to protected FPGA image (bitstream) to create the deployable bin file.
\begin{lstlisting}[label={header}, language=bash, numbers=none]
createBootImage <SYSTEM.bif> <FPGA_IMAGE_PATH> -o <BYOT.BIN_PATH>
\end{lstlisting}
\end{comment}

The developer uses the {\scshape SSAPacker}\xspace to generate protected SSA binaries:

\begin{lstlisting}[label={header}, language=bash, numbers=none]
SSAPACKER -d <SSA_BIN> -o <PROTECTED_SSA>
\end{lstlisting}

{\scshape BYOTee}\xspace also includes a tool to put the boot loader, PetaLinux image, UA, protected FPGA image, and protected SSA binaries on the SD card for the deployment and debugging on Xilinx SoC FPGA systems:

\begin{lstlisting}[label={header}, language=bash, numbers=none]
deploySoC <SD_DEVICE> <BYOTee_BIN> <FPGA_IMAGE> <PROTECTED_SSA> <UA> <IMAGE.ub>
\end{lstlisting}

\subsection{Security Evaluation}

To demonstrate the security of {\scshape BYOTee}\xspace, we report the software TCB size of the four example SSAs and evaluate the cold-boot attacks on DRAM and BRAM of the same board.

\subsubsection{Software TCB Size}
\label{sub:tcbAnalysis}

Table~\ref{t:FwLocRes} presents the size of the software TCB for the four example SSAs and their corresponding {\scshape Firmware}\xspace.
As the table shows, the size of {\scshape Firmware}\xspace increases as the SSA gets more complicated and needs more services.
Nevertheless, the run-time software TCB (SSA and {\scshape Firmware}\xspace combined) of SSA-3, which is a functional digital right management music player, has only 10,727 SLOC, representing a significant software TCB reduction from its counterpart implemented as a TrustZone, e.g., TF-M~\cite{TFMsrc} has over 117K SLOC, or SGX application, e.g., the Gramine library OS~\cite{gramine} has 83K SLOC.

\subsubsection{Cold-boot Attacks on DRAM and BRAM}

\begin{figure}[!t]
    \vspace{-0.2cm}
	\centering
	\subfloat[0 s]{\includegraphics[width = 0.19\linewidth]{images/0.png}}\hfill
	\subfloat[5 s]{\includegraphics[width = 0.19\linewidth]{images/5.png}}\hfill
	\subfloat[15 s]{\includegraphics[width = 0.19\linewidth]{images/10.png}}\hfill
	\subfloat[20 s]{\includegraphics[width = 0.19\linewidth]{images/15.jpg}}\hfill
	\subfloat[30 s]{\includegraphics[width = 0.19\linewidth]{images/20.jpg}}  \\
	\subfloat[0 s]{\includegraphics[width = 0.19\linewidth]{images/0.png}}\hfill
	\subfloat[1 m]{\includegraphics[width = 0.19\linewidth]{images/f1m.png}}\hfill

	\subfloat[10 m]{\includegraphics[width = 0.19\linewidth]{images/f10m.jpg}}\hfill
	\subfloat[13 m]{\includegraphics[width = 0.19\linewidth]{images/20.jpg}}    \hfill
	\subfloat[BRAM initialization]{\includegraphics[width = 0.19\linewidth]{images/sram.png}}\hfill

	\caption{Visualizing DRAM decay after power down and BRAM hardware initialization after power up on the same Z7-07S board. We loaded a bitmap image (150$\times$150 pixel; 90.1kB) on the DRAM and BRAM.
	We then measure 1) DRAM decay at room temperature (20$^{\circ}$C/68$^{\circ}$F) after (a) power reset (0 second) and (b) - (e) losing power for different intervals;
	2) DRAM decay at -18$^{\circ}$C/0$^{\circ}$F after (f) power reset and (g) - (i) losing power for different intervals; and 3) (j) BRAM hardware initialization after power up. The reconstructed image from the fully decayed DRAM is red because half of the cells are measured as 1s and the other half as 0s. The reconstructed image from the BRAM is transparent because all the bits are initialized to 0s.}
	\label{fig:cold-boot}
	\vspace{-.3cm}
\end{figure}

We evaluated the feasibility of cold-boot attacks on DRAM and BRAM on the same Cora Z7-07S board.
In these experiments, we loaded a bitmap image onto the DRAM and BRAM and measured the DRAM decay at room temperature (20$^{\circ}$C/68$^{\circ}$F) and -18$^{\circ}$C/0$^{\circ}$F after power reset (0 seconds) and losing power for different intervals.
We dumped the content of BRAM, for which the Xilinx Zynq-7000 FPGA has a non-bypassable hardware initialization mechanism after power up to clear all the bits to 0s.
As we discussed in~\ref{subsec:analysisColdBoot}, even if the BRAM is not initialized, cold-boot attacks on it are much more difficult than on DRAM.
Figure~\ref{fig:cold-boot} visualizes the cold-boot attack results, which clearly show cold-boot attacks on DRAM are feasible but not on BRAM.

\subsection{Performance on the Low-end MicroBlaze Softcore CPU}

We evaluate the performance of the low-end MicroBlaze-powered enclaves, which provides a lower-bound performance estimation of available softcore CPUs.
We first evaluate the performance using 12 Embench-IoT benchmark applications~\cite{embenchIOT}.
We then evaluate the {\scshape BYOTee}\xspace software run-time performance by measuring the time cost of different {\scshape Firmware}\xspace operations.

\subsubsection{Benchmark Performance Evaluation}
To show the performance of the MicroBlaze softcore compared to the Cortex-M4 hardcore, we use 12 applications from the Embench-IoT benchmarks~\cite{embenchIOT}.
As Table~\ref{t:embench} shows, the applications run comparatively faster on the low-end MicroBlaze softcore CPU than the hardcore Cortex-M4.
For better performance, users can choose more advanced softcore CPUs.

\subsubsection{{\scshape Firmware}\xspace Performance}
We evaluate the time {\scshape Firmware}\xspace spends on the dynamic loading, decrypting, integrity and authenticity verification, attestation, cleaning up, suspending, and restoring operations of three SSAs,
As Table~\ref{t:all} shows,
the time spent by the {\scshape Firmware}\xspace on these operations is linear to the size of the SSA and its data combined.
To copy the protected SSA and its input from DRAM to BRAM, {\scshape Firmware}\xspace running on the Z7-07S spends around 1.07 ms for every 10,000 bytes.
To decrypt the protected SSA and its data using 256-bit CBC mode AES, {\scshape Firmware}\xspace spends around 2182 ms for every 10,000 bytes.
The integrity and authenticity verification costs around 93.43 ms for every 10,000 bytes.
The BLAKE-based pre-execution and post-execution cost around 124.77 ms for every 10,000 bytes.
Cleaning up the BRAM takes around 0.45 ms for every 10,000 bytes.
The SHA512-HMAC and AES 256-bit with CBC mode based suspending and restoring cost roughly 2834 ms for every 10,000 bytes.

\begin{comment}
\emph{SSA Dynamic Load and Clean Up}

Table~\ref{t:all} includes the performance of {\scshape Firmware}\xspace with respect to dynamic load and clean-up of BRAM for different sizes of SSAs.
Dynamic load of SSA performance is directly related to the size, as before loading of SSAs {\scshape Firmware}\xspace will perform integrity and authenticity check operations on SSA modules.
For clean-up operation, the performance depends on reserved memory size for SSAs by the {\scshape Firmware}\xspace

\textbf{SSA Decryption}
\label{s:decrypt-res}
Table~\ref{t:all} represents the ciphertext size for each SSA and their decryption execution time inside the {\scshape Firmware}\xspace.
The ciphertext data for the SSAs is made up of the extracted code section, data sections, and metadata.
We are using symmetric encryption of SSAs, namely AES with developer-defined key.
For multiple invocation of the same SSA, the {\scshape Firmware}\xspace performs this decryption once.

\textbf{SSA Authentication}
SSA authentication data include the SSA binary, input data, any metadata from SEB of CTEEs.
From the table~\ref{t:all} we observe that the authentication performance of {\scshape BYOTee}\xspace depends on the size of data to be authenticated.
For SSA-3 application user inputs are put to SEB at runtime instead of load time by UA.
Authentication is not performed with the runtime data, but for files loaded from SD card such as encrypted song files are authenticated.
\textbf{Attesation on SEB}
\label{sub:attst}

Table~\ref{t:all} represents the {\scshape Firmware}\xspace attestation performance on the three SSAs.
For \emph{preExeAtt}, the attestation data is the sum of the extracted SSA modules, metadata, and inputs.
For \emph{postExeAtt}, we include the SSA outputs along with the \emph{preExeAtt} attestation data.
The \emph{preExeAtt} and \emph{postExeAtt} are performed on each invocation of the SSAs.
Although based on the result, it may seem the attestation is a slow process, this can be mitigated with a more powerful FPGA unit and more sophisticated processors.

\end{comment}

\begin{comment}
\textbf{CTEE communication performance}
{\scshape BYOTee}\xspace supports communication of the SSAs in different CTEEs through shared memory.
Thus, the communication speed is analogous to the access of local memory and no additional constraints are present except the bus speed.
While in traditional TEEs such as Intel SGX, two secure enclaves even in the same platform need to go through local attestation and requires the OS to perform additional key exchange, encryption operations to ensure secure communication~\cite{costan2016intel}.
In {\scshape BYOTee}\xspace as the BRAM is physically isolated from the PS (REE), the communication medium is trusted and can be used directly to communicate with other SSAs on different CTEEs.
\end{comment}

\begin{comment}
\subsection{Attacks and Mitigations}
\label{sub:atkDef}
{\scshape BYOTee}\xspace can be subject to different attack scenarios.
We demonstrate some attacks on {\scshape BYOTee}\xspace, which are within the scope in the following:

\subsubsection{SSA Integrity} As described in \S~\ref{subsec:attest}, the SEB between the PS and the CTEEs are not secure, and PS software has full access.
Before the {\scshape Firmware}\xspace copies the SSA modules onto BRAM, the PS software can inject or temper code and data sections of the SSA modules.
We demonstrate such an attack from the PS software, which disrupts the integrity of SSAs.
Firstly, the SSA modules are encrypted while it's in the SEB.
Thus, code or data manipulation to leak sensitive information is infeasible from the PS software.
Even if the malicious PS software manipulates the encrypted SSA modules, the \emph{preExeAtt} measurement makes those attacks detectable.
Besides, the \emph{postExeAtt} after each SSA execution provides PoX for the SSAs.

\subsubsection{Input and Output Data Integrity} Even though the SSA modules are encrypted in the SEB, the input and output data of the SSAs are not encrypted.
We construct an attack where the UA disrupts the integrity of the input or output data.
Our attestation mechanism detects the tempering of the I/O data similarly to the tempering of SSA modules.
Additionally, we include a \textit{chal} number for each measurement, which detects replay attacks from the UA.

\end{comment}

\section{Implementation and Evaluation}
\label{s:impleval}

In this section, we present an implementation of the {\scshape BYOTee}\xspace framework for the Xilinx SoC FPGA and evaluate it on a low-end Digilent Cora Z7-07S development board ($\approx$\$130) with a Zynq-7000 FPGA.

\subsection{Experiment Environment}

\begin{figure*}[!t]
    \centering
    \subfloat[Enclave-1 w/ debugger]{\includegraphics[width = 0.19\linewidth]{images/Impl_aes_with_mdm.png}}\hfill
    \subfloat[Enclave-1 wo/ debugger]{\includegraphics[width = 0.19\linewidth]{images/Iml_aes_without_mdm.png}}\hfill
    \subfloat[Enclave-2 wo/ debugger]{\includegraphics[width = 0.19\linewidth]{images/Impl_led_without_mdm.png}}\hfill
    \subfloat[Enclave-3 wo/ debugger]{\includegraphics[width = 0.19\linewidth]{images/Impl_secure_DRM_without_mdm.png}}\hfill
    \subfloat[Enclave-1 and Enclave-4 wo/ debugger]{\includegraphics[width = 0.19\linewidth]{images/Impl_aapp4_with_mdm.png}}
    \vspace{-0.2cm}
    \caption{Resource footprints of the enclaves with MicroBlaze softcore CPUs for the example SSAs on the Cora Z7-07S.
The yellow and red portions represent the MicroBlaze softcore CPU cells.
The purple portion represents the cells of BRAMs.
The pink rectangle represents the I/O ports. In (d), the rectangle on top of the I/O ports represents an analog to digital conversion module. The blue portions represent all other IPs, such as different peripherals, debugging modules, interconnects.}
    \label{fig:corr-res-iml}

\end{figure*}

\begin{table*}[t]
	\centering
	\footnotesize
	\caption{Hardware TCB size and resource utilization of enclaves for the example SSAs on Cora Z7-07S}
	\label{t:resApps}
	\begin{tabular}{c|c|c|c|c|c|c|c|c}
		\hline
		 & \multicolumn{2}{c|}{Enclave-1}   & \multicolumn{2}{c|}{Enclave-2}  & \multicolumn{2}{c|}{Enclave-3}& \multicolumn{2}{c}{Enclave-4}  \\ \hline
		Resource & w/ debugger   & w/o debugger  & w/ debugger   & w/o debugger & w/ debugger   & w/o debugger & w/ debugger   & w/o debugger \\ \hline
		LUT     & 5,255 (36.5\%)      &   2,232 (15.5\%)    & 6,385 (44.3\%) &  3,291 (22.9\%) & 10,781 (74.9\%) & 6,778 (47.1\%) & 5,302 (36.8\%) & 3,130 (21.7\%) \\ \hline
		LUTRAM   &  419 (7.0\%)        &    211 (3.5\%)    &   507 (8.5\%) & 282 (4.7\%)  & 725 (12.1\%) & 427 (7.1\%) & 319 (5.3\%) & 145 (2.4\%)\\ \hline
		Flip-flop    & 5,245 (18.2\%)       &  2,259 (7.8\%)    &     6,759 (23.5\%)  & 37,64 (13.1\%) & 11,363 (39.5\%)&   7,721 (26.8\%) & 5,497 (19.1\%) & 3,014 (10.5\%)\\ \hline
		BRAM  & 18 (36.0\%)  & 16 (32.0\%)    &     34 (68.0\%) & 32 (64.0\%)   &  48 (95.0\%)   & 45.50 (91.0\%)   & 28 (56.0\%)  & 26 (52.0\%)\\ \hline
		DSP    & 3 (4.5\%) & 0 (0.0\%)&  3 (4.5\%)     &   0 (0.0\%) & 3 (4.5\%)  & 0 (0.0\%) & 3 (4.5\%) & 0 (0.0\%) \\ \hline
		IOB     & 0 (0.0\%) & 0 (0.0\%)   &  6 (6.0\%)   & 6 (6.0\%)   &   28 (28.0\%) & 28 (28.0\%) & 0 (0.0\%) & 0 (0.0\%) \\ \hline

	\end{tabular}
	\vspace{0.25cm}
\end{table*}

The Cora Z7-07S board has a single-core 667MHz Arm Cortex-A9 processor with 512MB DDR3 memory, 32KB L1 cache, 512KB L2 cache and a Xilinx Zynq-7000 FPGA.
As shown in Table~\ref{t:fpgas}, the Zynq-7000 FPGA has 3,600 logic cells, 14,400 LUTs, 6,000 LUTRAM, 28,800 flip-flops, a 225KB BRAM, 66 Digital Signal Processing (DSP) slices, and 100 IOBs.
The development board also has an SPI header, two push-buttons, two RGB LEDs, a microSD card slot, two Pmod connectors, etc.
We connect a Pmod I2S2 stereo audio input and output device~\cite{pmod} to the board for SSA-3 evaluation.
Figure~\ref{fig:z7} (in Appendix) shows the top and bottom view of the board with the connected audio device.
We create two partitions, namely \verb|boot| and \verb|root|, on an SD card and use the Xilinx \verb|bootgen| tool to generate device boot images~\cite{bootgen}.
\verb|Bootgen| stitches a stock FSBL for the hardcore system and the protected FPGA image together to create a binary boot file.
The boot file, U-Boot as the SSBL for the hardcore system, and a PetaLinux image for the hardcore system are stored in the \verb|boot| partition, whereas the protected SSAs, UAs, and other application files are stored in the \verb|root| partition.

\subsection{{\scshape BYOTee}\xspace Implementation}
We implemented the {\scshape BYOTee}\xspace infrastructure and toolchain, which include {\scshape HardwareBuilder}\xspace, {\scshape SSAPacker}\xspace, and {\scshape Firmware}\xspace, for the Xilinx SoC FPGA.
The {\scshape HardwareBuilder}\xspace was developed in Python (2.5K SLOC).
The {\scshape SSAPacker}\xspace include Python (63 SLOC) and C code (420 SLOC).
The {\scshape Firmware}\xspace was developed in C and has an SSA loader and cleaner (1.1K SLOC), an attestation module (333 SLOC), an interrupt initialization and handling module (101 SLOC), and a linker script (212 lines).
The {\scshape Firmware}\xspace is linked against the vendor-provided HAL (7.9K SLOC) and libraries, e.g., libc (1.2MB), etc.
The {\scshape Firmware}\xspace, especially the HAL, can be customized to reflect an SSA's needs.
Our implementation of the SSA loader uses AES-256 for SSA encryption and SHA512-HMAC to protect the integrity and authenticity of SSAs.
We use the BLAKE2~\cite{Blk2} hash algorithm to implement the pre-execution- and post-execution-attestations of SSA applications in the attestation module.
On the hardcore system side, a userspace I/O interface is used for the UAs to access the shared DRAM regions between the hardcore system and FPGA.

\begin{table}[t]
	\centering
	\footnotesize
	\caption{Size of the example SSAs' software TCB}
	\label{t:FwLocRes}
	\begin{tabular}{c|c|c|c|c|c|c|c}
		\hline
		 & \multicolumn{2}{c|}{SSA} & \multicolumn{5}{c}{Corresponding {\scshape Firmware}\xspace} \\
			\hline
		& SLOC & Bytes & SLOC & .text & .data & .bss & Total \\
		\hline
		SSA-1 & 717 & 12,892& 3,143 &   27,296    &  3,236    &  448    &      30,532 \\
			\hline
		SSA-2 & 346 & 2,868 & 3,532&  30,748     &  2,800  &   440   &  33,988   \\
		\hline
		SSA-3 & 1,029 & 20,380 & 9,698 &     57,142  &    4,308  &    635  &  62,085   \\
		\hline
		SSA-4 & 622 & 31,088 & 3,235 & 28,377   &   3,608   &   528  &  35,748  \\
		\hline
	\end{tabular}
    \vspace{0.1cm}
\end{table}

\begin{table*}[t]
	\centering
	\footnotesize
	\caption{{\scshape Firmware}\xspace performance evaluation on the low-end softcore MicroBlaze CPU (Time in milliseconds; size in bytes). The experiments show SSA-3 can verify, decrypt, and play 48KHz WAV music smoothly on the very low-end softcore CPU.}
	\label{t:all}
	\begin{tabular}{c|c|c|c||c|c|c|c|c|c||c|c}
		\hline
		&  Binary & Input & Output & Loading & Decryption & Integrity and & pre-execution & post-execution  & Cleaning & Suspend & Restore \\
		&  size & size & size &  & & authenticity & attestation & attestation & up & and & and\\
		&  &  & &  & & verification & & & & export & execute\\
		 \hline
		SSA-1 & 12,892 & 64 & 64 & 1.39 & 2784.56 & 118.54 & 153.94  & 154.79 & 0.54 & 3694.55 & 3609.65\\ \hline
		SSA-2 &  2,596 & N/A & N/A & 0.30 & 579.11 & 29.15 & 30.90 & 30.90 & 0.11 & 741.71
		 & 729.69 \\ \hline
		SSA-3 & 20,152 & 128
		& 132 & 2.17 & 4414.32 & 185.30 & 258.30 & 260.50 & 0.90 & 5787.97 & 5638.76 \\ \hline

	\end{tabular}
    \vspace{-0.2cm}
\end{table*}

\begin{table}[t]
	\centering
	\footnotesize
	\caption{Embench-IoT performance evaluation in millisecond on  softcore MicroBlaze CPU (Zynq-7000 FPGA implemented, version 10.0, 100MHz, no cache, no FPU) and hardcore Cortex-M4 CPU (16MHz, no cache, no FPU, officially reported performance from ~\cite{embencm4}).}
	\label{t:embench}
	\begin{tabular}{l|p{3cm}|c|c}
	\hline
	 Application & Description & M4~\cite{embencm4} & MicroBlaze \\ \hline
	 aha-mont64 & Modulo generator& 4,004 &  501 \\ \hline
	 crc32 & 32 bit error detector & 4,010 & 193\\ \hline
	 huffbench & Data compressor & 4,120 &  111\\ \hline
	 minver&Floating point matrix inversion&3,998 & 327\\ \hline
	 nettle-aes & Low level AES library & 4,026 & 245\\ \hline
	 nsichneu & Computes permutation&4,001 & 237\\ \hline
	 primecount &Prime counter & n/a & 193\\ \hline
	 sglib-combined& Sort, search, and query on array, list, and tree &3,981 &  189\\ \hline
	 slre&Regex matching&4,010&113\\ \hline
	 statemate&Car window lift control & 4,001 & 139\\ \hline
	 tarfind&Archive file finder& n/a &163
	  \\ \hline
	 ud & Matrix factorization & 3,999 & 343\\ \hline
	 \multicolumn{2}{c|}{Geometric Mean}&4,015&208\\ \hline
	\end{tabular}
\end{table}

\subsection{Enclaves for the Example SSAs}
We specified the hardware description for each example SSA in~\S\ref{sub:apps} and used the {\scshape HardwareBuilder}\xspace and synthesizer to generate its enclave bitstream, e.g., Enclave-1 for SSA-1.
All the enclaves are configured with a 32-bit Microblaze CPU (version 10.0, 100MHz, no instruction/data cache, no FPU).
The Enclave-1, Enclave-2, Enclave-3 have a 128KB BRAM, whereas Enclave-4 has a 32KB BRAM as their main memory.
The peripherals that belong to an enclave are connected through a dedicated AXI Interconnect IP.
Figure~\ref{fig:corr-res-iml} shows the footprints of the four hardware designs on the Z7-07S device.
These figures demonstrate the configurable nature of the {\scshape BYOTee}\xspace hardware TCB and the physical isolation of the enclaves from each other and from the hardcore processor.
Figure~\ref{fig:corr-res-iml}(a) shows the hardware design with a debugger, which developers can use to debug the SSA and {\scshape Firmware}\xspace on the softcore,
whereas all other designs do not have a debugger for the minimum hardware TCB.
The orange portion of the footprint represents cells of the ZYNQ7 processing system, whereas yellow represents softcore cells.
The purple straight lines represent the cells of BRAMs.
The pink rectangle represents the I/O ports, and in Figure~\ref{fig:corr-res-iml}(d) the rectangle on top of the I/O ports represents an analog to digital conversion module.
Figure~\ref{fig:corr-res-iml}(e) represents hardware implementation of two enclaves, namely Enclave-1 (yellow) and Enclave-4 (red), to run SSA-1 and SSA-4, respectively.
There are three types of BRAM in this design.
The purple straight lines represent the contiguous memory of Enclave-1 (128KB), and the green straight lines represent the contiguous memory of Enclave-4 (32KB).
The ash straight lines represent the shared BRAM (8KB) between Enclave-1 and Enclave-4.
The blue portions in Figure~\ref{fig:corr-res-iml} represent the rest of the IPs used in the hardware design.
These include interconnects, interrupt controllers, etc.

Table~\ref{t:resApps} presents each enclave's hardware TCB and resource utilization on the Cora Z7-07S board with and without a debugger IP.
As the table shows, the debugger IP significantly increases the resource utilization of an enclave as it uses three DSP slices, two BRAMs, and many other resources.
Since SSA-1 and SSA-4 do not use peripherals, Enclave-1 and Enclave-4
do not have any IOB.
Figure~\ref{fig:corr-res} (in Appendix) shows the block diagrams of the enclaves generated by the {\scshape HardwareBuilder}\xspace for the four example SSAs.

\section{Introduction}

Existing hardware-assisted Trusted Execution Environments (TEE) on commodity computing devices make use of hardware security primitives offered by the CPU, such as Intel SGX~\cite{costan2016intel} and Arm TrustZone~\cite{pinto2019demystifying}, to guarantee code and data loaded inside to be protected, with respect to confidentiality and integrity, from the Rich Execution Environment (REE).
In recent years, we have witnessed unprecedented growth in using SGX and TrustZone in real-world products and academic projects,
which include real-time kernel protections~\cite{azab2014hypervision,ge2014sprobes}, securing containers and runtime libraries~\cite{arnautov2016scone,tsai2017graphene, santos2014using}, shielding applications from attacks~\cite{baumann2015shielding,lind2017glamdring,schuster2015vc3}, etc.

\textbf{Limitations of Existing TEEs.} The hardware and software of existing TEEs nevertheless have the following issues, making them either ineffective, inefficient, or untrustworthy:
i) existing TEE architectures, such as Intel SGX and Arm TrustZone, are proprietary and only work for specific computing architectures, which users have to trust blindly.
It is impossible for users to verify the correctness of the TEE designs or attest the hardware states at run-time;
ii) they only provide a static and fixed hardware Trusted Computing Base (TCB), which cannot be customized for different applications.
For TrustZone, it means the TEE has the highest privilege to control the REE and communicate with all peripherals.
The design violates the principle of least privilege by including unnecessary peripherals and buggy peripheral drivers in the software TCB~\cite{cerdeira2020sok} and exposing the TEE to malicious peripheral inputs~\cite{gross2019breaking}.
For SGX, it means applications in enclaves have to go through and trust the REE OS to communicate with peripherals~\cite{schneider2020pie}, bloating the size of software TCB by including a usually monolithic REE OS kernel, e.g., 27.8M Source Lines of Code (SLOC) of the Linux kernel~\cite{LinuxSLOC};
iii) the TEE shares a processor core with the REE in a time-sliced fashion, not only costing many CPU cycles~\cite{song2016enforcing, zhang2019icore} for the expensive context switches between the TEE and REE but also making it vulnerable to cache side-channel attacks~\cite{zhang2016truspy, cho2018prime, brasser2017software, zhang2016return, gutierrez2018cachelight}, which directly undermine TEE's security promises;
iv) the software TCBs in TEEs are large, creating big attack surfaces for run-time attacks that hijack the control or data flow~\cite{almakhdhubmurai2020, zhou2020silhouette}.
For example, Haven~\cite{haven} places a whole Windows 8 library OS inside the enclave.
OP-TEE, a Cortex-A secure world OS, has 277K SLOC~\cite{OP-TEE},
and TF-M~\cite{TFMsrc}, a trusted firmware for Cortex-M TrustZone, also over 117K SLOC.
Partly due to these reasons, in 2022 Intel has removed support for SGX in some future CPUs~\cite{sgxdepre}.
In addition,
TrustZone has many other limitations:
i) TrustZone only provides \emph{one} isolated execution environment;
ii) TrustZone does not encrypt the contents of Dynamic RAM (DRAM).
Therefore, it is subject to code and data disclosure from cold-boot attacks~\cite{halderman2009lest};
and iii) TrustZone does not have native hardware supports for remote attestations of code and data integrity~\cite{nunes2020apex, abera2019diat}
due to the lack of platform registers to accumulate measurements.

\begin{table}[t]
	\centering
	\footnotesize

	\begin{tabular}{c|c|c|c}
		\hline
		Vendor & {Intel}   & \multicolumn{2}{c}{Xilinx}   \\ \hline
		 &  Agilex F  & Zynq-7000 &  UltraScale+ EG  \\
		SoC FPGA  & R25A~\cite{Agilex}  & XC7Z007S~\cite{CoraZ7z}& ZU19EG~\cite{evUltra}  \\ \hline
		LE/LC & 2,692,760 & 3,600& 1,143,000 \\ \hline
		ALMs/LUT  & 912,800&14,400& 523,000\\ \hline
		PLLs & 28& 2 & 12\\ \hline
		DSP  & 8,528&66 & 1,968 \\ \hline
		BRAM (MB) & 259 &0.225 & 80.4\\ \hline
 		I/O & 624&49&347 \\ \hline

		Hardcore CPU &Cortex-A53 (Q)&Cortex-A9 (S) & Cortex-A53 (Q) \\ \hline
		DRAM (GB)& 16 & 0.512 & 8 \\ \hline

		Price in 2022 & $\approx$\$10K& $\approx$\$130& $\approx$\$4.3K\\ \hline
	\end{tabular}

	\caption{Example SoC FPGA products. LE: Logic Elements (Intel); LC: Logic Cells (Xilinx); ALM: Adaptive Logic Modules (Intel: Multiple LUTs, registers, adders, and multiplexers make up an ALM); LUT: Look Up Table (Xilinx); DSP: Digital Signal Processing blocks; PLLs: Fabric and I/O Phase-Locked Loops; I/O: Maximum User I/O Pin; S: Single-core; Q: Quad-core. We use a Zynq-7000 system in our evaluations.}
	\label{t:fpgas}
\end{table}
\begin{comment}

\begin{table*}[t]
	\centering
	\footnotesize

	\begin{tabular}{c|c|c|c|c|c|c}
		\hline
		Vendor & \multicolumn{3}{c|}{Intel}   & \multicolumn{3}{c}{Xilinx}   \\ \hline
		& Arria V GZ & Stratix V GX  & Agilex F  & Zynq-7000 &  UltraScale+ RFSoC&  UltraScale+ EG  \\
		SoC FPGA & 5AGZE5~\cite{arria} & 5SGXA7~\cite{stratix}  & R25A~\cite{Agilex}   &  XC7Z007S~\cite{CoraZ7z}  & ZU27DR~\cite{rfsoc} & ZY19EG~\cite{evUltra}  \\ \hline
		LE/LC  &400,000 & 622,000 & 2,692,760 & 3,600&  930,300& 1,143,000 \\ \hline
		ALMs/LUT  &150,960& 234,720 & 912,800&14,400&  425,280&523,000\\ \hline
		PLLs &24 & 28 & 28& 2 & 8 & 12\\ \hline
		DSP  &71,092& 256& 8,528&66&4,272 & 1,968 \\ \hline
		BRAM (MB) & 33.5& 50& 259 &0.225&  73.5&80.4\\ \hline
		I/O &674& 696& 624&49&120&347 \\ \hline

		Hardcore CPU &Cortex-A9 (D) &Cortex-A53 (D) &Cortex-A53 (Q)&Cortex-A9 (S) & Cortex-A53 (Q)&Cortex-A53 (Q) \\ \hline

		Price in 2022 (\$) & $\approx$5K& $\approx$8.5K& $\approx$10K& $\approx$130& $\approx$4.5K& $\approx$4.3K\\ \hline
	\end{tabular}

	\caption{Example SoC FPGA products in the market. LE: Logic Elements (Intel); LC: Logic Cells (Xilinx); ALM: Adaptive Logic Modules (Intel: Multiple LUTs, registers, adders, and multiplexers make up an ALM); LUT: Loop Up Table (Xilinx); DSP: Digital Signal Processing Blocks; PLLs: Fabric and I/O Phase-Locked Loops; I/O: Maximum User I/O Pin; S: Single-core; D: Dual-core; Q: Quad-core.}
	\label{t:fpgas}
\end{table*}
\end{comment}
While many software-based projects attempted to address one or more of the aforementioned issues by building software layers on SGX, TrustZone, or RISC-V, some issues, e.g., non-verifiable design, non-customizable hardware TCB, time-sharing of the processor, vulnerable to cold-boot attacks due to the slow DRAM decay, etc., are rooted in the design of their respective hardware system, business model and ecosystem, etc.; therefore they cannot be addressed by software alone (refer to Table~\ref{tab:comparison} for comparisons).
For example, {\scshape Sanctuary}~\cite{brasser2019sanctuary} uses the memory access controller to provide multi-domain isolation for sensitive applications on Cortex-A TrustZone,
and {\scshape Cure}~\cite{bahmani2021cure} enables the exclusive assignment of system resources, e.g., peripherals, CPU cores, or cache resources, to each enclave on RISC-V.
Even though they address some of the aforementioned issues, both solutions still suffer from cold-boot attacks and a large software TCB.
Other hardware-based solutions, such as HECTOR-V~\cite{nasahl2020hector}, Graviton~\cite{volos2018graviton}, etc., do not address all the issues either, and they cannot be deployed on commodity devices because hardware changes are required.

\textbf{Benefits of FPGA.}
Field-Programmable Gate Array (FPGA) is designed to be configured by users using a hardware description language after manufacturing.
Besides reconfigurability, it has advantages of high performance, fast development round, etc.
As shown in Table~\ref{t:fpgas}, a variety of FPGA products ranging from embedded systems, i.e., Xilinx Zynq-7000 with only 3,600 logic elements ($\approx$\$130), to data center devices, i.e., Agilex F R25A with 2.6 millions of logic elements ($\approx$\$10k), are available in the market.
Various FPGA-based application-specific accelerators, such as for deep convolutional neural networks~\cite{zhang2015optimizing, suda2016throughput}, recurrent neural networks~\cite{li2019rnn}, classic and post-quantum cryptographic algorithms~\cite{chelton2008fast, elkhatib2021high},  Memcached~\cite{lavasani2013fpga}, etc., have been proposed and deployed on such devices~\cite{qiu2016going, huang2016programming}.

More interestingly, FPGA can also be used to build general-purpose computing platforms, in which users can design and implement their own softcore CPUs or customize existing open-sourced~\cite{OpenCores, VexRiscv, Neo430, microwatt, a2i, a2o, OpenSPARC, libreSoc} or proprietary ones~\cite{lysecky2009design, nios}.
The available softcore CPUs range from the partially configurable (i.e., cache size, pipeline depth, memory management unit, etc.) and proprietary low-end 32-bit MicroBlaze~\cite{lysecky2009design}, to the fully customizable and open-sourced mid-end 32-bit RISC-V~\cite{VexRiscv, matthews2017taiga} and high-end 64-bit A2I POWER processor~\cite{a2i}.
While the low-end softcore CPUs are comparable to micro-controllers, their mid-end and high-end counterparts have performances comparable to hardcore micro-processors~\cite{heinz2019catalog}.
Because it is possible to formally verify the FPGA implementations of system-on-chip resources~\cite{sieh1997verify, breuer1997refinement}, including the softcore CPUs,
users do not have to blindly trust a whole proprietary CPU but just the FPGA configuration modules and verifiers.

\textbf{Customizing Enclaves Using FPGA.}
We present a hardware and software co-design framework to \underline{B}uild \underline{Y}our \underline{O}wn \underline{T}rusted \underline{E}xecution \underline{E}nvironments ({\scshape BYOTee}\xspace) on commodity FPGA devices.
Using the {\scshape BYOTee}\xspace toolchain, users can easily build multiple equally secure and customized enclaves on-demand to execute their Security-Sensitive Applications (SSA).
An enclave can include only the hardware and software functionality necessary for the SSA and exclude other hardware and software components on the system, minimizing the sizes of hardware and software TCB.

At the hardware layer, {\scshape BYOTee}\xspace utilizes commodity System-on-Chip (SoC) FPGA devices,
which integrate both hardcore CPU system, e.g., Cortex-A, RISC-V, etc., and FPGA programmable logic architectures.
The nature of FPGA enables on-demand configurations and reconfigurations of enclaves' hardware TCBs, which may include softcore CPUs, Block RAM based (BRAM; same as Static RAM on known SoC FPGA devices) main memory, and peripherals.
For example, an enclave may have a softcore A2I POWER CPU and a hardware GPU as its dedicated peripheral for acceleration.
Each enclave has its own isolated physical address space, which maps its own main memory, system configuration registers, and peripherals.
The software on the hardcore CPU and other enclaves cannot see an enclave's address space unless it is explicitly specified in the design.
By assigning the hardware resources to co-resident enclaves, {\scshape BYOTee}\xspace builds a multiprogramming environment, isolates software faults, and provides memory protection on FPGAs.
Because enclaves do not time-share any processor with each other and the hardcore system, the cache side-channel attack vector is removed.
Cold-boot attacks on these enclaves are also difficult due to the characteristics of BRAM.

FPGA alone, however, does not address all the issues of existing TEEs, and the SSAs need external libraries to execute and drivers to control the peripherals.
On the software front, the configurable {\scshape Firmware}\xspace component of {\scshape BYOTee}\xspace provides necessary software libraries, e.g., libc, and a Hardware Abstraction Layer (HAL), for a minimum software TCB.
Even though formally verifying the correctness and security of hardware designs, e.g., softcore CPU, is beyond our scope,
{\scshape BYOTee}\xspace offers mechanisms to bootstrap the trust and attest the integrity of the hardware implementations.
In particular, to bootstrap trust in enclaves statically and dynamically, {\scshape BYOTee}\xspace offers example paths building on the secure boot or a run-time trusted module, respectively.
Additionally, the {\scshape Firmware}\xspace offers two attestation mechanisms, namely pre-execution and post-execution attestations.
The former extends the traditional remote attestation of code and input to include hardware integrity, whereas the latter extends output data integrity attestation~\cite{abera2019diat}.

We implement a proof-of-concept {\scshape BYOTee}\xspace infrastructure and toolchain for the Xilinx SoC FPGA.
The toolchain includes {\scshape HardwareBuilder}\xspace, which takes developers' hardware resource need as input and automatically generates hardware modules and interconnections.
{\scshape HardwareBuilder}\xspace enables developers to focus on SSA development, increases the usability of {\scshape BYOTee}\xspace, and decreases the chances of developer-induced misconfigurations.
The system and toolchain also include {\scshape Firmware}\xspace, which provides a run-time environment, and {\scshape SSAPacker}\xspace, which encrypts and signs an SSA binary.
We demonstrate the usage of {\scshape BYOTee}\xspace with four example SSAs.
We emphasize none of the existing TEE solutions can meet all the security needs of the SSA-3, which is a secure music player with digital rights management.
We evaluate the security and performance of {\scshape BYOTee}\xspace with 12 benchmark applications and the four SSAs.
The evaluation results on the low-end Zynq-7000 system show {\scshape BYOTee}\xspace is effective in defeating attacks and efficient in executing SSAs.
The contributions of this paper are summarized as follows:

\begin{itemize}
\item We present {\scshape BYOTee}\xspace, a framework for building multiple general-purpose enclaves with configurable hardware and software TCBs utilizing commodity FPGA devices.
The idea of {\scshape BYOTee}\xspace can be implemented on FPGA systems from any vendor;

\item We present example paths to bootstrap trust in an enclave's hardware and software statically or dynamically. We design and implement two attestation mechanisms that capture the identifies of boot loaders, bitstream (hardware implementation), {\scshape Firmware}\xspace, and SSA;

\item We implement the {\scshape BYOTee}\xspace system and toolchain for the Xilinx SoC FPGA, which include the {\scshape HardwareBuilder}\xspace, {\scshape Firmware}\xspace, {\scshape SSAPacker}\xspace, and other helper tools, e.g., linker scripts, to facilitate developers to design, develop, and debug their SSAs.
\ifdefined
    We open-source the {\scshape BYOTee}\xspace system and toolchain\footnote{https://github.com/CactiLab/BYOTee-Build-Your-Own-TEEs};
\else
   To show the good faith of open-source, we put an anonymized copy of the source code online for artifact evaluations\footnote{https://anonymous.4open.science/r/byot-8011};
\fi

\item We choose the very low-end MicroBlaze softcore CPU and Zynq-7000 system to demonstrate {\scshape BYOTee}\xspace's usage, security, effectiveness, and performance with the Embench-IoT benchmark suite and four SSAs.
\end{itemize}

\begin{comment}
The rest of this paper is organized as follows.
In~\S\ref{s:back}, we introduce the background knowledge of SoC FPGA.
\S\ref{s:thread} presents the system and threat model of {\scshape BYOTee}\xspace.
\S\ref{s:arch} presents the architecture and workflow of the {\scshape BYOTee}\xspace framework and illustrates its components in detail.
\S\ref{sec:analysis} presents the security analysis.
In \S\ref{s:dev}, we discuss how to develop and build SSAs from developer's perspective and illustrate several examples of real-world applications that can be protected by {\scshape BYOTee}\xspace.
\S\ref{s:impleval} presents the implementation and evaluation results on the security and performance of {\scshape BYOTee}\xspace.
We discuss and compare {\scshape BYOTee}\xspace with related work in \S\ref{s:rel}.
\S\ref{s:conclusion} concludes this paper.
\end{comment}
\section{Future Research Directions}
\label{s:future}
After our current implementation of {\scshape BYOTee}\xspace, we realize there are several future research opportunities.
Some of the possible research directions are listed below:

\subsection{Compartmentalize Applications} Given a bare-metal  application, one future direction could be to compartmentalize them into different secure and non-secure modules.
The secure modules will be loaded onto the CTEEs dynamically, and the non-secure modules will execute on the PS side.
This automated process will make the system easier to use, and developers will develop one single application.
\subsection{Decouple {\scshape Firmware}\xspace from CTEEs} To reduce the software TCB further {\scshape Firmware}\xspace can be decoupled from the CTEE hardware.
Instead of individual {\scshape Firmware}\xspace for each CTEE, single {\scshape Firmware}\xspace to manage all the CTEEs will reduce the software TCB further.
\subsection{Dynamic Reconfiguration of CTEEs}
Using the PS interfaces such as DevC and PCAP, partial or full reconfiguration of the CTEEs will benefit from dynamic hardware TCB.
This dynamic hardware configurable feature for individual sensitive applications can further reduce the hardware TCB.
\end{comment}

\section{Acknowledgment}

We thank Paul Ratazzi, Zhenhua Wu, Kai Ni, and Arnob Paul for the discussions while this work was in progress.
We thank Sandro Pinto for bringing TEEOD to our attention.
We also thank the organizers of MITRE eCTF 2020 for inspiring this research.
This material is based upon work supported in part by the National Science Foundation under Grant No. 2037798 and by the Air Force Visiting Faculty Research Program.
Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of United States Government or any agency
thereof.

\bibliographystyle{abbrv}

\section{Related Work}
\label{s:rel}

\begin{table}[!tpb]
	\center
	\footnotesize
	\scalebox{1}{
	\begin{threeparttable}
	\begin{tabular}{l|p{3.2ex}|p{0.01ex}p{0.01ex}p{0.01ex}p{0.01ex}p{0.01ex}p{0.01ex}p{0.01ex}p{0.01ex}p{0.01ex}p{0.01ex}p{0.01ex}p{0.01ex}} \hline
	\multicolumn{1}{c}{Projects} & \multicolumn{1}{c}{} & \multicolumn{12}{c}{Benefits} \\ \hline\hline
	 & \rotatebox{90}{Underlying Hardware Primitive} & \rotatebox{90}{Multiple TEEs} & \rotatebox{90}{TEE with dedicated hardware} & \rotatebox{90}{Customizable CPU and Memory} & \rotatebox{90}{Customizable Peripheral Connections} & \rotatebox{90}{Minimum Software TCB}&\rotatebox{90}{Trust Bootstrap} & \rotatebox{90}{Remote Attestation} & \rotatebox{90}{Post-Execution Attestation} & \rotatebox{90}{Concurrent TEE and REE Execution} & \rotatebox{90}{Cache Side-channel Attack Resistant} & \rotatebox{90}{Cold-boot Attack Resistant} & \rotatebox{90}{Deployable on Commodity Devices} \\ \hline\hline
		Flickr~\cite{mccune2008flicker} & S/T &  & & &  & \cmark &\cmark & \cmark & \cmark & & & & \cmark \\
		TrustVisor~\cite{mccune2010trustvisor} & S/T & & &  &  & \cmark &\cmark& \cmark & \cmark & & & & \cmark \\
		Haven~\cite{haven} & X & \cmark & & & &  &\cmark& \cmark & \cmark &&&& \cmark \\
		SGXIO~\cite{weiser2017sgxio} & X+H & \cmark & & & \cmark & &  &&  &  &  &  & \cmark \\
		SGX-FPGA~\cite{xia2021sgx} & X+F & \cmark & & & \cmark && & \cmark &  &  &  &  & \cmark \\
		KeyStone~\cite{Keystone2020} & R & \cmark & & & & && \cmark & & & & &\cmark \\
		Sanctum~\cite{costan2016sanctum} & R & \cmark & &  &  \cmark & & & \cmark & \cmark & & \cmark & & \\
		{\scshape Cure}~\cite{bahmani2021cure} & R & \cmark & & & \cmark & & & & & & \cmark & & \cmark \\
		Composite Encl.~\cite{schneider2020pie} & R & \cmark & & & \cmark &&\cmark& \cmark & & & & & \cmark \\
		{\scshape Sanctuary}~\cite{brasser2019sanctuary} & A & \cmark & &  &  & \cmark && \cmark &  & & & & \cmark \\
		TrustICE~\cite{sun2015trustice} & A & \cmark & & &  & & & \cmark & \cmark &  & & & \cmark \\
		vTZ~\cite{hua2017vtz} & A+H & \cmark & & & && & &&&&&\cmark \\
		Ambassy~\cite{hwang2021ambassy} & A+F & & $-$ & $-$ & \cmark & $-$ && \cmark & & \cmark & \cmark & \cmark & \cmark \\
		uTango~\cite{oliveira2021utango} & M & \cmark & & & & && \cmark &  & & & & \cmark \\
		Graviton~\cite{volos2018graviton} & G & \cmark & & && &\cmark& \cmark & & \cmark & &\cmark& \\
		HECTOR-V~\cite{nasahl2020hector} & N & \cmark & \cmark & & \cmark & &&&& \cmark && \cmark &  \\
		TEEOD~\cite{teeod} & F & \cmark & \cmark &\cmark &  & \cmark& &&& \cmark &\cmark &  &\cmark \\
		\hline
		{\scshape BYOTee}\xspace & F & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark \\ \hline
	\end{tabular}
	\begin{tablenotes}
	\item[] S: AMD Secure Virtual Machine extension, T: Intel Trusted eXecution Technology, H: Hypervisor, A: Arm Cortex-A TrustZone, M: Arm Cortex-M TrustZone, X: Intel SGX, F: FPGA, R: RISC-V, G: GPU, N: New hardware design. $-$: not applicable.
	\end{tablenotes}
	\end{threeparttable}
	}
	\caption{Comparison with other TEE projects}
	\label{tab:comparison}
	\vspace{-0.4cm}
\end{table}

Many software or hardware solutions have been proposed to address one or more limitations of existing TEEs, but none tackle all the issues at the same time as {\scshape BYOTee}\xspace does.
Among them, TEEOD~\cite{teeod}, which Pereira et al. independently developed at roughly the same time, is most related to {\scshape BYOTee}\xspace.
Different from {\scshape BYOTee}\xspace, TEEOD implemented SSA loading, inter-enclave communications, etc., as hardware IPs.
{\scshape BYOTee}\xspace offers additional security features, such as trust bootstrap, attestation, etc.
Table~\ref{tab:comparison} lists the benefits {\scshape BYOTee}\xspace offers and compares related work with {\scshape BYOTee}\xspace.

To address the single TEE issue, vTZ~\cite{hua2017vtz} provides each guest virtual machine with a virtualized guest TEE by running a monitor within the secure world, which virtualizes memory mapping and world switching for the guest TEEs on Cortex-A TrustZone.
{\scshape Sanctuary}~\cite{brasser2019sanctuary} addresses the same drawback by utilizing the memory access controller to provide multi-domain isolation for sensitive applications.
TrustICE~\cite{sun2015trustice} creates multiple isolated
computing environments in the normal domain and runs a monitor in the secure world.
uTango~\cite{oliveira2021utango} use the secure attribution unit of Cortex-M to create multiple secure execution environments within the non-secure state.
The uTango kernel runs in the secure state privileged level, while other applications, services, OSes are isolated in their own non-secure state domains.
On RISC-V, KeyStone~\cite{Keystone2020} utilizes the Physical Memory Protection (PMP) feature to create multiple enclaves.
The TEE and REE in these solutions time-share the CPU and other hardware resources, resulting in side-channel attacks.

To enable isolated or trusted I/O paths between TEEs and peripherals, {\scshape Cure}~\cite{bahmani2021cure} enables the exclusive assignment of system resources, e.g., peripherals, CPU cores, or cache resources to single enclaves.
Composite Enclaves~\cite{schneider2020pie} builds on top of KeyStone and extends the TEE to several hardware components.
HECTOR-V~\cite{nasahl2020hector} uses a dedicated processor as a TEE with configurable peripheral access permissions for secure communication.
{\scshape Cure}, Composite Enclaves, and HECTOR-V were implemented on RISC-V.
SGXIO~\cite{weiser2017sgxio} presents a hypervisor-based trusted path architecture for Intel SGX.
SGX-FPGA~\cite{xia2021sgx} builds a secure hardware isolation path between CPU and FPGA.
To eliminate side-channel attacks,
Sanctum~\cite{costan2016sanctum} adopts isolation that combines minimally invasive hardware modifications with a trusted software security monitor on RISC-V.

There have also been attempts to utilize other hardware computing modules to build TEEs.
Graviton~\cite{volos2018graviton}, which requires some hardware change, enables applications to offload security and performance-sensitive kernels and data to a GPU, and execute them in isolation.
Ambassy~\cite{hwang2021ambassy} uses FPGA logic fabric to construct a second TEE and the secure world of TrustZone to control the secure loading of bitstream to the FPGA.
Different from {\scshape BYOTee}\xspace, Ambassy does not include softcore CPUs for developers to execute arbitrary code.
Other off-SoC dedicated processor solutions, such as Google Titan~\cite{johnson2018titan}, Samsung eSE~\cite{samsunGeSE}, and Apple SEP~\cite{appleSEP} uses external connections for communication between the REE and TEE, making them vulnerable to physical probing attacks.
{\scshape BYOTee}\xspace is also inspired by other isolated execution environment solutions, including Flickr~\cite{mccune2008flicker}, TrustVisor~\cite{mccune2010trustvisor}, and Haven~\cite{haven}, and other hypervisor-based protections, such as Overshadow~\cite{chen2008overshadow}, InkTag~\cite{hofmann2013inktag}, AppSec~\cite{ren2015appsec}, and Terra~\cite{garfinkel2003terra}.

\section{System, Threat and Deployment Model}
\label{s:thread}

\textbf{System Model.}
We assume the DRAM can be configured to connect to both of the hardcore system and FPGA, and peripherals can be connected to the FPGA without routing through the hardcore system.
The former enables the hardcore system and FPGA modules to communicate efficiently via shared memory, and the latter makes sure the software on the hardcore system cannot eavesdrop or tamper the data between the FPGA and peripherals.
We assume the hardcore system and Direct Memory Access (DMA) masters cannot dump the content in the FPGA.
All of the assumptions are realistic in that they are the standard configurations on most commercial systems~\cite{parno2011bootstrapping}.
Even though recent research~\cite{ender2020unpatchable} discovered dumping content from the FPGA is possible due to some implementation bugs, it was not intended to be a feature.
We assume secure or trusted boot, which loads the software images onto the hardcore system and the bitstream and {\scshape Firmware}\xspace onto the FPGA after integrity checks at boot-time.
We assume the cryptographic algorithms are secure.

\textbf{Threat Model.} We assume the adversary can compromise and take full control of the software system of the hardcore system at run-time, which means the user developed Untrusted Applications (UA), the kernel, and even privileged software in the secure world of a TrustZone-based system can be malicious.
The compromised software on the hardcore system can send arbitrary data to the {\scshape Firmware}\xspace and SSAs via shared DRAM regions and also to the enclave hardware pins, such as interrupts.
Therefore, the {\scshape Firmware}\xspace and SSA in an enclave can be compromised at run-time.
Attackers can also perform cold-boot attacks to dump the content in DRAM.

\textbf{Deployment and Key Management Models.}
{\scshape BYOTee}\xspace does not aim to address any specific deployment and key management model, as it is related to the user and application's policies and needs.
Instead, {\scshape BYOTee}\xspace provides the mechanisms required to build a secure system.

We nevertheless discuss two examples for local and cloud deployment and key management.
At the cryptographic layer, we assume each device has a device key,  e.g., AES ($k_{d}$), RSA ($sk_{d}, pk_{d}$), which are used to encrypt/decrypt and sign/verify software images, bitstream, and the {\scshape Firmware}\xspace.
The device keys are unique to each device and are programmed in the eFUSE or BBRAM using hardware interfaces with physical access.
We assume a developer can be identified by a developer key, e.g., RSA ($sk_{u}, pk_{u}$), which the developer uses to encrypt and sign the SSAs, and the {\scshape Firmware}\xspace uses it to decrypt and verify the SSAs.
Developer public keys are either embedded into the {\scshape Firmware}\xspace at the development stage or loaded into the {\scshape Firmware}\xspace at run-time.

An example solution for local deployment is that the device owner programs the device keys into the one-time programmable eFUSE, e.g.,  through the JTAG interface on a development board~\cite{xilinxSecure}.
The {\scshape Firmware}\xspace developer embeds the allowed the developers' public keys inside the {\scshape Firmware}\xspace at the development stage.
If SSA developers need to register or update a key, they will send the key public keys to the {\scshape Firmware}\xspace developers.

An example solution for cloud deployment is that the manufacturer programs the device keys into the one-time programmable eFUSE same as Intel creates root keys for any SGX-enabled CPUs.
The {\scshape Firmware}\xspace developers generate {\scshape Firmware}\xspace and bitstream, and they also encrypt them using the public part of the device key and embed developers' public keys in the {\scshape Firmware}\xspace.
The protected {\scshape Firmware}\xspace and bitstream along with the UA are sent to the untrusted cloud provider.
The developers' keys update process is similar to the process for local deployment.

\section{Related Work}
\label{s:rel}

\begin{table}[!tpb]
	\center
	\footnotesize
	\scalebox{1}{
	\begin{threeparttable}
	\begin{tabular}{l|p{3.2ex}|p{0.01ex}p{0.01ex}p{0.01ex}p{0.01ex}p{0.01ex}p{0.01ex}p{0.01ex}p{0.01ex}p{0.01ex}p{0.01ex}p{0.01ex}p{0.01ex}} \hline
	\multicolumn{1}{c}{Projects} & \multicolumn{1}{c}{} & \multicolumn{12}{c}{Benefits} \\ \hline\hline
	 & \rotatebox{90}{Underlying Hardware Primitive} & \rotatebox{90}{Multiple TEEs} & \rotatebox{90}{TEE with dedicated hardware} & \rotatebox{90}{Customizable CPU and Memory} & \rotatebox{90}{Customizable Peripheral Connections} & \rotatebox{90}{Minimum Software TCB}&\rotatebox{90}{Trust Bootstrap} & \rotatebox{90}{Remote Attestation} & \rotatebox{90}{Post-Execution Attestation} & \rotatebox{90}{Concurrent TEE and REE Execution} & \rotatebox{90}{Cache Side-channel Attack Resistant} & \rotatebox{90}{Cold-boot Attack Resistant} & \rotatebox{90}{Deployable on Commodity Devices} \\ \hline\hline
		Flickr~\cite{mccune2008flicker} & S/T &  & & &  & \cmark &\cmark & \cmark & \cmark & & & & \cmark \\
		TrustVisor~\cite{mccune2010trustvisor} & S/T & & &  &  & \cmark &\cmark& \cmark & \cmark & & & & \cmark \\
		Haven~\cite{haven} & X & \cmark & & & &  &\cmark& \cmark & \cmark &&&& \cmark \\
		SGXIO~\cite{weiser2017sgxio} & X+H & \cmark & & & \cmark & &  &&  &  &  &  & \cmark \\
		SGX-FPGA~\cite{xia2021sgx} & X+F & \cmark & & & \cmark && & \cmark &  &  &  &  & \cmark \\
		KeyStone~\cite{Keystone2020} & R & \cmark & & & & && \cmark & & & & &\cmark \\
		Sanctum~\cite{costan2016sanctum} & R & \cmark & &  &  \cmark & & & \cmark & \cmark & & \cmark & & \\
		{\scshape Cure}~\cite{bahmani2021cure} & R & \cmark & & & \cmark & & & & & & \cmark & & \cmark \\
		Composite Encl.~\cite{schneider2020pie} & R & \cmark & & & \cmark &&\cmark& \cmark & & & & & \cmark \\
		{\scshape Sanctuary}~\cite{brasser2019sanctuary} & A & \cmark & &  &  & \cmark && \cmark &  & & & & \cmark \\
		TrustICE~\cite{sun2015trustice} & A & \cmark & & &  & & & \cmark & \cmark &  & & & \cmark \\
		vTZ~\cite{hua2017vtz} & A+H & \cmark & & & && & &&&&&\cmark \\
		Ambassy~\cite{hwang2021ambassy} & A+F & & $-$ & $-$ & \cmark & $-$ && \cmark & & \cmark & \cmark & \cmark & \cmark \\
		uTango~\cite{oliveira2021utango} & M & \cmark & & & & && \cmark &  & & & & \cmark \\
		Graviton~\cite{volos2018graviton} & G & \cmark & & && &\cmark& \cmark & & \cmark & &\cmark& \\
		HECTOR-V~\cite{nasahl2020hector} & N & \cmark & \cmark & & \cmark & &&&& \cmark && \cmark &  \\
		TEEOD~\cite{teeod} & F & \cmark & \cmark &\cmark &  & \cmark& &&& \cmark &\cmark &  &\cmark \\
		\hline
		{\scshape BYOTee}\xspace & F & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark \\ \hline
	\end{tabular}
	\begin{tablenotes}
	\item[] S: AMD Secure Virtual Machine extension, T: Intel Trusted eXecution Technology, H: Hypervisor, A: Arm Cortex-A TrustZone, M: Arm Cortex-M TrustZone, X: Intel SGX, F: FPGA, R: RISC-V, G: GPU, N: New hardware design. $-$: not applicable.
	\end{tablenotes}
	\end{threeparttable}
	}
	\caption{Comparison with other TEE projects}
	\label{tab:comparison}
	\vspace{-0.4cm}
\end{table}

Many software or hardware solutions have been proposed to address one or more limitations of existing TEEs, but none tackle all the issues at the same time as {\scshape BYOTee}\xspace does.
Among them, TEEOD~\cite{teeod}, which Pereira et al. independently developed at roughly the same time, is most related to {\scshape BYOTee}\xspace.
Different from {\scshape BYOTee}\xspace, TEEOD implemented SSA loading, inter-enclave communications, etc., as hardware IPs.
{\scshape BYOTee}\xspace offers additional security features, such as trust bootstrap, attestation, etc.
Table~\ref{tab:comparison} lists the benefits {\scshape BYOTee}\xspace offers and compares related work with {\scshape BYOTee}\xspace.

To address the single TEE issue, vTZ~\cite{hua2017vtz} provides each guest virtual machine with a virtualized guest TEE by running a monitor within the secure world, which virtualizes memory mapping and world switching for the guest TEEs on Cortex-A TrustZone.
{\scshape Sanctuary}~\cite{brasser2019sanctuary} addresses the same drawback by utilizing the memory access controller to provide multi-domain isolation for sensitive applications.
TrustICE~\cite{sun2015trustice} creates multiple isolated
computing environments in the normal domain and runs a monitor in the secure world.
uTango~\cite{oliveira2021utango} use the secure attribution unit of Cortex-M to create multiple secure execution environments within the non-secure state.
The uTango kernel runs in the secure state privileged level, while other applications, services, OSes are isolated in their own non-secure state domains.
On RISC-V, KeyStone~\cite{Keystone2020} utilizes the Physical Memory Protection (PMP) feature to create multiple enclaves.
The TEE and REE in these solutions time-share the CPU and other hardware resources, resulting in side-channel attacks.

To enable isolated or trusted I/O paths between TEEs and peripherals, {\scshape Cure}~\cite{bahmani2021cure} enables the exclusive assignment of system resources, e.g., peripherals, CPU cores, or cache resources to single enclaves.
Composite Enclaves~\cite{schneider2020pie} builds on top of KeyStone and extends the TEE to several hardware components.
HECTOR-V~\cite{nasahl2020hector} uses a dedicated processor as a TEE with configurable peripheral access permissions for secure communication.
{\scshape Cure}, Composite Enclaves, and HECTOR-V were implemented on RISC-V.
SGXIO~\cite{weiser2017sgxio} presents a hypervisor-based trusted path architecture for Intel SGX.
SGX-FPGA~\cite{xia2021sgx} builds a secure hardware isolation path between CPU and FPGA.
To eliminate side-channel attacks,
Sanctum~\cite{costan2016sanctum} adopts isolation that combines minimally invasive hardware modifications with a trusted software security monitor on RISC-V.

There have also been attempts to utilize other hardware computing modules to build TEEs.
Graviton~\cite{volos2018graviton}, which requires some hardware change, enables applications to offload security and performance-sensitive kernels and data to a GPU, and execute them in isolation.
Ambassy~\cite{hwang2021ambassy} uses FPGA logic fabric to construct a second TEE and the secure world of TrustZone to control the secure loading of bitstream to the FPGA.
Different from {\scshape BYOTee}\xspace, Ambassy does not include softcore CPUs for developers to execute arbitrary code.
Other off-SoC dedicated processor solutions, such as Google Titan~\cite{johnson2018titan}, Samsung eSE~\cite{samsunGeSE}, and Apple SEP~\cite{appleSEP} uses external connections for communication between the REE and TEE, making them vulnerable to physical probing attacks.
{\scshape BYOTee}\xspace is also inspired by other isolated execution environment solutions, including Flickr~\cite{mccune2008flicker}, TrustVisor~\cite{mccune2010trustvisor}, and Haven~\cite{haven}, and other hypervisor-based protections, such as Overshadow~\cite{chen2008overshadow}, InkTag~\cite{hofmann2013inktag}, AppSec~\cite{ren2015appsec}, and Terra~\cite{garfinkel2003terra}.

\section{System, Threat and Deployment Model}
\label{s:thread}

\textbf{System Model.}
We assume the DRAM can be configured to connect to both of the hardcore system and FPGA, and peripherals can be connected to the FPGA without routing through the hardcore system.
The former enables the hardcore system and FPGA modules to communicate efficiently via shared memory, and the latter makes sure the software on the hardcore system cannot eavesdrop or tamper the data between the FPGA and peripherals.
We assume the hardcore system and Direct Memory Access (DMA) masters cannot dump the content in the FPGA.
All of the assumptions are realistic in that they are the standard configurations on most commercial systems~\cite{parno2011bootstrapping}.
Even though recent research~\cite{ender2020unpatchable} discovered dumping content from the FPGA is possible due to some implementation bugs, it was not intended to be a feature.
We assume secure or trusted boot, which loads the software images onto the hardcore system and the bitstream and {\scshape Firmware}\xspace onto the FPGA after integrity checks at boot-time.
We assume the cryptographic algorithms are secure.

\textbf{Threat Model.} We assume the adversary can compromise and take full control of the software system of the hardcore system at run-time, which means the user developed Untrusted Applications (UA), the kernel, and even privileged software in the secure world of a TrustZone-based system can be malicious.
The compromised software on the hardcore system can send arbitrary data to the {\scshape Firmware}\xspace and SSAs via shared DRAM regions and also to the enclave hardware pins, such as interrupts.
Therefore, the {\scshape Firmware}\xspace and SSA in an enclave can be compromised at run-time.
Attackers can also perform cold-boot attacks to dump the content in DRAM.

\textbf{Deployment and Key Management Models.}
{\scshape BYOTee}\xspace does not aim to address any specific deployment and key management model, as it is related to the user and application's policies and needs.
Instead, {\scshape BYOTee}\xspace provides the mechanisms required to build a secure system.

We nevertheless discuss two examples for local and cloud deployment and key management.
At the cryptographic layer, we assume each device has a device key,  e.g., AES ($k_{d}$), RSA ($sk_{d}, pk_{d}$), which are used to encrypt/decrypt and sign/verify software images, bitstream, and the {\scshape Firmware}\xspace.
The device keys are unique to each device and are programmed in the eFUSE or BBRAM using hardware interfaces with physical access.
We assume a developer can be identified by a developer key, e.g., RSA ($sk_{u}, pk_{u}$), which the developer uses to encrypt and sign the SSAs, and the {\scshape Firmware}\xspace uses it to decrypt and verify the SSAs.
Developer public keys are either embedded into the {\scshape Firmware}\xspace at the development stage or loaded into the {\scshape Firmware}\xspace at run-time.

An example solution for local deployment is that the device owner programs the device keys into the one-time programmable eFUSE, e.g.,  through the JTAG interface on a development board~\cite{xilinxSecure}.
The {\scshape Firmware}\xspace developer embeds the allowed the developers' public keys inside the {\scshape Firmware}\xspace at the development stage.
If SSA developers need to register or update a key, they will send the key public keys to the {\scshape Firmware}\xspace developers.

An example solution for cloud deployment is that the manufacturer programs the device keys into the one-time programmable eFUSE same as Intel creates root keys for any SGX-enabled CPUs.
The {\scshape Firmware}\xspace developers generate {\scshape Firmware}\xspace and bitstream, and they also encrypt them using the public part of the device key and embed developers' public keys in the {\scshape Firmware}\xspace.
The protected {\scshape Firmware}\xspace and bitstream along with the UA are sent to the untrusted cloud provider.
The developers' keys update process is similar to the process for local deployment.

redpajama_set_name: RedPajamaArXiv