extfieldsofstudy:
Computer Science
Engineering
Mathematics

text:
Sensor-Augmented Neural Adaptive Bitrate Video Streaming on UAVs

Recent advances in unmanned aerial vehicle (UAV) technology have revolutionized a broad class of civil and military applications. However, the designs of wireless technologies that enable real-time streaming of high-definition video between UAVs and ground clients present a conundrum. Most existing adaptive bitrate (ABR) algorithms are not optimized for the air-to-ground links, which usually fluctuate dramatically due to the dynamic flight states of the UAV. In this paper, we present SA-ABR, a new sensor-augmented system that generates ABR video streaming algorithms with the assistance of various kinds of inherent sensor data that are used to pilot UAVs. By incorporating the inherent sensor data with network observations, SA-ABR trains a deep reinforcement learning (DRL) model to extract salient features from the flight state information and automatically learn an ABR algorithm to adapt to the varying UAV channel capacity through the training process. SA-ABR does not rely on any assumptions or models about UAV's flight states or the environment, but instead, it makes decisions by exploiting temporal properties of past throughput through the long short-term memory (LSTM) to adapt itself to a wide range of highly dynamic environments. We have implemented SA-ABR in a commercial UAV and evaluated it in the wild. We compare SA-ABR with a variety of existing state-of-the-art ABR algorithms, and the results show that our system outperforms the best known existing ABR algorithm by 21.4% in terms of the average quality of experience (QoE) reward.

I. INTRODUCTION
Inexpensive commercially available unmanned aerial vehicles (UAVs) are rising rapidly, making drones a popular host of a wide class of applications, including environment monitoring [1], precision agriculture [2], photography [3], policing [4], firefighting [5], and package delivery [6]. An essential functionality enabling these applications is to record high-definition videos of superior quality and seamlessly share them with ground base stations or clients for manual inspection and further analysis.
Despite the excitement, today's UAVs are struggling to deliver high-quality video in real time to ground receivers. Today's commercial UAVs adopt fixed-bitrate video streaming strategies which may result in severe rebuffering under poor X. Xiao, W. Wang channel conditions [7]. In addition, many studies have adopted various kinds of adaptive bitrate (ABR) algorithms [8]- [30], including learning methods [20]- [30], under various network conditions on the ground. These algorithms make ABR decisions based on network observations and video playback states. However, it is challenging for them to fit well in UAV communications, as the channel capacity of air-to-ground links fluctuate dramatically and the primary causes come from factors including varying environments and dynamic motion states, such as flight velocities, intense vibrations and distances from the ground clients. These factors result in unique patterns in the channel capacity variances, which can hardly be learned from models built for ground-to-ground links. Consequently, the majority of ABR algorithms either fail to transmit higherquality video streaming [9] or exceed the channel capacity [11] due to unexpected situations. To break this stalemate, dedicated models tailored for air-to-ground links are required to cope with such unique variance patterns.
Instead of solely relying on the network observations and video playback states which are not sufficient enough to adapt to the highly dynamic air-to-ground links, we argue to incorporate more inherent sensor data that can reflect UAV flight states in the ABR algorithm. Through field tests and measurements, we observe that the UAV's flight-state-related sensor data can provide hints about channel variance patterns, which can guide the design of ABR algorithms. In addition, the temporal patterns in the past throughput variances can be further extracted and exploited to obtain valuable information about current channel conditions. Thus, we believe it is essential to incorporate the flight-state-related sensor data as side information to design video streaming strategies for UAVs.
In this work, we present SA-ABR, a new sensor-augmented system that generates ABR video streaming algorithms based on deep reinforcement learning (DRL) [31], [32], which aims at obtaining optimal bitrate selection strategies under varying UAV channel conditions. The goal of SA-ABR is to maximize the video quality of experience (QoE) [33], [34] for viewers through the training process. As illustrated in Fig. 1, apart from the state information of past throughput experience and video playback, SA-ABR also feeds the inherent sensor data, including GPS coordinates, acceleration and velocity, which indicates the flight state of the UAV, into the neural network. The sensor data is updated in real time at the beginning of each video chunk. Additionally, the average throughput of past video chunks is calculated by recording the number of packets. The focus of our model is on how to capture the salient features from the time-series throughput sequences and make full use of the sensor data to better estimate the future throughput trend. After the training process, our model can automatically adapt to the throughput dynamics and make optimal bitrate decisions for the next video chunks.
A key challenge in our design is how to make full use of the sensor data to extract features that are indicative of throughput dynamics. Although there are some relations between the sensor value and the throughput, it is prohibitively complex to describe these relations in analytical expressions or clear rules. For example, the acceleration pattern is generally very misleading due to the existence of the UAVs' own vibrations, making it intractable for the DRL model to distinguish whether the acceleration data comes from the change of flight states or vibrations. To overcome this hurdle, SA-ABR applies quantization-based preprocessing to sensor data before directly feeding it to the neural network. This not only ensures the full use of the sensor data that provide hints about channel conditions, but also eliminates irrelevant noise and disturbance. Another obstacle stems from how to effectively analyze the temporal characteristics of throughput when making predictions about future throughput. We incorporate the long short-term memory network (LSTM) [35] into the DRL model, which exploits the unique memory function of the LSTM to better capture the temporal properties and improve the accuracy of throughput forecasts.
We implement SA-ABR on a DJI Matrice 100 and compare the performance of our system with state-of-the-art ABR algorithms in the wild. The results show that SA-ABR achieves up to a 21.4% gain in the average QoE reward over the best known existing ABR algorithm [24].
The main contributions of this paper are summarized as follows.
• We conduct a comprehensive measurement study to explore the impact of UAV's motions on throughput, which provides hints to optimize the video streaming strategies from the flight-state perspective. • We propose a new DRL-based ABR architecture with the assistance of the sensor data. The model exploits the inherent sensor data that is used to pilot UAVs to better adapt to the highly dynamic air-to-ground links. • We implement our design on a commercial UAV and conduct a series of experiments in the wild to validate our system. The results demonstrate the feasibility of adapting to the air-to-ground channel dynamics, resulting in a 21.4% increase in the average QoE reward compared to the best known existing ABR algorithm. The remainder of this paper is structured as follows. We begin in Section II to explore the impact of the UAV's flight states. Section III describes the system design of our sensoraugmented DRL algorithm. Section IV describes the system implementation and Section V evaluates the performance of SA-ABR. Sections VI and VII present the related work and conclusion.

II. EXPLOITING FIGHT STATE AWARENESS
In this section, we start by giving a brief introduction of the flight-state-related sensor data available on UAVs. Then, we conduct experiments in controlled flight states to reveal the specific relationship between throughput and the sensor data. Finally, we proceed to take a deeper inspection of the complex relationship and summarize counter-intuitive observations and irregular phenomena, which motivates our DRL-based design.

A. Flight States of the UAV
UAVs such as multi-rotor drones need to identify their flight states at all times, including 3D position, 3D orientation, and their derivatives. Therein, the positions and velocities are important data for the UAV to determine whether it is hovering or moving. As UAVs normally adjust the postures to generate thrusts in certain directions, they also need to measure their own current 3D orientations in real time. Together with corresponding accelerations and angular velocities, there are a total of fifteen state quantities that are required to identify the flight states.
With various sensor technologies, including GPS, inertial measurement unit (IMU), barometer and geomagnetic compass, equipped on UAVs, we can obtain all fifteen state quantities that comprehensively reflect the flight conditions. To explore how the flight states affect the air-to-ground channel conditions, we collect the sensor data, including GPS coordinates, velocities, accelerations and further analyze the relationship between sensor data and throughput.

B. Impact of Flight States
As the capability of throughput forecast plays an essential role in the ABR video streaming algorithm to make bitrate selection meeting viewers' QoE requirements, we start with a series of tests to analyze the underlying relationship between the sensor data and the throughput in controlled flight states. In these tests, the transmitter is a DJI Matrice 100 UAV and sends the data file through the IEEE 802.11n protocol. The ground receiver uses the commercial WiFi network card with a USB interface embedded on the laptop.
The sites of measurements are selected to validate the generalization of our observations, including a playground, a plaza and a pool on campus, each with its own channel characteristics that cover a majority of different scenarios. The air-toground propagation on the playground can be described as an ordinary two-ray model [36]. Different from the unobstructed playground, the plaza is surrounded by buildings and trees. Under such circumstances, the propagation is affected by the    shadowing losses and the multipath diversities. Moreover, the surface reflectivity and roughness of the pool are different from the ground, resulting in varying parameters in propagation models. For each flight state, we collect data lasting more than 150 s in each place. Impact of distance. We perform a set of tests to analyze the impact of the distance between the ground client and the UAV on throughput. This set of experiments includes two groups of measurements according to the UAV's flight states, i.e., the hovering and moving states. In the hovering experiment, we collect throughput and sensor data when the UAV hovers at a height of 10 m and the link distances vary from 10 m to 60 m. In the moving experiment, we consider a higher altitude of 20 m for safety reasons, with various distances ranging from 20 m to 60 m. In addition, the UAV is controlled to fly at different distances at a constant velocity of 8 m/s.
The experimental results are presented in Fig. 2. From the mediums, the quartiles and the ends of each boxplot, we can conclude that the throughput decreases with distances no matter whether the UAV is moving or hovering. This rule provides a basis for SA-ABR to incorporate the distance value that affects the throughput, into the algorithm.
Impact of velocity. The experiments with different velocities are also conducted by dividing the testing process into two sets: one is at a distance of around 20 m and the other is around 50 m. We control the UAV to fly around the ground client at stable distances while the velocity sweeps from 0 to 16 m/s.
The throughputs achieved at different velocity ranges (in-cluding 0-4 m/s, 4-8 m/s, 8-12 m/s, >12 m/s) are shown in Fig. 3. The throughput diminishes quickly when the velocity range increases, no matter whether the link distance is around 20 m or 50 m. Based on the impact of velocity on throughput, we can exploit the velocity data in our algorithm design to better forecast throughput. Impact of acceleration. To obtain a better bitrate adaptation model, we further perform tests and investigate the impact of acceleration. The acceleration data includes three dimensions: a x , a y , a z . During the flight, the UAV continuously accelerates and decelerates while the distance from the ground client remains stable.
The results are demonstrated in Fig. 4. For the acceleration data, the low value is dominated by the UAV vibrations, while the high value can indicate the changes in flight states. Thus, we can pick out the high values of acceleration data to analyze their impact on throughput. Note that the variances in acceleration and velocity are synchronous to some extent, which may compromise the impact of acceleration data on throughput prediction. Nevertheless, when the acceleration increases rapidly at 8s, the velocity value has not reached large enough to determine if the channel quality becomes worse. Therefore, the utilization of acceleration data can make up for this weakness in our algorithm and serve as a supplement to velocity data to perform better throughput forecast.
Summary. The general relationships between the sensor data and the throughput in the controlled flight states can be summarized as follows: (1) the throughput monotonically decreases as the distance and the speed increase, and (2) the acceleration fluctuations are able to reflect the throughput changes, i.e., the larger the variance in acceleration data, the more likely the throughput will be affected. These rules exist due to complex latent factors including the path loss, the multipath effects, the modulation coding scheme (MCS) conversions and the fast changes in environments caused by the high-speed movements of the UAV. Thus, the flight-staterelated sensor data is indicative of channel state information, which provides hints for SA-ABR to enhance the forecast capability and achieve the best QoE from the flight-state perspective.

C. Deeper Inspection
The above experiments are conducted with preset or controlled flight states. In this part, we proceed to conduct experiments where the UAV is allowed to fly on random paths at arbitrary speeds in various environments (see Fig. 8), to take a deeper investigation of the relationship between throughput and sensor data. The analyses include three aspects: throughput vs. distance, velocity, and acceleration, respectively.
The detailed results of throughput vs. distance and throughput vs. velocity are visualized in Fig. 5. We first observe the general rule in Fig. 5(a), i.e., the throughput decreases with velocities. However, the throughput value at lower distances covers almost the entire range (0-16 Mbps), as indicated in the bottom and top edges of the boxplots. The reason for this phenomenon may be the uncertainties caused by the rapidly changing velocities of the UAV in the wild. In addition, the cumulative distribution function (CDF) in Fig. 5(b) further demonstrates that for lower distances, the throughput is more likely to be distributed in the higher numerical area and less in the lower area, while the throughput at high distances follows an opposite pattern. This phenomenon transforms the regular impact characteristics of distances into a probabilistic problem with uncertainty due to the changing velocities, which is more complex than what we observed in previous experiments. Similar phenomena are also observed in the experiments regarding velocity (see Fig. 5(c) and Fig. 5(d)).
After deeper inspection, some counter-intuitive observations can also be noted in the acceleration data. For example, during the deceleration phase (see Fig. 4 at around 38 s), the velocity curve drops and the throughput shows an upward trend. The acceleration data, however, shows a sharp fluctuation due to the sudden change in velocity. These irregular phenomena occasionally happen when the impact of acceleration data on throughput contends with other influencing factors, such as distance and velocity, which might result in a misjudgment for the throughput predictor.
Summary. Based on these irregular phenomena and counter-intuitive observations, we observe that the monotonicity in relationships between sensor data and throughput is not as clear as the experiments in controlled flight states. For example, the UAVs at low velocities do not necessarily lead to high-quality wireless channel due to the dominant influence of path loss at long link distance. In other words, various sensor data comprehensively affects the throughput, which makes the conventional ABR algorithms prohibitively complex to describe these relations in analytical expressions. This problem motivates us to exploit neural networks without  relying on preconfigured analytical expressions. Through the training process, the neural networks can gradually learn the experience to cope with the irregular phenomena and extract effective features from these complex relations to improve forecast capability.

III. LEARNING SENSOR-AUGMENTED ALGORITHMS
SA-ABR generates ABR algorithms based on the DRL model and LSTM network. The objective of SA-ABR is to maximize the expected rewards of user feedback over the whole video. Thus, instead of simply using neural networks to emphasize each temporal step of video playback, DRL enables SA-ABR to focus on the overall performance and generates the optimal bitrate selection policy over the whole video sequence. Furthermore, SA-ABR can take advantages of DRL to essentially improve the bitrate selection mechanism by forcing the agent to automatically learn better strategies without manual configuration about the throughput traces and video states.
We first design a training methodology to faithfully model the dynamics of video streaming in client applications, which accelerates the training process. Then, we introduce the quantization-based preprocessing performed on sensor data before directly feeding it to the networks. Finally, we characterize the DRL training process in various aspects. As shown in Fig. 6, the training algorithm and the networks are established based on the DRL policy. The networks receive a variety of inputs (e.g., the video states, the sensor data) through data sampling process and output bitrate selections for future chunks. The reward is an assessment of video quality, which motivates the network parameters constantly updated to achieve better video quality. The DRL training process is implemented as a prior task and our ABR model is running on the ground client.

A. Training Methodology
The first step is to design a training methodology that faithfully models the dynamics of video streaming. In fact, using actual video clients for training need to employ a video player to continuously download videos and observe video state changes. This process is very time-consuming since the download time is long and the model cannot be updated until all video chunks are downloaded. For example, the video in our experiment is set to have 41 chunks, with each chunk lasting two seconds. That is to say, the model needs to wait tens of seconds before being updated.
In order to save the video download time and accelerate the training process, our training methodology directly calculates the new video states (including the buffer occupancy and the rebuffering time) based on current information. The buffer occupancy, shown as the progress bar of video players, indicates how long the video can continue to play. Moreover, video transmissions from UAVs are mainly live streaming, which may have a time-to-live requirement of 15-45 s. This bounds the buffer level that video players can build [17]. Thus, we set the maximum buffer boundary and the maximum rebuffering time to 20 s. Once the buffer level or rebuffering time exceeds this boundary, our model will be punished and obtain a very low QoE reward. Additionally, there exists a half of the roundtrip-time (RTT) delay derived from the video requests sent by the client to the UAV. However, the delay is below the millisecond level which has been proved to have minimal impact on the chunk-level ABR systems [24].
At the end of each video chunk t's download, the update of video states can be divided into two cases. First, if the download time f t of the video chunk t is less than the beginning buffer value b t (b t < 20 s), no rebuffering will happen. We update the new buffer size b t+1 by subtracting the download time f t , and then adding a two-second duration of one chunk t chunk . The updated buffer size b t and the rebuffering time T t can be expressed as Therein, the download time f t solely relies on the video chunk's bitrate selection l t and network throughput traces x trace,t . The throughput traces x trace,t employed are collected in advance by keeping track of the wireless channel quality of UAVs in the wild. The available video bitrates in our experiment are {300, 750, 1850, 2850} Kbps that correspond to video types of {240, 360, 720, 1080} p.
For the second case where the download time f t of the current video chunk exceeds the beginning buffer value b t , the rebuffering occurs. In these scenarios, our model is configured to wait for 500 ms before retrying to request the playback of next video chunk. In addition, the video chunks cannot be played until they are downloaded. Thus, the buffer size and rebuffering time are updated by b t+1 = t chunk , After each chunk is downloaded, SA-ABR passes new observations of video states, including the buffer size b t+1 and rebuffering time T t , to the RL agent. The agent then assesses the video QoE and obtains the reward to periodically update the policy. Guided by the policy, the agent makes the next bitrate decisions based on the received states. Then, new video states are regenerated by Eq. (1)-(2) for the next round of update.

B. Preprocessing sensor data
Recall that the general relationships between the inherent sensor data and the throughput are summarized in Section II. We incorporate the sensor data into SA-ABR to provide hints about channel variance patterns. However, the raw sensor data (such as the acceleration patterns, shown in Fig. 4) keeps fluctuating. These noises, incurred by UAV vibrations, have little impact on throughput, while the resulting uncertainty in sensor data can interfere with the model's prediction of future throughput. Moreover, when the link distance is relatively short, the path loss is dominated by the constructive and destructive interference caused by the multipath effect [37]. That means, the distance increment within a closer area has little impact on throughput (shown in Fig. 2(a)). However, such a distance change may interfere with the model's throughput forecast. Therefore, in order to ensure the full use of the sensor data that is indicative of throughput dynamics while eliminating these noises and disturbances, we quantize the sensor data according to the degrees of its impact on throughput, instead of directly feeding it to the neural network.
The quantization scheme is designed as follows. Recall that the available video bitrates are {300, 750, 1850, 2850} Kbps. Thus, the throughput for video transmission can be roughly divided into <0.75 Mbps, 0.75-1.85 Mbps, 1.85-2.85 Mbps and >2.85 Mbps, each of which can satisfy the corresponding video bitrate, with an average interval of 0.95 Mbps. That is to say, when the throughput drops by 0.95 Mbps, the optimal bitrate selection may change. Then, we sort the throughput in ascending order of the corresponding sensor data and move a sliding window to obtain the average throughput. The experimental data is from Section II.C and the UAV is allowed to fly on random paths at arbitrary speeds. When the average throughput drops by 0.95 Mbps, the corresponding sensor value is set as a quantization threshold. This quantization process is for the overall value of the sensor data, and it is not necessary to consider the partitions of the 3D state space. After traversing the entire experimental data corpus, we can obtain the quantization results as follows.
• Preprocessing distance data. We encode distances over 50 m into "1", and distances within 50 m into "0". • Preprocessing velocity data. The first level "0" represents velocity below 8 m/s, and the second level "1" indicates velocity within 8-12 m/s, while the third "2" is velocity over 12 m/s. • Preprocessing acceleration data. The acceleration data which exceeds 18 m/s 2 is marked as "1", while the other is marked as "0".

C. DRL training process
We use the Advantage Actor-Critic [38], [39] method to generate ABR algorithms with the assistance of sensor data and LSTM networks. The training algorithm is based on onpolicy policy. Although both generated by DRL algorithm, SA-ABR is very different from Pensieve [24]. SA-ABR first incorporates the sensor data as input to accommodate the changes in the UAV channel. Then, it quantizes the sensor data to eliminate the noise and disturbance in the UAV channel. Moreover, it also explores the role of the LSTM network in analyzing past throughput sequences and predicting future changes. To summarize, SA-ABR takes into account the flight states of UAVs and is optimized to improve QoE in UAV video streaming, which are brand new compared to existing DRL-based solutions, including Pensieve [24].
Input and network. We take the current state s t as input to the two neural networks. As shown in Fig. 7, s t is defined as s t ( u t , v t , x t ) = (d t , p t , a t , b t , l t−1 , x t ). The subscript t indicates that the model has finished downloading chunk t − 1 and intends to choose chunk t. The UAV's state u t refers to its motion characteristics, obtained at the beginning of each video selection, which consists of distance d t , velocity v t and acceleration data a t . The video playback state v t consists of the buffer size b t and last bitrate selection l t−1 .
The vector x t represents the average throughput of past eight chunks. In short, our input consists of five parameters and a 1 × 8 vector. The typical input dimensions of RL-based ABR algorithms [20]- [25] range from two parameters [23] to 25 × 64 × 36 matrices [25].
As shown in Fig. 7, the past throughput data {x t−8 , ..., x t−1 } is fed into an LSTM network in time order for feature extraction. We choose the LSTM network because it can make full use of the temporal characteristics in the throughput sequence. That means, the earlier the throughput, the smaller its impact on the throughput forecast. This advantage improves the forecast capability and prompts us to choose the LSTM network instead of convolutional neural network (CNN). Then, the feature vectors obtained from the LSTM network are concatenated with other state information and finally fed into the hidden layers. Although using the same structure, the actor network and the critic network are separate and have different output.
Deep reinforcement learning policy. Almost all RL problems can be formulated as Markov decision processes (MDPs) [16], [40] to achieve the optimal solutions. Generally, the whole MDP consists of a state set S, an action set A and a reward set R, each of which can be expressed as a sequence tuple M = {s(t), a(t), r(t), s(t + 1)}. In our system, s(t) denotes the current state at video chunk t and a(t) indicates the selected video bitrate based on the current state. The reward r(t) entirely depends on the states and model's reactions and is expressed as r(s(t), a(t)). In the context of video streaming, each time a video chunk is downloaded, SA-ABR obtains a reward to evaluate the current state and action. The discounted cumulative reward can be expressed as where γ ∈ [0, 1] denotes the discounted factor, and R t represents the discounted cumulative reward from time chunk t to the end. As shown in Fig. 7, two neural networks are included in our proposed model. The goal of actor network is to find a strategy π : π θ (s, a) → [0, 1] to maximize the total reward. In our system, π θ (s, a) is the probability distribution over different video bitrate choices. With this distribution, each video bitrate is selected based on its probability via a stochastic policy, i.e., the one with the highest probability is the most likely to be picked up. The duty of critic network is to make an objective assessment V (s t ; w) for the current state s t .
Nevertheless, SA-ABR does not directly increase the discounted cumulative reward R t as the update direction [39]. Instead, R t is subtracted by a baseline b t , and R t − b t can be replaced by the advantage function A π θ (s t , a t ) = Q π θ (s t , a t ) − V (s t ; w) in the network. This represents the difference in the cumulative reward between the expected value and the actual value after selecting the action a t based on policy π θ at s t .
Training algorithm. The key step in the actor network is to calculate the advantage function A π θ (s t , a t ). In our system, we use the n-step Temporal-Difference (TD) method [41] to calculate the advantage function in the actor network, which is given as where γ ∈ [0, 1] denotes the discounted factor. Q π θ (s t , a t ) is not simply the discounted cumulative reward from chunk t. Instead, it sets the final reward as V (s t+n ; w), which is the assessment from the critic network of chunk t + n. In our experiment, the subscript t + n refers to the end chunk in a video. By performing the subtracting operation, we can finally obtain the advantage function A π θ (s t , a t ).
At the training phase, the goal of the actor network is to maximize the advantage function, i.e., making better decisions than the current policy. Thus, the parameter of the actor network θ is updated via a stochastic gradient ascent algorithm where α is the learning rate. ∇ θ log π θ (s t , a t ) shows how to change the parameter θ in order to achieve the goal. Additionally, in order to improve the generalization capability of the network, SA-ABR applies the dropout technique to reduce overfitting and add a regularization term to the update of the actor network. This term is the entropy of the probabilities over bitrate selections H(π θ (·|s t )), which encourages exploration and prevents overfitting. For the critic network, it aims at making an accurate assessment for all the states of experiments during training. We use the standard TD method to calculate the loss function of the critic network and minimize the value. Therefore, we can update the parameter of the critic network w through a stochastic gradient descent algorithm (6) where α is the learning rate, V (s t ; w) and V (s t+1 ; w) are respective assessments from the critic network of chunk t and chunk t + 1.
Data sampling. Based on the on-policy learning algorithm, the RL agent is updated periodically as data arrives, and then follows the updated strategy to sample the new data. To reduce the correlation between the data sampled from one agent and accelerate the training process, we run ten agents in parallel to experience different states, transitions and environments. These elements form a minibatch of {s t , a t , r t , s t+1 } tuples and are sent back to the central agent. The central agent then uses the actor-critic algorithm to compute the policy gradient (Eq. (5) and Eq. (6)) and updates the networks. Note that this algorithm does not require replay memory (e.g., Deep Q-Network (DQN)) and the extreme version can be directly trained on video clients to adapt to the varying UAV conditions, which shows the advantages of using actor-critic algorithm to adapt to the UAV video transmission.
Reward. Reward r is given after each chunk is downloaded. It reflects the performance of each bitrate selection according to whether the video quality meets the requirements of viewers. In our system, we consider a general QoE metric [14], [24] as a reward judging criterion, which is defined as where q(l t ) represents the user perception for video bitrate l t , T t the rebuffering time and |q(l t )−q(l t−1 )| the jitters between video chunks. Thus, the QoE is determined by three factors including the bitrate utility q(l t ), the rebuffering penalty µT t and the smoothness penalty |q(l t ) − q(l t−1 )|.
The QoE log is chosen because this kind of metric does not pursue excessive clarity, for the increase in reward gradually shrinks when switching to high bitrate selection. Therefore, it is more practical to decline the rebuffering time and improve the smoothness to maximize the reward. This preference is also very suitable in the video transmission scenario, since UAVs are often used for real-time video capturing and transmission.

IV. IMPLEMENTATION
This section encompasses three aspects. To begin with, we characterize the neural network architecture and all the hyperparameter settings during the training process. Next, we give a brief introduction of our UAV platform. Finally, we elaborate on the collected network traces used in our experiment.
Neural network architecture. In this part, we elaborate on the specific design of our neural network architecture and all the hyperparameters in the experiment. First, we feed a sequence of past eight chunks' throughput in time order to an LSTM network, which is constructed with two-layer LSTM cells, with 64 hidden unites each. Each input contains one throughput in the past and the step size is one. Then, the resulting vector from this LSTM network is flattened and combined with other inputs including video playback states and UAV's flight states before being imported to two fully connected layers, with 30 and 10 hidden units, respectively. The actor network and the critic network have the same input and structure, while they are different in network parameters and output. We add a softmax layer and obtain a probability distribution of the bitrate selections for the actor network, while setting the state evaluation as output for the critic network. Furthermore, the input sizes of both networks are 13. Within the training period, we set the discount factor γ to 0.99, and configure the learning rates α and α to 3 × 10 −5 and 1 × 10 −2 , respectively. Additionally, the reward factor µ is set to 2.26 according to the QoE metric we choose.
Hardware setup. As shown on the right part of Fig. 8, we build a wireless link between a UAV and a laptop on the ground through the IEEE 802.11n protocol. SA-ABR is a client-based model and the key ABR algorithm runs on the laptop. An Android smartphone is attached on our UAV platform, DJI Matrice 100, and transmits the video and sensor data to the laptop on the ground. Additionally, the data file is programmed to be transmitted through a WiFi channel, which is different from the control channel of the UAV to avoid interference. We use the 2.4 GHz band for transmission with a channel bandwidth of 20 MHz. During the transmission process, the laptop on the ground calculates the number of TCP packets in the application layer per second to obtain the average throughput, which tracks the wireless-link dynamics. Another function of the laptop is to emulating the video playback states, which we specifically describe in Section III. Based on these available messages, SA-ABR can gradually adapt to the throughput dynamics through the training process.
Network traces. In order to model the real-world wireless channel condition on UAV, we collect the throughput data by flying the UAV in the wild. Since the available video bitrates in our experiment are {300, 750, 1850, 2850} Kbps, the throughput is multiplied by weight to match the video quality, which is common in the scenes where the UAV transmits videos to multiple clients. The experimental sites includes a playground, a plaza and a pool. The flight trajectories are randomly distributed in these areas and the flying velocities range from 0 to 19.5 m/s. The height of the UAV is set at around 25 m for safety. We record a total of 1000 throughput traces, with each trace spanning 100 s. In our experiment, we use a random sample of 80% in the data corpus as the training set, while the remainder 20% is used as the test set. The throughput ranges from 0 to 20 Mbps.

V. EVALUATION
In this section, we conduct a series of experiments to evaluate SA-ABR from three aspects as described below.
• How does SA-ABR compare with state-of-the-art ABR algorithms in terms of video QoE? We test several ABR schemes including kinds of fixed control rules and RL algorithms, and then perform comparative experiments with SA-ABR. The result shows that SA-ABR always performs better compared with other algorithms and is able to outperform the best ABR schemes by 21.4% in terms of average QoE value (Figs. 9, 10, and 11). • Does SA-ABR benefit from LSTM network? In the framework design of SA-ABR, we exploit the LSTM network to extract the time-series features from the past throughput experience. To specifically evaluate the advantage, we propose a baseline model that is similar to our network architecture, but replacing the LSTM network with CNN. By comparing these two models, we find SA-ABR still presents its improvements in average QoE (Fig. 12). • What is the advantage of feeding various sensor data into the neural network and can the network effectively filter out the confusing information and extract useful features from it? We conduct an experiment to compare SA-ABR with the same network architecture without sensor assistance. Results in Fig. 14 show the performance improvement for SA-ABR with the assistance of sensor data.

A. SA-ABR vs. Existing ABR Algorithms
We compare SA-ABR with a variety of existing stateof-the-art algorithms which generate the ABR algorithm in completely different ways including fixed control rules and RL. These algorithms perform bitrate adaptations mainly based on the past throughput experience and video playback states without the assistance of sensor data. The detailed principles of these algorithms are illustrated below.
• Buffer-based policy [11] is an algorithm that chooses the video bitrate only based on the playback buffer occupancy. The goal is to reach balanced states that ensure the avoidance of unnecessary rebuffering while maximizing the average video bitrate. The model is manually configured to keep the buffer occupancy above five seconds and when the buffer occupancy exceeds 15 s, the highest available bitrate is automatically selected. • Rate-based policy [9] exploits the harmonic bandwidth estimator to compute the harmonic mean of the last five throughput samples, which provide robust bandwidth estimates for future chunks. Thus, this model can automatically select the highest bitrate that does not exceed the expected channel capacity. • MPC [14] proposes a concrete ABR workflow that can optimally combine the advantages of future throughput prediction and buffer-based functions. The algorithm uses the same approach as Rate-based models to provide throughput forecasts, based on the past throughput trajectory. Then, MPC can map the collected information including the throughput prediction value, previous bitrate and buffer occupancy to future bitrate selections of video chunks. • Pensieve [24], which is based on the RL algorithm, has been experimented to use in the video bitrate adaptation subject with no pre-programmed control rules or explicit assumptions of the environments. The model uses CNN to extract effective features from the input data, including the past throughput trajectories and video states, and learns automatically through the RL algorithm to make better ABR decisions. In our experiment, SA-ABR is trained to obtain the optimal policy for higher QoE metric rewards, using the training set described in Section IV. Although SA-ABR is generated from the limited training set, its performance can be extended to the test period in which SA-ABR can still make right decisions when encountering states that are never present in the training set. The reason is that what SA-ABR learns through DRL in the training process is not just the limited state-action pairs, but a continuous neural network function that maps the successive state space to actions. Moreover, the parameters of the aforementioned existing ABR algorithms are also adjusted accordingly to adapt to the varying UAV channel capacity. We evaluate the performance of all the ABR algorithms based on the same test set (Section IV). In addition, we use the same QoE function (Eq. (7)) to assess all the ABR algorithms. Besides, three components (Eq. (7)) of the QoE definition, including the bitrate utility, the rebuffering penalty and the smoothness penalty, are also analyzed to better evaluate the video performance. Fig. 9 shows the average and variance values of QoE for one video chunk. Note that the average QoE reward of SA-ABR is 21.4% higher than that of Pensieve, which presents as the best known ABR algorithm. The detailed causes of the QoE improvement are represented in the following histograms. The main reasons for the gain in the average QoE compared with Pensieve are improvements in bitrate utilization and smoothness. The average bitrate utility exceeds Pensieve by 10.8% while the smoothness penalty is reduced by 35.3%. As mentioned above, the characteristics of the QoE log metric lead to its slower growth in value as the bitrate increases, i.e., the reward of selecting higher bitrates (1850 Kbps and 2850 Kbps) may not counteract the rebuffering and smoothness punishment to some extent. However, keeping playing videos at lower bitrates (300 Kbps and 750 Kbps) all the time will undoubtedly affect the users' viewing experience. With the assistance of the LSTM network and sensor data, our RLbased system can better weigh the gains and losses of choosing higher video bitrates, i.e., increasing the average bitrate utility in smoother trend on the basis of no rebuffering increases. It is a step forward in ABR video streaming algorithm under the UAV channel environments.
In addition, as illustrated in Fig. 9, we observe that the bitrate utility of SA-ABR and Pensieve all present lower than MPC and buffer-based algorithms. The reason accounting for this phenomenon is that the DRL algorithm prompts SA-ABR to achieve performance gain in a more balanced way. Thus, although not achieving the maximum bitrate utility, SA-ABR has the minimum rebuffering time and the highest smoothness, which achieves a balanced state in three components of the QoE metric. In contrast, MPC and bufferbased algorithms perform poorly in decreasing rebuffering and improving smoothness. The distributions of three QoE components are also explicitly exhibited in Fig. 10 in the form of CDF, which show gaps between different ABR algorithms. Nevertheless, to further increase the average video bitrates is a constant challenge we need to overcome in the future. Fig. 11 (top subfigure) first depicts the network throughput traces and respective bitrate selections made by MPC, Pensieve and SA-ABR over a period of video in our data corpus. We can observe that MPC selects the highest bitrate (2850 Kbps) at 56 s. However, it immediately switches to a lower bitrate (750 Kbps) when the throughput begins to fall. Throughout the entire playback process, the bitrate selections of MPC fluctuate constantly with the throughput dynamics, which gives viewers bad experiences. As for Pensieve, the model does not make full use of the channel capacities and switches to a lower bitrate at 56 s due to the uncertainties in the future even though the current throughput is still very high. In contrast, SA-ABR is courageous to select the high bitrate (1850 Kbps) at 48 s and keep the bitrate for a while for video smoothness, which verifies the forecast accuracy and the ability to balance the bitrate utility and buffer occupancy. Moreover, Fig. 11 (bottom subfigure) further shows the changes in buffer occupancy corresponding to each chunk' s bitrate selection for these three ABR algorithms. Compared with other algorithms, SA-ABR can make full use of the buffer occupancy, while the other models waste a lot of buffer resources.

B. LSTM Network vs. CNN
SA-ABR applies the LSTM network to extract effective time-series features from the past throughput experience, which is described in detail in Section III. However, despite the fact that it outperforms all the listed state-of-the-art ABR algorithms, it still cannot certify whether SA-ABR benefits from the LSTM networks, as Pensieve lacks the assistance of sensor data to become the comparative experiment. Therefore, we set a baseline model which has the same architecture as SA-ABR but replaces the LSTM network with CNN.
The results of comparisons are shown in Fig. 12. SA-ABR's average QoE is 17.5% higher than the baseline and the gain comes from its ability to limit rebuffering and smoothness penalty. Therein, SA-ABR reduces rebuffering penalty by 88.6% through maintaining sufficient buffer occupancy to handle the risk of unpredicted fluctuations in channel capacity. This phenomenon indicates that the LSTM network masters the dramatically varying features of communication dynamics through analyzing past throughput experience. Additionally, with the assistance of the LSTM network, SA-ABR decreases the smoothness penalty by 49.4%, based on the robust predictive function, which provides a more comfortable viewing experience. Moreover, to analyze how many past throughput samples are necessary to be fed to the LSTM network for better throughput forecast, we conduct an experiment to compare several versions of SA-ABR, each with a different number of throughput samples. As shown in Fig. 13, compared to two throughput samples, eight throughput samples enable the LSTM network to extract more temporal information from the throughput sequence, which leads to a significant gain in the average QoE reward. However, when increasing the past throughput samples to 16, the QoE gain is marginal. That means, eight throughput samples are sufficient for the throughput prediction and we select the past eight throughput samples as input to the LSTM network.

C. Sensor Assistance vs. No Sensor Assistance
For the purpose of better understanding the QoE gains obtained from the assistance of sensor data, we analyze SA-ABR's performance on individual terms of the QoE metric. To avoid interference from other related factors, such as the network type, we present another baseline model for comparison which also has the same network architecture as SA-ABR but lacks the assistance of sensor data. This baseline model is also trained to obtain the optimal policy for higher QoE reward in the experiment.
Specifically, Fig. 14 shows the results of comparisons between SA-ABR and the baseline without sensor assistance. We observe that SA-ABR outperforms the baseline and the gain in the QoE reward ranges from 7.0% to 30.7%. The reason for such a variance in the performance gain is that the improvement from the sensor data is closely connected to the flight conditions of the UAV. When the UAV flies slowly and the channel is relatively stable, the benefit of the sensor data is marginal, resulting in a lower performance gain. Whereas, when the UAV flies at a high speed, it causes unpredicted fluctuations in the UAV channel capacity. In this case, the sensor data can indicate the throughput changes and provide hints for SA-ABR to enhance the forecast capability and achieve better QoE rewards.
Other evaluation metrics such as the bitrate utility and the rebuffering penalty also verify the important role of the sensor data. As shown in Fig. 14, SA-ABR increases the bitrate utility by 6.9%. Although the gain is not large, it still embodies the raising abilities to challenge high-definition video chunks, resulting from the enhanced predictive accuracy. Moreover, SA-ABR is able to minimize the rebuffering time while ensuring the bitrate utility of video chunks, even if the channel condition is poor. As depicted in Fig. 14, the reduction ratio of the rebuffering penalty reaches 57.1%. Furthermore, the change in the smoothness penalty is negligible, which indicates that the use of sensor data does not play a major role in maintaining sufficient smoothness.
To further verify the advantages of quantizing the sensor data, we conduct a comparative experiment in which we feed the raw sensor data into the networks. The results are shown in Fig. 13. Note that without the quantization process, the variance level of QoE increases while the average value declines, which indicates that the disturbance and noise from the raw sensor data prevent models from getting the optimal strategy. In other words, the quantization process eliminates the noise while ensuring full utilization of the sensor data.
VI. RELATED WORK ABR algorithm. The ABR algorithm essentially follows a dynamic selection mechanism that has been widely used in various fields [42], [43]. Existing ABR algorithms fall into two categories: fixed control rules [9]- [19] and learning method [20]- [30]. The majority of existing fixed control rules generate ABR algorithms based on the available bandwidth estimates (rate-based algorithms [9], [10]), playback buffer occupancy (buffer-based algorithms [11], [12], [18]) or the hybrid methods of combining these two approaches ( [13]- [16]). The rate-based algorithms first estimate the future available bandwidth according to past throughput experience, and then select the highest bitrate below the bandwidth. The bufferbased algorithms, however, only consider the current buffer conditions when making bitrate decisions. The hybrid methods integrate these two technologies, i.e., using future throughput estimates and buffer occupancy information to select the proper bitrates for future chunks. Based on these fixed control rules, Akhtar et al. [17] dynamically adjust the configurable parameters of rules to make ABR algorithms work better over a wide range of network conditions. Xie et al. [18] and Xu et al. [19] respectively propose a buffer-based ABR algorithm with dynamic threshold and a QoE-driven adaptive k-push algorithm for low-latency live streaming. However, all these ABR algorithms with fixed control rules need substantial preprogramming overhead, which is not suitable for adapting to the dramatically varying channel capacity of UAVs.
Recently, there has been a growing interest in developing the optimal learning-based ABR algorithms. Before Pensieve [24], A separate line of studies [20]- [23] propose to generate ABR algorithms based on reinforcement learning. However, all of these algorithms store the value function for all states instead of using value function approximation, which cannot generalize to large state and action space. Pensieve [24] is the first model to apply the actor-critic network to the ABR algorithm which learns the optimal policy automatically. In addition, QARC [25] employs a DRL model to select the bitrate by jointly considering the predicted video content and network states. NAS [26] directly applies neural networks (NN)-based quality enhancement on the received video content to maximize the user QoE. Moreover, Jiang et al. [27] leverage the correlations of video content to dynamically pick the best configurations for analytics pipelines. Furthermore, based on the DRL algorithms, HotDASH [28] considers the prefetching of user-preferred temporal video segments, while Guo et al. [29] perform dynamic resource optimization for wireless buffer-aware video streaming. Kan et al. [30] design a DRL-based rate adaptation algorithm for 360-degree video streaming. These algorithms have their own advantages in different video transmission applications. However, they are not specifically designed for the UAV video transmission, and thus are not optimized to adapt to the dramatically changing UAV environments.
Application of sensor data. The sensor data brings great convenience for IoT device communication [44], especially in mobile scenarios. The sensor data is widely used in mobile devices. Several studies [45]- [48] use sensor information to infer the motion states and surrounding environments of objects, which optimize wireless communications by adjusting protocols including client roaming, bit rate adaptation, frame aggregation and beamforming. Santhapuri et al. [46] employ light sensor readings on the phone to distinguish indoor or outdoor locations and exploit the on-phone accelerometers to identify the mobility states of users, such as walking patterns and vehicular motion, which is beneficial to improve the user experience. Zhang et al. [47] take as input users' location information through planned routes, and then predict the bandwidth along the route to make online transmission decisions. Furthermore, several studies use WiFi signal strength [49] or PHY layer information [50] to detect the users' motion states, which are contrary to the above studies. Asadpour et al. [51] analyze in detail the impact of the relative motion between two UAVs on throughput and Wang et al. [45] experiment to add the GPS information of the UAV to the ABR video streaming algorithm, which enables the model to indicate more UAV channel information.
VII. CONCLUSION SA-ABR exploits the flight-state-related sensor data that is readily-available on today's commercial UAVs to generate ABR algorithms that provide stable and better QoE under various flight scenarios. The sensor data we analyze and exploit in experiments include GPS coordinates, accelerations, and velocities. With the help of sensor data, our model can better infer the future channel condition and effectively mitigate the negative impact caused by sudden changes in flight states. The experimental results have verified that SA-ABR outperforms the best known ABR algorithm by 21.4% in terms of average QoE reward. SA-ABR can be seamlessly integrated into existing wireless protocols and commercial hardware. We believe that with these features, SA-ABR can provide some insights for future UAV transmission policy designs.


Profiling US Restaurants from Billions of Payment Card Transactions

A payment card (such as debit or credit) is one of the most convenient payment methods for purchasing goods and services. Hundreds of millions of card transactions take place across the globe every day, generating a massive volume of transaction data. The data render a holistic view of cardholder-merchant interactions, containing insights that can benefit various applications, such as payment fraud detection and merchant recommendation. However, utilizing these insights often requires additional information about merchants missing from the data owner's (i.e., payment company's) perspective. For example, payment companies do not know the exact type of product a merchant serves. Collecting merchant attributes from external sources for commercial purposes can be expensive. Motivated by this limitation, we aim to infer latent merchant attributes from transaction data. As proof of concept, we concentrate on restaurants and infer the cuisine types of restaurants from transactions. To this end, we present a framework for inferring the cuisine types of restaurants from transaction data. Our proposed framework consists of three steps. In the first step, we generate cuisine labels for a limited number of restaurants via weak supervision. In the second step, we extract a wide variety of statistical features and neural embeddings from the restaurant transactions. In the third step, we use deep neural networks (DNNs) to infer the remaining restaurants' cuisine types. The proposed framework achieved a 76.2% accuracy in classifying the US restaurants. To the best of our knowledge, this is the first framework to infer the cuisine types of restaurants by analyzing transaction data as the only source.

I. INTRODUCTION
Payment cards (primarily debit and credit), with billions of active users, are among the most popular payment methods across the globe. In the US alone, millions of people regularly use debit and credit cards to pay for goods and services. Payment processing companies handle these transactions and record their attributes (e.g., cardholder id, merchant name, and authorized amount), generating a massive volume of transaction data. Transaction data render a holistic view of cardholder-merchant interactions, containing insights into the behavior of cardholders and the business of merchants. The insights can benefit a variety of applications, such as payment fraud detection and merchant recommendation.
While transaction data accommodate valuable insights, utilizing these insights often require auxiliary information about * During this work, both authors were employed at Visa Research. merchants that are not present in raw data. For instance, in payment fraud detection, it is critical to know merchants' categories-the type of goods or services they provide-to classify and track purchases. Recommendation services also require characterizing merchant attributes, such as business category and sub-categories (the type of goods or services), market positioning (high-end vs. low-end), and business hours (when to visit). Broadly, incorporating merchant attributes can boost the performance of machine learning models that empower today's recommendation and anomaly detection services.
However, acquiring merchant attributes is challenging, especially for commercial purposes. Even payment processing companies that possess the most detailed view of transactions do not have access to all required attributes. One possible way to acquire the missing attributes is to collect them from crowdsourcing companies (e.g., Yelp, Foursquare), which generate merchant profiles from customer reviews. However, such data acquisition has several limitations. First, it is costly to acquire merchant information for commercial purposes. Second, debit and credit cards are commonly used in many countries where collecting merchant information through crowd-sourcing may not be possible. For example, Yelp currently operates in 32 countries, whereas Visa cards are accepted in more than 200 countries and territories. Third, the collected information may become outdated, as new merchants appear or old merchants go out of business. These limitations motivate us to generate merchant profiles by analyzing transactions. Unlike user reviews, transaction data do not contain a large corpus of texts or images to extract merchant attributes. Consequently, generating merchant profiles from transaction data requires inferring attributes through other means.
Present Work. The present work contributes to our broad goal of inferring latent merchant attributes from transaction data. Though raw transactions do not contain much information about merchants, the transaction data as a whole contains rich signals, which have the potential to reveal a multitude of merchant attributes. As proof of concept, we concentrate on restaurants and infer the cuisine types of restaurants from transactions. Restaurant is perhaps the most popular merchant category, with which most cardholders have regular transactions. However, while processing any restaurant transaction, a payment company does not know what type of cuisine the restaurant serves. This information can be useful for providing restaurant recommendations and improving fraud detection.
Restaurant Recommendation. Cuisine types play a critical role in restaurant recommendation. For example, consider a transaction-based restaurant recommendation system that provides recommendations without being aware of restaurants' cuisine types. A standard solution to develop such a recommendation system is to employ user-based collaborative filtering, which generates recommendations based on the similarity of transactions among cardholders. While practical, this solution comes with considerable risk-imagine recommending steakhouses to vegetarians who visited many Mediterranean restaurants, as Mediterranean cuisine is popular among both vegetarians and meat-lovers. Incorporating cuisine types of restaurants into the recommendation algorithm can prevent such misreckoning. Cusine types can also be useful for addressing the cold-start problem in restaurant recommendation.
Fraud Detection. Cuisine types of restaurants also have the potential to inform fraud detection systems. For example, suppose a fraud detection system has access to the cuisine types of restaurants. In that case, it can use this information in conjunction with transaction data to generate a cuisine signature for each cardholder. This signature captures how likely the cardholder is to visit different types of restaurants. Then, while processing a restaurant transaction, the system can examine if the restaurant aligns with the cardholder's cuisine signature, among other signals, to detect if the transaction is a fraudulent one.
This paper presents a framework to infer the cuisine types of restaurants from billion-scale payment card transactions. Please note that the intended use of the framework includes commercial purposes. For this reason, we can not use external sources (e.g., Google, Yelp) for data acquisition. The proposed framework has three steps: 1) weakly-supervised label generation, 2) statistical feature and neural embedding extraction, and 3) deep neural network based classification. In the first step, we programmatically generate cuisine labels for restaurants via weak supervision. We first compile a cuisine taxonomy containing the major cuisine types and a list of keywords for each cuisine. We then use the keywords as common patterns to label restaurants based on their names. For example, if the keyword "Peking" appears in a restaurant's name, we recognize it as a Chinese restaurant. Through bootstrapped label expansion, our weakly-supervised approach can generate cuisine labels for 35% of US restaurants. We infer the remaining US restaurants' cuisine labels by employing large-scale feature engineering followed by deep neural network based classification. In the second step, we extract a wide variety of statistical features and neural embeddings for half a million US restaurants, from four billion restaurant transactions. Our statistical features encompass a wide range of business information about restaurants, such as location, business hours, pricing information, tipping culture, serving capacity, expected party size, customer visitation patterns, and customer loyalty. We also model transaction data as a bipartite graph between customers and restaurants and then employ state-of-the-art neural language models to generate two sets of restaurant embeddings. The embeddings capture the semantic similarity between restaurants at two distinct levels: the semantic similarity between any two restaurants that an individual customer visits (micro-level), the semantic similarity between two restaurants based on their entire customer base (macro-level). We further apply neural language models on restaurant names (excluding the words used in labeling) to generate another set of restaurant embeddings. Overall, we generate three sets of restaurant embeddings to represent the latent characteristics of restaurants. In the third step, we design several deep neural networks (DNNs) to predict remaining restaurants' cuisine types based on the statistical features and embeddings generated in the second step. Our best performing network is a deep feedforward network with residual connections. The network achieved a 76.2% overall accuracy in classifying US restaurants, where per class accuracy is high for all classes, including minority cuisine types such as Mediterranean cuisine. To the best of our knowledge, this is the first attempt to profile restaurants in the mostly untapped space of payment card transactions.

II. PROBLEM SETTING
This work aims to develop a framework for inferring the cuisine types of restaurants from debit and credit card transactions. Cuisine type information is essential to payment processing companies for providing user services, notably, restaurant recommendation and payment fraud detection.
Feature Engineering. A secondary goal of this work is to extract general-purpose features that can benefit a variety of tasks beyond cuisine-based restaurant classification, such as business profiling and customer segmentation. Preferably, we want to extract interpretable features that can provide insights into the restaurant business.
Data Restriction. The intended use of this framework includes commercial purposes. Therefore, we can not use external sources (e.g., Google, Yelp) for label generation, feature engineering, and framework validation. For example, without explicit consent, Yelp API and content usage are restricted to non-commercial use [1]. Acquiring merchant information for commercial purposes can be costly.
In remaining sections we present a framework that meets the requirements above and achieves a high accuracy in inferring the cuisine types of US restaurants.

III. TRANSACTION DATASET
Our dataset contains four billion debit and credit card transactions in more than half a million US restaurants within three months. This dataset is a large subset of in-person restaurant transactions in the US (customer made payment at a restaurant using a VISA card). Each transaction in our dataset is an interaction between a cardholder and a merchant. Therefore, all attributes of each transaction come from the cardholder, the merchant, or their interaction. Specifically, our dataset contains the following information for each transaction.

IV. CUISINE INFERENCE FRAMEWORK
In this section, we describe our proposed framework for inferring the cuisine types of restaurants from debit and credit card transactions. Figure 1 presents an overview of the framework. The framework has three steps: 1) weakly-supervised label generation, 2) statistical feature and neural embedding extraction, 3) deep neural network based classification. We perform these three steps in a closed-loop to classify the restaurants in our transaction dataset.

A. Weakly-Supervised Label Generation
In the first step, we programmatically generate cuisine labels for restaurants via weak supervision. We compile a cuisine taxonomy along with keywords, extract new keywords from restaurant names via bootstrapping, and employ topic modeling to augment the keyword-based approach.
Cuisine Taxonomy Compilation. We compile a cuisine taxonomy based on the literature on food [2], [3] and the US restaurant industry [4]- [6]. Table I presents  Asian, East Asian, Grill & Steak, Fastfood, Bar, Dessert. Each of these major cuisine types (e.g., European) accommodates more fine-grained types (e.g., French). Though our taxonomy does not contain all cuisine types (e.g., Circassian), it has a high coverage for US restaurants. We compile a set of seed words for each cuisine type in our taxonomy. We use these words as common patterns to generate cuisine labels for restaurant names. For example, if the keyword "Peking" appears in a restaurant's name, we recognize it as a Chinese restaurant. Currently, we have a list of 225 seed words that represent the ten major cuisine types. Bootstrapped Label Expansion. To better utilize the cuisine information obtainable from restaurant names, we apply pattern-based bootstrapping [7]. The key idea of our approach is to extract new words (beyond seed words) from restaurant names, which can serve as highly accurate patterns to increase the coverage of labeled restaurants. We first determine the cuisine label distribution for each new word based on the labeled restaurants. We then select the words that meet three minimum criteria-i) frequency: the word needs to appear in θ f fraction of restaurant names; ii) precision: if we use the word and its majority label as a labeling rule, the rule needs to be true for θ p fraction of labeled restaurants; iii) significance: the ratio of labeled and unlabeled restaurants that contain the word should be θ s . We identify 198 words that satisfy the criteria mentioned above and use these bootstrapped words as secondary patterns to label more restaurants. Altogether, using the seed and bootstrapped keywords, we label 35% of restaurants in our dataset.
Topic Modeling for Weak Supervision. To augment the keyword-based approach of weak supervision, we apply a variety of topic models on restaurant names, treating each restaurant name as a document. Conventional topic models such as LDA [8] do not work well with restaurant names,

Pricing
The deciles of authorized amount in transactions Tipping culture The deciles of (settlement amount -authorized amount) in transactions Serving capacity The deciles of hourly transaction count Party size The proportion of transactions for different party size Party pricing The average authorized amount for different party size Temporal pattern I The distribution of number of transactions over days of the week Temporal pattern II The distribution of number of transactions over the hours of weekdays Temporal pattern III The distribution of number of transactions over the hours of weekends Customer revisitation The deciles of the number of revisits by the customers Customer loyalty The deciles of the number of restaurants visited by the customers Location The digits of restaurant zipcode and corresponding location granularity Micro embedding The restaurant embedding generated by applying word2vec on user documents Macro embedding The restaurant embedding generated by applying doc2vec on restaurant documents Name embedding The embedding based on restaurant name resulting in incoherent topics, as measured via UMass coherence score [9]. We observe three critical challenges in applying topic models-i) monolith: many restaurant names consist of a single word; ii) sparsity: sparse word co-occurrence patterns in restaurant names; iii) long-tail: long-tail distribution of words appearing in restaurant names. To address the monolith issue, we adopt the concept of sprinkling [10]adding artificial words (known cuisine types) to boost the association in the topic model. To address the sparsity issue, we adopt the bi-term topic model (BTM) [11] that learns topics by directly modeling the generation of word co-occurrence patterns known as biterms. BTM has two primary advantages over conventional topic models: i) BTM explicitly models word co-occurrence patterns, instead of word documents, which improves the topic learning process; ii) BTM uses the aggregated patterns in the whole corpus for learning topics, which solves the problem of sparse patterns at documentlevel. To address the long-tail issue, we perform stratified sampling on biterms before passing them to BTM. Using BTM as our topic model, we vary the topic model parameter k (the number of topics) from 5 to 50, while assigning the value 50 k to parameter α (document-topic density) and 0.1 to parameter β (topic-word density), as suggested by Griffiths et al. [12]. We then compute the UMass coherence score [9] for each set of topics. The resultant topics (cuisine classes) are coherent (high UMass score) and consistent with the cuisine classes generated by our keyword-based approach.

B. Statistical Feature and Embedding Extraction
In the second step, we perform extensive feature engineering for large-scale transaction data. Table II presents the description of the features that we extract from restaurant transactions. Broadly, our features fall into two categories: statistical features that capture business information about restaurants, and embeddings that capture restaurant's latent characteristics based on customer-restaurant interactions.
Statistical Feature Extraction. We extract a wide variety of statistical features from transactions based on the literature on the restaurant industry [13]- [25]. These features encompass a wide range of business information about restaurants, such as location, business hours, pricing information, tipping culture, serving capacity, expected party size, customer visitation patterns, and customer loyalty. The features provide insights into the restaurant business and benefit various tasks beyond restaurant classification, such as business profiling and customer segmentation. A key idea of our statistical feature engineering approach is to use distributions as features, enabling the framework to differentiate among cuisine types in terms of tail behavior.
Pricing: Price is one of the crucial features for characterizing a restaurant, which bears implications for cuisine type [23]. Roy [24] reported the price inequality among different cuisine types for restaurants located in New York City (NYC), e.g., people in NYC pay more for European cuisines compared to East Asian and South Asian cuisines. To capture the idiosyncrasies of pricing, we extract the deciles of the authorized amount for each restaurant. The 2 nd column in Table III shows how the median price of food varies across cuisine types, for all US restaurants (median of median). The median price is: a) high for European cuisines; b) medium for Bar, Grill & Steak, and Asian cuisines; c) low for Dessert, Fastfood, Latin American, and MMA cuisines.
Tipping Culture: Tipping norms vary across restaurant types, e.g., fine dining restaurants have different norms compared to fast food restaurants. Chen et al. [25] examined how the bias towards different types of cuisines affect gratuities. The study confirms the effect of cuisine types on tipping behavior. To capture customers' tipping behavior for a restaurant, we extract the deciles of the difference between the settlement amount and the authorized amount. The 3 rd column in Table III shows how the median of tips varies across cuisine types. The median tip is: a) high for Bar and European cuisines; b) medium for Grill & Steak, South Asian, and South East Asian cuisines; c) low for all other cuisines. Serving Capacity: Serving capacity is an important variable for studying a restaurant business [15], e.g., fast food restaurants typically have higher serving capacity compared to the fine dining restaurants [21]. To capture the serving capacity of a restaurant, we extract the deciles of the number of transactions per hour. The 4 th column in Table III shows how the average number of servings varies across cuisine types, for all US restaurants (median of average). Average serving capacity (number of customers per hour) is: a) high for Bar, and Grill & Steak; b) medium for Dessert, European, and Latin American cuisines; c) low for all other cuisines.
Party Size: Party size composition is an insightful feature for studying the customers of a restaurant [17], [18]. Qu et al. [14] showed that at Chinese restaurants in Indiana, the majority (57%) of customers dined with a party size of two. Estimating party size from transaction data is challenging. A bill for a dining party could be paid by one of the members or shared among the members. If one of the members pays the entire bill, there would be no explicit information about the party size. In contrast, if the members share the bill, there would be no association between their transactions. We overcome the first challenge by estimating party size from the authorized amount. We can not address the second challenge due to the weak association between transactions. To estimate party size, we apply a separate Gaussian mixture model (GMM) on each restaurant's authorized amounts. Our Gaussian mixture model is a weighted sum of K component Gaussian densities as given by the equation, where x is the vector of authorized amounts, φ i is the mixture weight (prior probability) of i-th component, and N (µ i , σ i ) is the Gaussian density of i-th component. We determine the optimal number of mixture components for GMM by using the Akaike information criterion (AIC). We use the proportion of transactions for different party sizes, and the corresponding average authorized amounts as party composition features. The 7 th column in Table III shows how the average expenditure per person varies across cuisine types. Consistent with pricing, the average expenditure per person is highest for European cuisine and lowest for dessert. The 8 th column in Table III shows how the percentage of single diner varies across cuisine types. Notice that the single diner percentage is lower for the Asian cuisines compared to other cuisine types. This finding is consistent with the norm that Asian restaurants are popular for group eating.
Temporal Patterns: Temporal patterns such as business hours and busy times are critical indicators for understanding a restaurant business [13], e.g., bars typically operate during night time, whereas brunch spots operate during day time. To capture the temporal patterns of a restaurant's operation, we extract the distribution of transactions over days-of-week and hours-of-day. We further divide the hours-of-day into hours-ofweekday and hours-of-weekend. The 9 th column in Table III shows how the percentage of transactions on weekends varies across cuisine types, for all US restaurants. Notice that Bars and European restaurants attract more of their customers during weekends.
Customer Revisitation: Researchers have acknowledged the importance of understanding the differences between firsttime customers and repeat customers in restaurants [19]. Barber et al. [20] showed a strong correlation between the food type and repeat visitation of customers. To capture the customer revisitation tendency in a restaurant, we extract the deciles of the number of revisits (repeat transactions) by the customers. The 5 th column in Table III shows how the number of revisits for top customers (on and above 90th percentile) varies across cuisine types. Notice that frequent customers have a strong revisit tendency to bars and dessert places.
Customer Loyalty: Customer loyalty can be useful for studying a restaurant business [16]. This feature is especially helpful for understanding the customers and their endemicity. To capture the loyalty of customers of a restaurant, we extract the deciles of the number of restaurants visited by the customers. The 6 th column in Table III shows how the median number of restaurants visited by customers varies across cuisine types. Notice that there is not much difference among the cuisine types in terms of customer base loyalty.
Location: Location has been a critical factor for characterizing a restaurant [22]. As a restaurant business is often established based on potential customers' preference in the locality, location plays a crucial role in restaurant profiling. We extract the digits of restaurant zip code and corresponding location granularity to capture a restaurant's locality.
Restaurant Embedding Generation. Restaurants in our dataset can be represented in a lower-dimensional space based on the inherent structure imposed by their characteristics, such as cuisine type, price range, and location. Cardholders can also be represented in a lower-dimensional space based on their traits, such as cuisine affinity, price sensitivity, and location. The transactions in our dataset are interactions between samples from these two spaces, where certain interactions are plausible while others are unlikely to happen. To capture the interactions between cardholders and restaurants, we model transaction data as a bipartite graph. We then propose two hypotheses that capture the plausibility of the interactions in this graph at the micro and macro level.
Hypothesis I: The plausibility of interaction between a customer and a restaurant is a function of the customer's preferences and the restaurant's attributes.
Hypothesis II: The plausibility of interaction between a group of customers and a restaurant is a function of the collective taste and the restaurant's attributes.
The subtle distinction between the two hypotheses lies in their application: the first hypothesis applies to individuals (micro), and the second hypothesis applies to groups (macro). The first hypothesis implies that the compatibility between a customer's preferences and a restaurant's attributes is a good predictor of whether the customer will visit the restaurant. For example, a vegetarian is likely to visit an Indian restaurant. The second hypothesis implies that the type of customers who visit a given restaurant is a good predictor of its attributes. For example, a restaurant is unlikely to be a steakhouse if many of its customers are vegetarian.
Based on the hypotheses mentioned above, we aim to represent the restaurants in our dataset in a low-dimensional space by applying state-of-the-art representation learning techniques.
There is a variety of methods that can be used for learning lower-dimensional representations-notably neural language models, such as word2vec [26] and its extension doc2vec [27]. These models have been successfully adopted in a variety of domains, including product representation in eCommerce [28]. In this work, we utilize word2vec and doc2vec to create two distinct sets of restaurant embeddings based on our micro and macro hypotheses. Details about the parameters of the micro (word2vec) and macro (doc2vec) model can be found in the experimental evaluation (impact of hyperparameters).
Micro Embedding: We represent each customer as a document and each restaurant as a word. Accordingly, each transaction by a customer corresponds to a word (restaurant) in a document (the customer's ledger of restaurants). The words appear chronologically in the documents, as per the day and time of transactions. After creating the documents, we apply word2vec on these documents. As per our representation, words refer to restaurants, and consequently, the word embeddings generated by word2vec are restaurant embeddings. These embeddings capture our micro hypothesis: the context of a restaurant in a customer's ledger is defined by other restaurants visited by the customer within the same time window.
Macro Embedding: We represent each restaurant as a document and each customer as a word. Accordingly, each transaction in a restaurant corresponds to a word (customer) appearing in a document (the restaurant's ledger of customers). These words appear chronologically, as per the day and time of transactions. After creating the documents, we apply doc2vec on these documents. As per our representation, documents refer to restaurants, and consequently, the document embeddings returned by doc2vec are restaurant embeddings. These embeddings capture our macro hypothesis: the context of a customer in a restaurant is defined by other customers visiting the restaurant within the same time window.
Name Embedding: Apart from the two sets of embeddings learned from the customer-restaurant interactions; we generate another set of embeddings from restaurant names. Name embedding has been successfully utilized in other domains, such as social media [29]. In this paper, we generate name embeddings to utilize the remaining (non-labeling) words in restaurant names. To this end, we first remove the labeling words (used in step 1) from restaurant names. We then use pre-trained word embeddings (GloVe [30]) to generate name embeddings for restaurants. Specifically, to generate the name embedding of a restaurant, we combine the embeddings corresponding to its words via max pooling.

C. DNN Based Classification
In the third step, we classify the unlabeled restaurants in our dataset. To this end, we create several deep neural networks (DNNs) to complement our extracted features. Some of these networks mimic prominent DNN models deployed in largescale industry setup, such as wide and deep [31], deep and cross [32]. Here we only discuss the architecture of our bestperforming DNN model: a deep feedforward network with residual connections. Figure 2 presents our best performing DNN model. The best performing DNN consists of several parallel paths, each corresponding to a different feature. First, we pass each of our features (vectors of real-numbers) through two hidden layers, where the layers have residual connections [33]. We then concatenate the resultant layers to create a single concatenated layer. Finally, we pass the concatenated layer through two more hidden layers and apply SoftMax activation at the output layer.

V. EXPERIMENTAL EVALUATION
In this section, we provide a comprehensive evaluation of our cuisine inference framework. We conduct extensive experiments on a large-scale transaction dataset to validate the effectiveness of the framework. Specifically, we focus on answering the following research questions.
1) How effective is our framework in inferring the cuisine types of restaurants? 2) How useful are different feature types for classifying restaurants? 3) What are the impacts of different hyper-parameters in overall performance?

A. Experimental Setup
We first describe our experimental setup: evaluation dataset and DNN models.
Evaluation Dataset. Our transaction dataset contains ∼4,000,000,000 transactions in ∼650,000 restaurants. A sizable fraction of these restaurants are chains, e.g., McDonald's, Subway, and KFC. For the purpose of evaluation, we exclude the chain restaurants. The final evaluation dataset contains ∼500,000 restaurants. We generate cuisine labels for 35% of these restaurants using our weakly-supervised method. We then split the labeled restaurants into training (80%) and testing (20%) sets. We used 5-fold cross-validation for tuning the hyperparameters. In addition to the weakly supervised restaurants, we manually label another 1000 restaurants to create a holdout set. DNN Models. To demonstrate the effectiveness of our framework, we develop several DNN models as follows.
(a) Wide & Deep: This is a variant of the wide and deep network [31] that captures the interaction between features. (b) Deep & Cross: This is a variant of the deep and cross network [32] that applies feature crossing. (c) Shallow Feedforward: This is a feedforward neural network with two hidden layers. We first pass each feature through a hidden layer, concatenate them, and finally pass them through another hidden layer. (d) Deep Feedforward: This is a feedforward neural network with four hidden layers. We first pass each feature through two hidden layers, then concatenate them, and finally pass them through two more hidden layers. (e) Deep Feedforward with Residual: This is a deep feedforward network with residual connections [33]. We first pass each feature through two hidden layers (with residual connection between the two layers), concatenate them, and finally pass them through two more hidden layers.

B. Effectiveness of Our Framework (RQ1)
We now discuss the effectiveness of our framework in inferring the cuisine types of restaurants.
DNN Performance Comparison. We compare the performance of our DNN models for inferring the cuisine types of restaurants. Table IV reports the accuracy of our DNN models. These scores are based on the best configuration of hyperparameters (dropout, batch size, and learning rate) determined through a grid search. From the results, we make the following observations.
1) The Deep Feedforward network outperforms the Shallow Feedforward network as expected. The performance gap between the two is significant. performing architecture may seem small in a relative scale, the gain is significant in absolute terms: 1% improvement implies correctly classifying another 5000 restaurants. Pre-trained Embedding Baseline. VISA previously generated a set of general-purpose merchant embeddings for all merchants (including restaurants). We used the pre-trained restaurant embeddings in conjunction with a deep feedforward network to develop a baseline for inferring the cuisine types of restaurants. This baseline achieved an accuracy of 0.51, implying that the proposed framework provides a 49% improvement over the baseline.
Non-Neural Baselines. For completeness, we also compare the performance of our DNN models with several non-neural baselines. Our non-neural baselines include logistic regression and SVM with different kernel functions (linear, polynomial, rbf, and sigmoid). The best performing non-neural baseline is the multi-class SVM with a linear kernel, which achieves an accuracy of 0.351.
Top-k Accuracy. We investigate if further improvement in accuracy can be achieved by increasing the number of predictions. We compute top-k accuracy for our best-performing model, where correct prediction is assumed if the ground truth label appears in the top-k predictions. The top-2 and top-3 accuracy scores for our best-performing model are 0.879 and 0.921, respectively.
Confusion Matrix. As our dataset is highly imbalanced (class imbalance ratio up to 16:1), we study per class accuracy to better understand performance. Figure 3 presents the normalized confusion matrix for the best performing model in the validation test. Notice that our framework leads to a balanced confusion matrix, where per class accuracy is high for all classes, including minority classes such as South Asian.   Holdout Performance. As we use the weakly-supervised labels as ground truth, the labels may introduce some bias.

Dessert
To investigate if such bias affects our final results, we use the manually labeled holdout set to review our framework's performance. We found that our framework's performance in the holdout set is comparable with the performance in the validation set.
Model Stability. As transaction data is time-varying, we investigate if our framework's predictions remain stable over time. To this end, we use another evaluation dataset (with six billion transactions) based on a different three months period. We generate predictions for unlabeled restaurants based on both datasets and compare the results. We observe that the predictions of our model remain stable over time.

C. Importance of Features (RQ2)
We perform an ablation study to understand the importance of different features in cuisine-based restaurant classification. Table V reports the change in accuracy scores obtained by removing one feature at a time. We observe that the neural restaurant embeddings are the most useful features for cuisine type prediction. This finding implies that the embeddings successfully capture the latent characteristics of restaurants. Among the embeddings, the contribution of name embedding is particularly noteworthy. It shows that the remaining words (that have not been used for labeling) in restaurant names contain strong signals about cuisine type. The micro embedding that captures compatibility between a customer and a restaurant is another strong indicator, implying that many customers have a strong preference towards certain cuisine types. The macro embedding is also useful in cuisine type prediction, confirming the taste similarity among the customers who visit a given restaurant. Among the statistical features, temporal patterns play a crucial role, indicating that cuisine types impact the business hours and busy times. Other prominent statistical features are price, location, and customer visitation patterns. As evidenced by the decline in accuracy score, all of the proposed statistical features are also useful in cuisine type prediction.  There are several sets of hyperparameters in our framework that may impact the overall performance, notably, the hyperparameters of the DNN model and the hyperparameters associated with the neural embeddings. To determine the impact of these hyperparameters, we perform a grid search over the space of hyperparameters. Table VI shows the optimal configuration of hyperparameters (in bold). We found that two hyperparameters have a profound impact on performance: batch size (smaller batch size leads to significant improvement in performance) and the dimension of micro embedding (the dimension size significantly affects its contribution).

VI. DISCUSSION
We make the following observations from our results: 1) names are informative in identifying the cuisine types of restaurants; 2) different cuisine types considerably differ across the business dimensions (such as pricing, tipping culture, and serving capacity); 3) it is difficult to distinguish certain cuisine types (say South Asian and South East Asian) from one another. Now, we qualitatively analyze these results and present the implications of our findings.
Cuisine Information in Restaurant Names. Restaurant names often contain rich information about the underlying cuisine types. Recall that, with just 423 keywords (225 seed, 198 bootstrapped), we could label 35% of US restaurants. Further, we validated the efficacy of the labeling through a holdout set. Altogether, the keywords provide high support and confidence in labeling restaurants. We also generated name embeddings form the remaining words in the restaurant names, which became the most informative feature in inferring the cuisine types. Overall, names have the potential to inform the latent characteristics of restaurants.
Cuisine Types across Business Dimensions. Our study reveals several insights about the US restaurant industry. Table III shows some of the insights derived from our statistical features. A careful inspection of these statistics presented reveals the inequalities across different ethnic cuisine types in the US. For example, across all US restaurants, the median tips for European cuisine are much higher compared to the Asian cuisines. These insights could open up new threads of research on the restaurant industry.
Similarity of Cuisines. We analyze the normalized confusion matrix to get an understanding of which classes are harder to separate. We observe that it's hard to separate the following classes: i) MMA from Fast food; ii) MMA from Grill & Steakhouse; iii) South Asian from South East Asian; iv) South East Asian from East Asian; v) European from Fast food; vi) Bar from Grill & Steakhouse; vii) Grill & Steakhouse from Fast food; viii) Grill & Steakhouse from Bar. Note that iii) and iv) are examples of kissing cuisines [2]: cuisines that are similar in terms of ingredients and flavors. In remaining cases, there are many restaurants that serve both types of cuisines. For example, many Mediterranean places serve gyros (a popular fast food), bars often include grills.

VII. RELATED WORK
Our work draws from and improves upon, prior works on restaurant recommendation.
Restaurant Profiling and Recommendation. There is a recent body of work on restaurant profiling and recommendation [34]- [38]. These works have typically relied on user ratings and reviews for generating restaurant profiles and subsequent recommendations. Bakhshi et al. [37] studied the effects of restaurant attributes, local demographics, and local weather conditions at the date of visit on online customer reviews. Zhang et al. [36] designed a unified Collective Implicit Explicit Preference Model (CIEPM) to combine the implicit (check-in) and explicit (review) preference of customers for restaurant recommendation. Lian et al. [34] used restaurant attributes, check-in data, and review text to perform a survival analysis of restaurants in China. We explore the problem of restaurant profiling in an untapped space-from debit and credit card transactions. Unlike user reviews, transaction data does not contain a large corpus of texts or images to infer attributes. Accordingly, we perform an extensive feature engineering for transaction data to infer restaurant attributes.
Studies on Restaurant Industry. Our work draws from the rich literature on the restaurant industry. In addition to the works already mentioned, our study is motivated by the seminal work of Susan Auty [39], and its followups [40].
The key finding of Auty's work is that food (cuisine) type is a primary factor in restaurant selection by consumers. Our proposed work builds upon this premise, where we identify cuisine types of US restaurants to provide meaningful restaurant recommendations

VIII. CONCLUSION
In this paper, we present a framework for inferring the cuisine types of restaurants from debit and credit card transactions, drawing from the literature on the restaurant industry, and applied machine learning. Our framework involves three steps-programmatically generating cuisine labels for restaurants, extracting statistical features and neural embeddings from transactions, and finally, deep neural network based classification. Our framework achieved a 76.2% accuracy in classifying the US restaurants. Our findings have implications for feature engineering for large-scale transaction data, inferring latent merchant attributes, and studying the US restaurant industry. To the best of our knowledge, this is the first framework that performs cuisine inference by analyzing transaction data as the only source.