Reinforcement Learning in DASH

---------------------------------------------------------------------- ABSTRACT ------------------------------------------------------------Reinforcement Learning is an important class of optimization which has recently been used in the area of dynamic adaptive streaming over HTTP (DASH). Though DASH is very popular method of video delivery in recent years it is plagued with problems when multiple players share a bottleneck link. Thus, this area has become a very active area of research. Two works which implement Reinforcement Learning in DASH are selected and their performances compared against the Conventional DASH player. It is shown that SDP works well for various network conditions.


I. INTRODUCTION
At the end of 1979, we first focused on what is now known as strengthening learning. This was simply the idea of a learning system that wants something, adapting its behavior to maximize a particular signal from its environment. While reinforcement learning had clearly motivated some of the earliest computational learning studies, most of these researchers had gone on to other things, such as classification of patterns, supervised learning, and adaptive control, or had completely abandoned learning [33]. As a result, relatively little attention was paid to the special issues involved in learning how to get something from the environment. Since then, the field has come a long way, developing and maturing in various directions. Reinforcement learning in machine learning, artificial intelligence, and neural network research has gradually become one of the most active research areas [36]. The contributions to establishing and developing relationships to the theory of optimum control and dynamic programming were particularly important. The general issue of learning to attain objectives from interaction is still far from being solved, but our knowledge of it has been considerably enhanced. We can now position constituent concepts, such as temporal-difference learning [8], dynamic programming [6], and function approximation [4], within a meaningful standpoint to the entire problem. The Internet traffic [37] due to video applications is increasing thanks to the diffusion of new devices such as tablets, smart phones, Smart TVs which are connected to the Internet through broadband wired and wireless connections. Video streaming, in the form of usergenerated video distributionsuch as in the case of Youtubeor to deliver movies and TV seriessuch as in the case of Netflixis the application that is driving this growth.
Even though the TCP has been regarded in the past as unsuitable to transport video flows, today videos are streamed through HTTP with the TCP, and web browsers implementing the HTML5 [15] standard are now able to reproduce videos without the use of any external plug-in.
The most common approach to distribute video is the progressive download streaming: the video content is encoded at a given bitrate and it is sent to the user as any other file through a HTTP connection [27]. The issue with this approach is that, even though TCP connections [2] are elastic, the video content transported through the TCP socket is not elastic; thus, a persistent mismatch between the encoding bitrate and the net-work available bandwidth may result in playout interruptions. Another drawback of such an approach is that mobile devices, such as tablets or smart phones, may not be able to play a high definition video due to their limited computational resources.
To tackle these issues, the video content must be made adaptive [24], [18], [23], [22], [20], [21]. Among the approaches proposed so far, the stream-switching is gaining momentum due to its deployment and implementation simplicity. With this approach, the video is encoded at different bitrates and resolutions, the video levels, and the encoded videos are logically or physically divided into segments of fixed durations. The streamswitching controller decides, for each video segment, the video level to be streamed, see Figure 1. From the architectural standpoint, two different approaches can be used to implement a stream-switching algorithm: the client-side, that places the controller at the client, and the server-side, that implements the controller at the server. It has been shown that the client-side algorithms proposed so far generate an on-off traffic pattern at steady-state that can lead to unfairness when many video flows share a bottleneck (see Figure 2) [14], [25], [16]. Moreover, in [1] it has been established that the adaptive video players of three popular video streaming services were not able to get a fair share when coexisting with a TCP greedy flow. Authors name this issue the "downward spiral effect" and ascribe its cause to the onoff traffic pattern described above; authors suggest increasing the segment size and filter bandwidth estimates to tackle this issue.

Fig. 2. Bottleneck Links in video distribution
In the presence of competing HTTP-based adaptive streaming (HAS) clients the TCP throughput does not always faithfully represent the fair-share bandwidth [25]. Three performance issues that can take place when two or more adaptive streaming players share a network bottleneck and compete for available bandwidth are instability, unfairness and utilization [25] It is shown that in the case of two competing video flows Adaptive video streaming players provide a received video rate that oscillates around the fair share, but with an increased number of video level switches [14]. Depending on the temporal overlap of the ON-OFF [16] periods among competing players, they may not estimate their fair share correctly. In the case where both players overestimate their fair share, they may request a video representation with a higher bitrate than the fair share, which causes network congestion. Consequently, the players measure that their TCP throughput is lower than their previous fair share estimate, and so switch down to a lower video bitrate level. This creates a repeating oscillatory scenario, so inducing instability.
A scenario can also occur where some players are requesting chunks with lower bitrates than the other players. This can occur as some players observe a throughput lower than the fair share, while others observe a throughput that is more than the fair share. This means that some players overestimate its fair share. When some players overestimate their fair share, it can be that the system of players converge to a stable equilibrium, but unfair [1]. This occurs as the players with the larger fair share estimates request higher bitrate video levels. Even in the case where two players estimate their fair share correctly, bandwidth underutilization can still be prevalent. This occurs as both players request the same lower video bitrate level, which causes underutilization, even though stability and fairness still exist. In reality, several other factors can play an important role in the appearance and extent of instability, unfairness and underutilization, such as the exact player adaptation algorithm, TCP dynamics, bandwidth fluctuations, and the variability of the video encoding rate.
We group these problems into three categories: The first relates to the stability of the players in terms of requested bitrates and video quality. The second is the unfairness among competing players. The third is the potential bandwidth underutilization when multiple adaptive players compete. They are described as follows: Instability: The instability [25] for player at time is given in Equation below, where ( ) =is a weight function that puts more weight on more recent samples.
is selected as 20 seconds.
be the Jain fairness index (cf. Equation 10) calculated on the average received rates [14], , (cf. Equation below) at time over all players. The unfairness at time t is defined as √1 − . A lower value implies a fairer allocation.
The utilization metric [16] is defined as the aggregate throughput during an experiment divided by the available bandwidth in that experiment (cf. Equation below, where is the throuput at time and is the experimental available bandwidth).
A discussion on Reinforcement Learning is given in Section II. DASH approaches using Reinforcement Learning is explored in section III. Experimental setup is given in section IV. Results are given in section V and finally, the conclusion is given in Section VI.

II. REINFORCEMENT LEARNING
Reinforcement learning is learning what to dohow to link scenarios to actionsto maximize numerical reward amount(s). The learner is not told what actions to take, as in most forms of machine learning, but instead, by trying them, he must find out which actions give the most reward. In the most exciting and difficult cases, scenarios can influence not only the present reward but also the next scenario and all the rewards that follow. The fundamental concept is merely to capture the most significant elements of the actual problem that a learning agent is confronting in order to attain a objective. The purpose of the agent is to include only these three aspectssensation, action, and goalin their easiest possible forms without trivializing any of them, see Figure 3. The crux of reinforcement learning is capturing essential parts of a problem, when a learning agent interacts with its surroundings, while trying to accomplish a goal. The agent should be able to "sense" its surroundings and make decisions, on the conditions it observes. The agent desires to complete one or more goals. In achieving its goals, the agent attempts to maximize some reward given the actions it takes. Thus, the agent discovers most fruitful actions by checking the rewards for each goal. On a stochastic task or assignment, each action attempt is done multiple times to gain a reliable expected reward estimate [33] considers the whole problem of a goal-directed agent interacting with its surroundings. Reinforcement learning agents: (a) have explicit goals, (b) can sense aspects of their environments and (c) can choose actions to influence their environments. The trade-off between exploration and exploitation is one of the challenges that arise in reinforcement learning and not other types of learning. A reinforcement learning agent must prefer actions he has attempted in the past and discovered to be efficient in generating reward in order to achieve its goal(s). To achieve reward, the agent must exploit what he already knows, but he must also explore to create choices for better action in the future. The agent must attempt a variety of actions and gradually favor those that seem to be the best. However, each action must be attempted many times in a stochastic assignment to obtain a credible estimate of its anticipated reward. Another key feature of reinforcement learning is that it specifically considers entire problem of engaging with an uncertain environment by a goal-directed agent [5]. All reinforcement learning agents have specific objectives, can sense elements of their settings and can choose activities to impact their environments. Four main sub-elements of the reinforcement learning system can be identified: a policy, a reward function, a value function and, optionally, an environment model [3]. A policy defines the behavior of the learning agent at a particular time. A policy is a mapping of actions to be taken in those states from perceived environmental states. Policies can generally be stochastic. A reward function describes the target of a problem of reinforcement learning. It maps each perceived state (or state-action pair) of the environment to a single number, a reward, indicating that that state's intrinsic desirability. The sole goal of a reinforcement learning agent is to maximize the combined long-term reward it gets. The value function specifies the long-term usefulness. The value of a state is the total percentage of reward that an agent can expect to collect from that state into the future. Values indicate the long-term desirability of states and the rewards available in those states after taking into account the states that are likely to follow [7]. Rewards are primary in a sense, while values are secondary as indicators of rewards. There could be no values without rewards, and the only purpose of estimating values is to obtain more reward. These are the values that we are most concerned with when taking decisions and evaluating them. Action choices are made based on value judgments [33]. We are looking for actions that bring the highest value, not the highest reward, because these actions receive the greatest reward for us in the long run. The derived amount called value in decision-making and planning is the one we are most interested with. Regretfully, determining values is much harder than determining rewards. The environment basically gives rewards directly, but values must be estimated and reestimated from the sequences of observations that an agent makes throughout his lifetime. A method for the efficient estimation of values is the most important component of almost all reinforcement learning algorithms. A model of the environment is the fourth and final element of certain reinforcement learning systems. This is something that imitates the environment's behavior. For example, the model could predict the resulting next state and next reward given a state and action. Modern reinforcement learning covers the range from low-level, trial-and-error learning to deliberative, high-level planning [33].

III. REINFORCEMENT LEARNING DASH-BASED APPROACHES
Researchers in [28], [32], [17] apply reinforcement learning in quality selection algorithms for streaming video. In multi-player competitive environments these reinforcement learning approaches are able to achieve fairness. A coordination proxy facilitates co-ordination among players. Thus, these approaches fall under networkassisted DASH. These approaches learn and adapt their policy depending on network conditions, yielding low overhead. [11] progressively maximize a pre-defined QoErelated reward function. By utilizing this action, players are able to learn an optimal request strategy. An autonomous RL-learning agent [26] provides adaptive video streaming in best effort networks. The agent learns an optimal control strategy regarding user-QoE, without the need for implementation of a complex heuristics. Researchers in [9] use an original reinforcement learning approach to develop better adaptation agents. Agents gradually improve, by taking into account, both user's behaviour and usage context.

A. Method A [35]
HTTP Adaptive Streaming (HAS) is becoming the defacto standard for Over-The-Top video streaming. A HAS video consists of multiple segments, encoded at multiple quality levels. Allowing the client to select the quality level for every segment, a smoother playback and a higher Quality of Experience (QoE) can be perceived. Although results are promising, current quality selection heuristics are generally hard coded. Fixed parameter values are used to provide an acceptable QoE under all circumstances, resulting in suboptimal solutions. Furthermore, many commercial HAS implementations focus on a video-ondemand scenario, where a large buffer size is used to avoid play-out freezes. When the focus is on a live TV scenario however, a low buffer size is typically preferred, as the video play-out delay should be as low as possible. Hard coded implementations using a fixed buffer size are not capable of dealing with both scenarios. In this paper, the concept of reinforcement learning is introduced at client side, allowing to adaptively change the parameter configuration for existing rate adaptation heuristics. Bandwidth characteristics are taken into account in the decision process, thus allowing to improve the client's bandwidth-awareness. Focus in this paper is on actively reducing the average buffer filling. Authors show that using the proposed learning-based approach, the average buffer filling can be reduced by 8.3% compared to state of the art, while achieving a comparable level of QoE.

B. Method B [10]
HTTP Adaptive Streaming (HAS) is becoming the de facto standard for Over-The-Top (OTT)-based video streaming services such as YouTube and Netflix. By splitting a video into multiple segments of a couple of seconds and encoding each of these at multiple quality levels, HAS allows a video client to dynamically adapt the requested quality during the playout to react to network changes. However, state-of-the-art quality selection heuristics are deterministic and tailored to specific network configurations. Therefore, they are unable to cope with a vast range of highly dynamic network settings. In this letter, a novel Reinforcement Learning (RL)-based HAS client is presented and evaluated. The self-learning HAS client dynamically adapts its behaviour by interacting with the environment to optimize the Quality of Experience (QoE), the quality as perceived by the enduser. The proposed client has been thoroughly evaluated using a network-based simulator and is shown to outperform traditional HAS clients by up to 13% in a mobile network environment.

IV. EXPERIMENTAL SETUP
A virtual network is setup on the same host machine creating a custom emulation framework. Our setup consists of client players, video servers, and a bottleneck link. The server resides on a Windows 10 machine. All experiments are performed on a Windows 10 client with an Intel(R) Core(TM)i7-5500U CPU 2.40GHz processor, 16.00 GB physical memory, and an Intel(R) HD Graphics processor. It serves video data to the client(s) who are on a Ubuntu operating system hosted on VMware. The virtual machine is allocated 12GB of physical memory.
TAPAS [13] is installed on Ubuntu 15.04 Linux. The TAPAS Adaptive Video Controller client makes different video segment bitrate level requests to the Apache server. TAPAS allow multiple instances of the player to be created enabling multi-client scenarios. This work involves the interaction between adaptive streaming algorithm at the controller and TAPAS player (cf. Figure  6). All traffic between clients and servers go through the bottleneck, which uses VMware settings which allow bandwidth limits to be set during the experiment. TAPAS support both the HTTP Live Streaming (HLS) and Dynamic Adaptive Streaming over HTTP (DASH) format. Algorithms that uses Method A and Method B was tested and shown to work on both MPEG-DASH [34], and Apple HTTP Live Streaming (HLS). This makes it useful for video on demand (VOD) [29] and live streaming [12], for example, real-time video chats. However, the MPEG-DASH standard is used for testing in this research paper, because it makes the experiments more comparable to the ones in the research literature, for example, [30]. The ten-minute-long MPEG-DASH video sequence "Elephant's Dream" 1 is encoded at twenty different bitrates, between 46 Kbps to 4200Kbps and five different resolutions, between 320x240 to 1920x1080, is used to run the experiments (cf. Table II). The video is encoded at 24 frames per second (fps) using the AVC1 codec [31]. Fragment duration of 2s is used and is recorded in the mpd playlist accordingly. All the DASH files (.m4s fragments and .mpd playlists) are placed on the Apache server. We implemented three client-side algorithms in the TAPAS controller. The conventional approach is present by default and is used as a baseline in which to compare against other algorithms. TAPAS is lightweight in built, thus allowing the same receiving host to run a large number of separate video player instances at the same time at different command line interfaces. Thus, it allows the multi-client scenarios which are essential to the work in this paper.

V. RESULTS
The first experiment illustrates five players competing at a 20Mbps bottleneck link. Table 1 gives the results. Method A outperforms Method B and the Conventional.

VI. CONCLUSION
Reinforcement Learning is an important class of optimization which has recently been used in the area of dynamic adaptive streaming over HTTP (DASH). Though DASH is very popular method of video delivery in recent years it is plagued with problems when multiple players share a bottleneck link. Thus, this area has become a very active area of research. Two works which implement Reinforcement Learning in DASH are selected and their performances compared against the Conventional DASH player. It is shown that Reinforcement Learning works well for various network conditions. However, one method outperforms the others in the experiments conducted. In future work multipath schemes [19] should be used together with reinforcement learning techniques to better DASH-based approaches.