Body: <p>Many Q-learning techniques have been developed to capture discrete state(observation), actions like a robot in a grid world, and even continuous (state or action) spaces. But I am wondering how we can model the states/space in a time-dependent environment. Please, consider the following example:</p>
  
  <p>There is one smartphone (client) and five compute servers that are addressing/serving many clients (smartphones) at the same time. The smartphone transfers some raw data (e.g, sensor data) to one of those five servers (e.g., every t seconds) and gets the results. Suppose the server computes the stress-level of the client in real-time based on the collected data. </p>
  
  <p>Now, a q-learning agent should be deployed to the smartphone to be able to select the best server with minimum response time (i.e., the goal is to minimize the execution/response time). Note that servers are serving different clients and their load is a function of time and varies from time to time.  </p>
  
  <p>So in the above scenario, I am wondering what would be our "states" and how we can model the "environment"? </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-29T12:30:44.937Z
Comments.Text: Does the client that needs to make a decision know the current load on the server, or have state data that would allow it to approximate it? Is it allowed to have state such as response statistics for its last few requests, or some statistical data from the servers for past equivalent time periods?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-29T12:33:04.173Z
Comments.Text: Architecturally, in the real world, it is more normal to have server allocation managed inside a cluster by a load balancer server rather than by clients, since the service will be able to measure and compare load/responses across multiple services whilst the clients cannot do this. If you are trying to solve a real world problem to create a stabel and fast service, you should probably consider this approach first as opposed to AI at the client, but the AI issue is likely interesting and on-topic here regardless of that.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-29T12:44:38.127Z
Comments.Text: Well, the smartphone can log and store its own last few requests.
Comments.UserDisplayName: user2867237
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-29T12:46:56.027Z
Comments.Text: Regarding load balancer: yes that is correct. I just simplified the problem. I am working on an offloading technique for IoT, in which the and edge device is connected to a fog node, and the fog node is connected to the cloud (with several servers). Now the edge should decide to run the task itself, or offload the task to the fog nodes, or to the cloud.
Comments.UserDisplayName: user2867237
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-29T12:49:37.867Z
Comments.Text: According to the system model, three different sites are considered
  as a feasible platform for offloading computation (1) the
  mobile fog in close proximity of end-user devices, i.e. site L1 (2)
  the adjacent mobile Fog (or distant mobile Fog) to handle mobility
  and load balancing issues, i.e. site L2 (3) the remote public cloud to
  manage huge traffic and computing requirements and archiving,
  i.e. site L3. So the actions are: 1) run in the device,, offload to L1, offload to L2, or offload to L3.
Comments.UserDisplayName: user2867237
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-29T12:53:13.493Z
Comments.Text: it would be great if we could have multiple agents (collaborating with each other) e.g, one agent in the IoT device, one agent in each fog nodes, and one agent in the cloud. so we could offer more actions such as migration within L1, migration from L1 to L2, etc.
Comments.UserDisplayName: user2867237
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-29T12:55:37.777Z
Comments.Text: but let's focus on the simplest case which I explained in my first question.
Comments.UserDisplayName: user2867237
Tags: <reinforcement-learning><q-learning>
Title: Is Q-Learning suitable for time-dependent spaces?
UserDisplayName: user2867237
ThreadText: # Is Q-Learning suitable for time-dependent spaces?
  
  (Asked by: username_0 on 2020-04-29)
  
  Many Q-learning techniques have been developed to capture discrete state(observation), actions like a robot in a grid world, and even continuous (state or action) spaces. But I am wondering how we can model the states/space in a time-dependent environment. Please, consider the following example:
  
  There is one smartphone (client) and five compute servers that are addressing/serving many clients (smartphones) at the same time. The smartphone transfers some raw data (e.g, sensor data) to one of those five servers (e.g., every t seconds) and gets the results. Suppose the server computes the stress-level of the client in real-time based on the collected data.
  
  Now, a q-learning agent should be deployed to the smartphone to be able to select the best server with minimum response time (i.e., the goal is to minimize the execution/response time). Note that servers are serving different clients and their load is a function of time and varies from time to time.
  
  So in the above scenario, I am wondering what would be our "states" and how we can model the "environment"?
  
  **Tags:** reinforcement-learning, q-learning
  
  > **Comments:**
  > **username_1**: Does the client that needs to make a decision know the current load on the server, or have state data that would allow it to approximate it? Is it allowed to have state such as response statistics for its last few requests, or some statistical data from the servers for past equivalent time periods?  
  > **username_1**: Architecturally, in the real world, it is more normal to have server allocation managed inside a cluster by a load balancer server rather than by clients, since the service will be able to measure and compare load/responses across multiple services whilst the clients cannot do this. If you are trying to solve a real world problem to create a stabel and fast service, you should probably consider this approach first as opposed to AI at the client, but the AI issue is likely interesting and on-topic here regardless of that.  
  > **username_0**: Well, the smartphone can log and store its own last few requests.  
  > **username_0**: Regarding load balancer: yes that is correct. I just simplified the problem. I am working on an offloading technique for IoT, in which the and edge device is connected to a fog node, and the fog node is connected to the cloud (with several servers). Now the edge should decide to run the task itself, or offload the task to the fog nodes, or to the cloud.  
  > **username_0**: According to the system model, three different sites are considered as a feasible platform for offloading computation (1) the mobile fog in close proximity of end-user devices, i.e. site L1 (2) the adjacent mobile Fog (or distant mobile Fog) to handle mobility and load balancing issues, i.e. site L2 (3) the remote public cloud to manage huge traffic and computing requirements and archiving, i.e. site L3. So the actions are: 1) run in the device,, offload to L1, offload to L2, or offload to L3.  
  > **username_0**: it would be great if we could have multiple agents (collaborating with each other) e.g, one agent in the IoT device, one agent in each fog nodes, and one agent in the cloud. so we could offer more actions such as migration within L1, migration from L1 to L2, etc.  
  > **username_0**: but let's focus on the simplest case which I explained in my first question.  
  
  ---
  
  ## 0 answers
  
  