Solving Multi-Period Financial Planning Models: Combining Monte Carlo Tree Search and Neural Networks

This paper introduces the MCTS algorithm to the financial world and focuses on solving significant multi-period financial planning models by combining a Monte Carlo Tree Search algorithm with a deep neural network. The MCTS provides an advanced start for the neural network so that the combined method outperforms either approach alone, yielding competitive results. Several innovations improve the computations, including a variant of the upper confidence bound applied to trees (UTC) and a special lookup search. We compare the two-step algorithm with employing dynamic programs/neural networks. Both approaches solve regime switching models with 50-time steps and transaction costs with twelve asset categories. Heretofore, these problems have been outside the range of solvable optimization models via traditional algorithms.

Introduction
Monte Carlo tree search (MCTS) is a heuristic based multi-purpose search algorithm which searches for optimal decision by combining the concepts of tree search and reinforcement learning. The MCTS algorithm dates back to 2006, when a Computer Go program won the 10th KGS computer-Go tournament (Coulom 2007). Since then, MCTS has been applied in a variety of fields, from security evaluation methodology of image-based biometrics authentication systems (Tanabe et al. 2009) to murder mystery generation (Jaschek et al. 2019).
The algorithm has gained popularity over the past several years, especially due to the success of DeepMind's AlphaZero, a program that beats the reigning world champion programs in games like chess, shogi and go (Silver et al. 2017) .
The original MCTS algorithm is mainly designed to simulate and move forward in time in a multi-period decision process, that is the algorithm slowly builds a search tree depending on the simulation results where the depth of a node in the tree represents how far away into the future it is. However, depending on the problem, the convergence can be prohibitive. In our LMCTS (Lookup Monte Carlo Tree Search) algorithm, we aim to remedy this by combining the strengths of the MCTS algorithm and dynamic programming. We start solving our multi-period problem from the end and store the results of the MCTS algorithm on a lookup table. Then we move backward in time and solve a bigger problem by using the results of the lookup table as our rollout policy in the simulation phase of the MCTS algorithm. This niche algorithm can only be employed in a problem where the number of state-action pairs are tractable, otherwise the time it would take to calculate the lookup table and its size would explode. The advantage of the algorithm over its counterpart, dynamic programming, is its flexibility thanks to its simulative nature. The new algorithm is able to handle complex problem definitions, objective functions and constraints on the state space.
MCTS was originally designed as to deal with problems that have finite discrete action spaces. Different policies are available within the algorithm, but most famous is the clas-sical UCT method (Kocsis and Szepesvári 2006) which tackles the exploration-exploitation dilemma that occurs in problems with stochastic outcomes. Multiple variants have been developed to extend the MCTS algorithm to continuous action spaces. Progressive widening (Coulom 2007) or progressive pruning (Chaslot et al. 2008) adopts a growing set of discrete actions to handle continuous action spaces. cRAVE (Couëtoux et al. 2011) uses a similar idea applied to the RAVE algorithm. HOOT (Mansley et al. 2011) applies the HOO (Bubeck et al. 2011) algorithm in UCT instead of UCB. However, for our problem setting, we have chosen KR-UCT (Yee et al. 2016) to be the most suitable variant of the MCTS that handles continuous action spaces. It is advised for the reader to check the aforementioned paper which analyzes the KR-UCT algorithm and compares it empirically to other algorithms for continuous action spaces. Although the algorithm is mainly developed for "actions with execution uncertainty" it works similarly for our financial decision model under uncertainty (In contrast, a problem setting without any action execution or outcome uncertainty is chess).
The distinguishing traits of KR-UCT are explained in (Yee et al. 2016) as follows: a) information sharing between all actions under consideration, b) identification of actions outside of the initial candidates for further exploration, c) selection of actions outside of the candidate set. The benefits of these traits become clearer once the problem setting is established in the next section. In short, due to time constraints, we may not have adequate computational resources to run enough simulations at each time step. Thus, the algorithm benefits from information sharing between all actions under consideration. And since the action space under consideration is massive, identification of actions outside of the initial candidates and selection of actions outside of the candidate set are traits which are extremely valuable to us.
A multi-period asset allocation problem is the focus of this study. Multi-period problems enjoys its farsightedness over single-period investment planning, where economic regime dynamics as well as investors' life cycle goals are better cooperated in the model. Elegant as single-period investment models, they fail to address the issues such as intermediate cash flows, transaction costs and trade-off between short-term versus long-term benefits. In the early days of financial mathematics, multi-period asset allocation problems are often solved with analytical solutions or by numerical methods like dynamic programs. When numerical methods are employed, the sizes of problems are usually constrained by computing powers.
Though powerful in solving multi-stage optimization problems, vanilla dynamic program suffers the curse of dimensionality, a phenomenon that the required running time grows exponentially in the complexity of the problem. Modern algorithms are developed over the years to overcome the curse of dimensionality, Monte Carlo Tree Search and neural networks among the most known ones (Silver, David, and Aja Huang, and Chris J. Maddison, and Arthur Guez, and Laurent Sifre, and George Van Den Driessche, and Julian Schrittwieser and et al. (2016)).
The Merton portfolio problem is among the earliest and most famous inter-temporal investment problems. (Merton 1969) proposes and finds a closed form solution for a continuoustime asset allocation problem where an investor maximizes her utility function for both finite and infinite lifetime. Since then, portfolio management over long horizon has been widely studied, and various extensions are made to the original Merton's problem to better describe the real economic environment. Literature finds that historical financial data have heavier left tails than normal distribution, and it is therefore not representative to model the returns with a symmetric distribution such as normal distribution. In particular, the returns form volatility clusters when crash happens, in which period asset returns behave substantially different to normal periods. (Nystrup, Peter, and Henrik Madsen, and Erik Lindström 2018) argue that investors benefit from dynamic strategies that weigh assets differently in crash versus in normal regime. In this paper, we follow (Li and Mulvey 2021) and analyze the trading strategies in a market switching between two possible regimes -normal and crash -with a Markov process. They present results for a 50-period 11-asset problem involving multiple regimes solved by a combined method with dynamic program and neural network, the size of which is untraceable with traditional methods alone. Here, we compare the performance and efficiency of several numerical methods, under various constraints on budgeting, transaction costs and utility functions. We will be tackling a 50-period stochastic optimization problem, which is quite an ambitious task to say the least.
The main contribution of the paper is to introduce the benefits of the MCTS algorithm to the financial portfolio optimization literature. To our knowledge, MCTS was not used to tackle a multi-period financial portfolio optimization problem under a regime-switching framework before. In addition, we extend the vanilla MCTS algorithm via novel version of it called LMCTS. We also employ a variant of the UCT algorithm called KR-UCT which is more suitable for the problem at hand due to reasons explained in the previous paragraphs.
We solve our multi-period problem without transaction costs and use the solution of the LMCTS algorithm as a starting point for a neural network which handles the transaction cost of the problem. We compare the results with another neural network which uses a dynamic programming solution as a starting point.
The paper is organized as follows. Section 2 describes the underlying model. Here we define our general multi-period portfolio allocation problem which utilizes a Markov regime switching framework, and specify the utility functions and transaction costs that are used.
Section 3 explains the methodologies used to tackle the problem at hand, which are MCTS, neural networks, and dynamic programming. Firstly, the Monte Carlo tree search algorithm and its four main steps (selection, expansion, simulation and backpropagation) are described.
Next, the LMCTS algorithm, the KR-UCT function and our specific implementation of the Monte Carlo tree search for the problem is explained thoroughly. Then, general neural networks, recurrent neural networks and the way our neural networks are trained for the problem at hand are described. The section concludes with a brief discussion of dynamic programming. Section 4 presents our empirical results. The main goal is to compare the performances of starting algorithms for the recursive neural network, a Monte Carlo tree search solution and a dynamic programming solution as starting points respectively. We explore cases where we allow shorting and do not allow shorting separately. Also we observe algorithmic performance under different utility functions, namely probability of reaching a goal and terminal utility maximization. Section 5 contains the concluding remarks.

Model
In our multi-period portfolio optimization problem, we assume n ≥ 1 risky assets and one risk-free asset in the market (the risk-free asset can be considered as cash) where the prices of risky assets follow a set of correlated geometric Brownian motions processes. The parameters of the Brownian motions (the means and the covariance matrix) depend on the regime. The values of these parameters are inferred from historical values of real assets. A hidden Markov model is employed for the regimes, meaning that the investors do not know the regime that they are in, but they can infer the probability of being in a regime based on the asset returns they observe. The goal of the investor is to maximize their terminal utility and to do that she may rebalance her portfolio at the end of each period.

The general multi-period portfolio allocation problem
The general model is as follows: subject to where T is the number of periods, x 0 , x 1 , ..., x T −1 ∈ R n are the decision variable for the asset allocations at the beginning of each period, 1 ∈ R n is the vector of all ones, W t is the wealth at the beginning of period t, W → t is the wealth at the end of period t, r t ∈ R n is the vector of returns in period t, π → t ∈ R n is the allocation vector at the end of period t, is the element-wise multiplication operator, and C(W ; π → , π) is the dollar value of transaction and market impact costs when the allocation is rebalanced to π → from π with current wealth being W , and F t is the cash flow at time t. We assume that the initial wealth, W 0 , is given.
The distribution of asset returns r t and the function of transaction cost C(·; ·, ·) depend on the problem at hand.
The objective (1) is to maximize a utility function which is most likely to be a function of the terminal wealth W T , such as expected terminal wealth or probability of reaching a goal with the terminal wealth. Constraint (2) ensures that the total percentage allocation to assets is equal to 1. Equation (3) updates the wealth vector according to realized returns and asset allocations. Equation (4) updates the asset allocation vector according to the returns.
Equation (5) calculates the wealth in the next period using wealth at the end of this period and the transaction cost.

Utility Function
The investor in our model will be aiming to maximize the expected value of her utility function, which is a function of the terminal wealth: The two different utility functions that we will be looking at are the CRRA(constant relative risk aversion) utility function: log(W ), γ = 0 and the probability of reaching a goal, G: The CRRA utility function (7) and the non-convex probability of reaching a goal utility function (8) will be used to compare DP-DNN and MCTS-DNN methods.

Regimes and Returns
As mentioned, we assume a Markov regime switching framework. The r t in our model follows a set of correlated Geometric Brownian motions where the parameters are: S t represents the regime at time t. The returns on the risk-free rate r f are fixed, but the value may depend on the regime.

Transaction Cost
The transaction cost function C(·; ·, ·) can take any form. We assume a linear function in this paper. Selection: The selection step corresponds to moving down the tree and selecting a node based on a tree policy. The tree policy is basically an evaluation function according to which the algorithm decides on which path to follow. Different tree policies can be adopted, but the most famous and common one is the Upper Confidence Bounds Applied to Trees (UCT) (Kocsis and Szepesvári 2006). UCT is a function of the mean return of the node and the number of times the node is visited, and with the help of a tunable constant, it allows the user to tackle the exploration-exploitation dilemma. The UCT function is as follows: where v is the mean reward of the node, N is the total number of simulations done from the parent node, n is the number of simulations done through the node at hand and C is the tunable exploration parameter. This function is calculated for every child node of the parent node and the one with the highest function value is selected.
Expansion: Expansion is the creation of a new leaf node in the tree. The expansion can be made at a node which already has child nodes, but is not fully expanded (This means that not all possible actions have been taken at that node yet). Expansion can be made from a leaf node as well, in which case the newly created node would be the first child of the parent node.
Simulation: Simulation is the process of simulating from the chosen/newly created node. This process is also called rolling out. Many different rollout policies can be used, such as the uniform random rollout policy in which the actions taken during the simulation phase are chosen at random with equal probability.
Backpropagation: Backpropagation is the act of updating values of the nodes that were along the simulation path. After the simulation is over and a reward is obtained, backpropagation algorithm updates the value of all the nodes along the traversed path. The values that are of interest are the number of times the nodes are visited and the mean rewards of the simulations that the respective nodes participated in.

LMCTS
In the original MCTS algorithm the simulation phase is guided by what is called a rollout policy, which determines the actions taken during the simulations. We need to use a rollout policy that we do not yet know what the optimal action is in the future time steps. The hope is that the algorithm will still converge with enough simulations, regardless of how inaccurate the rollout policy is. There are different types of rollout policies. If there is no domain knowledge, the rollout policy generally implemented is the uniform random rollout policy. That is, we choose our action randomly with equal chances for each possible action at each node during the rollout phase. However, if we possess domain knowledge that can guide us during the rollout phase, we aim to outperform the uniform random rollout policy (James et al. 2017).
If a superior rollout policy improves performance, the best performance should be obtained using the ultimate best rollout policy, which is the optimal policy at each time step.
This is the motivation behind the LMCTS (Lookup Monte Carlo Tree Search) algorithm.
As mentioned before, this cannot be implemented in the original algorithm since the optimal policy for the future time steps are unknown. LMCTS starts solving the problem from the end of the horizon, rather than the present time. First, we solve a one period problem and store the results to a lookup table. This lookup table will have the LMCTS solutions for all possible states at time n-1 for an n-period problem. Next we proceed to solve the two period problem. Now, we need to follow a rollout policy to complete our simulations. Instead of using a random rollout policy, or any other arbitrary rollout policy, we employ the results of the LMCTS lookup table as our rollout policy. We again store our results to our lookup If we were to solve the problem with transaction cost, we wouldn't be able to use LMCTS directly. In this case, the optimal portfolio at any time step depends on the previous time step's portfolio. This means that we would have to construct a table that stores the LMCTS value for every possible belief and portfolio duos. This would be practically impossible to do. In our two-step algorithm to solve the multi-period portfolio allocation problem with transaction costs, we implement the LMCTS solution for the no transaction cost case (and dyanmic programming for comparison), and let the neural network handle the transaction cost. Hence, the LMCTS algorithm is indeed viable in this algorithm pipeline.
Note that the LMCTS algortihm only modifies the simulation phase of the MCTS algorithm. It is compatible with any sort of selection, expansion and back-propagation methods.

Kernel Regression
Kernel Regression is a nonparametric technique that estimates the conditional expectation of a random variable. In 1964, Nadaraya and Watson (Nadaraya 1965) (Watson 1964) came up with the idea of estimating the expected value of a point based on a weighted average of the values of other points in the data set where the weights were inversely correlated with the distance between the points. The Nadaraya-Watson estimator for the expected value of a point is: where (x i , y i ) represents the data and K(., .) is the kernel function. Different kernel functions can be used, but in this paper, a relatively popular function, the radial basis function (RBF), will be employed. The denominator of the Nadaraya-Watson estimator (11) is also called the kernel density which is a measure of relavant data to the point of interest: The notations in this section are chosen to be the same with (Yee et al. 2016).

KR-UCT in LMCTS and the Portfolio Allocation Problem
The KR-UCT is an extension to the original UCT, which applies the kernel regression values This new KR-UCT function is basically UCT where v is replaced with E[v|a] and n is replaced with W (a). The expected value of each action is now a function of the mean rewards of every single action node we have, weighted by their distance to the node in consideration.
A closer node, where closeness is defined as having a relatively larger kernel function value, has a larger weight compared to a distant node and hence has a bigger effect on the expected value of the action node at hand. This allows the algorithm to share information between the nodes and gives us a better estimate of the potential value of the node.
Expansion: As mentioned, this algorithm applies a variant of the progressive widening.
Whether a new node will be added in the expansion phase or not is determined by a linear function of the number of visits to the node. In the case of an addition of a new node, we want a new node that is not too far away from our current best node for efficiency purposes, but we also want it to be relatively distant so that we can explore a new region in the action space. This trade-off is balanced by choosing the farthest node in terms of kernel density (argminW (a)) out of the nodes which are at least τ close to the current best action (K(action, a) > τ ). τ is a hyper-parameter that can be tuned according to the data. There are some specific changes made to the algorithm in this part using our domain knowledge to get a faster algorithm. Domain knowledge in general consists of our knowledge of the problem at hand which allows us to come up with more efficient algorithms. We have identified two considerations. First, it is expected for solutions of neighboring time steps to be similar to each other, for the same regime belief. To take advantage of this, the first node coming out of any node is manually coded to be the solution of the next timestep, which is retrieved from the lookup table of LMCTS. For the other expansions, we employ an approximation for the optimization described in this part as in (Yee et al. 2016). This is mainly done for computational efficiency. Instead of solving a fully fledged optimization problem, we determine a set of actions and select from these actions the one with minimal kernel density.
The set of actions to be considered are the feasible portfolio allocations around the current best portfolio.
We have also implemented breadth wise expansion only, that is we are essentially employing trees with depth one at each time step. The point of building a deeper tree is to utilize better actions in the further time steps to more accurately measure the performance of the current actions in the tree. Since we are already using the optimal actions in the next time steps in LMCTS, there is no point in building a larger tree. The algorithm itself is totally compatible with a larger tree, but it would only cost unnecessary computational time.
Simulation: The main logic of the simulation phase is the same. We simulate until the end of our defined horizon and achieve a reward at the end. "ROLLOUT" represents the rollout phase in the pseudo-code (line 16). The rollout policy followed here is actually the LMCTS rollout policy, which is the deterministic policy described in the previous sections.
We are adopting a Markov switching framework for the regimes, so our belief of the current regime is constantly updated after we get new return values each period. This holds true for the original problem of the multi-asset portfolio allocation, but also for the individual simulations within the MCTS algorithm. The beliefs are updated: and the wealth is updated for the original problem and for the simulations which is done as follows: Backpropagation: The backpropagation step is identical to the original MCTS, the relevant variables are updated along the traversed path using the reward obtained at the simulation step.
As mentioned, LMCTS uses information sharing across the nodes for each instance because we know that similar allocations should lead to similar returns. Another domain knowledge we have is that the optimal allocations in an asset at nearby time steps should not be too far away from each other. Depending on the size of the problem and the computational power at hand, a simulation based algorithm such as LMCTS may not fully converge, especially for time steps that are closer to the present time. We employ a commonly used signal filtering algorithm called Savitsky-Golay filter (Savitzky and Golay 1964) for each asset independently across 50 timesteps to denoise the final result to give a more robust starting point for the neural network. The final results are normalized to ensure that we still obey the budget constraint.

General neural networks
Artificial neural networks, referred as neural networks in this paper, are inspired by the way biological neural systems process information. They are widely employed in predictive modeling and adaptive controlling. A neural network is based on a connected set of artificial neurons, where each connection is associated with a real number called weight that represents the relative strength of the connection. Figure 2   When training, the neural network is fed with examples, and it adjusts the weights by gradient method to minimize the loss function. The process is called backpropagation. In each training step, the weights are updated according to the gradients of the loss function with respect to the weights as well as to the learning rate. Selecting learning rate is a nontrivial problem in training a neural network. On one hand, learning rate being too small causes the neural network to learn slowly and may result in unexpectedly long training time; on the other hand, if the learning rate is too large, the neural network may miss the global optimum.

Recurrent neural networks
where W hh and W xh are the matrices of weights respectively, and θ is the activation function. Herein, we will employ recurrent neural network to learn the trading strategy of a multi-period asset allocation problem. In particular, when given complex problems, neural networks do not always converge to optimality in a fast and accurate way. However, it helps the neural networks to converge to global optima if the starting point is relatively close to the optimal solution. A good starting point does not only improves the performance overall, but also shortens the time it takes to train the neural network. For our multi-period asset allocation problem involving transaction costs, we will take advantage of methods such as MCTS and dynamic program to find an approximate solution within reasonable running time, and then utilize neural networks to tune the allocation decision.

Optimal no-trade zones via recurrent neural networks
The optimal trading strategy under geometric Brownian motion(GBM) assets and proportional transaction cost is to form a no-trade zone (Davis, Mark HA, and Andrew R. Norman 1990). If we are in the no-trade zone, no action is required. If not, the agent is required to rebalance the portfolio to the closest point in the no trade zone. We are following the neural network structure in (Li and Mulvey 2021) to come up with no-trade zones. The no-trade zone is parameterized as follows: where p and τ represent our belief and time step respectively. π(p, τ ) is the optimal solution under no transaction cost(embedded into the neural network as DP/LMCTS solution).
u(p, τ ) and l(p, τ ) are the upper and lower no-trade zone boundaries. The goal of the neural network is to essentially learn f u (p) and f l (p).
The computational graph of the neural network can be seen in (Figure 3) which is based on (Han, Jiequn, and Weinan E 2016) and Mulvey, John M., and Yifan Sun, and Mengdi Wang, and Jing Ye 2020) (2020). In the computational graph, S t is the unobservable real regime of market, r t is the vector of realized returns of the risky assets, p t is the agent's belief in the current market regime, h l and h u are the hidden layers in the neural network.
W (t − ) and π (t − ) are the wealth and asset allocations at time t respectively before rebalancing.
π (t + ) is chosen based on the no-trade zone and the wealth after rebalancing is calculated as Finally, the loss function of the neural network is −E[U (W T ]. We encourage readers to refer to (Li and Mulvey 2021) to have a deeper understanding of the neural network structure and no-trade zones. Figure 3: Computational graph of the neural network used.

Training neural network to maximize the probability of reaching goal
We train the weights of neural networks via gradient propagation. When the objective is a CRRA utility function, there is a natural non-zero derivative of the objective with respect to weights. On the other hand, when the objective is the probability of reaching goal, the slope is zero almost everywhere. In addition, since transaction costs deteriorate the chance of reaching goal in a non-continuous manner, the choice of starting point is less obvious in this case. In this subsection, we provide a strategy to tackle the issues associated with this objective function.
To maximize the probability of reaching a given goal, we choose to employ the DP/MCTS solution for CRRA utility under zero transaction costs as the starting point to feed into neural networks. The reason is two-fold: First, to maximize directly on probability, the current wealth needs to be considered in state space and therefore slows down the calculation of starting point. Secondly, the allocation strategy gets suboptimal as transaction costs are considered, and in particular, one shall almost always invest more heavily in early stages because the transaction costs consumes part of the profit and diminishes the probability of reaching goal. This makes the solution for maximizing probability deviate from the center of no-trade zone once transaction costs are added.
To address the issue that current wealth would affect the allocation decision, we feed the current wealth along with the regime estimation into the neural networks. In addition, an extra neural network is placed to adjust the starting solution so that the no-trade zone is centered at the adjusted solution instead of the CRRA solution. Recall that the neural network is trained based on gradient propagation, whereas the objective of probability of reaching goal has gradient zero almost everywhere. To overcome this issue, we replace the objective with an approximated function who has positive slope everywhere (Figure 4).
Please refer to Section 4 to the empricial problem setup to have a better understanding of ( Figure 4).

Dynamic Program
Dynamic programming is a algorithmic technique for solving optimization problems, where the original problem is break down to simpler subproblems, and the solutions to subproblems are utilized for solving the original problem. It is widely used in sequential problems, where subproblems are often well-defined by its nature. To fully describe a dynamic program, one defines the state space that includes all information needed for decision making, the action space that contains all possible actions to take, a transition function that tells the outcome of certain actions under some state, and a value function that quantifies the goodness of an action under a state. For example, in our setting, a value function returns the expected terminal utility of taking an action given the current market environment.
It is a natural algorithm for our multi-period asset allocation problem, if one has sufficient computational power to solve for the whole system. However, when the size of state space or that of the action space grows, dynamic program faces the curse of dimensionality, meaning that the running time would grow exponentially with the complexity of the problem. In our allocation example, if there are no transaction costs, we may simplify the state space to a two dimensional space that includes the probabilistic estimation of underlying regime and the time until horizon. The prices of risky assets does not need to be included in the state space when CRRA utility is considered, as one can easily scale the wealth. On the other hand, if transaction costs are taken into consideration in a dynamic program, an allocation decision must depend on the current weights in each asset, and therefore the state space grows with the number of risky assets. Practically we find it intractable to directly apply a vanilla dynamic program to the multi-period asset allocation problem involving transaction costs.
Instead, we propose it to be one of the methods that finds allocation decisions under zero transaction costs, and could help us to gain an advance starting point for neural networks.

Empirical Results
The problem setting for the empirical experiments is defined in the previous sections. We The neural network and the dynamic programming follows the same specifications in (Li and Mulvey 2021). For the LMCTS, the exploration/exploitation tradeoff parameter is taken to be 5. The belief probabilities are discretized to be multiples of 5%. For the expansion phase of the LMCTS, the candidate portfolios that we consider are the feasible portfolios that deviate 5%, 10%, and 20% away in a single asset from the current best portfolio. We have used a first order Savitsky Golay filter with a window size of 11 to smoothen the results. 10000 simulations are used for each belief/time pair and the reward at each iteration is calculated as the mean of 50000 return paths. A set of return paths starting from each possible belief state is simulated beforehand to expedite the computations.

Terminal utility maximization without shorting
First, we examine at the case where shorting the assets is not allowed. Note that being able to short an asset causes the state space for the asset allocation to increase immensely for the LMCTS algorithm. The reason for the increase in complexity is that since we discretize the asset allocation space for the LMCTS, the budget constraint for the no shorting scenario helps us to shrink the state space considerably, which does not quite work effectively for the scenario where shorting is allowed.
A lower bound of the optimal expected utility is provided with the optimal trading strategy under zero transaction costs. We call the strategy "adjusted DP" because it deals with the no-shorting constraint with a penalty term of shorting in the objective function.
This strategy is learned by a dynamic program, and involves rebalancing at each time period which leads to high transaction costs. The summary of the results can be seen in ( Figure 5). The utility path of numerous simulations have been gathered to produce the comparison graph. We see that LMCTS+NN and DP+NN are the best performing algorithms overall, which was expected. Remember that LMCTS and DP are actually solving the question for zero transaction cost and their results are only used for starting points for the neural network. We see that the LMCTS result significantly outperforms the adjusted dp result without the neural networks. This translates into the LMCTS+NN very slightly outperforming DP+NN, which definitely shows us that LMCTS is the better algorithm to use in the case of a constrained state space.

Terminal utility maximization with shorting
Now we explore the case where shorting the assets are allowed. For each asset, we are allowing an allocation between -100% and 100% of our total capital. The results can be seen in (Figure 6). In the case of shorting, we again see that LMCTS has comparable performance to the optimal solution under zero transaction cost(DP). Of course, one of the reasons for this is that the LMCTS algorithm works with a discretized space, so it rebalances less than the DP, which results in less incurred transaction cost. DP+NN does perform better than LMCTS+NN though after the neural network deals with the transaction costs.
The main difference between shorting and no-shorting experiments is that in the latter case, the state space is significantly larger. Not only we can short each asset, but once we do so, we end up with a higher capital that can be allocated into the other assets. Similarly, longing asset creates more combinations of shorting for other asset. It is still promising to see that LMCTS gives a very competitive good result under a setting which is challenging for its structure.

Maximizing the probability of reaching a goal
Another subject of interest is the ability of the algorithms to discover good solutions under different objective functions. In the previous sections, maximizing the terminal utility was our objective function. Now we examine algorithmic performance when the objective function is maximizing the probability of reaching a goal, a non-convex optimization model.  This time we present two different graphs when the goal is $1580 without shorting, one for the probability of reaching a goal (Figure 7), which is the main objective, and one for the wealth paths of the algorithms (Figure 8). Similar to previous sections, DP+NN and LMCTS+NN perform significantly better than the standalone algorithms, with goal-based neural networks (which optimizes on probability of reaching goal) outperforming CRRAbased neural networks (which optimizes on the CRRA utility). We see that the LMCTS+NN provides the best performance overall.  We see that as the transaction cost decreases, we can reach higher terminal utility values at the end of the horizon,as expected. For higher transaction cost values, there is a significant decrease in the utility at the beginning of the time horizon. The algorithms render an early adjustment at the expense of some wealth at the beginning of the horizon to reach a favorable portfolio. Both algorithms recover from the initial loss rapidly and reach a higher terminal value at the end.

Conclusions
The MCTS is an algorithm that can be employed in real-world finance problems. The computational advantage of combining neural networks with a MCTS or DP algorithm becomes evident as the size of the problem grows. When traditional algorithms suffer from the curse of dimensionality, the combined methods strive to offer an efficient way in finding solutions. In this paper, we explored a multi-period financial portfolio optimization problem under a regime-switching hidden Markov model and shown that the MCTS, more specifically LMCTS, algorithm yields sufficient starting points for a deep neural network algorithm to achieve excellent results. In select cases, the results are better than the DP+NN duo which serves as a viable benchmark to test performance. A noticeable advantage of the algorithm is its adaptability due to the simulation environment, which makes it a general purpose tool. A challenge of the MCTS algorithm is exponential growth in run time with regard to the structure of the model. This suggests that alternative models can be exploited going forward. The introduced LMCTS algorithm together with the KR-UCT function aims to reduce this dependency. By using a lookup table and a more efficient selection/expansion function LMCTS manages competitive results in larger state spaces, as compared to its dynamic programming benchmark.

Future Work
As mentioned, we have used the CRRA utility function solutions of the LMCTS and DP algorithms as the starting points for the case of maximizing the probability of reaching a goal objective function. LMCTS is in fact able to handle such an objective function by itself, a reward of 1 if we reach the goal and 0 if not, but we haven't used it for two reasons.


Learning Agent-Aware Affordances for Closed-Loop Interaction with Articulated Objects

Interactions with articulated objects are a challenging but important task for mobile robots. To tackle this challenge, we propose a novel closed-loop control pipeline, which integrates manipulation priors from affordance estimation with sampling-based whole-body control. We introduce the concept of agent-aware affordances which fully reflect the agent's capabilities and embodiment and we show that they outperform their state-of-the-art counterparts which are only conditioned on the end-effector geometry. Additionally, closed-loop affordance inference is found to allow the agent to divide a task into multiple non-continuous motions and recover from failure and unexpected states. Finally, the pipeline is able to perform long-horizon mobile manipulation tasks, i.e. opening and closing an oven, in the real world with high success rates (opening: 71%, closing: 72%).

I. INTRODUCTION
In the future, autonomous mobile robots could relieve humans of tedious, repetitive manual tasks in a wide variety of environments like hospitals, homes or laboratories. Many daily tasks require interaction with articulated objects, for example to open the door of a dishwasher. This is especially challenging for a robotic agent, because of the complex system dynamics caused by the object's degrees of freedom and kinematic constraints.
A common approach is to estimate the kinematic and semantic properties of articulated objects from visual data [1]- [4] and leverage this information for planning and control [5], [6]. This two-staged approach often requires heuristics, for example defining the grasping point on the handle, limiting the flexibility to deal with unseen articulation types and object geometries. A more generic approach using affordances, i.e. where and how an agent can interact with an object, was recently explored [7], [8]. Given a point cloud of an articulated object, they use neural networks to predict point-wise interaction scores (actionability), which are used as priors for a downstream robotic controller. While these approaches show promising initial results, they neglect that affordances are always dependent on the agent's capabilities, which are defined by the hardware and the controller. Instead, the state-of-the-art models are trained on a disembodied gripper, disregarding the robot kinematics and joint limits. This can lead to the predictions of motions that are infeasible This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 101017008 (Harmony). *   for the real robot. Furthermore, these pipelines query the affordance module only once and then keep the interaction pose and planned trajectory fixed. Due to this open-loop perception and planning setup, they cannot perform longterm tasks requiring a change of interaction pose when the robot reaches kinematic or joint limits.
To overcome these limitations, we propose a novel closedloop pipeline combining agent-aware affordance perception with sampling-based whole-body control. Concretely, this work deals with the non-prehensile manipulation of articulated objects with one degree of freedom using a mobile manipulator equipped with a single, fixed cylindrical finger. Taking inspiration from the Where2Act framework [7], we train an artificial neural network to estimate point-level affordances from visual data, indicating the success likelihood for interactions at each point. The pose proposal with the highest affordance score is passed on to a sampling-based controller (based on [9], [10]), which then iteratively determines the best interaction trajectory. This setup is well suited for non-prehensile manipulation since the interaction location and trajectory are continuously adapted based on real-time feedback, enabling adaptive and robust task execution. In contrast to previous end-effector-aware approaches, we train our affordance inference network with data collected using the full model of our target robot platform, making our predictions fully agent-aware. Additionally, unlike previous approaches with a fixed interaction pose, our pipeline is able to re-evaluate the affordance model at any point during task execution, which allows the robot to execute long-term tasks where a change of interaction pose is required. In this work, we show in ablation experiments that agent-aware training significantly increases the quality of pose proposals, and that allowing the agent to change the interaction pose during the task increases the robustness of the integrated affordance-control pipeline. Additionally, we benchmark our method against VAT-Mart [8] as a state-of-the-art work and we perform experiments in the real world ( Fig. 1), reaching a success rate of more than 70% for both fully opening and closing an oven door using a mobile manipulator. In summary, our contributions are: • We formulate the concept of agent-aware object affordances which are conditioned on the full shape and kinematics of the robot. • We propose a novel closed-loop manipulation framework that combines the concept of visual affordances with a sampling-based controller.

II. RELATED WORK
Control strategies for manipulating articulated objects: A common approach for the manipulation of articulated objects is to extract object properties like part segmentation and joint kinematics from visual data [1]- [4], [11] and use this to plan an interaction trajectory [12]- [14]. Mittal et al. [5] recently successfully used this approach by implementing a Model Predictive Controller (MPC) to track the feed-forward trajectory plan. However, MPC struggles with discontinuities caused by contact dynamics, such that a simplification of the problem is required, e.g. splitting the task into first achieving a stable grasp at a fixed interaction position and then executing a predefined motion based on known articulation kinematics. To alleviate these limitations, samplingbased control has recently emerged, allowing a more taskspecific and complex interaction representation [9], [15], [16]. In previous work by Rizzi et al. [10], a samplingbased controller was successfully applied as a whole-body closed-loop controller for the non-prehensile manipulation of articulated objects. However, to decrease the size of the sampling space, a fixed end-effector interaction pose still has to be provided a priori for each object.
Affordances for manipulating articulated objects: An affordance is defined as the ability of an agent to perform an action with a target object in a given environment [17], [18]. In robotics, the concept of affordances is commonly applied either on an object-level, e.g. a dishwasher is openable [19], [20], or on a point-level to encode information about the object geometry and grasp possibilities [21]- [24]. In the current literature, affordances are usually learned from labeled visual data [25]- [27] or from self-supervised interactions [19], [20], [24], [28]. Recent works have applied this concept to the manipulation of articulated objects. VAT-Mart [8] and Where2Act [7] propose frameworks which learn affordances for robotic manipulation from interactions generated in a photo-realistic simulator [29]. A network is then trained to infer interaction points and trajectory proposals for downstream manipulation tasks from visual data. However, these approaches generate training data by simulating the interaction between a disembodied end-effector and a unitsphere scaled object. This results in affordances that are not only unaware of the kinematics and control of the actual robot platform, but also of the realistic object size. Consequently, the trajectories derived from these approaches might be infeasible in real life, e.g. due to joint limits or collisions of the arm with the object. Additionally, they only query the visual model at the beginning of the interaction in an open-loop fashion, keeping the planned grasp location and trajectory constant during the interaction. UMPNet [30] on the other hand proposed a closed-loop affordance-based manipulation framework that can update the interaction direction during task execution. However, it still uses a fixed interaction point, determined at the start of the interaction using only end-effector-aware (agent-agnostic) affordances.

III. PROBLEM FORMULATION
We consider a physical system consisting of a mobile manipulator and an articulated object which is composed of two rigid bodies connected by a rotational or translational joint. We define the state x = [q, o,q,ȯ] as the configurations of the robot and object and their time derivatives, where q consists of the position of the mobile base relative to the object as well as the joint angles of the manipulator. The robot kinematics and the object geometry and articulation are assumed to be provided a priori. Given a desired object configuration o * , the point cloud from the robot's point of view C and the current state observation x, the overall objective is to find a robot control policy that generates joint velocity commands u such that o = o * .

IV. AGENT-AWARE AFFORDANCE LEARNING
We propose a novel control pipeline combining an agentaware affordance network with a sampling-based whole-body controller (Fig. 2). Given an object point cloud C, from which the movable object part is extracted by applying a segmentation mask, and target configuration o * , we use an affordance neural network to select a favorable end-effector interaction pose (point p ∈ R 3 and orientation R ∈ SO(3)). The interaction pose is then used by the sampling-based controller κ to generate joint velocity commands u = κ(x | o * , p, R). When the interaction fails or stops leading to any improvement of o, a task scheduler triggers a pose update as described in Section IV-B.  and its target configuration o * , we define the affordance A of an interaction pose (p, R) as the expected value of J over all the state-input trajectories induced by the controller κ(x | o * , p, R) from initial state x 0 . Inferring the optimal interaction pose by maximizing the affordance function A is impractical, as the set of all possible interaction positions and orientations is extremely large. We therefore follow [7], [8] and define two auxiliary functions. The orientation proposal function Q generates end-effector orientation proposals given an interaction point. The actionability function α models the point-wise expected affordance A over a distribution of interaction orientations. At test time, we first select the interaction point that maximizes α. We then generate multiple orientation proposals and score their affordance. Finally, the highest-scoring pose proposal (p, R) is selected and passed on to the sampling-based controller.
Network architecture: We use a neural network based on [7] to learn the functions required by the optimization problem, namely affordance A, orientation proposal Q and actionability α. The network architecture, shown in Fig. 3, combines a PointNet++ [31] feature encoder with three Multilayer Perceptron (MLP) decoding heads, one for each function. All three networks take as input the point position p, its feature vector f p and the target object configuration o * . The orientation proposal module additionally takes a sampled Gaussian noise vector z ∈ R 10 ∼ N (0, 1) as input to generate proposals R which are then passed on as an input to the affordance prediction module.

Data collection:
The training data is collected in a simulation setup where we simulate the complete control pipeline as well as the kinematic chain, collision bodies and dynamics of the target robot platform in order to create agent-aware affordances. A baseline agent samples a random interaction pose (p, R) on the movable part of the object, where the orientations are sampled only within a 45 • cone of the surface normal at point p (based on [8]). Given the sampled pose and the desired configuration o * , the samplingbased controller attempts to interact with the object for 10s. A binary reward J rates the interaction as successful if the object configuration was changed by more than 5 • towards the target o * . Finally, we balance the dataset to contain the same number of successful and unsuccessful samples. One advantage of using the sampling-based controller to generate training data is the relatively high rate of successful interactions (13.6%), which makes it much more data efficient than first having to train a control policy (as done in [8]).
Training and losses: Given the point cloud and the interaction sample (p, R, o * , J), we train the components of the pose module jointly where the affordance module learns to predict the reward realization J using a binary crossentropy loss, while the orientation proposal module uses the mean cosine similarity loss on quaternion predictions. The actionability module is trained on an L1 loss, where the ground truth actionability for a given pose proposal can be calculated using predictions from the affordance and orientation modules. We use an Adam optimizer and employ learning rate scheduling and early stopping to ensure convergence. Training a network takes around 3 hours on a low-grade GPU (NVIDIA GTX 1050 Ti with 4 GB RAM).

B. Control Pipeline
Sampling-based controller: The sampling-based controller uses a physics engine to forward simulate the system dynamics of the robot and object to iteratively refine the motion trajectory [10]. At every time step, it samples multiple robot joint velocity references u around the current best guess and simulates the resulting system dynamics for a time horizon of 1s. The best trajectory is then selected based on a cost function, which is made up of multiple components. The object cost encodes the offset from the target o * , the collision cost punishes robot-object collision, the joint limit cost and the arm reach cost prevent mechanically unreachable configurations. Finally, the pose reach cost is defined as the distance between the end-effector and the reference pose (p, R), which is required to make the optimization problem feasible. A low-level proportional-integral controller then converts the velocity reference u to the executed joint torques commands.
Task scheduler: To increase the robustness of the system, we implement a closed-loop affordance inference setup, allowing the robot to change its interaction pose when the current one is no longer favorable. In practice, the task scheduler triggers a pose update whenever the object state cost stagnates. To update the pose, the robot moves to a configuration from where it can observe the full object, records a point cloud, segments it and passes it through the pose module to obtain the new interaction pose.

V. EXPERIMENTS
To evaluate the performance of our pipeline, we conducted three different experiments in simulation and additionally validated our approach in real life. For training, we collect for each task 12000 simulations with eleven different object models from the PartNet-Mobility dataset categories oven and dishwasher, which both have a revolute joint [32]. First, we compared affordances conditioned only on the end-effector (as in [7], [8]) against affordances conditioned on the full robot. Second, we evaluated the advantage of allowing the agent to update the interaction pose when the current one is no longer advantageous, i.e. split one task into multiple non-continuous motions. These two experiments were performed with seven unseen object models from the training categories since they are especially challenging in terms of reach and collision and are therefore well suited to study the advantages of our method. Finally, we show that our approach is also capable of generalizing to unseen object categories and articulation types. To this end, we tested our pipeline on 140 object models from the categories safe, table and washing machine containing both revolute and prismatic joints and compare it to VAT-Mart [8] as a state-of-the-art benchmark.

A. Agent and Environment
The robot platform consists of a seven degree-of-freedom manipulator mounted on a mobile base. The manipulator hand is equipped with a single, fixed cylindrical finger. One of the advantages of this simple finger design is that it allows the physics engine RAISIM to simulate the contact dynamics, which is a computationally demanding task [33], smoothly in real-time with a simulation timestep of 0.0015 s. To generate the pointclouds, the robot and the object are rendered in the SAPIEN environment [29], where the object's initial position and scale are sampled uniformly at random within predefined, category-specific bounds.

B. Baselines
As a minimum baseline, we employ the same random agent used during training for data collection. To evaluate the importance of agent-aware affordances, we compare against a purely end-effector-aware baseline where we collected the training data with a disembodied, freely moving gripper initialized directly at the interaction pose and thus ignoring the issue of reach. This is similar to the affordance-learning paradigm in Where2Act [7]. As an additional baseline, we enhance the end-effector-aware network with a reachability filter using a Jacobian-based inverse kinematics (IK) check as well as with a robot-object collision filter. These filters are implemented such that each interaction pose proposed by the end-effector-aware network has to pass the feasibility check before being executed. Next, to evaluate the effect of enabling a change of interaction pose, we created for both the end-effector-aware and the agent-aware network one version where the pose stays fixed during the interaction and a closed-loop setup that allows pose updates.

C. Results
Agent-aware vs end-effector-aware: We first evaluate the quality of the proposed agent-aware interaction poses with a simplified version of the pipeline which keeps the interaction pose fixed. As in [7], we report the sample success rate as the percentage of interaction poses creating movement towards the target o * , i.e. |∆o| ≥ 5 • , as well as the sample reach rate defined as the percentage of pose references that are reachable by the agent. The considered tasks are to open or close articulated objects from the testing dataset using the full robot model. Over the 500 trials of each task, the initial object configuration o 0 is such that half of the trials start with the object fully closed or fully open (for the open and close tasks, respectively), the other half start with the object open to an angle sampled randomly over the full joint range.
In the open task, both networks learned to interact with a similar set of points (Fig. 4a), leading to similar average performance (Table I). In the close task on the other hand, while the end-effector-aware network considers the entire outer surface of the door to be actionable, the agent-aware network strongly prefers interaction points close to the top edge (Fig. 4b). As the end-effector-aware network was trained on a disembodied gripper, it often predicts interaction poses that are unreachable for the full agent due to kinematic or collision constraints. This results in a significantly lower reach and success rate compared to the agent-aware version. Table I shows that even though the feasibility checks progressively improve the reachability and success rate of poses, the performance is still lower than for our proposed agent-aware network. From this experiment we conclude that there is no simple heuristic filter that can restore the agent-awareness a posteriori. The failure cases of the agentaware model on the other hand are mostly due to suboptimal actionability predictions of the network, i.e. it proposes poses that can be reached but do not result in successful interactions. Additionally, we compared multiple network instances trained with different random seeds and found that Fig. 4: a) Open task: both networks learn to interact with a similar set of points. b) Close task: the agent-aware network strongly prefers reachable points while the end-effector-aware network, which was trained on a disembodied gripper, also predicts points on the door surface which are unreachable for the full agent.  while stochasticity has a visible effect on the affordance map, the resulting performance is consistent across networks.
In general, we observe that the sampling-based controller provides a certain robustness against small errors of the predicted interaction pose, as it locally adapts the pose while minimizing the controller cost.
Closed-loop pose update: In this experiment we evaluate the performance on the full opening (o 0 = 0 • , o * = 90 • ) and closing (o 0 = 90 • , o * = 0 • ) tasks and report the task success rate. Over 1000 trials we compare our closed-loop pipeline, which allows the robot to change its interaction pose during the task, with an ablated fixed pose version. The agent-aware closed-loop pipeline achieves an extremely good task success rate of 88.6% on both open and close tasks (Table II). We observe two advantages of the closedloop setup: Firstly, when an initially successful interaction pose becomes infeasible at an intermediate state, the task scheduler triggers an interaction pose update. Secondly, when the sampling-based controller is unable to perform the task at all by interacting with the current reference pose, the agent is able to recover by trying again at a different reference pose. In this simulated setup, we predominantly observed  the second effect, because the sampling-based controller is able to locally adapt the interaction pose such that the task can be fulfilled in one motion, e.g. by sliding or rotating. Finally, the closed-loop setup also improves the results of the end-effector-aware network, but it is still outperformed by the agent-aware network. Generalization capabilities: To benchmark against VAT-Mart [8], we evaluated our pipeline on their test objects (safe, table and washing machine) and use their task definition which is to sample the target o * and the initial object configuration o 0 randomly over the full joint space. Qualitative results in Fig. 5 show that our pipeline learned interaction strategies that generalize well to unseen object categories despite the limited object variability seen during training. Table III shows that we outperform the VAT-Mart benchmark in all tasks and categories, even when only using a fixed interaction pose. Our pipeline is explicitly designed to profit from available information about the object and the robot which VAT-Mart does not use, namely the collision shape and articulation model as well as the realtime feedback of robot joints and articulation angle which are required by the controller. For our targeted service robotics applications, object CAD models are often readily available [34] [8] as within a tolerance of 15% of the task, of our framework on unseen object categories compared to an ablated fixed pose version of our method and the VAT-Mart benchmark.

D. Real-world experiments
We deployed our pipeline on a mobile manipulator interacting with an oven (Fig. 1). Collision bodies and articulation type of the oven are provided to the controller a priori, while the articulation joint angle is obtained by integrating the angular velocity measurement of an IMU mounted on the back of the oven door. The pose of the robotic platform is tracked by an OptiTrack motion capture system and the point cloud data is collected from an onboard Azure Kinect sensor. The reflective glass surface of the oven door was covered to make it visible to the RGB-D camera. For part segmentation, the Iterative Closest Point (ICP) algorithm is used to match the real-world point cloud to the point cloud rendered in simulation, allowing a projection of the segmentation mask.  Table IV, real-world results are compared to simulation trials with the same setup and testing object. We report the task success rate (task s. r.), the average number of interaction attempts (n. int.) and the average number of successful motions (|∆o| ≥ 5 • ) needed to complete a task (n. mot.).
Results: The network is able to produce reasonable actionability maps and interaction poses from real-world point cloud data ( Fig. 1 and supplementary video). For the close task, the behavior transfers very well from simulation to the real world: The robot is able to complete the task 72% of the time and often in one motion (Table IV). In the open task, unlike in simulation, the real robot typically needs multiple attempts to slide the finger into the handle due to errors introduced by the state estimation and model mismatches. Once the end-effector is placed such that it can exert enough force, it opens the door to an intermediate state. This end-effector pose usually becomes unfavorable at a certain articulation angle (as seen in Fig. 1b), such that a pose update is triggered and the robot uses a second and sometimes third motion to fully open the door. The recovery capabilities of the closed-loop pipeline are therefore essential to overcome the described sim-to-real gap, such that we achieve an overall task success rate of 71% for opening.

VI. CONCLUSIONS
We introduced a novel closed-loop manipulation pipeline combining point-level affordance inference from visual data with whole-body control. The pipeline is based on samplingbased control, using affordances to estimate the end-effector interaction pose. Current state-of-the-art methods only condition affordance predictions on end-effector geometry. In contrast, we show that our agent-aware network is able   to exploit the full robot model and low-level controller to improve the quality of the inferred interaction priors. Our pipeline also enables the agent to re-evaluate the pose model and update the interaction pose at any time during task execution. This allows the agent to split a task into two or more non-continuous motions and recover from failure and unexpected states. We find this to be crucial especially in the real-world experiments, partially compensating for the effects of the sim-to-real gap and allowing our pipeline to perform long-horizon mobile manipulation tasks with high success rates.
Limitations and future work: The sampling-based controller used in our pipeline requires precise measurements of the object joint state, currently provided by an external sensor (IMU). While this is not an issue in simulation and in controlled real-world environments, for general deployment the object pose estimation should be performed using onboard sensors, e.g. from RGB-D data [2], [35]- [38]. The controller also requires a precise object model, which could be estimated from visual data and optionally refined using interaction data [3], [39], [40]. Additionally, in future work the pipeline could be extended to allow for grasping, since not all tasks can be solved through non-prehensile manipulation.

APPENDIX
A. Controller a) Sampling-based controller: In this section, we state the equations of our sampling-based controller based on [10]. Let q ∈ R nq andq ∈ R nq be the configuration of the agent robot and its time derivative, where n q ∈ N >0 is the agent's degrees of freedom (DOF) -in our implementation n q = 10. Similarly, let o ∈ R no andȯ ∈ R no be the configuration of the object and its time derivative, where n o ∈ N >0 is the object's DOF -in our implementation n o = 1. The controller state vector is defined as: The controller defines several stage cost function components. The first component is an object cost, which encodes the object manipulation objective: where o * is the target object configuration and W o ∈ R no×no is a diagonal weight matrix. The second component is the pose reach cost, which encodes the distance of the end-effector to the reference interaction pose T * = (position p * , orientation R * ): where the actual end-effector pose T(q) is computed via forward kinematics from the joint configuration vector q and W t ∈ R 6×6 is a diagonal weight matrix.
A number of surrogate objectives are defined to encode agent limits and constraints: • Collision cost: Wj ) where q upper and q lower are the upper and lower joint limits, w j ∈ R and W j ∈ R nq×nq . • Arm reach cost c a (q; w a , w as ) = 1[r(q) > r max ](w a +w as (r(q)−r max )) where the current reach r is calculated from forward kinematics from the joint state q and w a , w as ∈ R are weights. The controller defines two modes of operation m ∈ {1, 2}, corresponding to two stages in object manipulation reaching and interacting. In the first phase, the end-effector reaches the target interaction pose while avoiding contact with the object. This is achieved by setting the stage cost as: In the second phase, the agent interacts with the object and moves it towards the target state: b) Low-level controller: We use a dynamically compensated low-level proportional-integral controller to convert the velocity references u from the sampling-based controller to joint torque commands τ . Let q * be the desired agent configuration, calculated by integrating the velocity references over the time interval dt: Let M(q) ∈ R nq×nq be the matrix of inertia of the agent. The Coriolis and gravity terms of the system dynamics are denoted as Co(q,q) and g(q), respectively. The joint torque vector is calculated as: with the auxiliary error variableq = q − q * and K D and K I being positive diagonal gain matrices.
c) Task scheduler: The task scheduler selects the appropriate mode of the sampling-based controller, which can either be reaching (m = 1) or interacting (m = 2). It also regulates updates in the interaction pose T. Given the current state observationx, the target object configuration o * and the current pose reference T, the task scheduler implements the following simple heuristic: • At the beginning of the interaction, update the reference pose and go into reaching mode (m = 1). • During the interaction: -If in reaching mode and the current reference pose was reached by the end-effector, change into interacting mode (m = 2). -If in interacting mode and the object configuration did not improve for a set number of seconds (5s in simulation, 1s in real-life testing), lift the robot arm to prevent camera occlusion while recording a new point cloud, trigger an update of the reference pose and return to reaching mode. -If |o − o * | ≤ 5 • , stop the interaction as the task is fulfilled. The task scheduling described above allows the agent to change the interaction pose when the current one is not advantageous anymore as well as to try again if the current interaction pose failed to lead to any improvement in the object configuration. As both the sampling-based controller and the pose module are non-deterministic and the agent moves in between the pose updates, the same initial conditions of task and object might result in different behavior, i.e. success where before the agent failed.

B. Affordance Definitions
This section gives an overview of the equations used for the pose module. Let J(x 0:N , u 0:N −1 ; o * ) be a reward function that, given the the target object configuration o * , maps a trajectory of system states {x 0:N } and controller inputs {u 0:N −1 } to a binary reward value: where θ ∈ R >0 is a success threshold (in our implementation θ = 5 • ).
Given an object, a target configuration o * and an initial state x 0 , we define the affordance A J of an interaction pose (p, R) as the expected value of J over all the state-input trajectories induced by the controller κ(x; o * , p, R) on the agent-object system from initial state x 0 . In our method, the object and the initial state are implicitly represented by the point cloud which is passed through the PointNet++ encoder [31] to generate point-wise feature vectors f p (p, x 0 , object). Therefore we use the following definition of affordance A J : Following this affordance definition, the optimal interaction pose (p * , R * ) is given by maximizing the affordance function A J : Solving this equation for all possible interaction poses is computationally intractable, therefore we define auxiliary functions to optimize the position and orientation in a hierarchical manner. The orientation proposal distribution Q J generates orientation samples R ≈ R * that approximate the optimum for a given interaction point p: The actionability function α J is defined as the point-wise expected value of the affordance over interaction orientations sampled from an orientation proposal distribution Q J : This allows us to first obtain the interaction pointp by sampling the actionability: Next, a set of orientation proposals are sampled from Q J . Finally the affordance function A J is used to score the orientation proposals and sample the highest-scoring one: This hierarchical procedure does not necessarily output the globally optimal interaction pose. Instead, the quality of the solution heavily depends on the orientation sampling function Q J . If R ∼ Q only and always samples the optimal interaction orientation, the procedure will yield the global pose optimum.
C. Simulation Settings and Training a) Simulation settings: Dynamics are simulated using a physics simulation based on the commercial software RAISIM. We set the simulation timestep to 0.0015 s. The range of motion of the object articulation joint is set to 90 • . To increase simulation speed, we do not check for collisions between the two object links. We set a damping coefficient of 20 and friction of 40 at the object articulation joint. Between surfaces of contact bodies, we set a low friction coefficient (0.01).
b) Training data collection: Each object in the PartNet Mobility dataset is scale-normalized within a unit sphere. For each simulation instance, we sample an object from the category oven or dishwasher and apply a scale factor ∈ [45%, 55%] of the normalized scale. This ensures a realistic relative scale between our object and agent models, and serves as augmentation. The position of the object, measured from the world origin to the center of the object, is sampled uniformly between the following bounds (x = 0 m, y ∈ [−0.3 m, 0.3 m], z ∈ [0.75 m, 0.95 m]). We initialize the object articulation joint at rest 50% of the time (i.e. 0 • for the open task, 90 • for the close task) and in an intermediate state for the other 50% of initializations. We sample the initial and target configuration within the articulation bounds and according to the task, with a minimum task distance The agent is initialized facing the world origin, with a uniformly sampled distance ∈ [1.75 m, 3.0 m] and base rotation angle arctan(y, x) ∈ [−20 • , 20 • ]. During the simulated interaction, the agent is given 30s to move to the pose reference (i.e. mode m = 1 of the sampling-based controller) and 5s to interact with the object (i.e. m = 2). A simulation is considered successful if the object configuration was moved by at least 5 • towards the task. Figure 6 shows our simulation environment, complete with axes for reference. We collect 12000 simulations for each task, which takes around 15 hours per task on a commercial CPU (Intel Core i7-7700HQ quadcore processor). c) Network training: We use an Adam optimizer with an initial learning rate of 1e−4 and batch size of 10. To ensure convergence, we employ both learning rate scheduling (patience of 5 epochs, discount factor 0.2, minimum improvement of 2%, minimum learning rate of 1e−6) and early stopping (patience of 12 epochs, minimum improvement of 5%). Fully training a network usually requires between 30 and 50 epochs, and takes around 3 hours on a low-grade GPU (NVIDIA GTX 1050 Ti, with 4 GB of dedicated RAM). d) Evaluation setup: For evaluation, the simulation is initialized in the same manner as during training data collection. In the first experiment of the paper, the agent is given 40s to move to the pose reference and 30s to interact with the object. The sample success rate is evaluated, which is the percentage of successful pose references and success is defined as a movement of the object configuration o by at least 5 • towards the target o * : sample success rate = n successful proposals n proposals .
We also report the sample reach rate, which is the percentage of pose references that are reachable by the agent if given as a target pose to the sampling-based controller: sample reach rate = n reached proposals n proposals .
In the second experiment, which evaluates the effect of the closed-loop dynamic pose update, the agent is given 40s to move to the pose reference and 300s to interact with the object. We report the task success rate, where a task is considered successful if the final object configuration o is within 5 • of the target o * : task success rate = n successful tasks n tasks .
In the third experiment, the pipeline is tested on objects of the category safe, washing machine and table, applying a scale factor of 50%, 50% and 100% respectively to ensure a realistic proportion between agent and object. Following VAT-Mart [8], we first uniformly sample an object category, and then sample a shape from this category. We sample a task ∆o = |o * − o 0 | in a range of [10 • , 70 • ] for revolute joints and [0.1, 0.7] for prismatic joints. The initial object configuration o 0 is randomly sampled such that it lies within the articulation joint bounds given the task. We report the task success rate, where we define a successful articulation joint value equivalently to VAT-Mart within a tolerance of 15% of the task ∆o = |o * − o 0 |.

D. Network Sensitivity Study
We identify three sources of randomness in the results of our interaction trials. Firstly, the random seed used for initialization of the artificial neural networks causes variability in the output of the network. Secondly, in our testing protocol the initial state and target object configuration are sampled at random. The third source of randomness lies in the stochastic nature of the sampling-based controller. To analyse these effects, we use the training data from the first experiment to train multiple networks initialized with different random seeds. The results are evaluated on 500 interaction trials for each task and model (end-effector-aware and agent-aware). Figure 7 reports the sample success rate of the two models for the open task across 5 training runs and for the close across 8 training runs. We observe that for the open task, the performance of the networks is repeatable. In the close task on the other hand, the end-effector-aware networks are far more inconsistent. This could be caused by the fact that behavior learned during training with a disembodied gripper doesn't transfer consistently to the testing task with the full agent. Overall, this analysis shows that the performance of the agent-aware pipeline is consistent over multiple training runs, and that therefore the influence of the stochasticity in controller and neural network is minimal. We analyzed the predictions of networks trained with different random seeds and observed that while the testtime performance is consistent across different networks, the spread of the predicted actionabilities varies (c.f. Fig. 8). This suggests that the networks converge to different local minima, which could be caused by the fact that our training data is relatively sparse and has a stochastic component due to the controller. Additionally, it should be noted that at test time only the points with the highest actionability are sampled, therefore the distribution of the intermediate-level actionability scores does not have a direct influence on the executed interaction.

E. Hardware experiments
The hardware experiments were conducted with an omnidirectional platform (Clearpath Ridgeback) with a 7 degreeof-freedom robotic arm (Franka Emika Panda) mounted on top. The selected target object is an oven which was placed on a workbench, as seen in Fig. 9. As it is required by our control pipeline, we build an articulation and collision model of the oven from real-world measurements.
The sampling-based controller is run on an onboard computer (Intel Core i7-8550U quad-core processor) at a frequency of 66Hz, and a proportional-integral velocity controller computes joint torque commands for the arm at 1000Hz. The Ridgeback base is controlled directly in velocity-space at a frequency of 50Hz. The task scheduler is run at 30Hz. The pose module is only run when a pose update is triggered, where each pose update requires roughly 1s of computational time. The pose module and task scheduler are run on a separate laptop (Intel Core i7-7700HQ quad-core processor). The two devices communicate wirelessly over the network using the ROS framework.
The sampling-based controller requires an accurate state estimation. The pose of the Ridgeback mobile base is tracked using an OptiTrack motion capture system, while for base velocity we use odometry data. At startup, we calibrate the oven position in the global reference frame. To obtain the articulation joint angle, filtered linear acceleration and angular velocity are continuously extracted from an inertial measurement unit (IMU) mounted on the back of the oven door. The joint velocity is estimated by projecting the angular velocity measurements onto the articulation rotation axis, while joint position is estimated by integrating velocity.
Point cloud data is collected by an Azure Kinect sensor mounted on the robot base ( Figure 9). While state-of-theart object segmentation, e.g. a neural network such as Mask R-CNN [41], could be implemented for segmentation, we used a simple heuristic which exploits the knowledge about the position and size of the object. The segmentation of the object from the scene is implemented using bounding boxes. For part segmentation, the Iterative Closest Point (ICP) algorithm is used to closely match the real-world point cloud to an expected point cloud obtained from our simulator/rendering setup [42]. Once the two point clouds are matched, the simulated point cloud is converted to a rough voxel map with a voxel size of 50mm and the realworld point cloud is segmented by checking if its points are contained in the voxel map. Example images from this process can be found in Figure 10.
While the real-world experiment shows that our proposed system performs well on the oven, we would also like to showcase its generalization capabilities. Fig. 11 depicts the point cloud of multiple articulated objects with the corresponding affordance prediction. It shows that despite the partial and noisy point cloud of unseen objects, the affordance prediction module creates reasonable interaction scores.