The equilibrium of venture capital incentive contract: Optimization and Q-learning approaches

. The current study presents an incentive contract model to allocate income to venture projects. In this respect, Venture Capital (VC), as one of the main sources of (cid:12)nancing innovative projects, faces several challenges such as moral hazards, information asymmetry, and interest con(cid:13)icts, often referred to as three agency problems. In addition to the identi(cid:12)cation of the factors that may a(cid:11)ect the income of venture projects and elaboration of cost functions, this study presented an optimal incentive contract model from the perspectives of venture capitalists and entrepreneurs. In this model, a venture capitalist, as an active investor, provides entrepreneurs with managerial and training assistance. The results revealed that the higher the initial ability of the entrepreneur was, the less money the venture capitalist would pay for training. Of note, in case the venture contract was not accepted, the wealth that the contract parties would obtain would become an in(cid:13)uential factor in the contract payment function. This model was studied considering the bounded rationality hypothesis and implemented using the Q-learning algorithm. In addition, the results obtained from the Q-learning approach were found to be reasonably convergent with the Nash equilibrium.


Introduction
Entrepreneurial equity investments by venture capitalists, corporate venture capitalists, angel investors, and more recently, crowd funders and accelerators are the key sources of capital that contribute to fuelling innovation and development [1]. A risk nance and Venture Capital (VC) environment are essential to wealth creation revolution and rapid economic growth that can leverage innovation, promote technology, and harness knowledge-based ideas [2]. In other words, VC is an important intermediary that enables transforming the capital into innovations in a highly productive manner [3]. Among all kinds of entrepreneurial nance, VC mostly address the growth in the revenue of companies, business models, and current investors [4]. In the nancial area, VC is regarded as the frontier of innovation, liberated from the regulations and precedents of traditional markets [5]. The VC market involves earlystage and, consequently, high-risk investments, whose main members are active investors [6].
In VC investment, capital and competence are transferred from the venture capitalist to the entrepreneurial rm at the value-adding phase, and the capital transformation is considered as the nal ending of the investment decision phase [7]. Venture capitalists are mainly nancial intermediaries who not only provide nancing but also o er advice and monitoring, and they play a very active role in negotiating with suppliers, helping recruit and compensate key individuals, and providing strategic advice and access to consultants and lawyers [8]. Therefore, the management team is an important factor for VCs [9].
VCs face four generic concerns in the investment process: 1. The entrepreneur will not work hard to maximize the value after the investment is made; 2. The entrepreneur is more aware of his ability than the VC; 3. After the investment, there will be circumstances when the VC disagrees with the entrepreneur and VC will claim the right to make decisions; 4. The entrepreneur can \hold up" the VC by threatening to leave the venture when the entrepreneur's human capital is particularly valuable to the company [10].
In uncertain environments, VCs fail to exploit even available information due to two obstacles, namely information asymmetry and agency problems [5]. The agency problem arises when the principal (venture capitalist) and agent (entrepreneur) pursue two di erent objectives and when the principal faces di culty understanding what the agent is actually doing [11]. Asymmetric information refers to a situation where one party to a contract has access to the relevant information that is not available to the other party. A venture capitalist needs to trade o the bene ts and costs when attempting to attenuate the agency problems in the investment process [12]. Kaplan and Stromberg reviewed the con icts of interest between an entrepreneur and a venture capitalist and identi ed a number of possible ways that the investor could mitigate these con icts. They found that the investor could rst engage in information collection before deciding whether to make investment in order to screen out the ex-ante unpro table projects and bad entrepreneurs. Second, the investor could engage in information collection and monitoring once the project is under way. Third, nancial contracts, i.e., allocation of the cash ow and control rights between the entrepreneur and investor, would be designed to provide incentives for the entrepreneur to behave optimally [13]. VC investments are made under conditions with potentially extreme information asymmetry and agency cost that can motivate the parties to design nancial contracts [14]. The contract tends to create a mutual understanding between the parties, hence forming a basis for successful cooperation. This justi es the time spent on negotiation in the VC investment process [15].
Contracting and monitoring e orts can improve the portfolio performance by reducing the possible losses caused by moral hazard and adverse selection [16]. Optimal contract arrangement can overcome the revenue con ict between the venture capitalists and entrepreneurs and help them reach mutually motivating conditions [17].
Bitler et al. developed a principal-agent model in an entrepreneurial setting and tested the predictions made by the model based on unique data derived from entrepreneurial e ort and wealth in privately held rms. They found that entrepreneurial ownership shares would decrease upon increase in the rm risk; the entrepreneurial e ort increased with ownership and it increased rm performance [18]. Fluck et al. proposed a model to study the design of nancial contracts between entrepreneurs and venture capitalists. They demonstrated that staged nancing would alleviate the e ort provision problem [19]. Considering the fact that both capitalists and entrepreneurs aim to maximize their own revenue, Zou and Zhou presented an optimal contract model. They also designed an optimal contract for short-and long-term VCs from the venture capitalists and entrepreneurs' perspectives [17].
The advisory, networking, and experience services provided by VCs are complemented by the entrepreneurs' technological and innovative skills. These complementary skills will in turn create a partnership that has a signi cant impact on the company value [20]. Gonzalez-Uribe, referring to direct evidence based on patent citations, reported that venture capitalists di used knowledge about their existing patented innovations among the portfolio companies [21]. Helmers et al. highlighted the signi cant positive e ect of information transmission through the interlocking boards of directors on innovation [22]. A venture capitalist can transfer valuable knowledge to an entrepreneur and facilitate innovation. He/she can also communicate the entrepreneur's innovative knowledge to other portfolio companies. In fact, venture capitalists take on a new role as knowledge intermediaries [23].
Common sense says that to grasp investment opportunities, people do not consider only economic factors; instead, they take into account psychological factors. In fact, individuals face bounded rationality in their decisions. Bounded rationality models aim to answer the question \How do people with bounded time and knowledge make decisions?" In this regard, Rodriguez-Fernandez et al. pointed out that adequate analysis of the past actions of market players improved the decision-making process of negotiators and allowed them to maximize their outcomes. They proposed a new model based on an adaptation of the Qlearning reinforcement learning algorithm to estimate the expected prices given in contracts in a speci c context. They also found that the learning method could identify the best scenario for each situation since the behavior of negotiators would change depending on the negotiation environment [24].
According to the study conducted by Vergara et al. [20], the model proposed in this study took into account several issues such as the e ect of the potential market value on the revenue function, constraints of e ort and capability levels, and exponential growth of these features. This model was designed to allocate the proceeds from the innovative projects with the objective of reducing the con icts of interest between the contract parties by providing su cient incentive for them and leading them to an optimal contract. Of note, the venture capitalists, as active investors, played a key role in the success and accumulation of the project revenue by transferring experiences to entrepreneurs and training them. This model was designed from the venture capitalists and entrepreneurs' perspectives from which the parameters of optimal contract arrangement were derived separately for each case. Further, equilibrium solutions were extracted from optimal solutions to achieve a uni ed strategy.
Followed by this introductory section, Section 2 describes the proposed mathematical investment model from the venture capitalists and entrepreneurs' points of view and o ers optimal solutions for each party. The rest of this section calculates the equilibrium answers so that the parties can reach agreement. Finally, it examines the problem from the perspective of bounded rationality using the Q-learning method and reports the obtained results. Section 3 presents the most relevant conclusions of this research.

The proposed model
In the VC sphere, the entrepreneur's e orts and abilities are the determinants of income function [17]. In addition, the potential market value is believed to be a ecting the income functions of the VC as well as the concerning e ort and ability; however, it is not possible to increase the level of e ort and ability inde nitely. At early stages, increase in the e ort and ability is observed at a higher rate; however, from the level one onwards, it takes much capital to make a small change in the level of ability or e ort. Therefore, some nonlinear functions in exponential forms were taken into account, indicating the entrepreneur's e ort and ability. The ability function is developed based on the desired properties in Eq. (3). Accordingly, the VC income function (venture capitalist's observable income) can be de ned as: where f s (x) and f (T ) represent the entrepreneur's e ort and ability functions, respectively. In addition, V stands for the potential market value of the company that depends on the environment and market conditions of the innovative project. Finally, ", as the random error of VC income function, follows the normal distribution: " N(0; 2 ).
Realization of the entrepreneurs' innovative ideas requires their e ort. In the e ort function, their e ort level is denoted by x which takes a value in the of range of [0; 1). The e ort function is regarded as a positive but descending function, i.e., f s (0) = 0, f 0 s > 0, and f 00 s < 0. The e ort function is de ned as: where ; is a constant coe cient that indicates the degree of convexity of f s (x). In the discussed function, the greater the entrepreneur's e ort level, the closer the e ort function to 1. Of note, in the case of in nite e ort level, it takes the value of (0 f s (x) 1) [19].
It should be noted that the venture capitalists actively invest their funds in the existing plans to obtain the ability function, meaning that in addition to investment, they increase the knowledge of entrepreneurs and improve their ability level by holding training courses and providing management assistance. One of the advantages of VC companies over other methods of nancing is the management and training assistance provided by venture capitalists. Hence, entrepreneur's ability to function varies with respect to the level of cost (T ) paid by the venture capitalist to enhance the former ability. Therefore, it can be stated that: where 0 is the ability level of the entrepreneur.
As obviously observed in the de ned function, if the investor (venture capitalist) does not invest in entrepreneur training, the initial ability level of the entrepreneur will not change. On the contrary, if the venture capitalist invests in entrepreneur training (T ), the ability level will be greater than 1. However, learning can compensate for lack of e ort; in other words, the ultimate value of the learning function can be greater than 1: 0 f (T ).
Knowing the e ort and ability functions, we can de ne the revenue function of the plan as follows: This function acts such that if an in nite level is attributed to the entrepreneur's e ort and no money is allocated by the venture capitalist, both parties will reach the maximum value of the company. However, it is obviously costly and sometimes impossible to make in nite e orts. In this situation, the investor attempts to improve the entrepreneur's ability level by spending money on training in order to achieve the maximum predicted value for the company. To ensure the e ort level of the entrepreneur and maximize the venture project, both parties should work out an incentive contract where the incentive payment function to the entrepreneur is de ned as follows: S(; ) = () + (); (5) where () stands for the xed component and () the variable component of the payment [17].
Followed by the contract approval, the VC parties incur costs. The cost incurred by the entrepreneur depends on his e ort and ability. The venture capitalist pays for managing, monitoring, and training. Accordingly, the cost functions of venture capitalist and entrepreneur are de ned as follows: Entrepreneur: Venture capitalist: where e x is the e ort cost with x representing the effort level of entrepreneur, and the constant coe cient certi ed based on each person's characteristics. Subsequently, the monetary income of the contract parties is described as: Monetary income of the venture capitalist: (9) In this case, the certainty equivalent wealth for the venture capitalist and entrepreneur, respectively, are given as follows: Venture capitalist: Entrepreneur: where v is the risk aversion coe cient of the venture capitalist, 2 v (1 ()) 2 where m is the wealth that the entrepreneur can obtain if the venture contract is not accepted.

Entrepreneur
The optimum investment model from the entrepreneur's perspective is: s.t.: where q is the wealth that the investor can obtain if the venture contract is not accepted.

Optimal solution to the model
Based on the model presented in Section 2.1, this issue falls into the category of constrained multi-objective optimization problems. To solve this model, the Karush, Kuhn, Tucker (KKT) method was employed.
In addition, to implement the conditions of the KKT, the model must be written as a maximization problem and the constraints must be ( 0). The conditions of the KKT are: In the following, we will solve the model through these conditions.
Upon solving IC and obtaining the optimal value of x, the IC constraint is removed from the model and then, According to the rst condition of KKT: @f @ ; @f @ = 1 @g 1 @ ; @g 1 @ ; 2 4 @f @ @f @  x   Finally, we came up with the following optimal solutions for venture capitalist that are listed in Table 1.

Entrepreneur
As described in the previous section, the model proposed in Subsection 2.1.2 is solved from the entrepreneur's point of view and the obtained optimal values are listed in Table 2.

Equilibrium in the optimal solutions of VC parties
A comparison of the optimal results obtained for the contract parties of the VC reveals that the optimum values of the e ort level (x) and constant component of the contract ( ) are di erent for the parties. In this situation, the parties will agree if their optimum answers are close.
The equilibrium values of the parties are obtained in this section and listed in Tables 1 and 2. An important question may arise here: how much does the venture capitalist pay (T ) for training the entrepreneur and increasing his/her ability to equalize the e ort level of the contracting parties? In addition, in the case of non-participation in the plan (m + q), how much should the total amount of income of the parties be for the constant component of the contract to make an equilibrium between the parties? 2.3.1. Parametric equilibrium values In order to obtain the parametric equilibrium solution, the optimal values of the e ort level, from both parties' perspective, as well as the optimal values of the constant component of the contract must be equalized. To equalize the optimal e ort level of the parties, rst, it should be determined that how much the capitalist should pay to train the entrepreneur in order to balance the optimal e orts level of the parties. Consequently:

Stating numeric values
Once tending to fund through the VC, entrepreneurs must present their innovative plan(s) to the venture capitalist for assessment and examination. The capitalist's assessment illustrates that the entrepreneur's innovative plan, if successful, could bring a value of 15 units (V = 15) for the venture company. Moreover, the entrepreneur's proposed plan was estimated to be 0.4 at the total investment required in the plan (r = 0:4). The entrepreneur's initial ability level was placed at the closed interval of [0; 1] (in this study, it was ( 0 = 1); however, it can vary while the capitalist holds training courses). The standard deviation, as a measurement of risk aversion, equals 1 ( = 1). In all studies, the entrepreneur was considered to be more risk averse than the venture capitalist. Therefore, in this study, it is assumed that entrepreneur's and venture capitalist's risk aversion coe cients are 3 and 1, respectively ( u = 3; v = 1). There are also constant coe cients in the functions allocated to e ort and cost which are used to determine the degree of convexity and gradient of the functions. In this numerical study, according to the personal characteristics of the entrepreneur and capitalist, the following values are estimated for these coe cients: Finally, once the optimal value for T is determined, the optimal and equilibrium values of the e ort level, from the perspective of the contract parties, are as follows: x v = x u = 4:216: By determining the equilibrium values of and x as well as the desired cost of holding the training courses (T ), we can create equilibrium in the values of the constant component of the contract ( ). By substituting the numbers into Eq. (28), the following result will be obtained: m + q = 1:5313: Therefore, if the total saved capital of the capitalist and entrepreneur (if not participating in the plan) is 1.5313 units, equilibrium in the constant component of the contract will be reached.
Ultimately, with regard to the assumptions of the problem and values de ned for the variables, the optimal values from the parties' perspectives are summarized in Table 3. Of note, the optimal values given in Table 3 were calculated, assuming that T = 0:6018.
As shown, , income, and pro t are sensitive to m and q. The e ects of m and q values on and pro t are illustrated in Figures 1 and 2, respectively.

Bounded rationality vs. game theory
Here, the venture capitalist and entrepreneur, as the two parties in the contract, are consequently considered as parties to the game, facing numerous strategic decisions to cooperate with each other. Given the nature of the model and variables used in it, the values of Table 3. Optimal values of numerical example from the perspective of the parties.  this is that if the investor proposes a low , a higher cost will be allocated to training the entrepreneur since the proposed is proportional to the ability of the entrepreneur, and choosing a low indicates that the investor believes in the entrepreneur's low ability, hence forced to provide more educational services. On the contrary, there are two views concerning the entrepreneur: 1. Entrepreneurs increase their e ort with the increase in the proposed (Table 4), 2. Entrepreneurs decrease their e ort with the increase in the proposed (Table 5). In this matrix, the entrepreneur's and venture capitalist's strategies are shown in the rst row and column, respectively. The values written in the matrix represent the players' income where the upper value is related to the venture capitalist and the lower value belongs to the entrepreneur. In each of the matrix cells, the numbers on the left correspond to the individuals' pro t, while those on the right (the larger number) correspond to their income regardless of the costs.   Individuals' incomes and bene ts are calculated based on the items such as the cost of risk-taking in the cost section as well as the non-real and tangible costs in order to clarify the results.
To meet the conditions of bounded rationality in the model, assume that in the initial stage, the contract parties with bounded knowledge and incomplete recognition of each other will make one of the strategic decisions. With additional information and a greater understanding of the environment over time, the parties will update their strategies. As a result, learning will happen during the implementation of the contract and will a ect the decisions of individuals. Therefore, learning algorithms are used in the model. This study also employed Q-learning algorithm as one of the reinforcement learning algorithms.
In this learning algorithm, the player number i performs the activity a i t at moment t and receives the reward R i t (a i t ; a i t ) based on the activity a i t , which is played by the rival. Then, function Q is updated based on the following recursive formula: where is the reducing factor and t = C i t 1 (a i ) the learning rate variable. Then, the next activity, a i t+1 , is randomly chosen from the probabilities determined through the following exponential decision-making formula: The results of this algorithm depend on many factors including the parameters used in the model such as , , and that have a great impact on the results. Other factors are the frequency of learning the model and its run time. In this regard, numerous studies have been carried out to nd proper values for these parameters. The values = 0:96, = 0:63, and = 6 were taken into account in this study [25]. Among the most important and in uential factors are the condition of stopping the algorithm (the degree of probability convergence of the chosen strategy) and random choices in the algorithm. This study took into account the probability of higher than 0.999 for the players simultaneously. On the other hand, random choices of the algorithm made the software output results not the same. To solve this problem, the algorithm should be repeated until learning takes place. In doing so, the probability of choosing each matrix cell and each strategy by the parties can be determined. The results of the two previously presented games are given in Tables 6 and 7. In the given matrix, the upper and lower numbers in each cell represent the number of repetitions and the corresponding probability, respectively.
The created matrices can be examined based on game theory. This game has two players who decide at the same time. Here, each player must maximize his/her expected income based on his/her perceptions of the other player. Based on interpretations, Nash  equilibrium in the generated game matrices can be calculated. The most important feature of the Nash equilibrium is that the outcome is not the greatest for the players. In this balance, the assumption of the rationality of the players is a necessary condition, and the correctness of each player's belief in choosing the opponent is a su cient condition. The distinct cell in Tables 6 and 7 indicates the Nash equilibrium.  Tables 8 and 9 show the probability of choosing each strategy for the investment parties based on the rst and second views of the entrepreneurs. The basis of the calculations is the pro t earned so that the results can be compared with those of the mathematical solution.

Conclusion
This paper attempted to design a model for venture capital contracts to prove that despite di erent expectations of the contract parties, equilibrium points exist that could convince the parties to sign the contract. As mentioned earlier, the optimal values of the models from the perspectives of each party varied, thus making it impossible to achieve a single strategy  and sign a contract in the real world. To nd the equilibrium values and sign the contract, the optimal values of the e ort level (x) and xed component of the incentive payment function ( ) from the perspective of each party should be equalized. Then, the values of T (the cost of entrepreneur training) and m+q (the total wealth that both parties to the contract could obtain if the contract was not accepted) were determined. According to our expectations, the higher the initial ability of the entrepreneur is, the less money the venture capitalists will pay for their training. In addition, the higher the value of q is, the lower the value of ( xed component of the entrepreneur's payment function) will be suggested by the venture capitalist; in addition, the higher the value of m is, the greater the value the entrepreneur will call for ; the results con rm these expectations.
As is clear, both entrepreneurs and venture capitalists seek to optimize the model from their own perspectives. This study shows that if the values of T (the cost of entrepreneur training) and m + q (total wealth that the parties to the contract can obtain if the contract is not accepted) are close to the equilibrium points obtained in the models, the optimal values of decision variables in the models of each party will be equal. In addition, as the distance from the equilibrium values increases, the probability of concluding a contract is reduced because of the con ict of interest.
In the learning algorithms section, the computational results of the Q-learning algorithm indicate that learning has been completely done for the investors because they only choose the strategies that generate the highest income in the rst and second rounds, but learning has not been well done for the entrepreneurs; thus, they select all of their strategies with di erent possibilities (higher than 10%). In the second scenario, the investors also learn very well and are more likely to choose pro table strategies. Finally, learning happens for the entrepreneurs and they select the pro table strategies with higher probabilities.
According to Table 6, in the rst scenario, the rst matrix cell is introduced as the Nash equilibrium. Based on Table 6, which shows the results of 100 iterations of the Q-learning algorithm, we nd that the parties most likely choose the Nash equilibrium point. That is, Q-learning outcomes converge with Nash equilibrium results. Also, according to Table 7, in the second scenario, the cell 1 5 is selected as the Nash equilibrium. This cell would be most likely selected by the parties compared to other cells. Therefore, the results of the Q-learning are reasonably convergent with Nash equilibrium.