title: Python: How to optimize function parameters?
body: <p><strong>Background:</strong></p>
  <p>I'd like to solve a wide array of optimization problems such as asset weights in a portfolio, and parameters in trading strategies <strong>where the variables are passed to functions containing a bunch of other variables as well</strong>.</p>
  <p>Until now, I've been able to do these things easily in Excel using the Solver Add-In. But I think it would be much more efficient and even more widely applicable using Python. For the sake of clarity, I'm going to boil the question down to the essence of portfolio optimization.</p>
  <p><strong>My question (short version):</strong></p>
  <p>Here's a dataframe and a corresponding plot with asset returns.</p>
  <p><em>Dataframe 1:</em></p>
  <pre><code>                A1      A2
  2017-01-01  0.0075  0.0096
  2017-01-02 -0.0075 -0.0033
  .
  .
  2017-01-10  0.0027  0.0035
  </code></pre>
  <p><em>Plot 1 - Asset returns</em></p>
  <p><a href="https://i.stack.imgur.com/HrxcG.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/HrxcG.png" alt="enter image description here" /></a></p>
  <p>Based on that, I would like to find the weights for the optimal portfolio with regards to <code>risk / return (Sharpe ratio)</code>, represented by the green dot in the plot below (the red dot is the so-called minimum variance portfolio, and represents another optimization problem).</p>
  <p><em>Plot 2 - Efficient frontier and optimal portfolios:</em></p>
  <p><a href="https://i.stack.imgur.com/QBSSE.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/QBSSE.png" alt="enter image description here" /></a></p>
  <h2><strong>How can I do this with numpy or scipy?</strong></h2>
  <p><strong>The details:</strong></p>
  <p>The following code section contains the function <code>returns()</code> to build a dataframe with random returns for two assets, as well as a function <code>pf_sharpe</code> to calculate the Sharpe ratio of two given weights for a portfolio of the returns.</p>
  <pre><code># imports
  import pandas as pd
  import numpy as np
  from scipy.optimize import minimize
  import matplotlib.pyplot as plt
  
  np.random.seed(1234)
  
  # Reproducible data sample
  def returns(rows, names):
      ''' Function to create data sample with random returns
      
      Parameters
      ==========
      rows : number of rows in the dataframe
      names: list of names to represent assets
      
      Example
      =======
      
      &gt;&gt;&gt; returns(rows = 2, names = ['A', 'B'])
      
                    A       B
      2017-01-01  0.0027  0.0075
      2017-01-02 -0.0050 -0.0024
      '''
      listVars= names
      rng = pd.date_range('1/1/2017', periods=rows, freq='D')
      df_temp = pd.DataFrame(np.random.randint(-100,100,size=(rows, len(listVars))), columns=listVars) 
      df_temp = df_temp.set_index(rng)
      df_temp = df_temp / 10000
  
      return df_temp
  
  
  # Sharpe ratio
  def pf_sharpe(df, w1, w2):
      ''' Function to calculate risk / reward ratio
          based on a pandas dataframe with two return series
      
      Parameters
      ==========
      df : pandas dataframe
      w1 : portfolio weight for asset 1
      w2 : portfolio weight for asset 2
      
      '''
      
      weights = [w1,w2]      
      
      # Calculate portfolio returns and volatility
      pf_returns = (np.sum(df.mean() * weights) * 252)
      pf_volatility = (np.sqrt(np.dot(np.asarray(weights).T, np.dot(df.cov() * 252, weights))))
         
      # Calculate sharpe ratio
      pf_sharpe = pf_returns / pf_volatility
      
      return pf_sharpe
  
  # Make df with random returns and calculate
  # sharpe ratio for a 80/20 split between assets
  df_returns = returns(rows = 10, names = ['A1', 'A2'])
  df_returns.plot(kind = 'bar')
  
  sharpe = pf_sharpe(df = df_returns, w1 = 0.8, w2 = 0.2)
  print(sharpe)
  
  # Output:
  # 5.09477512073
  </code></pre>
  <p>Now I'd like to find the portfolio weights that optimize the Sharpe ratio. I think you could express the optimization problem as follows:</p>
  <pre><code>maximize:
      pf_sharpe()
  
  by changing:
      w1, w2
  
  under the constraints:
      0 &lt; w1 &lt; 1
      0 &lt; w2 &lt; 1
      w1 + w2 = 1
  </code></pre>
  <p><strong>What I've tried so far:</strong></p>
  <p>I found a possible setup in the post <a href="https://stackoverflow.com/questions/26882087/python-scipy-optimization-minimize-using-slsqp-showing-maximized-results">Python Scipy Optimization.minimize using SLSQP showing maximized results</a>. Below is what I have so far, and it addresses a central aspect of my question directly:</p>
  <blockquote>
  <p>[...]where the variables are passed to functions containing a bunch of other variables as well.</p>
  </blockquote>
  <p>As you can see, my initial challenge prevents me from even testing if my bounds and constraints will be accepted by the function <code>optimize.minimize()</code>. I haven't even bothered to take into consideration the fact that this is a maximization and not a minimization problem (hopefully amendable by changing the sign of the function).</p>
  <p><strong>Attempts:</strong></p>
  <pre><code># bounds
  b = (0,1)
  bnds = (b,b)
  
  # constraints
  def constraint1(w1,w2):
      return w1 - w2
  
  cons = ({'type': 'eq', 'fun':constraint1})
  
  # initial guess
  x0 = [0.5, 0.5]
  
  # Testing the initial guess
  print(pf_sharpe(df = df_returns, weights = x0))
  
  # Optimization attempts
  
  attempt1 = optimize.minimize(pf_sharpe(), x0, method = 'SLSQP', bounds = bnds, constraints = cons)
  attempt2 = optimize.minimize(pf_sharpe(df = df_returns, weights),  x0, method = 'SLSQP', bounds = bnds, constraints = cons)
  attempt3 = optimize.minimize(pf_sharpe(weights, df = df_returns), x0, method = 'SLSQP', bounds = bnds, constraints = cons)
  </code></pre>
  <p><strong>Results:</strong></p>
  <ul>
  <li>Attempt1 is closest to the scipy setup <a href="https://meta.stackoverflow.com/questions/365646/is-it-ok-to-use-youtube-links-in-a-question/365647?noredirect=1#comment571606_365647">here</a>, but understandably fails because neither <code>df</code> nor <code>weights</code> have been specified.</li>
  <li>Attempt2 fails with <code>SyntaxError: positional argument follows keyword argument</code></li>
  <li>Attempt3 fails with <code>NameError: name 'weights' is not defined</code></li>
  </ul>
  <p>I was under the impression that <code>df</code> could freely be specified, and that <code>x0</code> in <code>optimize.minimize</code> would be considered the variables to be tested as 'representatives' for the weights in the function specified by <code>pf_sharpe()</code>.</p>
  <p>As you surely understand, my transition from Excel to Python in this regard has not been the easiest, and there is plenty I don't understand here. Anyway, I'm hoping some of you may offer some suggestions or clarifications!</p>
  <p>Thank you!</p>
  <p><strong>Appendix 1 - Simulation approach</strong>:</p>
  <p>This particular portfolio optimization problem can easily be solved by simulating a bunch of portfolio weights. And I did exactly that to produce the portfolio plot above. Here's the whole function if anyone is interested:</p>
  <pre><code># Portfolio simulation
  def portfolioSim(df, simRuns):
      ''' Function to take a df with asset returns,
          runs a number of simulated portfolio weights,
          plots return and risk for those weights,
          and finds minimum risk portfolio
          and max risk / return portfolio
      
      Parameters
      ==========
      df : pandas dataframe with returns
      simRuns : number of simulations
      
      '''  
      prets = []
      pvols = []
      pwgts = []
      names = list(df_returns)
      
      for p in range (simRuns):
          
          # Assign random weights
          weights = np.random.random(len(list(df_returns)))
          weights /= np.sum(weights)
          weights = np.asarray(weights)        
      
          # Calculate risk and returns with random weights
          prets.append(np.sum(df_returns.mean() * weights) * 252)
          pvols.append(np.sqrt(np.dot(weights.T, np.dot(df_returns.cov() * 252, weights))))
          pwgts.append(weights)
              
      prets = np.array(prets)
      pvols = np.array(pvols)
      pwgts = np.array(pwgts)
      pshrp = prets / pvols
      
      # Store calculations in a df
      df1 = pd.DataFrame({'return':prets})         
      df2 = pd.DataFrame({'risk':pvols})    
      df3 = pd.DataFrame(pwgts)
      df3.columns = names
      df4 = pd.DataFrame({'sharpe':pshrp})
      df_temp = pd.concat([df1, df2, df3, df4], axis = 1)
      
      # Plot resulst
      plt.figure(figsize=(8, 4))
      plt.scatter(pvols, prets, c=prets / pvols, cmap = 'viridis', marker='o')
      
      # Min risk
      min_vol_port = df_temp.iloc[df_temp['risk'].idxmin()]   
      plt.plot([min_vol_port['risk']], [min_vol_port['return']], marker='o', markersize=12, color=&quot;red&quot;)
      
      # Max sharpe
      max_sharpe_port = df_temp.iloc[df_temp['sharpe'].idxmax()]    
      plt.plot([max_sharpe_port['risk']], [max_sharpe_port['return']], marker='o', markersize=12, color=&quot;green&quot;)
  
  # Test run
  portfolioSim(df = df_returns, simRuns = 250)
  </code></pre>
  <p><strong>Appendix 2 - Excel Solver approach</strong>:</p>
  <p>Here is how I would approach the problem using Excel Solver. Instead of linking to a file, I've only attached a screenshot and included the most important formulas in a code section. I'm guessing not many of you is going to be interested in reproducing this anyway. But I've included it just to show that it can be done quite easily in Excel.
  Grey ranges represent formulas. Ranges that can be changed and used as arguments in the optimization problem are highlighted in yellow. The green range is the objective function.</p>
  <p>Here's an image of the worksheet and Solver setup:</p>
  <p><a href="https://i.stack.imgur.com/j28j9.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/j28j9.png" alt="enter image description here" /></a></p>
  <p><strong>Excel formulas:</strong></p>
  <pre><code>C3  =AVERAGE(C7:C16)
  C4  =AVERAGE(D7:D16)
  H4  =COVARIANCE.P(C7:C16;D7:D16)
  G5  =COVARIANCE.P(C7:C16;D7:D16)
  G10 =G8+G9
  G13 =MMULT(TRANSPOSE(G8:G9);C3:C4)
  G14 =SQRT(MMULT(TRANSPOSE(G8:G9);MMULT(G4:H5;G8:G9)))
  H13 =G12/G13
  H14 =G13*252
  G16 =G13/G14
  H16 =H13/H14
  </code></pre>
  <p><strong>End notes:</strong></p>
  <p>As you can see from the screenshot, Excel solver suggests a <code>47% / 53%</code> split between A1 and A2 to obtain an optimal Sharpe Ratio of 5,6. Running the Python function <code>sr_opt = portfolioSim(df = df_returns, simRuns = 25000)</code> yields a Sharpe Ratio of 5,3 with corresponding weights of <code>46% and 53%</code> for A1 and A2:</p>
  <pre><code>print(sr_opt)
  #Output
  #return    0.361439
  #risk      0.067851
  #A1        0.465550
  #A2        0.534450
  #sharpe    5.326933
  </code></pre>
  <p>The method applied in Excel is <code>GRG Nonlinear</code>. I understand that changing the <code>SLSQP</code> argument to a non-linear method would get me somewhere, and I've look into <a href="https://docs.scipy.org/doc/scipy-0.13.0/reference/optimize.nonlin.html" rel="nofollow noreferrer">Nonlinear solvers in scipy</a> as well, but with little success.
  And maybe Scipy even isn't the best option here?</p>