Answers.Body: <p>The missing values might be an issue indeed. You might want to use imputation methods, e.g.</p>
  <pre><code>from sklearn.impute import KNNImputer
  from sklearn.experimental import enable_iterative_imputer
  from sklearn.impute import IterativeImputer
  
  def impute_df(df, feature_columns_to_apply, strategy):
      if strategy == 'mean':
          for col in feature_columns_to_apply:    
              df.loc[:, col] = df.loc[:, col].fillna(df[col].mean())
      elif strategy == 'median':
          for col in feature_columns_to_apply:    
              df.loc[:, col] = df.loc[:, col].fillna(df[col].median())
      elif strategy == 'knn':
          knn_imputer = KNNImputer(n_neighbors=5)
          df_tmp = pd.DataFrame(knn_imputer.fit_transform(df[feature_columns_to_apply]), columns=feature_columns_to_apply)
          df.loc[:, feature_columns_to_apply] = df_tmp[feature_columns_to_apply]
      elif strategy == 'iterative':
          iterative_imputer = IterativeImputer()
          df_tmp = pd.DataFrame(iterative_imputer.fit_transform(df[feature_columns_to_apply]), columns=feature_columns_to_apply)
          df.loc[:, feature_columns_to_apply] = df_tmp[feature_columns_to_apply]
      return df
      
  df_all = impute_df(df_all, feature_columns_to_apply, 'mean')
  </code></pre>
  <p>(IterativeImputer is heavy and slow)
  These are just sample strategies, you might want to use other methods, see e.g. <a href="https://www.biorxiv.org/content/biorxiv/early/2023/04/08/2023.04.07.535980.full.pdf" rel="nofollow noreferrer">here</a>.<br />
  Also consider other models such as random forest / XGBoost, they outperform linear regression in many cases.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-22T12:59:54.740Z
Answers.LastActivityDate: 2024-03-17T13:39:17.793Z
Answers.LastEditDate: 2024-03-17T13:39:17.793Z
Answers.UserDisplayName: rkellerm
Answers.Body: <p>The gist looks fine ..</p>
  <ul>
  <li><code>pandas</code> - looks fine</li>
  <li><code>SelectKBest</code> - cool</li>
  <li>transformation (obviously there's a lot of them)</li>
  <li>standard ML</li>
  </ul>
  <p>A common ML situation is, the classifier working but the accuracy is notably lower than expected. My personal <em>first thing to check</em> is the use of <em>linear regression</em> as a baseline model. The scatterplots give a gist, we don't know in particular what data points belong to what classification - thats crucial. What's helpful are coloured dots to assess how they sit against the regression line.</p>
  <p>What I think has happened is there's been a PCA via <code>SelectKBest</code>, unsupervised learning, which doesn't account for much of variance (<em>cf</em> 0.3), i.e. this is not an accuracy score.</p>
  <p>I've three suggestions:</p>
  <p>1/ An acceptable baseline model often used is <code>k-nearest neighbour</code> ..</p>
  <pre><code>from sklearn.model_selection import train_test_split
  from sklearn.neighbors import KNeighborsClassifier
  clf = KNeighborsClassifier(n_neighbors=3)
  clf.fit(X_train, y_train)
  print(&quot;Test set accuracy: {:.2f}&quot;.format(clf.score(X_test, y_test)))
  </code></pre>
  <p>I think this will produce much better results with an accuracy of &gt;0.7</p>
  <p>2/ However,  I would check the following (please see <strong>caveat</strong>),</p>
  <pre><code>for drug in df.keys():
      df ....
      X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
  </code></pre>
  <p>Just make sure that X_train, etc ... is supposed within the loop, rather than immediately outside it (four spaces to the left). Looking at your outputs they look fine.</p>
  <p>The thing to check is its not:</p>
  <pre><code>for drug in df.keys():
      df....
  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
  </code></pre>
  <p>The above loop makes sense to me.</p>
  <p>The original loop position <code>train_test_split</code> looks like the <code>key</code> loop might repeatedly over-ride the <code>X_train, X_test, y_train, y_test</code> outputs.</p>
  <p>3/ Very final point <code>SelectKBest</code> followed by <code>fit</code>. What about <code>fit_transform</code> instead? If the issue is the features are resulting in low description of the variation, maybe thats key here?</p>
  <hr />
  <p><strong>Caveat</strong> I do stress that its hard to comment beyond general terms because a good biological understanding is needed of the goal of the ML training. In context its not my area and I haven't read any of the  papers.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-07T07:16:55.507Z
Answers.Text: Many thanks for this. Apologies there was an error in the indentention in the code I posted, I have amended this in the edit.
Answers.UserDisplayName: LJM
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-07T07:28:02.480Z
Answers.Text: Regarding the the use of KNN, I should of mentioned I have approached this as a regression task, and the y variable is a measure of the concentration required to induce cell death) I'll update this. Though changing this to a classification task could simplify it and allow me to look at feature contribution to classification.
  
  For the fit/fit_transform point, I thought that fit_transform did the same function performed the same as .fit() followed by .transform(), is there a subtle difference?
Answers.UserDisplayName: LJM
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-06T22:16:56.163Z
Answers.LastActivityDate: 2022-09-07T02:28:35.597Z
Answers.LastEditDate: 2022-09-07T02:28:35.597Z
Answers.UserDisplayName: M__
Body: <p>I want to create a model to predict proteins which could be associated with drug response in cancer cell lines. I have cell line proteomics data, with compound screening data they have gained for a number of these cells. My aim is to relate these proteins to drug response. I am using this exercise as an introduction in the use of ML in biology.</p>
  <p>To this end, I have created dictionaries, with selected compounds as the keys, and their respective datasets. Cells are rows, proteins are columns, and the drug response is the <code>y</code> variable.</p>
  <p>I have approached the task as a regression task. The aim being to predict the ln(IC50) - a measure of the concentration required to induce cell death - and subsequently pull out the features (proteins) which have made the largest contribution to this prediction. As my baseline model, I have attempted linear regression. However, my model performance has consistently been very poor. A problem with this dataset is the large number of missing values (heatmap below - yellow = missing). However, even when trying a simple feature selection strategy with sklearn (first function), the explained variance is around 0.3 for the best performing compound. I have plotted the residuals in attempt to dig into this further, but the spread looks quite distributed.</p>
  <p>Any advise/suggestions would be appreciated. Or any further reading/papers relating to ML in biology which could help a beginner build and use effective models.</p>
  <p>Training and test functions shown below.</p>
  <pre><code>    def process_training_and_test_data_for_LG(df):
          training_datasets = {}
          test_datasets = {}
          for drug in df.keys():
              df[drug].dropna(axis=1,inplace=True)
              print(&quot;at &quot;+drug)
              X = df[drug].drop([&quot;cell_line_name&quot;, &quot;ln_IC50&quot;,&quot;putative_target&quot;], axis=1)
              y = df[drug][&quot;ln_IC50&quot;]
              X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
  
              feature_selector = SelectKBest(score_func=f_regression, k=50)
              feature_selector.fit(X_train, y_train)
          
              X_train = feature_selector.transform(X_train)
              X_test = feature_selector.transform(X_test)
          
              #Add data to relevant dictionaries
              training_datasets[drug] = [X_train, y_train]
              test_datasets[drug] = [X_test, y_test]
          return training_datasets, test_datasets
  
  def train_models(training_data):
      trained_models = {}
      for drug in training_data.keys():
          #initiate model
          lm = LinearRegression()
          #extract drug specific training data
          X_train = training_data[drug][0]
          y_train = training_data[drug][1]
          #fit model
          model = lm.fit(X_train, y_train)
          #Append dict of trained models
          trained_models[drug] = model
          print(&quot;Model for {} trained&quot;.format(drug))
      return trained_models
  
  def test_models(model_dict, test_data):
      prediction_dict = {}
      for drug in model_dict.keys():
          drug_model = model_dict[drug]
          X_test = test_data[drug][0]
          predictions = drug_model.predict(X_test)
          prediction_dict[drug] = predictions
      return prediction_dict
  </code></pre>
  <p><a href="https://i.sstatic.net/ON9TG.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ON9TG.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/mNmy6.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/mNmy6.png" alt="Residual plot per compound" /></a></p>
  
Tags: <python><machine-learning><proteomics><modelling><linear-regression>
Title: Advise on building an effect ML model for predicting important proteins for drug response
ThreadText: # Advise on building an effect ML model for predicting important proteins for drug response
  
  (Asked by: username_0 on 2022-09-06)
  
  I want to create a model to predict proteins which could be associated with drug response in cancer cell lines. I have cell line proteomics data, with compound screening data they have gained for a number of these cells. My aim is to relate these proteins to drug response. I am using this exercise as an introduction in the use of ML in biology.
  
  To this end, I have created dictionaries, with selected compounds as the keys, and their respective datasets. Cells are rows, proteins are columns, and the drug response is the `y` variable.
  
  I have approached the task as a regression task. The aim being to predict the ln(IC50) - a measure of the concentration required to induce cell death - and subsequently pull out the features (proteins) which have made the largest contribution to this prediction. As my baseline model, I have attempted linear regression. However, my model performance has consistently been very poor. A problem with this dataset is the large number of missing values (heatmap below - yellow = missing). However, even when trying a simple feature selection strategy with sklearn (first function), the explained variance is around 0.3 for the best performing compound. I have plotted the residuals in attempt to dig into this further, but the spread looks quite distributed.
  
  Any advise/suggestions would be appreciated. Or any further reading/papers relating to ML in biology which could help a beginner build and use effective models.
  
  Training and test functions shown below.
  
  ```
      def process_training_and_test_data_for_LG(df):
          training_datasets = {}
          test_datasets = {}
          for drug in df.keys():
              df[drug].dropna(axis=1,inplace=True)
              print("at "+drug)
              X = df[drug].drop(["cell_line_name", "ln_IC50","putative_target"], axis=1)
              y = df[drug]["ln_IC50"]
              X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
  
              feature_selector = SelectKBest(score_func=f_regression, k=50)
              feature_selector.fit(X_train, y_train)
          
              X_train = feature_selector.transform(X_train)
              X_test = feature_selector.transform(X_test)
          
              #Add data to relevant dictionaries
              training_datasets[drug] = [X_train, y_train]
              test_datasets[drug] = [X_test, y_test]
          return training_datasets, test_datasets
  
  def train_models(training_data):
      trained_models = {}
      for drug in training_data.keys():
          #initiate model
          lm = LinearRegression()
          #extract drug specific training data
          X_train = training_data[drug][0]
          y_train = training_data[drug][1]
          #fit model
          model = lm.fit(X_train, y_train)
          #Append dict of trained models
          trained_models[drug] = model
          print("Model for {} trained".format(drug))
      return trained_models
  
  def test_models(model_dict, test_data):
      prediction_dict = {}
      for drug in model_dict.keys():
          drug_model = model_dict[drug]
          X_test = test_data[drug][0]
          predictions = drug_model.predict(X_test)
          prediction_dict[drug] = predictions
      return prediction_dict
  
  ```
  
  [![enter image description here](https://i.sstatic.net/ON9TG.png)](https://i.sstatic.net/ON9TG.png)
  
  [![Residual plot per compound](https://i.sstatic.net/mNmy6.png)](https://i.sstatic.net/mNmy6.png)
  
  **Tags:** python, machine-learning, proteomics, modelling, linear-regression
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-02-22. Score: 1
  
  The missing values might be an issue indeed. You might want to use imputation methods, e.g.
  
  ```
  from sklearn.impute import KNNImputer
  from sklearn.experimental import enable_iterative_imputer
  from sklearn.impute import IterativeImputer
  
  def impute_df(df, feature_columns_to_apply, strategy):
      if strategy == 'mean':
          for col in feature_columns_to_apply:    
              df.loc[:, col] = df.loc[:, col].fillna(df[col].mean())
      elif strategy == 'median':
          for col in feature_columns_to_apply:    
              df.loc[:, col] = df.loc[:, col].fillna(df[col].median())
      elif strategy == 'knn':
          knn_imputer = KNNImputer(n_neighbors=5)
          df_tmp = pd.DataFrame(knn_imputer.fit_transform(df[feature_columns_to_apply]), columns=feature_columns_to_apply)
          df.loc[:, feature_columns_to_apply] = df_tmp[feature_columns_to_apply]
      elif strategy == 'iterative':
          iterative_imputer = IterativeImputer()
          df_tmp = pd.DataFrame(iterative_imputer.fit_transform(df[feature_columns_to_apply]), columns=feature_columns_to_apply)
          df.loc[:, feature_columns_to_apply] = df_tmp[feature_columns_to_apply]
      return df
      
  df_all = impute_df(df_all, feature_columns_to_apply, 'mean')
  
  ```
  
  (IterativeImputer is heavy and slow)
  These are just sample strategies, you might want to use other methods, see e.g. [here](https://www.biorxiv.org/content/biorxiv/early/2023/04/08/2023.04.07.535980.full.pdf).  
  Also consider other models such as random forest / XGBoost, they outperform linear regression in many cases.
  
  
  ---
  
  ### Answer by username_2 on 2022-09-06. Score: 0
  
  The gist looks fine ..
  
  * `pandas` - looks fine
  * `SelectKBest` - cool
  * transformation (obviously there's a lot of them)
  * standard ML
  
  A common ML situation is, the classifier working but the accuracy is notably lower than expected. My personal *first thing to check* is the use of *linear regression* as a baseline model. The scatterplots give a gist, we don't know in particular what data points belong to what classification - thats crucial. What's helpful are coloured dots to assess how they sit against the regression line.
  
  What I think has happened is there's been a PCA via `SelectKBest`, unsupervised learning, which doesn't account for much of variance (*cf* 0.3), i.e. this is not an accuracy score.
  
  I've three suggestions:
  
  1/ An acceptable baseline model often used is `k-nearest neighbour` ..
  
  ```
  from sklearn.model_selection import train_test_split
  from sklearn.neighbors import KNeighborsClassifier
  clf = KNeighborsClassifier(n_neighbors=3)
  clf.fit(X_train, y_train)
  print("Test set accuracy: {:.2f}".format(clf.score(X_test, y_test)))
  
  ```
  
  I think this will produce much better results with an accuracy of >0.7
  
  2/ However, I would check the following (please see **caveat**),
  
  ```
  for drug in df.keys():
      df ....
      X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
  
  ```
  
  Just make sure that X\_train, etc ... is supposed within the loop, rather than immediately outside it (four spaces to the left). Looking at your outputs they look fine.
  
  The thing to check is its not:
  
  ```
  for drug in df.keys():
      df....
  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
  
  ```
  
  The above loop makes sense to me.
  
  The original loop position `train_test_split` looks like the `key` loop might repeatedly over-ride the `X_train, X_test, y_train, y_test` outputs.
  
  3/ Very final point `SelectKBest` followed by `fit`. What about `fit_transform` instead? If the issue is the features are resulting in low description of the variation, maybe thats key here?
  
  ---
  
  **Caveat** I do stress that its hard to comment beyond general terms because a good biological understanding is needed of the goal of the ML training. In context its not my area and I haven't read any of the papers.
  
  > **Comments:**
  > **username_0**: Many thanks for this. Apologies there was an error in the indentention in the code I posted, I have amended this in the edit.  
  > **username_0**: Regarding the the use of KNN, I should of mentioned I have approached this as a regression task, and the y variable is a measure of the concentration required to induce cell death) I'll update this. Though changing this to a classification task could simplify it and allow me to look at feature contribution to classification.  For the fit/fit_transform point, I thought that fit_transform did the same function performed the same as .fit() followed by .transform(), is there a subtle difference?  
  
  ---
  
  