Body: <p>I'm designing a NLP model to extract various kinds of &quot;hidden&quot; expenses from 10-K and 10-Q financial statements. I've come up with about 7 different expense categories (restructuring costs, merger and acquisitions, etc.) and for each one I have a list of terms/synonyms that different companies call them. I'm new to NLP would like some advice on the best approach for extracting them.</p>
  <p>Values are usually hidden in two different areas of the document:</p>
  <h3>Type 1: Free-form text (footnotes)</h3>
  <p>Values are nested in sentences. Here are some examples, with the <em>Expense Type</em> and <strong>Monetary value</strong> indicated.</p>
  <blockquote>
  <p><em>Exploratory dry-hole costs</em> were <strong>\$12.7 million</strong>, \$1.3 million, and \$1.0 million for the years ended December 31, 2012, 2011, and 2010, respectively.</p>
  </blockquote>
  <blockquote>
  <p>2012 includes the recognition of a <strong>$3,340 million</strong> <em>impairment charge</em> related to the carrying value of Citi's remaining 35% interest in the Morgan Stanley Smith Barney joint venture</p>
  </blockquote>
  <blockquote>
  <p>During the year ended December 31, 2017, we decided to discontinue the internal development of AMG 899, resulting in an <em>impairment charge</em> of <strong>$400 million</strong> for the IPR&amp;D asset</p>
  </blockquote>
  <h3>Type 2: Table data</h3>
  <p>SEC statements also contain &quot;structured&quot; data in HTML tables. Some line items, like the first row below, correspond to the expense type I'm looking for:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Item</th>
  <th>2020</th>
  <th>2019</th>
  <th>2018</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>impairment related to real estate assets(2):</td>
  <td>398.2</td>
  <td>200</td>
  <td>0</td>
  </tr>
  <tr>
  <td>research and development</td>
  <td>100</td>
  <td>200</td>
  <td>300</td>
  </tr>
  <tr>
  <td>other expenses</td>
  <td>20</td>
  <td>30</td>
  <td>40</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>Correct value = <code>398.2</code></p>
  <hr />
  <p>I'm thinking about a two-model approach:</p>
  <ol>
  <li><p>Define a new NER model based off the terms I already know (e.g. &quot;dry-hole costs&quot;, &quot;impairment charges&quot;). I would need to manually annotate extracts from historic statements that contain these terms for the training set.</p>
  <ul>
  <li>For free-form text, it would match the sentence and pass it on for further processing (see 2).</li>
  <li>For table data, I would loop over each row using beautifulsoup and pandas, check the first column for a match (e.g. using spaCy's comparison function), and then grab that year's value from the dataframe and finish.</li>
  </ul>
  </li>
  <li><p>For free-form matches, I still need to grab the monetary value for the correct year (sometimes multiple values are given for various years, see the first example above).</p>
  </li>
  </ol>
  <p>One potential problem here is that sentences like this would cause problems:</p>
  <blockquote>
  <p>We gained $100 million this year, despite facing restructuring charges.</p>
  </blockquote>
  <p>If the NLP algo is split into the above two-model process, model 1 would pass (because it contains a known term like &quot;restructuring charges&quot;), and model 2 would extract <code>$100 million</code>, which is incorrect because it doesn't actually correspond to the expense itself.</p>
  <p>Is there a better solution here? As I said, I'm new to NLP and data extraction so would really appreciate any advice or resources to learn more about solving these types of key/value problems.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-12T14:05:34.913Z
Comments.Text: A lot of ML when applied to financial analysis has the potential to be worth millions. I'm just interested in general strategies that might help solve this kind of problem, not a full implementation.
Comments.UserDisplayName: hohner
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-13T04:25:15.990Z
Comments.Text: well why can't hidden also be obvious? other expenses just labeled as "restructuring charges" match (e.g. (2) = $398.2 million applied over year 2020. or am i getting too far off of this one?
Comments.UserDisplayName: An0n173
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-03T22:47:22.707Z
Comments.Text: any update on what you tried?
Comments.UserDisplayName: drerD
Tags: <natural-language-processing><named-entity-recognition><spacy>
Title: Extracting "hidden" costs from financial statements using NLP
UserDisplayName: hohner
ThreadText: # Extracting "hidden" costs from financial statements using NLP
  
  (Asked by: username_0 on 2021-04-09)
  
  I'm designing a NLP model to extract various kinds of "hidden" expenses from 10-K and 10-Q financial statements. I've come up with about 7 different expense categories (restructuring costs, merger and acquisitions, etc.) and for each one I have a list of terms/synonyms that different companies call them. I'm new to NLP would like some advice on the best approach for extracting them.
  
  Values are usually hidden in two different areas of the document:
  
  ### Type 1: Free-form text (footnotes)
  
  Values are nested in sentences. Here are some examples, with the *Expense Type* and **Monetary value** indicated.
  
  > *Exploratory dry-hole costs* were **\$12.7 million**, \$1.3 million, and \$1.0 million for the years ended December 31, 2012, 2011, and 2010, respectively.
  
  > 2012 includes the recognition of a **$3,340 million** *impairment charge* related to the carrying value of Citi's remaining 35% interest in the Morgan Stanley Smith Barney joint venture
  
  > During the year ended December 31, 2017, we decided to discontinue the internal development of AMG 899, resulting in an *impairment charge* of **$400 million** for the IPR&D asset
  
  ### Type 2: Table data
  
  SEC statements also contain "structured" data in HTML tables. Some line items, like the first row below, correspond to the expense type I'm looking for:
  
  | Item | 2020 | 2019 | 2018 |
  | --- | --- | --- | --- |
  | impairment related to real estate assets(2): | 398.2 | 200 | 0 |
  | research and development | 100 | 200 | 300 |
  | other expenses | 20 | 30 | 40 |
  
  Correct value = `398.2`
  
  ---
  
  I'm thinking about a two-model approach:
  
  1. Define a new NER model based off the terms I already know (e.g. "dry-hole costs", "impairment charges"). I would need to manually annotate extracts from historic statements that contain these terms for the training set.
  
     * For free-form text, it would match the sentence and pass it on for further processing (see 2).
     * For table data, I would loop over each row using beautifulsoup and pandas, check the first column for a match (e.g. using spaCy's comparison function), and then grab that year's value from the dataframe and finish.
  2. For free-form matches, I still need to grab the monetary value for the correct year (sometimes multiple values are given for various years, see the first example above).
  
  One potential problem here is that sentences like this would cause problems:
  
  > We gained $100 million this year, despite facing restructuring charges.
  
  If the NLP algo is split into the above two-model process, model 1 would pass (because it contains a known term like "restructuring charges"), and model 2 would extract `$100 million`, which is incorrect because it doesn't actually correspond to the expense itself.
  
  Is there a better solution here? As I said, I'm new to NLP and data extraction so would really appreciate any advice or resources to learn more about solving these types of key/value problems.
  
  **Tags:** natural-language-processing, named-entity-recognition, spacy
  
  > **Comments:**
  > **username_0**: A lot of ML when applied to financial analysis has the potential to be worth millions. I'm just interested in general strategies that might help solve this kind of problem, not a full implementation.  
  > **username_1**: well why can't hidden also be obvious? other expenses just labeled as "restructuring charges" match (e.g. (2) = $398.2 million applied over year 2020. or am i getting too far off of this one?  
  > **username_2**: any update on what you tried?  
  
  ---
  
  ## 0 answers
  
  