title: Efficiently find "duplicates" between two lists with dict-elements comparing just a subset of the dict-fields
body: <h1>What makes my question different...</h1>
  
  <p>...from similar ones here on StackOverflow. In the <a href="https://en.wikipedia.org/wiki/Minimal_working_example" rel="nofollow noreferrer">MWE</a> below, the <code>dict</code> has 4 fields; but only 2 of them are relevant for comparing.</p>
  
  <h1>The problem...</h1>
  
  <p>...is easy, and I always provide a solution in my MWE below, but I would like to know <strong>if there is a more efficient, pythonic way</strong> to handle that -- efficient in the meaning of CPU time.</p>
  
  <p>Two <code>list()</code> with <code>dict()</code> elements need to be check for "duplicates." A <em>duplicate</em> is defined as if the fields <code>title</code> and <code>date</code> are equal. The content of the other fields (<code>flag</code> and <code>misc</code>) do not matter. To add another factor: the <code>dict</code> elements can have different number of fields, but <code>title</code> and <code>date</code> are always present.</p>
  
  <h1>My solution</h1>
  
  <p>I simply translate my human words into python words (<code>dupli</code> and <code>data</code> are the two lists):</p>
  
  <pre><code>for d in dupli[:]:
      for e in data:
          # compare by 'title' and 'date'
          if e['title'] == d['title'] and e['date'] == d['date']:
              print('Duplicate found: {}'.format(d))
              # remove duplicates
              dupli.remove(d)
  </code></pre>
  
  <h1>My full MWE</h1>
  
  <p>This MWE generates the list <code>data</code> with 100 random elements. The second list <code>dupli</code> has 10 elements where 3 of them have "duplicates" (by <code>title</code> and <code>date</code> BUT NOT <code>misc</code> or <code>flag</code>) in the first list.</p>
  
  <pre><code>#!/usr/bin/env python3
  import random
  import string
  
  
  # helper function
  def random_string(n):
      return ''.join([random.choice(string.ascii_letters) for i in range(n)])
  
  
  # Create persistent data
  def create_data(n):
      container = []
      for idx in range(n):
          # create an element with title, date and some misc data
          element = {
             'title': random_string(random.randrange(35)),
             'date': [2020, 5, random.randint(1, 30)],
             'flag': (random.random() &lt; 0.5),  # bool
             'misc': random_string(random.randrange(5)),
          }
  
          container.append(element)
      return container
  
  
  # persistent data
  data = create_data(100)
  
  # new data elements with possible duplicates
  dupli = create_data(10)
  
  # generate 3 duplicates
  for idx in [2, 5, 8]:
      dupli[idx]['title'] = data[idx*10]['title']
      dupli[idx]['date'] = data[idx*10]['date']
  
  # MY APPROACH to check for duplicates
  for d in dupli[:]:
      for e in data:
          # compare by 'title' and 'date'
          if e['title'] == d['title'] and e['date'] == d['date']:
              print('Duplicate found: {}'.format(d))
              # remove duplicates
              dupli.remove(d)
  
  # add the rest to persistent data
  data.extend(dupli)
  </code></pre>
  
  <h1>Background information</h1>
  
  <p>This is only an MWE. The real world data is more complex. That is why efficiency is important for me. The following numbers depending on my users of course, but just to give you and idea:</p>
  
  <ul>
  <li>A <code>dict</code> element has 4 to 12 fields.</li>
  <li>A <code>list</code> have 200 to 20.000 elements.</li>
  <li>There are 30 to 300 lists in my application.</li>
  <li>On the other side for each list there is a list with new elements: 10 to 40 elements.</li>
  </ul>