<p dir="auto">I have a dataframe looks like this, and its column 2 is missing:<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/6269369/8245984/6ff1798c-1669-11e5-88b1-2c27a6b5f2fb.png"><img src="https://cloud.githubusercontent.com/assets/6269369/8245984/6ff1798c-1669-11e5-88b1-2c27a6b5f2fb.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto">When I try to select the max date in each row, I got all NaN in return:<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/6269369/8246090/eea64bbc-166a-11e5-9fb3-668a37a2cf3a.png"><img src="https://cloud.githubusercontent.com/assets/6269369/8246090/eea64bbc-166a-11e5-9fb3-668a37a2cf3a.png" alt="img2" style="max-width: 100%;"></a></p>
<p dir="auto">However, If the dataframe's type is float64, the selection work as expected.</p>
<h4 dir="auto">Code Sample, a copy-pastable example if possible</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="df.to_sql('TableNameHere', engine, if_exists='append', chunksize=900, index=False)
"><pre class="notranslate"><span class="pl-s1">df</span>.<span class="pl-en">to_sql</span>(<span class="pl-s">'TableNameHere'</span>, <span class="pl-s1">engine</span>, <span class="pl-s1">if_exists</span><span class="pl-c1">=</span><span class="pl-s">'append'</span>, <span class="pl-s1">chunksize</span><span class="pl-c1">=</span><span class="pl-c1">900</span>, <span class="pl-s1">index</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)</pre></div>
<h4 dir="auto">Problem description</h4>
<p dir="auto">I am trying to append a large DataFrame to a SQL table. Some of the rows in the DataFrame are duplicates of those in the SQL table, some are not. But <code class="notranslate">to_sql()</code> completely stops executing if even one duplicate is detected.</p>
<p dir="auto">It would make sense for <code class="notranslate">to_sql(if_exists='append')</code> to merely warn the user which rows had duplicate keys and just continue to add the new rows, not completely stop executing. For large datasets, you will likely have duplicates but want to ignore them.</p>
<p dir="auto">Maybe add an argument to ignore duplicates and keep executing? Perhaps an additional <code class="notranslate">if_exists</code> option like <code class="notranslate">'append_skipdupes'</code>?</p>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<details>
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.0.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: English_United States.1252
<p dir="auto">pandas: 0.19.2<br>
nose: None<br>
pip: 9.0.1<br>
setuptools: 28.8.0<br>
Cython: None<br>
numpy: 1.12.0<br>
scipy: None<br>
statsmodels: None<br>
xarray: None<br>
IPython: 5.3.0<br>
sphinx: None<br>
patsy: None<br>
dateutil: 2.6.0<br>
pytz: 2016.10<br>
blosc: None<br>
bottleneck: None<br>
tables: None<br>
numexpr: None<br>
matplotlib: None<br>
openpyxl: None<br>
xlrd: None<br>
xlwt: None<br>
xlsxwriter: None<br>
lxml: None<br>
bs4: None<br>
html5lib: 0.999999999<br>
httplib2: None<br>
apiclient: None<br>
sqlalchemy: 1.1.9<br>
pymysql: None<br>
psycopg2: None<br>
jinja2: 2.9.5<br>
boto: None<br>
pandas_datareader: None</p>
</details>