title: Found duplicate column in one of the json when running spark.read.json even though there are no duplicate columns
body: <p>I am getting very strange error in PySpark and also in Synapse data flow.</p>
  <p>I am reading JSON file with below query but getting duplicate column error <strong>even though there is no duplicate column</strong>. I can read it using other tools and JSON validator and also with data flow but not in PySpark.</p>
  <p>PySpark query is as below:</p>
  <pre class="lang-py prettyprint-override"><code>df = (
      spark.read.option(&quot;multiline&quot;, &quot;true&quot;)
      .options(encoding=&quot;UTF-8&quot;)
      .load(
          &quot;abfss://&lt;Container&gt;]@&lt;DIR&gt;.dfs.core.windows.net/export28.json&quot;, format=&quot;json&quot;
      )
  )
  </code></pre>
  <p>This is stacktrace I get:</p>
  <blockquote>
  <p>AnalysisException: Found duplicate column(s) in the data schema: <code>amendationcommentkey</code>, <code>amendationreasonkey</code>, <code>amendationregulatoryproofkey</code>
  Traceback (most recent call last):</p>
  <p>File &quot;/opt/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py&quot;, line 204, in load
  return self._df(self._jreader.load(path))</p>
  <p>File &quot;/home/trusted-service-user/cluster-env/env/lib/python3.8/site-packages/py4j/java_gateway.py&quot;, line 1304, in <strong>call</strong>
  return_value = get_return_value(</p>
  <p>File &quot;/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py&quot;, line 117, in deco
  raise converted from None</p>
  <p>pyspark.sql.utils.AnalysisException: Found duplicate column(s) in the data schema: <code>amendationcommentkey</code>, <code>amendationreasonkey</code>, <code>amendationregulatoryproofkey</code></p>
  </blockquote>