Question: I have a dataset "app\_metadata.csv" with three column: item\_id, category, description. item\_id is integer number, category is a string, and description is a string. I loaded the dataset with the following code

```
app_metadata_df = pd.read_csv(app_metadata_csv_path)

```

However there is broken data in the dataset, for example there exist a row at which item\_id is not a number but a text. I want to remove rows with invalid item\_id value and convert the item\_id column datatype to int. The following is what i have tried, first i call pd.to\_numeric with errors='coerce'

```
app_metadata_df.loc["item_id"] = pd.to_numeric(app_metadata_df["item_id"], errors='coerce')

```

Then i drop NA value

```
app_metadata_df.loc["item_id"] = app_metadata_df.loc["item_id"].dropna()

```

And finally call the astype(int) to convert the datatype to int:

```
app_metadata_df.loc["item_id"] = app_metadata_df["item_id"].astype(int)

```

However, it throws the following error

```
invalid literal for int() with base 10: 'So'

```

It looks like to\_numeric does not convert some invalid value to NAN. Why is this happening and how do i fix this ?

Answer: 
`\u2019` is not a UTF-8 character, but a Unicode escape code. It's valid JSON and when read back via `json.load` will become `’` (RIGHT SINGLE QUOTATION MARK).

If you want to write the actual character, use `ensure_ascii=False` to prevent escape codes from being written for non-ASCII characters:

```py
with open('output.json','w',encoding='utf8') as f:
    json.dump(data, f, ensure_ascii=False, indent=2)

```
You didn'T paste your code, so I don't kwon how you converted XHTML to JSON. I assume that you ended with hex value characters in Python objects. This `\u2019` is a single character with a 16-bit hex value. The JSON module can handle this by default. For example, the `json.loads` method can fix that:

```
x = '''{
  "p": {
    "@class": "para-p",
    "#text": "I\\u2019m not on Earth."
  }
}'''

print(x)
x_json=json.loads(x)
print(x_json)

```

Output shows:

```
{
  "p": {
    "@class": "para-p",
    "#text": "I\u2019m not on Earth."
  }
}

{'p': {'@class': 'para-p', '#text': 'I’m not on Earth.'}}

```