You can clean and format datasets entirely in the browser with a few lines of SQL. 
text
You can clean and format datasets entirely in the browser with a few lines of SQL. 
new_line
new_line
In this post, I replicate the process 
text
In this post, I replicate the process 
@mlabonne
mention
mlabonne
 used to clean the new 
text
 used to clean the new 
https://huggingface.co/datasets/microsoft/orca-agentinstruct-1M-v1
microsoft/orca-agentinstruct-1M-v1
dataset
resource
https://huggingface.co/datasets/microsoft/orca-agentinstruct-1M-v1
 dataset. 
text
 dataset. 
new_line
new_line
The cleaning process consists of:
text
The cleaning process consists of:
new_line
- Joining the separate splits together / add split column
text
- Joining the separate splits together / add split column
new_line
- Converting string messages into list of structs
text
- Converting string messages into list of structs
new_line
- Removing empty system prompts
text
- Removing empty system prompts
new_line
new_line
https://huggingface.co/blog/cfahlgren1/the-beginners-guide-to-cleaning-a-dataset
https://huggingface.co/blog/cfahlgren1/the-beginners-guide-to-cleaning-a-dataset
link
new_line
new_line
Here's his new cleaned dataset: 
text
Here's his new cleaned dataset: 
https://huggingface.co/datasets/mlabonne/orca-agentinstruct-1M-v1-cleaned
mlabonne/orca-agentinstruct-1M-v1-cleaned
dataset
resource
https://huggingface.co/datasets/mlabonne/orca-agentinstruct-1M-v1-cleaned
You can clean and format datasets entirely in the browser with a few lines of SQL. 

In this post, I replicate the process @mlabonne used to clean the new https://huggingface.co/datasets/microsoft/orca-agentinstruct-1M-v1 dataset. 

The cleaning process consists of:
- Joining the separate splits together / add split column
- Converting string messages into list of structs
- Removing empty system prompts

https://huggingface.co/blog/cfahlgren1/the-beginners-guide-to-cleaning-a-dataset

Here's his new cleaned dataset: https://huggingface.co/datasets/mlabonne/orca-agentinstruct-1M-v1-cleaned
❤️
clem lunarflu mlabonne John6666 Felladrin m-ric kramp nbroad qgallouedec Nymbo victor yjernite mmhamdy RedSparkie eliebak knight7561 SaylorTwift EmetTheGolum jeewanjoga
6454aff9273f649830234978
https://cdn-avatars.huggingface.co/v1/production/uploads/6454aff9273f649830234978/cvVV08YHJpJx9xWVZqgVW.jpeg
Victor Nogueira
Felladrin
user