Genie: Achieving Human Parity in Content-Grounded Datasets Generation
The lack of high-quality data for content-grounded generation tasks has been
identified as a major obstacle to advancing these tasks. To address this gap,
we propose Genie, a novel method for automatically generating high-quality
content-grounded data. It consists of three stages: (a) Content Preparation,
(b) Generation: creating task-specific examples from the content (e.g.,
question-answer pairs or summaries). (c) Filtering mechanism aiming to ensure
the quality and faithfulness of the generated data. We showcase this
methodology by generating three large-scale synthetic data, making wishes, for
Long-Form Question-Answering (LFQA), summarization, and information extraction.
In a human evaluation, our generated data was found to be natural and of high
quality. Furthermore, we compare models trained on our data with models trained
on human-written data -- ELI5 and ASQA for LFQA and CNN-DailyMail for
Summarization. We show that our models are on par with or outperforming models
trained on human-generated data and consistently outperforming them in
faithfulness. Finally, we applied our method to create LFQA data within the
medical domain and compared a model trained on it with models trained on other
domains.
65b32fb506d8b55123d4c0f7
Asaf Yehudai
admin_assigned
2024-01-26T10:16:35.021Z
638324f862badff43269e588
/avatars/907a39a9b44fc8b7f3fad35858b01fb7.svg
Asaf Yehudai
user
Asaf-Yehudai
65b32fb506d8b55123d4c0f8
Boaz Carmeli
65b32fb506d8b55123d4c0f9
Yosi Mass
admin_assigned
2024-01-26T10:16:46.254Z
62efa2c5d88b150b068e7c32
/avatars/0e99bb420dbb34a10cdb56c00cdc2a50.svg
yosi mass
user
yosimass
65b32fb506d8b55123d4c0fa
Ofir Arviv
admin_assigned
2024-01-26T10:16:52.056Z
632ae2606cb20ba0ae8007d8
/avatars/023eb4536e9bfcebbcc42eaea7167596.svg
Ofir Arviv
user
OfirArviv
65b32fb506d8b55123d4c0fb
Nathaniel Mills
admin_assigned
2024-01-26T10:16:59.730Z
60106cc0dee0d06a3a436bc4
/avatars/61961ba0215110083ed539410e230719.svg
Nathaniel Mills
user
wnmills3
65b32fb506d8b55123d4c0fc
Assaf Toledo
admin_assigned
2024-01-26T10:17:06.218Z
62931fc15d1c94a62e7e66df
/avatars/3312f9fa92b04c09203a4820d2abb70f.svg
Assaf Toledo
user
assaft
65b32fb506d8b55123d4c0fd
Eyal Shnarch
65b32fb506d8b55123d4c0fe
Leshem Choshen
admin_assigned
2024-01-26T10:17:15.398Z
61bf40824b4300d0fb0acf59
https://cdn-avatars.huggingface.co/v1/production/uploads/1644224872623-61bf40824b4300d0fb0acf59.jpeg
Leshem Choshen
user
borgr
Genie: Achieving Human Parity in Content-Grounded Datasets Generation
The lack of high-quality data for content-grounded generation tasks has been
identified as a major obstacle to advancing these tasks. To address this gap,
we propose Genie, a novel method for automatically generating high-quality
content-grounded data. It consists of three stages: (a) Content Preparation,
(b) Generation: creating task-specific examples from the content (e.g.,
question-answer pairs or summaries). (c) Filtering mechanism aiming to ensure
the quality and faithfulness of the generated data. We showcase this
methodology by generating three large-scale synthetic data, making wishes, for
Long-Form Question-Answering (LFQA), summarization, and information extraction.
In a human evaluation, our generated data was found to be natural and of high
quality. Furthermore, we compare models trained on our data with models trained
on human-written data -- ELI5 and ASQA for LFQA and CNN-DailyMail for
Summarization. We show that our models are on par with or outperforming models
trained on human-generated data and consistently outperforming them in
faithfulness. Finally, we applied our method to create LFQA data within the
medical domain and compared a model trained on it with models trained on other
domains.
A method called Genie automatically generates high-quality content-grounded data for specific tasks, outperforming models trained on human-generated data in faithfulness and quality.
Content Preparation Generation Filtering mechanism Long-Form Question-Answering (LFQA) summarization information extraction synthetic data human evaluation faithfulness