Deepak Sahu commited on
Commit
bd396f2
·
1 Parent(s): 0720e54
Files changed (4) hide show
  1. .resources/eval7.png +3 -0
  2. .resources/preview.png +2 -2
  3. README.md +4 -4
  4. app.py +1 -1
.resources/eval7.png ADDED

Git LFS Details

  • SHA256: 9b6fe8f989149bbb095ed133bf619670616842ce39bb5362327bd6a0334d468b
  • Pointer size: 130 Bytes
  • Size of remote file: 21.4 kB
.resources/preview.png CHANGED

Git LFS Details

  • SHA256: a3d938839b6f20377cb870e9ed9373a59e1069d349b0e67558008bea7f61d94f
  • Pointer size: 131 Bytes
  • Size of remote file: 125 kB

Git LFS Details

  • SHA256: deb5bde342ac1dbdc52563bcb159dc17fd6fe5c3e296e2e182d8cf52428861d4
  • Pointer size: 131 Bytes
  • Size of remote file: 104 kB
README.md CHANGED
@@ -236,14 +236,12 @@ The generation is handled by functions in script `z_hypothetical_summary.py`. Un
236
 
237
  ![image](.resources/eval5.png)
238
 
239
- ### Evaluation Metric
240
 
241
  So for given input title we can get rank (by desc order cosine similarity) of the store title. To evaluate we the entire approach we are going to use a modified version **Mean Reciprocal Rank (MRR)**.
242
 
243
  ![image](.resources/eval6.png)
244
 
245
-
246
-
247
  Test Plan:
248
  - Take random 30 samples and compute the mean of their reciprocal ranks.
249
  - If we want that our known book titles be in top 5 results then MRR >= 1/5 = 0.2
@@ -254,12 +252,14 @@ Test Plan:
254
  python z_evaluate.py
255
  ```
256
 
257
- ![image](https://github.com/user-attachments/assets/d2c77d47-9244-474a-a850-d31fb914c9ca)
258
 
259
  The values of TOP_P and TOP_K (i.e. token sampling for our generator model) are sent as `CONST` in the `z_evaluate.py`; The current set of values are borrowed from the work: https://www.kaggle.com/code/tuckerarrants/text-generation-with-huggingface-gpt2#Top-K-and-Top-P-Sampling
260
 
261
  MRR = 0.311 implies that there's a good change that the target book will be in rank (1/.311) ~ 3 (third rank) **i.e. within top 5 recommendations**
262
 
 
 
263
  ## Inference
264
 
265
  `app.py` is written so that it can best work with gradio interface in the HuggingFace, althought you can try it out locally as well :)
 
236
 
237
  ![image](.resources/eval5.png)
238
 
239
+ ### Evaluation Metric & Result
240
 
241
  So for given input title we can get rank (by desc order cosine similarity) of the store title. To evaluate we the entire approach we are going to use a modified version **Mean Reciprocal Rank (MRR)**.
242
 
243
  ![image](.resources/eval6.png)
244
 
 
 
245
  Test Plan:
246
  - Take random 30 samples and compute the mean of their reciprocal ranks.
247
  - If we want that our known book titles be in top 5 results then MRR >= 1/5 = 0.2
 
252
  python z_evaluate.py
253
  ```
254
 
255
+ ![image](.resources/eval7.png)
256
 
257
  The values of TOP_P and TOP_K (i.e. token sampling for our generator model) are sent as `CONST` in the `z_evaluate.py`; The current set of values are borrowed from the work: https://www.kaggle.com/code/tuckerarrants/text-generation-with-huggingface-gpt2#Top-K-and-Top-P-Sampling
258
 
259
  MRR = 0.311 implies that there's a good change that the target book will be in rank (1/.311) ~ 3 (third rank) **i.e. within top 5 recommendations**
260
 
261
+ > TODO: A sampling study can be done to better make this conclusion.
262
+
263
  ## Inference
264
 
265
  `app.py` is written so that it can best work with gradio interface in the HuggingFace, althought you can try it out locally as well :)
app.py CHANGED
@@ -93,4 +93,4 @@ demo = gr.Interface(
93
  description=GRADIO_DESCRIPTION
94
  )
95
 
96
- demo.launch()
 
93
  description=GRADIO_DESCRIPTION
94
  )
95
 
96
+ demo.launch(share=True)