Error in evaluation script: def eval_triva, and reproducted value of HumanEval-infill is not matching with paper

[JiHoonLee9898]

Hello, I'm reproducing the paper results and found 2 weird things:

1. there is an error in eval script: I can't reproduce def eval_triva because the function is using ds = load_dataset("rajpurkar/squad", split='validation')
2. Reproduced HumanEval-infill value is not matching the reported value of paper (See the last column of the second screenshot, not matching with the last column of your table)

We’re able to reproduce the results well for Lambada and ROCStories. However, could you update the script just once more for the other evaluations? I’d like to exactly reproduce all the values reported in the paper.

Also, If possible, could you integrate GSM8K evaluation in the eval-diffullama.py? I’d like to be sure that I’m evaluating things in line with your intentions, and it seems the best way to ensure that is to just follow your code exactly as it is.

[summmeer]

Hi,

1. I think you should use "mandarjoshi/trivia_qa"? Squad is actually another dataset. I might plan to use it but forget to change it back.
2. It seems that your score is much higher than ours. I'm not really sure about this, but maybe you can try to remove if len(x0) > 1000: block? I filtered this for DiffuGPT which trained with 1024 context length but DiffuLLaMA uses 2048.
3. GSM evaluation is based on LLaMA-factory code, please check LLaMA-Factory/examples/inference/llama2_full_ddm-gsm-inf.yaml.