Getting bad results when fine-tuning GLiNER #294
Description
Hi everyone,
I am trying to fine-tune GLiNER but the results are consistently poor.
At first, I used an artificially generated dataset, but after fine-tuning, the model completely lost its ability to recognize and correctly label entities.
To verify whether the problem was with my data, I ran the official examples/fine-tuning.ipynb notebook with the sample dataset provided by the authors. However, even with that dataset, the model produces very bad results. (It is unclear to me how it previously gave good results, since now it consistently fails, even though I am using the same code and the same dataset.)
I also read in an another issue that the model may suffer from catastrophic forgetting, and that merging the Pile-NER dataset with the target dataset during fine-tuning could help. I tried that approach with various dataset sizes (small, medium, large), various ratios between my dataset and the Pile-NER one (1:2, 1:5) and various hyperparameters, but the results did not improve.
After fine-tuning, the model tends to over-predict by labeling every token, or sometimes even entire phrases, as entities. In many cases, it assigns the same label to all of them, and even when the labels vary, they are consistently incorrect. Here is a sample output:
Whiplash => treatment
, => treatment
a => treatment
soft => treatment
tissue injury to the neck, is also called neck sprain => treatment
or => treatment
strain => treatment
. => treatment
Treatment depends on the cause => treatment
, => treatment
but => treatment
may include => treatment
applying => treatment
ice => treatment
, => treatment
taking => treatment
pain relievers, getting physical therapy or wearing => treatment
a => treatment
cervical collar => treatment
. => treatment
You => treatment
rarely => treatment
need => treatment
surgery => treatment
. => treatment
Has anyone else experienced this issue?
Is there a recommended approach to improve this?