Pointwise V-usable information (PVI) is a recently proposed metric for measuring the hardness of individual instances. It is estimated by fine-tuning supervised models. (2/🧵) #EMNLP2023

In our paper we show that in-context PVI exhibits similar characteristics to the original PVI but is more time-efficient. The reason is that it requires only a few exemplars and does not need fine-tuning. (3/🧵) #EMNLP2023

Our findings show a lower prediction accuracy for low in-context PVI (see in the 🟦 box); and a higher average in-context PVI for correct predictions than incorrect (see in the 🟥 box). This matches what we see in the original PVI estimates. (4/🧵) #EMNLP2023

Major insight : in-context PVI estimates are more consistent across similar models (e.g., models that have similar architecture or similar training data. (5/🧵) #EMNLP2023

We also show that, comparable to the original PVI, the in-context PVI threshold at which instances start being predicted incorrectly is similar across datasets. However, in-context PVI estimates made by smaller models are much noisier than those made by larger models. (6/🧵)

Besides, in-context PVI estimates can be used to identify mislabeled instances. This is a very practical feature and demonstrates the reliability of the in-context PVI. (7/🧵) #EMNLP2023

Consider following our authors Sheng Lu ( @UKPLab ), @shan23chen ( @BrighamResearch ), Yingya Li ( @Bos_CHIP ), @dbittermanmd ( @aim_harvard / @BrighamWomens ), Guergana Savova ( @Bos_CHIP ) and @IGurevych ( @UKPLab ). See you in Singapore 🇸🇬! 🧵 (7/7) #EMNLP2023 📑