A New Semantic Similarity Based Measure For Assessing Research Publication’s Contribution

The use of the current research publications performance metrics (citations, altmetrics, webometris, etc) is, in our opinion, based on a false premise that the impact (or even quality) of a research paper can be assessed purely based on external data without considering the manuscript of the publication itself. Such assumption resembles the idea of judging a lawsuit without the suspect having the oppor- tunity of being in court and is consequently flawed in the same way.

However, citations are one of many attributes surrounding a publication and by themselves provide an insufficient evidence of impact, quality and research contribution. This is due to a wide range of characteristics they exhibit, including the variations in sentiment (positive, negative), the semantics of the citation (comparison, factual information, definition, etc), the context of the citation (hypothesis, analysis, result, etc) and the motives for citing [Nicolaisen,2007], the popularity of topics and the size of research communities [Brumback,2009; Seglen,1997], the time delay for citations to show up [Priem and Hemminger,2010], the skew-ness of their distribution [Seglen,1992], the difference in the types of research papers (theoretical paper, experimental paper, case, survey) [Seglen,1997] and finally the ability to game/manipulate citations [Arnold and Fowler,2010; Edit- ors,2006].

Ever since the idea of using citations for research evaluation was introduced [Garfield,1955], citation analysis has received a lot of attention and many theories and measures based on citations have been produced.

important feature of this idea is that our method does not require as long delay for assessment as the widely used cita- tion counts (typically decades) and can be therefore applied also to fairly young researchers.

Our hypothesis states that the added value of publication p can be estimated based on the semantic distance from the publications cited by p to the publications citing p.

The underlying idea is that, for example, in the case of a survey paper, it is natural that publications within the set A and also within the set B will be spread quite far from each other. However, this is not a sign of the paper’s contribution, but rather a natural feature of a survey paper. On the other hand, we believe that if a paper uses ideas from a narrow field, but has an impact on a very large field, it is a sign of the paper’s contribution.

In this paper, we present an approach for assessing the impact of a paper based on its full-text (Section 2). In this context, we use the term impact to refer to the research contribution to the discipline, which we believe is independent of the number of interactions in a scholarly communication network, but depends primarily on the content of the manuscript itself.

A publication has a high contribution if it creates a “long bridge” between more distant areas of science.

The sum in the equation is used to calculate the total distance between all combinations of publications in the sets A and B. It is expected that the distance is estimated using semantic similarity measures on the full-text of the publications, such as with cosine similarity on tf-idf document vectors.

Overall, we believe this situation demonstrates the need for supporting Open Access to research publications not only for humans to read, but also for machines to access.

a paper with high impact does not need to be extensively cited, however it needs to inspire a change in its domain or even define a new domain. This can be manifested by the changes in the vocabulary which are the result of a specific publication. Consequently, a very active scholarly debate about a survey paper in a specific subject generating many citations will have a lower impact than a paper developing a new strand of research.

Furthermore, we have demonstrated the importance of developing datasets on which this class of measures can be tested and explained the challenges in developing them. The primary issue is the citation data sparsity problem, which is a natural consequence of publications referencing work from different disciplines and across databases.