Skip to Main Content

Evaluating scholarly publications

This guide introduces essential tools and indicators for evaluating research.

Indicators used to evaluate journals

Indikaattorit kuvassa: Julkaisufoorumi, Impact Factor, JOurnal Immediacy index, Eigenfactor, Article Influence, H-index, SCImago Journal Rank (SJR) & Source Normalized Impact per Paper (SNIP)

Indicators in detail

Publication Forum

Publication Forum (Julkaisufoorumi in Finnish, JUFO for short) is a classification of publication channels created by the Finnish scientific community to support the quality assessment of academic research.

  • Four-level classification: 1 = basic, 2 = leading, 3 = top, 0 = channels that do not yet fill the requirements of level 1
  • Ratings re-evaluated at four-year intervals.
  • Created to evaluate the research output not only quantitatively but also qualitatively.
  • Comprehensive because covers publication practices in various academic fields.
  • Used as a quality indicator within the university funding model of the Ministry of Education and Culture since 2015.

Publication Forum is meant for evaluating large publication volumes, it is not suited for evaluating the merits of an individual researcher!

Impact factor (IF)

Originally developed for comparing journals, impact factor is still the oldest and most used indicator for measuring the impact of journals. Clarivate Analytics has exclusive rights to the impact factor indicator, and IF-numbers are only available in the Journal Citation Reports (JCR) database which is updated yearly. JCR uses the data found in Web of Science.

Impact factor is calculated as follows: 
The number of times a journal’s articles have been cited over a period of two years is divided by the number of articles published in the journal during the same time period.

 

Restrictions and criticism:

  • Impact factors on different disciplines are not comparable!
  • Impact factor favours disciplines where the publication rate and reaction to new articles is fast. The impact factor value of a journal should always be compared to citation coefficients of journals of the same discipline.
  • Journals that publish many review articles usually have higher IF values. Broader disciplines are commonly more likely to have more articles with many citations than narrower disciplines. This is why journals in specialised disciplines have low IF values.
  • When calculating the IF, only the journal citations in Clarivate Analytis' JCR database are taken into account. Thus, citations of other journals and monographic publications are not included in the IF value but self-citations in journals are.
  • It is better to favour IF values of a longer time period than two years (e.g. a 5 year impact factor).

H-index

The Hirsch index, developed by Professor Jorge E. Hirsch in 2005, is used to evaluate journals.

The h-index is calculated as follows:
The journal has index h, if number h of the articles (Np) published in the journal have at least h citations each and rest of articles (Np - h) have no more than h citations each.

 

  • When evaluating journals, the h-index is calculated over a certain time period, for example, one year.
  • The advantage is that individual articles with many citations do not affect the h-index value.
  • H-indexes of different disciplines are not usually comparable.
  • Usually higher for journals that publish many of articles.
  • Can also be used to evaluate researcher's publication efficiency and the impact of publications.

Eigenfactor Score (EF)

  • Depicts the journal’s total citation impact: the impact value is divided by the number of citations the journal has received.
  • Takes into account the differences in citation practices in different disciplines and emphasises citations made by prestigious journals.
  • Citations are calculated over a five-year period, self-references within the journal are not included.
  • Can also be calculated for books, theses etc.
  • Available in the Journal Citation Reports (JCR) database which is based on the data in Web of Science or free online at Eigenfactor.org.
  • If a journal's Eigenfactor value is 0,95, it means that the journal receives 0,95 % of all the citations in JCR.

Article Influence Score (AI)

  • Measures the relative importance of a journal per article.
  • Calculated by diving the Eigenfactor Score by the number of articles published in the journal.
  • Comparable with the impact factor (IF).
  • If a journal's AI value is 2,51, the articles published in the journal are appr. 2,5 times more influential that the average article in the JCR database.

SCImago Journal Rank (SJR)

  • Measures journal's prestige by focusing on the discipline, quality and reputation of the journal.
  • Emphasises citations differently depending on the journal’s SJR value. If journals A and B are equally popular (for example both have 100 citations), journal A has a higher SJR value if its citations are from more prestigious journals compared to journal B.
  • Calculated over a three-year period.
  • Self-references within the journal lower the SJR value.
  • Calculated based on the data in Elsevier's database Scopus, available in Scopus or free online on website SCImago Journal & Country Rank.

Source Normalized Impact per Paper (SNIP)

  • Measures the journal’s citation impact within the citation context focusing on the citing publications.
  • Takes into account the characteristics of various disciplines and citation practices.
  • Calculated over a three-year period.
  • Calculated based on the data in Elsevier's database Scopus, available in Scopus or free online on website CWTS Journal Indicators.

 

Comparing SJR and SNIP:

  • SJR is suitable especially for disciplines where citing is done quickly and that have a distinct group of core journals.
  • SNIP, on the other hand, is better suited for evaluating journal's impact on heterogeneous disciplines where journals are not the main publication channel.
  • Highest SJR values are in bio and health sciences, highest SNIP values in technology and social sciences.
  • It should be taken into consideration that the SNIP values of small and multidisciplinary journals can vary significantly on different years.

Other tools for evaluating journals

Predatory journals

As open access publishing has become more common, the number of predatory journals has also increased. These journals take advantage of the pressure of publishing pace by promising faster peer reviews. They have high Author Processing Charges (APCs) but do not offer proper peer review and editorial support. The credibility of a research published in a predatory journal suffers because the crucial process of scientific quality control is absent. Instead of creating a new publication channel, scammers might also create an illegal copy of a journal already in a citation database or legally acquire it.

In Publication Forum's JUFO portal you can check a journal's reliability. OA journals can be evaluated in the Directory of Open Access Journal (DOAJ). University of Vaasa has access to the database Cabells Predatory Reports, which helps identify questionable scientific journals.

Source: Open Science

Accessibility statement — Guide is licensed under CC BY 4.0Creative commons.Attribution.