Against the Impact Factor

San Francisco Declaration on Research Assessment

There is a pressing need to improve the ways in which the output of scientific research is evaluated by funding agencies, academic institutions, and other parties.

To address this issue, a group of editors and publishers of scholarly journals met during the Annual Meeting of The American Society for Cell Biology (ASCB) in San Francisco, CA, on December 16, 2012. The group developed a set of recommendations, referred to as the San Francisco Declaration on Research Assessment. We invite interested parties across all scientific disciplines to indicate their support by adding their names to this Declaration.

The outputs from scientific research are many and varied, including: research articles reporting new knowledge, data, reagents, and software; intellectual property; and highly trained young scientists. Funding agencies, institutions that employ scientists, and scientists themselves, all have a desire, and need, to assess the quality and impact of scientific outputs. It is thus imperative that scientific output is measured accurately and evaluated wisely.

The Journal Impact Factor is frequently used as the primary parameter with which to compare the scientific output of individuals and institutions. The Journal Impact Factor, as calculated by Thomson Reuters, was originally created as a tool to help librarians identify journals to purchase, not as a measure of the scientific quality of research in an article. With that in mind, it is critical to understand that the Journal Impact Factor has a number of well-documented deficiencies as a tool for research assessment. These limitations include: A) citation distributions within journals are highly skewed; B) the properties of the Journal Impact Factor are field-specific: it is a composite of multiple, highly diverse article types, including primary research papers and reviews; C) Journal Impact Factors can be manipulated (or "gamed") by editorial policy; and D) data used to calculate the Journal Impact Factors are neither transparent nor openly available to the public.

Below we make a number of recommendations for improving the way in which the quality of research output is evaluated. Outputs other than research articles will grow in importance in assessing research effectiveness in the future, but the peer-reviewed research paper will remain a central research output that informs research assessment. Our recommendations therefore focus primarily on practices relating to research articles published in peer-reviewed journals but can and should be extended by recognizing additional products, such as datasets, as important research outputs. These recommendations are aimed at funding agencies, academic institutions, journals, organizations that supply metrics, and individual researchers.

A number of themes run through these recommendations:

• the need to eliminate the use of journal-based metrics, such as Journal Impact Factors, in funding, appointment, and promotion considerations;
• the need to assess research on its own merits rather than on the basis of the journal in which the research is published; and
• the need to capitalize on the opportunities provided by online publication (such as relaxing unnecessary limits on the number of words, figures, and references in articles, and exploring new indicators of significance and impact).

We recognize that many funding agencies, institutions, publishers, and researchers are already encouraging improved practices in research assessment. Such steps are beginning to increase the momentum toward more sophisticated and meaningful approaches to research evaluation that can now be built upon and adopted by all of the key constituencies involved.

The signatories of the San Francisco Declaration on Research Assessment support the adoption of the following practices in research assessment.

General Recommendation

1. Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist's contributions, or in hiring, promotion, or funding decisions.

For funding agencies

2. Be explicit about the criteria used in evaluating the scientific productivity of grant applicants and clearly highlight, especially for early-stage investigators, that the scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published.

3. For the purposes of research assessment, consider the value and impact of all research outputs (including datasets and software) in addition to research publications, and consider a broad range of impact measures including qualitative indicators of research impact, such as influence on policy and practice.

For institutions

4. Be explicit about the criteria used to reach hiring, tenure, and promotion decisions, clearly highlighting, especially for early-stage investigators, that the scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published.

5. For the purposes of research assessment, consider the value and impact of all research outputs (including datasets and software) in addition to research publications, and consider a broad range of impact measures including qualitative indicators of research impact, such as influence on policy and practice.

For publishers

6. Greatly reduce emphasis on the journal impact factor as a promotional tool, ideally by ceasing to promote the impact factor or by presenting the metric in the context of a variety of journal-based metrics (e.g., 5-year impact factor, EigenFactor, SCImago, h-index, editorial and publication times, etc.) that provide a richer view of journal performance.

7. Make available a range of article-level metrics to encourage a shift toward assessment based on the scientific content of an article rather than publication metrics of the journal in which it was published.

8. Encourage responsible authorship practices and the provision of information about the specific contributions of each author.

9. Whether a journal is open-access or subscription-based, remove all reuse limitations on reference lists in research articles and make them available under the Creative Commons Public Domain Dedication.

10. Remove or reduce the constraints on the number of references in research articles, and, where appropriate, mandate the citation of primary literature in favor of reviews in order to give credit to the group(s) who first reported a finding.

For organizations that supply metrics

11. Be open and transparent by providing data and methods used to calculate all metrics.

12. Provide the data under a licence that allows unrestricted reuse, and provide computational access to data, where possible.

13. Be clear that inappropriate manipulation of metrics will not be tolerated; be explicit about what constitutes inappropriate manipulation and what measures will be taken to combat this.

14. Account for the variation in article types (e.g., reviews versus research articles), and in different subject areas when metrics are used, aggregated, or compared.

For researchers

15. When involved in committees making decisions about funding, hiring, tenure, or promotion, make assessments based on scientific content rather than publication metrics.

16. Wherever appropriate, cite primary literature in which observations are first reported rather than reviews in order to give credit where credit is due.

17. Use a range of article metrics and indicators on personal/supporting statements, as evidence of the impact of individual published articles and other research outputs.

18. Challenge research assessment practices that rely inappropriately on Journal Impact Factors and promote and teach best practice that focuses on the value and influence of specific research outputs.

Experiments 2013 look promising …

Four fundamental experiments of four months of 2013:

Matter, antimatter, we all fall down—right?

Scientists perform the first direct investigation into how antimatter interacts with gravity #

Dark Matter Signals Recorded in Minnesota Mine

Detectors at the Cryogenic Dark Matter Search have recorded three events that may represent collisions from weakly interacting massive particles #

OPERA snags third tau neutrino

For the third time since the OPERA detector began receiving beam in 2006, the experiment has caught a muon neutrino oscillating into a tau neutrino #

New results indicate that particle discovered at CERN is a Higgs boson #

What is Gauge Gravitation Theory about?

Classical field theory admits a comprehensive mathematical formulation in the geometric terms of smooth fibre bundles. For instance, Yang – Mills gauge theory is theory of principal connections on principal bundles.

Gauge gravitation theory as particular classical field theory also is formulated in the terms of fibre bundles.

Studying gauge gravitation theory, one believes reasonable to require that it incorporates Einstein's General Relativity and, therefore, it should be based on Relativity and Equivalence Principles reformulated in the fibre bundle terms.

In these terms, Relativity Principle states that gauge symmetries of classical gravitation theory are general covariant transformations. It should be emphasized that these gauge symmetries differ from gauge symmetries of the above mentioned Yang – Mills gauge theory which constitute a gauge group of vertical automorphisms of a principal bundles. Fibre bundles possessing general covariant transformations constitute the category of so called natural bundles.

Let Y->X be a smooth fibre bundle. Any automorphism of Y, by definition, is projected onto a diffeomorphism of its base X. The converse is not true. A fibre bundle Y->X is called the natural bundle if there exists a monomorphism of the group of diffeomorphisms of X to the group of bundle automorphisms of Y->X, called general covariant transformations of Y. 

The tangent bundle TX of X exemplifies a natural bundle. Any diffeomorphism f of X gives rise to the tangent automorphisms Tf of TX which is a general covariant transformation of TX.The associated principal bundle is a fibre bundle LX of frames in the tangent spaces to X also is a natural bundle. Moreover, all fibre bundles associated with LX are natural bundles. Principal connections on LX yield linear connections on the tangent bundle TX and other associated bundles. They are called the world connections.

Following Relativity Principle, one thus should develop gravitation theory as gauge theory of principal connections on a principal frame bundle LX over a four-dimensional manifold X, called the world manifold. A key point however is that this gauge theory also is characterized by spontaneous symmetry breaking in accordance with geometric Equivalence Principle.

Though spontaneous symmetry breaking is quantum effect, spontaneous symmetry breaking in classical gauge theory on a principal bundle P->X with a structure Lie group G is characterized as a reduction of this structure group to its closed Lie subgroup H. By virtue of the well-known theorem, such a reduction takes place if and only if there exists a global sections h of the quotient bundle P/H->X which are treated as a classical Higgs field.

There are different formulations of Equivalence Principle in gravitation theory. In particular, one separates weakest, weak, middle-strong and strong Equivalence Principles. All of them are based on the empirical equality of inertial mass, gravitational active and passive charges. The weakest Equivalence Principle is restricted to the motion law of a probe point mass in a uniform gravitational field. Its localization is the weak Equivalence Principle that states the existence of a desired local inertial frame at a given world point. This is the case of equations depending on a gravitational field and its first order derivatives, e.g., the equations of mechanics of probe point masses, and the equations of electromagnetic and Dirac fermion fields. The middle-strong Equivalence Principle is concerned with any matter, except a gravitational field, while the strong one is applied to all physical laws.

The above mentioned variants of Equivalence Principle aim to guarantee the transition of General Relativity to Special Relativity in a certain reference frame. However, only the particular weakest and weak Equivalence Principles are true. To overcome this difficulty, Equivalence Principle can be formulated in geometric terms as follows. In the spirit of Felix Klein's Erlanger program, Special Relativity can be characterized as the Klein geometry of Lorentz group invariants. Then geometric Equivalence Principle is formulated to require the existence of Lorentz invariants on a world manifold X. This requirement holds if the tangent bundle of X admits an atlas with Lorentz transition functions, i.e., a structure group of the associated frame bundle LX of linear tangent frames in is reduced to the Lorentz group SO(1,3). By virtue of the above mentioned theorem, this reduction takes place if and only if the quotient bundle LX/SO(1,3) possesses a global section, which is a pseudo-Riemannian metric on X.

Thus geometric Equivalence Principle provides the necessary and sufficient conditions of the existence of a pseudo-Riemannian metric, i.e., a gravitational field on a world manifold. Based on geometric Equivalence Principle, gravitation theory is formulated as gauge theory where a gravitational field is described as a classical Higgs field responsible for spontaneous breakdown of world gauge symmetries which are general covariant transformations.

The character of gravity as a Higgs field responsible for spontaneous breaking of general covariant transformations is displayed as follows. Given different gravitational fields, the representations of holonomic coframes dx by Dirac matrices acting on Dirac spinor fields are nonequivalent. Consequently, Dirac operators in the presence of different gravitational fields fails to be equivalent, too. 

It follows that, since the Dirac operators in the presence of different gravitational fields are nonequivalent, Dirac spinor fields fail to be considered, e.g., in the case of a superposition of different gravitational fields. Therefore, quantization of a metric gravitational field fails to satisfy the superposition principle, and one can suppose that a metric gravitational field as a Higgs field is non-quantized in principle.

References:

WikipediA: Gauge gravitation theory

G.Sardanashvily, Classical gauge gravitation theory, Int. J. Geom. Methods Mod. Phys.8 (2011) 1869-1895.

Facebook: Gravitation Gauge Theory

I. How we developed gauge gravitation theory (from my Scientific Biography)

II. How we developed gauge gravitation theory (from my Scientific Biography)