The flawed evaluation of Portuguese research units conducted by the ESF and FCT

Portuguese research units are funded by the Fundação para a Ciência e Tecnologia (FCT), the only national public funding agency for science. FCT has been carrying out periodic evaluations for more than 20 years, conducted by scientists from foreign institutions with its outcome determining the levels of funding.

The latest (and still ongoing) evaluation was outsourced to the European Science Foundation (ESF).   The practical outcome of this evaluation, if carried out to full completion, will be a huge cut in the number of viable research units and the concentration of most of the financial resources in a very small number of institutions. At first sight this might seem a legitimate option, but it is not.

First of all, this “shrink and concentrate” policy was not announced by the Portuguese government, and as a consequence was not submitted to a public discussion with the relevant institutions such as universities, who host the research units and pay the salaries for the majority of the senior researchers in the system.  Instead, it was disguised as an evaluation based on merit only, passing the full responsibility of its outcome to ESF. However, ESF conducted what was, as shown below, a severely flawed evaluation process, in which the absolute classifications of the research units had to fit into predefined quotas.

Contrary to all previous experience and even the law, the evaluation was conducted in two stages, the first based on documents only (with no visits to laboratories) and the second based on visits and interviews with the unit director and its researchers. Only units that made it to the second stage will have access to the major part of the funding to be awarded. The first stage suffered, for instance, from the lack of proper coverage of the different areas by the panels who made the decisions, the change of rules when the process was already underway, and the usage of incorrect bibliometric data. These serious flaws  alone, denounced by a large number of scientists and associations in Portugal ( http://goo.gl/KHYDwZ ) and many foreign scientists, of which Amayo Moro-Martin is but one example (article in Nature: http://goo.gl/U8OU9a), would be enough reason to suspend this evaluation. ESF announced a legal action against Moro-Martin, in view of her opinion piece in Nature, but after this became public ESF backed down.

But, even more significantly, the evaluation had a hidden quota from the very beginning. After strong legal pressure, FCT finally made public the contract signed with ESF from which we learnt that ESF had been instructed to a priori exclude 50% of the units from the second stage. Indeed, as may be read in the work plan for the panels: "Stage 1 evaluation will result in a short list of half of the research units that will be selected to proceed to stage 2." The evaluation had thus to abide by this hidden quota while it was being publicized as being driven by quality alone.

It should be stressed that this was not a relative evaluation to find the 50% best units, in some kind of “excellence” program,  but the financing of the basal science system and there is virtually no alternative funding from other Portuguese sources. Units which did not make it to the second stage will most likely not be able to continue. This alone would already be cause for concern, but it does become totally unacceptable when based on a flawed evaluation. Several complaints have been presented and, given the number and seriousness of formal and scientific irregularities, it may be expected that new independent panels will respond positively to the appeals.

These were the quotas for the first stage. In a recently disclosed document ( http://www.fct.pt/apoios/unidades/avaliacoes/2013/docs/FCT_Final_Meeting_guidelines_S2.pdf ,

FCT defined the quota for the classifications of the second stage. Only 10% of the units assessed in the second stage are allowed to have the highest classification (“exceptional”). And only up to 35% of the units assessed in the second stage are allowed to have the second higher classification (“excellent”). As in the first phase, very few evaluators visited the units and some of them were not specialists in the areas being examined.

We have to bear in mind that of a total of 322 units, only 178 have made it to the second stage and that those classified as “exceptional” will have 10 times more funding as the units classified as “good”. This means that a large part of the Portuguese public funds for research will be awarded to only 17-18 units, several of which will probably be private (private foundations will be subsidized with public money).

This evaluation has been highly contested by many scientists, organizations and national scientific societies (Physics, Chemistry, Philosophy, Mathematics, etc.). More significantly, the Council of Rectors of Portuguese Universities (CRUP) has completely demolished the process. On the 24th of October the 15 rectors of CRUP released a consensus position which was sent to the Minister of Education and Science, Nuno Crato To this day there has been no reaction from the Minister, denoting that the open letter was a serious political embarrassment for the government. A translation of the letter may be found here:

http://dererummundi.blogspot.pt/2014/11/the-portuguese-rectors-on-european.html

The minister has not replied to any of these issues so far.

In short, this evaluation is politically driven, meant do downsize science in Portugal and concentrate all the resources in a lucky few, selected by what many scientists deem to be a ridiculous evaluation process. Furthermore, it is illegal in various aspects, and indeed a legal action has already been filed by a researchers union (probably others will follow soon).

Despite all this, the Portuguese Secretary of State for Science and Technology, has stated recently that the evaluation "is running smoothly and according to plan.” Preliminary results of the second phase are about to be announced, but a bad plan and a flawed methodology can only produce bad results.

Carlos Fiolhais, Professor of  Physics, University of  Coimbra (Portugal)