SERIOUS FLAWS IN THE SCIENCE ASSESSMENT IN PORTUGAL: AN UPDATE

Just before Christmas the Portuguese Foundation for Science and Technology - FCT - announced the final results of the assessment of all Portuguese science units, outsourced to the European Science Foundation - ESF. No representatives of the ESF were present at the final press conference, in contrast with the announcement of the results of the first stage, last June, in which almost half of the research centers were either excluded from obtaining any funding or given very scarce means.

The final results, after the two evaluation stages, for the 322 units are:

Classified as:	Number of units	Annual funding for all units (M€)
Poor	32	0
Fair	33	0
Good	90	1
Very Good	104	23
Excellent	52	34
Exceptional	11	13

Units classified as “Poor” or “Fair” will have no funding. Units evaluated with “Good” will have very limited funding (1M€ for 28 units, which means an average funding of about 11 000€/unit /year) that will not allow them to be continue their research activities. Some of these units classified as Good are eligible for an extra 6 M€ “strategic restructuring” fund, which is offensive, because many of them do not need to be “restructured”; they simply need to be properly evaluated.

In practice the 155 units classified with less than “Very Good” are condemned. These are about 50% of all research units in Portugal. On the other hand, ample funding will be awarded to a small number of units. And this is where things are really strange: the funding does not have a direct relationship with the classification. Here are the top 5 research centers with the highest funding and respective classifications:

	Classification	Annual funding (€)
Institute for Research and Innovation in Health Sciences	Exceptional	6 195 503
Instituto de Medicina Molecular	Excellent	2 910 485
CNC.IBILI, Universidade de Coimbra (UC)	Excellent	2 623 710
INESC Technology and Science	Excellent	2 574 135
CICECO-Aveiro Institute of Materials	Excellent	1 891 383

The evaluation process that yielded these results is, in the opinion of many researchers, seriously flawed. And a flawed process can only give flawed results. More details on this may be found here: http://dererummundi.blogspot.pt/2014/12/the-flawed-evaluation-of-portuguese.html

Many researchers understood immediately after the results announcement that there was no clear correlation between the grade and the funding, not to mention the lack of correlation between grade and scientific output, measured by the number and impact of scientific publications. The ESF remained silent (it seems that they do not want to be involved in the funding distribution, claiming that this is entirely an FCT question). Given the many doubts and questions that followed,  FCT made public a posteriori a set of “funding rules” trying to explain what cannot be rationally explained: they are unable to provide a simple explicative algorithm. Instead they claim that funds have been distributed primarily not according to the assessment results but according to the funding request, with some "modulation" (read subtractions) based on grading. There are indeed very strange things: the unit with more funding per capita is a a small private research group doing psychology, which was virtually unknown.

In fact, there are many protests pending both from the first and from the second stage of the science evaluation process in Portugal. . The Portuguese reference daily newspaper "Publico" devoted a whole page to the more recent complains:

http://www.publico.pt/ciencia/noticia/ninguem-entende-os-criterios-de-atribuicao-de-financiamento-da-fct-1685285

The title may be translated as: "Nobody understands the FCT funding criteria".

The President of a top Portuguese Engineering School, Instituto Superior Técnico - IST, published an article in another national newspaper (“Expresso”) saying that, from the money given to the centers in the top rank, while only 5% had gone to engineering, more than 50% had gone to biomedicine. There is therefore a strong bias towards biomedicine, the area of both FCT's President and one of its Vice-Presidents, and of the Secretary of State for Science and Technology. These last two are associated with two of the five units mentioned above.

There are also  recent news about the first stage appeals. The Rectors held a meeting with the Prime Minister (in the presence of the Minister for Science and Technology). The Prime Minister was sensitive to the objections raised and said that a different multidisciplinary panel should be formed to consider the appeals of the first stage. Furthermore, this should be independent of ESF. This very sound decision is, in itself, is a criticism of the whole process and of the conduct of the directive council of FCT. But, in fact, a new decision can take time and all of the 2015 science budget seems to have been already allocated. There are elections next October and probably the new government will inherit the problem.

The deadline for second stage appeals was at the end of January and it seems that many units did complain. However, FCT refuses to make this number public. One suspects that a large portion, if not the majority, did hand in a complaint. Although in the second stage almost all units were rewarded, some did not agree with the grade while others did not agree with the funding (there is of course an overlap: some disagreed with both).

There are no known deadlines for the response to the appeals from either the first or the second stages (and it is not known whether ESF is handling the second stage appeals or not).

Let us present some political, administrative and scientific flaws. The classifications given to the research units are the result of an unusual quota system to exclude some scientists from the research system (this is nowadays seen as a strong political error). There was a contractually defined restriction of the number of units allowed to proceed to the second stage of evaluation (and have a possibility to be classified above “Good”). By contract between FCT and ESF only 50% of the units would be allowed to proceed to the second stage. Strangely enough ESF complied with this request. In the second stage there were quotas too. In this case, the grades could be "exceptional", "excellent" or "very good", and certain percentages of each rank was established a priori.

Between the two stages the Conference of the Portuguese Rectors - CRUP (encompassing the Presidents of all Universities) wrote a letter to the minister of Education and Science, whose contents may be read here:

http://dererummundi.blogspot.pt/2014/11/the-portuguese-rectors-on-european.html

The Portuguese press gave echo to this strong position and, given the lack of a public response by the Minister, the national public opinion made its own judgment.

Based on the many formal irregularities, non-observance of the rules and other flaws, a case has been submitted to court by a teachers' and researchers' union and some units sentenced to death in the first stage have started to do the same.

The ESF-FCT assessment was an opaque process. Very few specialists were involved in the final decisions of the first and second stages. No site visits were done during the first stage, contrary to legal dispositions. Site visits were done with very few panel members in the second phase. Not only do these facts denote a total lack of respect for proper procedures and even for the law, but they also clearly undermined any possibility of a fair and minimally rational evaluation.

hase, in many cases from scientific areas other than those of the visited units.

The Spanish astrophysicist Amaya-Morin was completely right when she wrote in "Nature" that the process was flawed: it falls short of being clean. See about the ESF threat to Amaya-Morin:

http://retractionwatch.com/2014/10/12/european-science-foundation-demands-retraction-of-criticism-in-nature-threatens-legal-action/#more-23216

“Nature" refused to continue this discussion on its pages. They refused to publish a letter explaining why science was not being correctly handled in Portugal. One of the reasons why Europe and the world should care is that two of the Portuguese connected to the present government have relevant jobs in Europe: Miguel Seabra, the FCT President, is President of Science Europe, which took over some of the ESF missions, and Carlos Moedas is the Science, Research and Innovation Commissioner of the European Union, responsible for the whole Horizon 2020 program.

We should be absolutely clear about this: what is happening is the equivalent of closing down half of the institutes and laboratories of a given country on the basis of a quick, irregular and unfair assessment process. In other words, it is tantamount to scientific genocide. It was not only a lack of a decent peer review: the whole process was conducted in an amateurish way, to say the least. And, since there are many appeals pending, still is

Carlos Fiolhais (University of Coimbra) on behalf of a group of reserachers