Statement of the CLA – Council of Associate Laboratories on the ongoing evaluation procedure of the FCT Research Units

We publish a quick English tranlation of the Statement of the Associate Laboratories (dated 07/25/2014), since the posts on the ERS/FCT evaluation are now being accessed by many foreign readers:

For more than two decades Portuguese research centers  have been evaluated by experts of foreign research centers, organized in specialized panels. This practice became widespread to all research centers for nearly 15 years. The names and resumes of appraisers were public and announced in advance  to allow  to fix bugs, add skills, avoid conflicts of interest or monolithic opinion. The scientific evaluation system  in Portugal was gradually improved and became one of the main factors of the external credibility of the Portuguese scientific development and the trust of scientists in Portugal in their institutions. Several leaders and various political majorities helped to consolidate it. The evaluation thus became the basis for the regular funding of scientific institutions, including research centers of universities.

The system of institutional support in force until allowed to balance a concentration of resources in strategic programs in the best institutions with a distribution of sufficient resources for the functioning of all institutions of recognized quality. Thus, so far, all scientific institutions classified as good, very good or excellent had a multiannual stable basis, indexed to their size (essentially the number of PhD researchers).

Against the opinion of the Universities and the CLA itself, the proven system was  replaced by a service contract  between the FCT and an external entity that only this week was finally known. The first consequences of changes to the evaluation system have sparked outrage and bewilderment in many Portuguese scientific institutions, and critical  positions of  CRUP (Rectors Council), the coordinators of Scientific Councils of the FCT and other entities. The main breaks introduced were:

a) Subverting the relationship between assessment and funding basic research centers. Under the new rules, an institution that has been judged only good will receive funding of residual base (which can reach a tenth of what it had before). That is, that institution is in practice extinguished. This methodology, kept as hidden as possible, ignored the widespread criticism from scientists and universities. Can a scientific system operate without a great number of good research centers which are not of exceptional quality? It can not, in any part of the world. This is the very definition of exceptionality. Furthermore: can a country allow, according to this logic, entire scientific areas disappear, as it now happening in Portugal, because their institutions are only good (but not exceptional)? Of course not. Thus, the political irresponsibility of this rupture, if materialized, will  belittle the country and deprive him of skills that laboriously managed to create.

b) Destroy a model of panels based on the expertise of its members. While until now the evaluation was done by international high level specialized panels (about 25, so that each panel would bring together experts able to comment in depth the scientific work of Portuguese institutions) now six (6) panels structured by large areas are asked to assess our specialized institutions. In many cases, the panel fails to have a single expert in the area of the institution under review. There are cases where the only expert with this curriculum would not even be hired by the institution that he will evaluate.

c) Undermining confidence in the assessment system in place so far. The evaluators, as well as studying the written documentation provided to them by the individual units, visited all institutions to review so that they could better know them and clarify the  information directly with the evaluated. It stopped being so: henceforth the evaluators become just to visit the centers which, on paper, have been assessed as better than "good",  even though some centers may come in the 2nd phase to be  rated with good or less. The prior evaluation, done only on paper, based on advices from anonymous experts and the final decision of a general panel, is eliminatory.

IN CONCLUSION:

 The CLA believes that the glaring anomalies in the current evaluation process of scientific research in Portugal should be urgently corrected. Namely: all research units should be subject to evaluation visits; these visits should be conducted by specialist panels, unlike announced; the duration of the visit should take into account the size of the unit; and continuous funding base must be guaranteed funding to institutions evaluated at least as Good.

The CLA warns all political responsibles for wasting resources and energy, and the damage in trust, difficult to recover, which is the attempt to impose to Science of an evaluation process significantly rejected by the scientific community and therefore without no stable future and calls once again for the constructive dialogue that has always asked for.

The CLA will join efforts with all other scientific and academic institutions, including universities and their research centers, to safeguard rigor, competence and transparency of scientific assessment in Portugal. The CLA calls attention to the low priority to science, visible, among other indicators, in the drastic reduction of multiannual funding for research units observed since 2011 (a 40% reduction), aggravating the forced emigration of scientists and reducing the national scientific capabilities.

The CLA alerts the country to the gravity of the current situation of science in Portugal and the consequences that would have a continued setback in this critical area for the future of the country. Forced emigration of scientists now joins the instability of institutions and the rupture consensus between scientists and policymakers. For the first time since Portugal joined the European Union (EU), scientific institutions and higher education were not even called to say a word on the government's proposals on the fate of EU structural funds in their areas.

CLA, 07/25/2014

OS REITORES, A FCT E O MINISTRO

A tentativa em curso de destruição de metade do sistema científico português mostrou a fragilidade do Conselho de Reitores da Universidades Portuguesas (CRUP), que não quis ou não foi capaz de contrariar o Presidente da FCT e o ministro. O modo como o protesto dos Reitores foi acolhido assim como o modo como foram indeferidas as reacções dos Reitores quanto a novos cortes que se somam aos antigos mostram a impotência do CRUP tal como está hoje organizado e dirigido. Já alguém notou que a debilidade da reacção do CRUP à FCT talvez se explique por o Presidente do CRUP ser colega de Faculdade do Presidente da FCT. Quanto à continuada aceitação dos cortes orçamentais impostos pelo Ministério da Educação e Ciência não consigo encontrar explicação. Protestam, protestam e depois...

Contudo, o processo de "avaliação" da FCT mostrou uma interessante mudança do poder nas universidades portugueses. António Cruz Serra, reitor da Universidade de Lisboa (que resultou da fusão da Clássica e da Técnica, uma das maiores mudanças no ensino superior português nesta legislatura, uma mudança que veio das bases e que o governo devidamente encaminhou) não teve papas na língua no que respeita à prepotência da FCT. Em Agosto, a grande subida da Universidade que dirige no ranking de Pequim veio reforçar o seu poder entre nós (os rankings valem o que valem, mas valem!). Há, portanto, a Universidade de Lisboa, uma grande universidade de investigação, e há poucas outras, como Porto e Coimbra, que competem por isso. Talvez essas - e não todo o CRUP, que obviamente não tem poder - possam dizer o que pensam sobre a questão, bastante relevante, da avaliação da investigação e, já agora, da relação que deve existir entre a ciência e ensino. A FCT quer ser reguladora única e exclusiva da avaliação da ciência, deixando os reitores a falar sozinhos. Nesse quadro, quer obrigar à viva força um número considerável de docentes-investigadores a  abandonarem a investigação. Era bom saber o que os reitores de Lisboa, Porto e Coimbra pensam sobre essa posição, claramente expressa por António Coutinho, mas que é provavelmente secundada por Miguel Seabra e Nuno Crato.

Old Cabinet of Physics of the University of Coimbra

The microscope sent from London by Jacob Castro Sarmento FTS, one of the masterpieces of the Cabinet collection

The Old Cabinet of Physics of the University of Coimbra (also known as Physics Museum, part of the Museum of Science of the University) is located at the city academic center  at the Colégio de Jesus, one of the oldest Jesuit colleges in the world (it was founded in 1542). The building has been adapted by the Marquis of  Pombal, who expelled the Jesuits, ca. 1772 to serve the purpose of transmitting Natural Philosophy, in particular Newtonian science. At that time with the University Reform the Faculties of Philosophy and Mathematics were created. The Cabinet houses an unrivalled collection of scientific and didactic instruments from the 18th and 19th centuries. This heritage consists exclusively of instruments used in the Physics Cabinet of the University of Coimbra since its very origin in 1772. It is today one of the most complete Cabinets for the experimental study of Physics which is on public display. Thanks to its unique characteristics, this collection of instruments is among the most notable and rare in the world. The instruments from the 18th century are considered by experts true art pieces, having been shown in Brussels (Europalia exhibition), Lisbon (Gulbenkian Foundation)  and S. Paulo, Brazil. The designers and constructors are among the best in the world, in particular owners and artists of London workshops (George Adams, Benjamin Martin, John Dollond, Edward Nairne, Edmond Culpeper, etc.). The book of the Dutch Musschenbroek  Introductio ad philosophiam naturalem had been the guide for selecting and constructing the instruments that equipped the Cabinet. In its turn, the instruments from the 19th century, in turn, are well representative of the evolution of Experimental Physics along that century (some of the constructors were Breguet, Bianchi, Koenig, Ruhmkorff, Ernecke, Muller-Unkel, Geissler, Siemens & Halske, etc.). The collection is well representative of the evolution of experimental physics in the 18^th and 19^th centuries and is certainly worth a visit by anyone interested in science. A lecture hall from the 18^th century may also be visited. Nearby there is the Laboratorio Chimico, Chemical Laboratory, from the same period, which is probably the first building all over the world which was constructed for Chemistry studies.

Some of the instruments of the rich collection are publicly exhibited in two large rooms, which keep the original atmosphere of the Physics Cabinet, even with the original shelves, table and chairs. Indeed, One of the rooms is a true recreation of a Physics Cabinet of the second half of 18th century, where a professor of Physics Giovanni Dalla Bella (1730-1823) was teaching (he was called by the Marquis of Pombal from Padova, Italy, where he teached, together with other science professors). In the other room, instruments from the 18th century, mainly from the first quarter, complete the exhibition. The thematic areas of the instruments are mainly Newtonian mechanics; mechanics of the continua media, thermodynamics, optics and electromagnetism.

A virtual visit to the Cabinet may be done here:

http://fisica.uc.pt/ax/mf/mf_vv.php?qob=1

Some instruments are displayed here:

http://fisica.uc.pt/ax/mf/mf_all_vobjs.php~
See, e.g., http://fisica.uc.pt/ax/mf/mf_vobjs.php?qob=43

The University of Coimbra is since June 2013 a site of the UNESCO World Heritage (http://candidatura.uc.pt/pt/ ).  Its Science Museum (http://www.museudaciencia.org/ ), has received some international awards licke the Micheletti Prize for the best new science and technology museum in Europe.

To know more see D. Martins and C. Fiolhais, “A place of pilmigrage” in Europhysics News (2003) vol. 34, n. 4, p. 154.
http://www.europhysicsnews.org/articles/epn/pdf/2003/04/Whole_issue.pdf

Carlos Fiolhais and Décio Martins

Universidade de Coimbra: Antigo Gabinete de Física Experimental nomeado Sítio Histórico Europeu

Trancrevo em baixo notícia emitida hoje pela Lusa. Comentário que acrescento ao que fiz à agência: Ao contrário dos dirigentes do país no século XVIII, que criaram este Gabinete, os actuais não ficarão na história, pois nada deixarão que passados séculos valha a pena visitar. Desprezam a ciência e a história da ciência, pelo que não terão lembrança. O seu objectivo declarado de terminar com metade das unidades de investigação do país, incluindo a investigação em Física em Coimbra, contrasta com o reconhecimento internacional e o prestígio agora obtido numa nomeação por especialistas. Lembro que alguns avaliadores contratados pela FCT afirmaram, num assomo de incultura, que o estudo da história da ciência neste sítio Património Mundial da Humanidade não interessa". 

"O antigo Gabinete de Física Experimental da Universidade de Coimbra (UC) foi nomeado Sítio Histórico Europeu, pela Sociedade Europeia de Física, sendo o primeiro local em Portugal e o segundo na Península Ibérica a receber esta distinção.

Em comunicado, a UC refere que a nomeação resulta de uma candidatura proposta pela Sociedade Portuguesa de Física (SPF), "em particular do seu grupo de História da Física, recentemente criado pela atual direção e preparada pelos físicos Carlos Fiolhais e Décio Martins". ´

Integrado no Museu de Ciência, o Gabinete de Física Experimental foi criado pela Reforma Pombalina, com o objetivo de colocar a UC entre as melhores da Europa, e "afirma-se pela riqueza dos seus instrumentos e aparelhos utilizados para realizar experiências científicas - mais de meio milhar - que constituem uma coleção rara no mundo".

"Temos razões para estar muito orgulhosos em Coimbra e em Portugal. De entre os sítios históricos da Física na Europa selecionados até à data, o Gabinete de Física Experimental da Universidade de Coimbra é um dos mais antigos", sublinha Carlos Fiolhais, citado no comunicado. O físico e investigador salienta que, "hoje, os turistas podem ver e admirar essa impressionante herança", fazendo votos de que "os responsáveis pela ciência portuguesa de hoje se inspirem nos exemplos antigos".

A presidente da Sociedade Europeia de Física, também citada no comunicado, considera que a distinção vai permitir "refletir para o resto da Europa a riqueza do espírito científico das luzes do século XVIII em Portugal". "É bom para a imagem de Portugal no mundo. Os espaços de ciência desse século são espaços de imaginação e criatividade, muito úteis para a construção do futuro", frisou Teresa Peña.

 A Sociedade Europeia de Física, que reúne as sociedades de físicos de todos os países europeus, criou a distinção de Sítio Histórico Europeu para reconhecer e promover uma lista muito restrita de locais ligados a episódios fundamentais na história da ciência, como descobertas e criação e utilização de instrumentos. A designação de Sítio Histórico Europeu é uma das mais importantes da sociedade e do setor.

Entre os locais distinguidos, "encontram-se o CERN [Centro Europeu para a Investigação Nuclear, na Suíça], onde têm sido feitas várias descobertas em física de partículas, incluindo a partícula de Higgs, o Instituto Niels Bohr (Copenhaga), berço da moderna física atómica e nuclear, e o Blackett Laboratory do Imperial College, em Londres, palco de numerosos avanços em física fundamental", refere a UC."

 11:55 - 30 de Agosto de 2014 | Por Lusa

Sociedade Americana de Física preocupada com o fim de metade da Física portuguesa (article in Physics Today on the science cuts in Portugal)

A revista Physica Today da Sociedade Americana de Física, num longo artigo no seu número de Setembro, faz eco das preocupações dos cientistas portugueses e internacionais com a liquidação de metade das unidades de investigação portuguesas: 

"Half of Portugal’s research centers could see their funding plunge

Shock and anger have pervaded Portugal's scientific community since early this summer, when the results of an evaluation organized by the Science and Technology Foundation (FCT) threatened nearly half of the country's research laboratories with death by financial starvation. The evaluation was the first of a two-part look at some 320 research groups across all areas of science and humanities. A second phase is under way, with final grades expected at the end of the year; the grades determine core lab funding for the six year period 2015–20.

Although it’s widely accepted that funding is tied to such evaluations, researchers say that the current process is unfair. Among other complaints, they say that not all sites were visited, the process didn’t permit fair comparisons across research centers, the expertise of the more than 650 outside evaluators represented an inadequate range of subdisciplines, there were irregularities in scoring, and the results are skewed geographically. “This evaluation is flawed,” says Carlos Fiolhais, a physicist at the University of Coimbra. “We are asking for a suspension of this

process and a reanalysis.”

The uproar is making national headlines as university rectors,the Portuguese Physical Society and other scholarly societies, and major laboratories protest the process. The complaints reached fever pitch in mid-July, when reports emerged that the FCT had specified from the start that only half the research groups should score well enough to proceed to the second phase of the evaluation.

The FCT, the main government funding agency, denies that it made such a specification. The research laboratories, or "units," being evaluated can be based at a single university or research center, or have members from multiple institutions.The units range in size from a handful

of researchers to hundreds. The first phase of the evaluation was outsourced to the European Science Foundation (ESF) and used bibliometrics, performance reports, and future plans.

(...)  Toni Feder

More: http://scitation.aip.org/content/aip/magazine/physicstoday/article/67/9/10.1063/PT.3.2508

DIPLOMAS ACADÉMICOS, EDUCAÇÃO E INSTRUÇÃO

“É fundamental que o estudante adquira uma compreensão e uma percepção nítida de valores.”
Albert Einstein

Por servir de mote a este novo texto, começo por transcrever este comentário ao meu post “Ensino Universitário e Investigação Científica”: “Imagine-se, a loucura que seria, em Portugal, reduzir os exames e substitui-los por exercícios, feitos em casa!!!" (Ildefonso Dias, 24/08/2014).

Nem sequer são necessários grandes rasgos de imaginação! Basta trazer à colação o exame de inglês enviado por uma universidade privada para o computador do então primeiro-ministro José Sócrates e respondido por ele por essa mesma via. Com leves semelhanças,  este caso passa-se em muitas das nossas escolas neste Admirável Mundo Novo da era computacional Vejamos a que este respeito escreveu a escritora Alice Veira, num artigo do Jornal de Notícias, intitulado Copiar não vale”(03/02/2008). Depois de, em jeito preambular, ter reconhecido que “antigamente, muito antigamente, copiar era coisa muito feia”, escreveu esta festejada escritora:

“Hoje em dia são os professores que ensinam os alunos a copiar, que os incentivam a copiar. Hoje em dia a cópia está institucionalizada. Está a fazer-se uma revolução silenciosa. Hoje em dia os alunos nem entendem que possa ser doutra maneira. Chamem-lhe o que quiserem ‘descarregar’, ‘fazer download’, o que quiserem: nunca deixará de ser uma cópia. Eu chego a uma escola e ouço ‘Os alunos fizeram muitos trabalhos a seu respeito’. E encontro 50, 100, 200 trabalhos rigorosamente iguais, iguais, por sua vez, aos que já tinha encontrado na escola anterior, e na outra, e na outra, com os mesmos erros (nem a Wikipedia nem o Google são infalíveis), com as mesmas desactualizações, com palavras difíceis de que nenhum deles sabe sequer o significado, etc. Os meninos são ensinados a mexer num computador, a carregar nos botõezinhos necessários para que o texto apareça – mas depois ninguém lhes ensina que isso não basta, e que trabalhar e pesquisar não é isso. Isso é, pura e simplesmente, copiar”. E como se dizia no meu tempo, copiar não vale, etc.,etc”

Longe de mim atrever-me a dizer, ou apenas a deixar subentendido, que o plágio é situação  apenas da era dos computadores. Mesmo em épocas anteriores, seria  exigência utópica  uma sociedade  perfeita em que os alunos portugueses se não aproveitassem de todas as oportunidades para, como diria João Lobo Antunes, procurar uma calha que lhes permitisse deslizar sem atrito,  infringindo  as regras ditadas pela aceitação ética e tácita de regras de conduta que condenam o copianço.

Os testes com cruzinhas de escolha múltipla (se a memória me não falha, com o nome de testes do tipo americano, surgidos como novidade em Portugal no meu curso de oficial miliciano, em Mafra/52) facilitam, também eles, o copianço, tornando-se mais difícil nas respostas de desenvolvimento e mais difícil, ainda, quando, no ensino universitário é permitida a consulta de manuais por as respostas às questões equacionadas exigirem a pesquisa rápida de várias temáticas dispersas em páginas de difícil acesso para quem não esteja  bem senhor da matéria.

Tudo isto é reflexo de uma sociedade em que só os chamados tótós não copiam ou tentam copiar, distanciando  esta prática do que se passa em outras sociedades como, por exemplo, o relatado pelo  nosso compatriota Ricardo Reis, professor da universidade americana de Princeton, num artigo publicado no Diário Económico (3/Abril/2007), com o sugestivo título Copianço. Escreveu ele:

“Em Princeton, o professor é obrigado a deixar os alunos sozinhos na sala durante o exame. Vigiá-los seria uma falta de confiança, até porque todos assinam no topo da folha de resposta uma jura de que se vão comportar de uma forma honrada. Mas se alguém é apanhado a copiar (ou porque foi denunciado por um colega ou porque as respostas o tornam óbvio) então a punição é muito severa: pelo menos suspensão por um ano e talvez expulsão”.

Em Portugal o chico-espertismo de alunos do básico, secundário e mesmo ensino universitário (quantas teses de mestrado, ou mesmo doutoramento, não são cópias de textos colhidos na NET?) traduz-se numa execrável  falta de respeito por aqueles que dedicam noites  insones de estudo e são vítimas do copianço dos respectivos colegas cabulões. De igual modo este chico-espertismo vigora, ainda que de modo diferente,  nos estacionamentos de centros comerciais, que exemplifico com o que se passa, por exemplo, no Coimbra Shopping, em que  indivíduos, sem distinção de sexo ou idade:

1. Batem nos carros estacionados e “dão às de Vila-Diogo”.

2. Abrem as portas, de propósito, dos seus automóveis em mau estado de pintura  para fazerem mossas nas carrocerias de carros arrumados a seu lado, acabados de sair do stand ou com pinturas em bom estado de conservação.

3. Entram com o carro em sentido proibido para ocupar um lugar vago mais perto de si.

4. Estacionam, com a maior desfaçatez, em lugares destinados a idosos.

5. Arrumam, pletóricos de vigor físico, as suas “bombas”, ou carros a cair de podre, em lugares para deficientes.

6 Estacionam, apesar de serem do sexo masculino e viajarem sozinhos, espantai-vos leitores!, as respectivas viaturas no lugar destinado a grávidas (nunca me passou pela cabeça vir a assistir ao espantoso e inédito espectáculo de homens grávidos, pese embora os avanços feitos no domínio da gestação em barrigas de aluguer ou de bebés-proveta!). Etc., etc.

Por último, receio  que a escolha do título deste meu post possa induzir o leitor a pensar que os diplomas académicos impliquem  o respeito por regras e normativos da vida em sociedade. Nada mais falso, quando esses diplomas académicos se alicerçam em caboucos sólidos de instrução e frágeis alicerces  de Educação. Em vez de ambas terem o mesmo cordão umbilical  são geradas em ventres diferentes, em nítida contravenção com o princípio defendido por Einstein: “É fundamental que o estudante adquira uma compreensão e uma percepção nítida de valores”.

Desse divórcio, entre instrução e educação, nos deu conta Gustave Le Bom (1841-1931), psicólogo social e sociólogo francês:

“Grande número de políticos ou universitários,carregados de diplomas, possuem uma mentalidade de bárbaros e não podem, portanto, ter por guia de vida senão uma alma de bárbaros!”

Parafraseando o historiador Oliveira Martins, quando escreve que Portugal e Espanha morreram como nações vivas em 1580, a convivência  entre educação e instrução é apenas formal neste rectângulo ibérico. Por falta de plasticidade do nosso actual sistema educativo!?

Carlos Poças Falcão

Dois Poemas de Carlos Poças Falcão, extraídos do livro “O Número Perfeito”, editado na década de 80. Alguns poemas do poeta estão também no livro “Verbo — Deus como interrogação na poesia portuguesa”, editado recentemente pela  Assírio e Alvim.

A obra do aedo está, em grande parte, reunida em "Arte Nenhuma", livro que tem a chancela da Opera Omnia. O amor, a religião e a ciência são temas basilares na sua arte.

1

Alguém se aproxima, alguém me vai falar.

Procuro as formas de me dar. Que sabemos

das duplas estrelas? Que sabemos

de toda a gravitação? Nem a luz

podemos ver de outro cérebro, nem

a cintilação interior de outro olhar.

Dois cofres de pele. Nós os dois. Um abismo

atmosférico. Palavras rápidas a abrir

uma úlcera no ar, Uma voragem.

Eis o verbo abrir. Eis o verbo fechar.

2

Pesar-me. Ter sessenta quilos

e perguntar porquê. Silêncio de balança.

Porque sinto o peso daquilo que me cansa

o mundo que se expande na cabeça

os dias adiados, longos anos

a lentidão que avança com os anos

a respiração, o esforço de viver

para cima quando tudo está a ceder.

Porque dentro peso muito. E ao contrário

da força de atracção do corpo pela terra.

Facts about the FCT/ESF science evaluation: the story so far

The purpose of these notes is to present an objective view of the evaluation of Portuguese research units that is taking place and which, if not halted, will destroy a large part of a system that has taken years to build.

The Portuguese system

Funding systems may differ greatly between countries. In Portugal research units are, to a large extent, independent from other units such as departments and universities. The funding of these research units is mostly done by the Foundation for Science and Technology (FCT), the only public national funding agency for science in Portugal. Thus, unlike in other countries like France, where there exists more than one source of funding such as the ANR and the CNRS, for instance, in Portugal practically all the public funding for science is channeled via FCT. Universities do not play a role in research funding  – their purpose from a financial perspective is providing salaries for staff members and carry out the administration of research funds coming from FCT.

This means that a research unit that did not make it to the second stage of the undergoing evaluation (with site visits) will inevitably stop existing as such – in fact, not even in the least demanding of areas from a financial point of view can a 30-researcher strong unit survive with a total budget of 5000 Euros per year, as it has been announced. This possibility cannot be taken seriously. For comparison, the same unit with the same classification received after the 2007 evaluation 2750 Euros per researcher per year, that is, 82,500 Euros per year. Many researchers do agree that a funding per head is not a good system since in that case units have an advantage in increasing their numbers irrespective of quality. There are, however, other ways of addressing this issue while still ensuring that the amounts involved do allow the active researchers in the units to continue developing their work, which will not be possible with the proposed funding (or, better, lack of it) in the present evaluation.

Previous FCT evaluations

No researcher in Portugal objects to the dividing of units into slots which will determine their funding, the best being allocated more funds than the others. That is how science funding should work and, in fact, this is how things have been working in Portugal until now. And although there have always been some (minor and isolated) complaints after each of the previous evaluations, nothing on this scale had been witnessed before.

In previous evaluations we had 5 grades, namely: Poor, Fair, Good, Very Good and Excellent. Units below the Very Good mark were not entitled to strategic funding. However, units with Good still received funding which allowed them to work to improve their standing. In particular, between the 2003 and the 2007 evaluations, while the percentage of units with Excellent remained at 21%, those with Very Good saw an increase from 31% to 38% - these numbers exclude the so-called Associate Labs.

It should also be made clear that the panels which carried out the previous evaluations were, as they are now, international, and that nearly all of the units being evaluated agree that it should remain that way. In fact, the only voices who have disagreed publicly are those who work in fields where the Portuguese language plays a strong role, like some of the Humanities. These do accept that involving Brazilian researchers or other specialists in Portuguese language and culture abroad would solve the problem.

The current evaluation

In this evaluation there are now 6 grades: Poor, Fair, Good, Very Good, Excellent and Outstanding. The lowest three levels are not entitled to any strategic funding, while, as described above, those with Good will still receive some funding but which is ridiculous and definitely not enough to carry on research at any level.

Those in the higher three levels will be visited by the panels and will be entitled to strategic funding. However, it has not been stated how this strategic funding is to be allocated as a function of the classification nor how much of what was requested by the unit will be conceded – in particular, there were no rules on how much could be asked by each unit.

Regarding panels, the main difference between this and the 2007 evaluation was that, before, there were 25 panels divided into scientific areas containing a total of 256 experts and, with some very specific exceptions, each  panel had at least 8 members. This allowed for a wide covering of the different areas and all the experts had a global view of the area they were evaluating.

In the current evaluation, there were only 6 panels plus one multidisciplinary panel with a total of 73 experts. A list of the panels and the names of the researchers involved in the 2007 evaluation may be found here, while the corresponding lists for the current evaluation which may be viewed in Table 1 may be found here.

Exact Sciences	Engineering Sciences	Health and Life Sciences	Natural and Environmental Sciences	Social Sciences	Humanities
Paolo Decuzzi The Methodist Hospital Research Institute, Houston, United States	Laurie Boswell City University, London, United Kingdom	Charles Auffray Université de Lyon, France	Grant Bigg University of Sheffield, United Kingdom	Bjørn T. Asheim Lund University, Sweden	Susan Bassnett University of Warwick, Coventry, United Kingdom
Eric Eliel Leiden University, the Netherlands	Anthony Bull Imperial College London, United Kingdom	William Cushley (Chair) University of Glasgow, United Kingdom	Huub De Groot University of Leiden, the Netherlands	Ben Chigara Brunel University West London, United Kingdom	Nicholas Canny National University of Ireland, Galway, Ireland
Jordi Jose Universitat Politecnica de Catalunya (UPC), Barcelona, Spain	Pasquale Chiacchio Universita degli Studi di Salerno, Italy	Hinrich Gronemeyer Université Louis Pasteur, Illkirch, France	Arnold J.M. Driessen University of Groningen, the Netherlands	Rosemary Deem (Chair) Royal Holloway, University of London, United Kingdom	Peter Michael Cronin Dublin City University, Ireland
Theofanis Kitsopoulos Foundation for Research and Technology-Hellas (FORTH), Heraklion, Crete	Barry Clarke University of Leeds, United Kingdom	Günter Ollenschläger University of Cologne, Germany	Mark Johnson National University of Ireland, Galway, Ireland	Juan Diez Medrano University of Barcelona, Spain	Gerhard Jaritz Central European University, Budapest, Hungary
Peter Kloeden J.W. Goethe University, Frankfurt, Germany	Fabio Gori University of Rome “Tor Vergata’, Italy	Giovanni Pacini National Research Council (CNR), Padova, Italy	Francesco Loreto Consiglio Nazionale delle Ricerche (CNR), Rome, Italy	Javier Esparcia Pérez University of Valencia, Spain	Janis Jefferies University of London, United Kingdom
Antonino La Magna Consiglio Nazionale delle Ricerche, Catania, Italy	Philippe Olivier Université de Toulouse, France	Martine Raes Faculté Universitaire Notre Dame de la Paix, Namur, Belgium	Raul Madariaga Ecole Normale Superieure, Paris, France	Francesco Francioni European University Institute, Florence, Italy	Janusz Krzysztof Kozlowski Jagellonian University, Kraków, Poland
Shahn Majid Queen Mary, University of London, United Kingdom	wp@soton.ac.ukWilliam Powrie University of Southampton, United Kingdom	Narender Ramnani Royal Holloway University of London, United Kingdom	Giuseppe Scarascia-Mugnozza Universita degli Studie della Tuscia, Viterbo, Italy	Akily Gürsoy Yeditepe University, Istanbul, Turkey	Svend Erik Larsen University of Aarhus, Denmark
Nigel Mason (Chair) The Open University, Milton Keynes, United Kingdom	André Preumont (Chair) Ecole Polytechnique de Bruxelles, Belgium	Martin Röllinghoff Erlangen-Nuremberg Universität, Germany	Peter Suhadolc (Chair) Università degli studi di Trieste, Italy	Lynn Jamieson University of Edinburgh, United Kingdom	Pasqualina Perrig-Chiello University of Bern, Switzerland
Tomas Obsil Charles University, Prague, Czech Republic	Wolfgang Reisig Humboldt-Universität zu Berlin, Germany	Shlomo Sasson The Hebrew University, Jerusalem, Israel		Erik Jones Johns Hopkins University SAIS, Bologna, Italy	Riccardo Pozzo Consiglio Nazionale delle Ricerche (CNR), Rome, Italy
Kenneth Ruud University of Tromsø, Norway	Juha Röning University of Oulu, Finland			Rainer Kattel Tallinn University of Technology, Estonia	David Robey (Chair) University of Oxford, United Kingdom
Nizar Touzi Centre de Mathématiques Appliquées, Palaiseau, France	Peter Van Roy Université Catholique de Louvain, Belgium			Anne Kovalainen University of Turku, Finland	Matti Sintonen University of Helsinki, Finland
	John Villadsen Technical University of Denmark, Lyngby			Volkmar Lauber University of Salzburg, Austria	Martin Stokhof Universiteit van Amsterdam, The Netherlands
				Romain Martin University of Luxemburg, Walferdange	Ann Thomson European University Institute, Florence, Italy
				Elisabeth Müller Frankfurt School of Finance & Management, Frankfurt/Main, German	Koenraad Van Balen KU Leuven, Belgium
				Claire O’Malley University of Nottingham, United Kingdom	Peter Weinreich University of Ulster, Newtownabbey, United Kingdom
				Constantinos N. Phellas University of Nicosia, Cyprus	Milena Žic Fuchs University of Zagreb, Croatia
				Christian Schultz University of Copenhagen, Denmark

Table 1. Panels responsible for the final decisions as to whether research units made it to the second stage or not. Not making it to the second round implies either no funding whatsoever or only a very limited amount which is not enough to keep any research going on. There are 48% of units in this situation.

Each of these 6 panels brought together several different areas such as Chemistry and Mathematics (Exact Sciences), Chemical and Civil Engineering (Engineering Sciences), Public Health and Experimental Biology (Health and Life Sciences), Forestry Sciences and Marine Sciences (Natural and Environmental Sciences), Educational Sciences and Economics (Social Sciences) and History and Psychology (Humanities). With an average of about 12 members per panel and numbers actually raging from 8 to 17, it is not clear how these panels could cover all areas in an appropriate way, not to mention not introducing significant and serious biases. Although FCT claims that using 585 remote reviewers allowed for a proper covering of all areas, it was in fact the reduced panels who made the decisions, in many cases overriding the external experts opinions – see point 5 in the section (entitled What are researchers objecting to?) below; in fact, while the FCT's President claims that the evaluation was “robust”, he also stated that a 19 given by a reviewer could really just be worth a 13 or 14 (sic), although it is not clear who is supposed to do this re-interpretation of the grades.

How did we reach this situation?

FCT's proposal laying out this evaluation's rules was put out for public discussion in the first trimester of 2013. From the start it was clear that this would be a complete departure from previous evaluations. Several serious criticisms were made by the Council of Rectors of Portuguese Universities (CRUP), the Council of Associate Labs (CLA), many universities and several researchers on an individual basis. The main issues raised were ignored by FCT, in spite of the fact that the points raised at the time are now proving themselves to have drastic consequences.

We thus started an evaluation with rules that most, if not all, units objected to.

On June 27 a public presentation of the general results was made by the head of FCT and the head of the Health and Life Sciences panel, Professor William Cushley.

When the actual results were made public later that day, at first researchers were taken aback by what was happening. However, it soon became clear that something was not right. A too large number of units which had been considered to be Very Good or even Excellent in previous evaluations had now been relegated to the status of Good or even Fair. In some cases, even units which had received positive reports from all referees prior to the rebuttal phase were now eliminated.

Using statistics and some reverse engineering, a group of researchers put forth a very strong case that quotas had been used. FCT denied this, stating that there had only been some standardization of grades between areas.

However, and under (legal) pressure, on July 18 FCT eventually had to make public the contract made with ESF, which several entities had been demanding for many months. Here the number of centres to pass on to the 2nd stage is indicated explicitly. In spite of this, to this day FCT still denies the existence of quotas and claims that this number was given as an estimate only, based on what had happened in the 2007 evaluation, for the purpose of calculating the budget. It should be stressed that even this number is actually incorrect, as in the calculations of the 2007 evaluation FCT did not take into account the Associate Labs, which are now being evaluated simultaneously with all other units. The actual percentage of Centres above the threshold is in fact not 50% but 58% and this does not take into account that many units merged or ceased to exist (the total in 2007 was 404, while the total of units applying this time was only 322), or of natural improvements to the quality of the existing units.

Furthermore, in another part of the contract (Work Plan, General Principles – Stage 1 Assessment), where nothing related to the budget is being mentioned, but instead indications to the panels on how the process should be handled are being issued, the “magic” number of 50% exclusions also appears. There it is stated explicitly that

“Stage 1 will result in a shortlist of half of the research units that will be selected to proceed to stage 2.”

FCT's official position is still that there were no quotas and that it had no influence whatsoever in the evaluation results. This position is, in fact, that all the decisions were made independently by the panels, who are then made responsible for everything that is happening.

What are researchers objecting to?

Apart from the fact that researchers feel cheated with the hidden quotas rule, they are complaining about the following issues.

1. The document that was put up for discussion by FCT in 2013 describing the evaluation process and which, with some modest changes, laid out the whole process, was updated at the end of April 2014, that is, with the evaluation already in full-swing, by another document referred to euphemistically  as Additional Information. In spite of the name, this in fact changed at least two fundamental aspects of the process completely, to wit:

i) while the first mentioned that at most 5 reports would be produced, the new document reduced this number to only two external (anonymous) reports plus one internal report.

ii) in the first document, the referee picked from among those proposed by the unit would take part in the discussion leading to the writing of the consensus report; this aspect is absent from the procedure describing the elaboration of the final report; in fact, and, as far as we can tell, the referee mentioned by the unit was never consulted regarding the consensus report.

Apart from the fact that it is completely unacceptable to change the rules once the process has started, it is clear that having 5 reports provides a more robust basis from which to proceed. This is definitely not guaranteed with only three reports. In particular, the data a group of analysts gathered so far (about 1/3 of all units) indicates that the difference between the maximum and the minimum marks
of the three referees is at least 5 points (out of 17) in 50% of the cases.

2. Many of the units which did not make it to the 2nd stage feel that this was the case because there were quotas and by this alone. In other words, had panels been left to their own devices to decide whether or not a given unit satisfied the criteria indicated, some of the units which are now staying behind would have made it to the second stage. This implies that there are units at very similar levels where one made it to the second stage and the other stayed behind. In particular, many think this from what was written in the consensus reports where, although some of the keywords which appear pointed to a classification of at least Very Good, the result was only Good.

In short, if this evaluation is allowed to go ahead, it will introduce unacceptable discontinuities between the quality of research units and their funding, effectively killing off nearly half of the research in the country. This was one of the many points that was already raised during the public discussion of the evaluation rules but one to which FCT paid no attention to.

3. Many experimental units have equipment that requires funds not only for running but also for maintenance; the actual funding provided to units classified with Good is totally ridiculous and the implication is that the corresponding equipment will either be lost or, at best, left unused. This amounts to a terrible loss, both scientifically and financially. In fact, very relevant and expensive scientific equipment is going, if nothing happens, to be sold or simply abandoned. The scientific production will, no doubt, be affected.

In any case, what should be made very clear is that this distinction made between two similar units implies that one will get funding while the other, very close to it, will cease to exist, its researchers being forced to leave research. It is also not clear that the best researchers in units which will disappear will be able to join other units, both for geographical reasons and because it is not clear that units will be willing to receive more researchers without an increase, albeit modest, of their funding – in fact, these extra researchers were not counted in the strategic development of the unit. Concerning the geographical issue, it is also feared that certain areas will cease to exist completely in certain parts of the country, as is the case of Physics which will be mainly concentrated in the capital.

4. A detailed statistical analysis has shown that the size of the units has been the main decisive indicator as to whether a unit made it to the second stage or not. Although it is not clear whether this was intentional or not, it had the effect of eliminating small units on non-scientific grounds.

A second statistical analysis has also shown that there is no evidence of an improvement in any of the indicators with respect to scale, that is, larger units do not benefit from being large when it comes to outputs.

5. In many cases, the external referees' reports were ignored by the panel in the elaboration of the final consensus report, sometimes in completely unexplained ways. With the data currently available to the analysts group (which have collected data about 1/3 of all units), it is estimated that over 20% of the cases fall into situations where the average of the external reports is contrary to that of the internal referee, but the latter's opinion prevailed in the consensus report. Only about 10% fall in the reverse situation.

6. In all previous evaluations, all units were visited by the panel, giving them the possibility to present their work, see the labs, and answer all questions that evaluators might have. Now this will only be the case for those units that made it to the second stage.

7. A complaint that has been voiced for some time now is that it is not clear how the funding will be distributed among all units that made it to the second stage. This was raised, for instance, by the Council of Associate Laboratories as early as March 2013. We recall that the mechanism by which this will be carried out by FCT has never been made public, a fact which is felt as unacceptable. Rules should be well defined and kept from the very beginning.

Who has made strong criticisms of the evaluation process publicly?

Here is a list, by no means complete:

1. The Council for Associate Laboratories

Associate Laboratories are some of the major research units in the country, representing over 2500 researchers and being transversal across almost all areas. There are 20 associate laboratories, of which 19 made it to the second stage. Their statement, which may be read here, covers many of the main complaints of Portuguese researchers, namely, those points referred to in the previous section.

In fact, the CLA had already alerted FCT and the research community as early as March 2013 to the many dire consequences that would ensue should this evaluation go ahead with the regulations proposed by FCT.

2. The Council of Rectors of Portuguese Universities (CRUP) has, after having met with the Secretary of State for Science and Technology, issued a statement where it considers that units which have had 14 in the evaluation or which had Excellent or Very Good in the previous evaluation should also make it to the next round. This proposal has been turned down by FCT, while the Minister for Education and Science claims that the evaluation should go ahead as it is and rules should not be changed in the middle – we recall, however, that FCT did just that in April this year (see point 1. in the previous section).

3. Following CRUPs statement, the Rector of the University of Lisbon (by far the largest in Portugal, and with a rate of approval of 73% units to the second stage) distanced himself from that position and requested the prime-minister to suspend the evaluation. In a public interview, where he made severe criticisms to the FCT-ESFs methodology, he stated that this type of policy will destroy an important part of the national scientific system. He also criticized the discontinuities that are being introduced in the system, where all of a sudden units with many active researchers with international careers will cease to exist.

4. The Portuguese Societies of Chemistry, Physics and Philosophy all produced statements which may be found here, here and here. The National Mathematics Commission, which includes representatives from all of the research units in mathematics, of the Portuguese Statistical Society and the Portuguese Mathematical Society, has also criticized the process in a statement which may be found here.

5. The very own Scientific Councils of FCT have made a public statement mentioned here, where they criticize many of the evaluation aspects.

6. A group from the Social Sciences, including researchers from both units who made it and those who did not make it to the next stage wrote an open letter to the president of FCT which may be found here.

7. Many centres who did not make it to the second stage have made their opinions known; they have also made their evaluation reports public to let their peers evaluate by themselves. In some instances, this also includes a statement by the accompanying panel of the centre, an international entity recognized by FCT made up, in general, by top scientists.

8. Several researchers who have seen the whole system grow and develop within the last 30 years, such as retired Professor Maria de Sousa, a eminent immunologist, in an article in one of the most important Portuguese daily newspapers (articles may be found here, here, here, here, here, here and here).

Some reports have also started to appear in the international specialised press (Nature, Physics World, The Conversation, etc.). Note that some of these were written before issues such as the quotas and the changing of the rules became known. Physics Today, the journal of the American Physical Society, is in the process of writing an article on the subject, giving international voice to the Portuguese physicists.

Who has defended the evaluation process publicly?

Except for FCT (and the ruling coalition MPs, when they voted against the opposition proposal to review the evaluation), only one voice so far has made itself heard to defend the evaluation. This is a retired, although well-known, researcher (António Coutinho), who stated that mediocrity should not be funded, only excellence, implying that there is nothing in between. There are in this position  echoes of the statements made by Vince Cable in the UK in 2010, in particular that only about half of research should be funded. We recall that this position was heavily criticised at the time by both researchers and commentators.

What will happen?

Most likely the whole process will end up in court, with many units and institutions having already pledged to do so. Some researchers' unions are preparing a large scale court action aimed at stopping the evaluation on its tracks. Since the general impression is that the direction of FCT is not listening to anyone, it is feared this might be the only way out, with all the wear and tear it will cause, not to mention the precious wasted research time.

But isn't this just a local  issue?

We do not see it like that for several reasons:

1. As soon as ESF became involved, this became a European issue. Although it is FCT's evaluation, ESF did have to agree to several issues such as reduced panels, the imposition of quotas, and the changing of the rules already after the process was well under way.

2. This type of funding, where all is given to very few might become the norm; we recall that the current head of FCT has been elected to be President of Science Europe, taking office as of the 1st September.

3. This is also the philosophy behind the statements made by the British Secretary of State for Business, Innovation and Skills, Vince Cable, back in 2010. At the time, this caused quite an uproar in the UK and many scientists and commentators spoke out against it. Since we believe that the situation is similar, we record here three of the very many reactions that took place. Robert M. May (former Chief Scientific Adviser to the UK Government and President of the Royal Society) declared that:

“He [Vince Cable] was clearly badly briefed, and it's a shame he didn't care to get all the facts beforehand. In particular, his claim that public money should not be made available to research that 'is neither commercially useful nor theoretically outstanding' is just plain stupid."

while the President of the Academy for Medical Sciences, John Bell, stated that

"A long term commitment to publicly funded research is vital if we are to harness the competitive advantage previous investment has generated."

Finally, Mark Henderson, science editor of the Times at the time, pointed out that

“His [VC's] claim that 45 per cent of research fails to pass muster is as credible as Blair’s claim that Iraq could launch WMDs in 45 minutes.”

4. What is at stake here is science, which is universal. Can science progress if only what is deemed to be excellent or exceptional at a certain point by a controversial process is funded? Recall that there are no other national sources of funding for almost all of the researchers involved.

5. Finally, we recall that although the outcome and most of the problems were caused by FCT's demand that 50% of the units should be eliminated, its official position is that it had nothing to do with the evaluation so far and that the outcome is the work of ESF's evaluation and the panels of experts alone. It is thus also ESF's and the panel members' reputations that are at stake here.

Carlos Fiolhais
(on behalf of a group of researchers, who have been analyzing the ESF/FCT evaluation process)