Am 4. November 2025 luden der Wiener Wissenschafts-, Forschungs- und Technologiefonds (WWTF) und die Plattform fteval zu einer Veranstaltung über Panelevaluierung, um die Erfahrungen in der Community in einen Austausch zu bringen.

Für diese Podcastfolge haben wir die Diskussion für unsere Zuhörenden eingefangen: Magdalena Wicher und Michael Strassnig vom WWTF fassen mit Katharina Warta, Leiterin des österreichischen Büros von Technopolis Group, die wichtigsten Punkte zusammen und kommen am Ende sogar noch in eine richtige Kontroverse!

Für die Beteiligten ist klar, was die Vorteile von Panels sind, weil man beispielsweise rasch die kritischen Punkte zum Evaluationsobjekt identifiziert werden können und die Rolle der "Critical Friends" sehr wertvoll sein kann. Gleichzeitig ist es herausfordernd und ressourcenintensiv, ein ausgewogenes und hochkarätiges Panels zu etablieren und Ownership über den Evaluationsgegenstand und Vertrauen zu generieren. Die Bedeutung von persönlichen Treffen wird besonders herausgestrichen. Katharina teilt ihre Erfahrung aus der Gruppendynamik und erklärt anhand des Johari-Fensters, wie man blinde Flecken, Unbekanntes und Geheimes aufdecken kann. 

Links: 

• Foliensatz des WWTF zu Panelevaluierung mit zwei Fallbeispielen
• Fallbeispiele von Technopolis Group: 
  • Evaluation Methodology der Evaluation der wissenschaftlichen Institutionen Lettlands: https://www.izm.gov.lv/lv/media/29432/download?attachment 
  • Presentation des Evaluationsberichts für Universität Luxembourg: https://www.chd.lu/fr/meeting/13645 (ab Minute 5:50)

Musik: Urbana-Metronica (wooh-yeah mix) (ft. Morusque, Jeris, CSoul, Alex Beroza) by spinningmerkaba http://ccmixter.org/people/jlbrock44; SouljaUnit Remix of DeadRobot Music’s Surfy via https://freesound.org/people/SouljaUnit/sounds/640175/
Foto: Isabella Wagner

podcast(at)fteval.at - twitter.com/fteval - www.fteval.at

Markus Wanko is the founding force behind the XISTA innovation ecosystem, empowering science innovators. Today, Markus leads XISTA’s venture fund (Managing Partner), science park (MD), and tech transfer arm (MD). With his work he helps ISTAs research to develop it's full societal potential. In this interview we learn how start up trainings must be sector specific to be most effective, where the venture capital for the science start-ups typically comes from and what Markus recommends to the evaluation community: To take evaluation seriously!

Links: 

• ISTA: https://ista.ac.at/en/home/
• XISTA: https://xista.com/
• XISTA science ventures: https://xista.vc
• XISTA innovation: https://xista.io/
• XISTA science park: https://xista-park.com/

Music: Urbana-Metronica (wooh-yeah mix) (ft. Morusque, Jeris, CSoul, Alex Beroza) by spinningmerkaba http://ccmixter.org/people/jlbrock44; SouljaUnit Remix of DeadRobot Music’s Surfy via https://freesound.org/people/SouljaUnit/sounds/640175/
Photo: The Gentlemen Creatives

podcast(at)fteval.at - twitter.com/fteval - www.fteval.at

Anastasia Lijadi is Executive Assistant to the Deputy Director General and Gender Equality and DEI Officer at the International Institute for Applied Systems Analysis (IIASA). We hear how IIASA's international set-up allows research perspectives at a global systemic scale to tackle our time's most pressing problems. 

More information about IIASA's Gender Equality Plan can be found here: https://iiasa.ac.at/about-iiasa/institute/equality-diversity-and-inclusion/iiasa-gender-equality-plan-gep

Music: Urbana-Metronica (wooh-yeah mix) (ft. Morusque, Jeris, CSoul, Alex Beroza) by spinningmerkaba http://ccmixter.org/people/jlbrock44; SouljaUnit Remix of DeadRobot Music’s Surfy via https://freesound.org/people/SouljaUnit/sounds/640175/
Photo: IIASA

podcast(at)fteval.at - twitter.com/fteval - www.fteval.at

Erika Hajdu and Stefan de Jong of the Evaluating Societal Impact team at the Erasmus University Rotterdam developed a card game to structure discussions around complex impact pathways, and to enable teams to draft early versions of their Theory of Change in a more interesting, time-efficient way. 

They presented their approach at the REvaluation Conference, and we thought this tool might be useful for the fteval community. 

In this interview we learn how the game works and how it can be helpful for different settings.

Links:  

• Journey of Progress card set | Erasmus University Rotterdam 
• Paper publication as part of the Conference Proceedings

This episode was made possible with the friendly support of the Austrian Ministry for Women, Science and Research. Thank you for this cooperation!

Music: Urbana-Metronica (wooh-yeah mix) (ft. Morusque, Jeris, CSoul, Alex Beroza) by spinningmerkaba http://ccmixter.org/people/jlbrock44; SouljaUnit Remix of DeadRobot Music’s Surfy via https://freesound.org/people/SouljaUnit/sounds/640175/
Photo: Erika Hajdu, Erasmus University Rotterdam 

podcast(at)fteval.at - twitter.com/fteval - www.fteval.at

Diesmal waren wir zu Gast bei WPZ Research und konnten uns dort einiges über die jüngste Ausgabe des European Innovation Scoreboards (EIS) und Österreichs Performance darin erfahren. Brigitte Ecker und Sascha Sardadvar erklärten uns geduldig, was uns der EIS sagen kann und was nicht und wie es sonst so um die Zukunftsfähigkeit der österreichischen Forschungslandschaft bestellt ist. WPZ Research ist als kleines unabhängiges Forschungsinstitut auch deshalb besonders, weil es zur DNA gehört, dass alle Forschenden gleichzeitig auch in der Lehre aktiv sind. 

Diese Podcastfolge wurde mit freundlicher Unterstützung des Bundesministeriums für Frauen, Wissenschaft und Forschung (BMFWF) ermöglicht. 

Musik: Urbana-Metronica (wooh-yeah mix) (ft. Morusque, Jeris, CSoul, Alex Beroza) by spinningmerkaba http://ccmixter.org/people/jlbrock44; SouljaUnit Remix of DeadRobot Music’s Surfy via https://freesound.org/people/SouljaUnit/sounds/640175/
Foto: Isabella Wagner

podcast(at)fteval.at - twitter.com/fteval - www.fteval.at

There is a generation shift in the community going on and in the dynamic landscape of research and innovation, the perspectives of the next generation are crucial for shaping future policies and practices. In this panel we tried to bring together younger researchers and evaluators and senior counterparts to discuss the challenge, demands and future directions of innovation research and policy evaluation.

Panellists

• Florian Bayer, Department of Science and Technology Studies, University of Vienna
• Luyu Du, Center for Organization Research and Design (CORD), Arizona State University, United States
• Nino Gogadze, CWTS, Leiden University
• Jakob Kofler, Austrian Institute for SME Research

Moderators

Philippe Larédo & Julia Melkers 

Music: Urbana-Metronica (wooh-yeah mix) (ft. Morusque, Jeris, CSoul, Alex Beroza) by spinningmerkaba http://ccmixter.org/people/jlbrock44; SouljaUnit Remix of DeadRobot Music’s Surfy via https://freesound.org/people/SouljaUnit/sounds/640175/
Photo: Martin Jordan

Thanks to our partners and co-organisers that helped realising the REvaluation Conference 2024: Ministry of Innovation, Mobility and Infrastructure, Ministry of Women, Science and Research, Ministry of Economy, Energy and Tourism, Vienna Science and Technology Fund, COST Action PROFEEDBACK (CA ), Meeting Destination Vienna, FH Campus Vienna, Austrian Science Fund, Austrian Research Promotion Agency, ZSI - Centre for Social Innovation, Fraunhofer ISI, Institut Francilien Recherche, Innovation et Société, Joanneum Research, the whole fteval Community. 

podcast(at)fteval.at - twitter.com/fteval - www.fteval.at

Motivated by evidence based scientists doing gut-based science policy, and by science policy maker acknowleding the faults of the system when they leave it rather than when they can still do something about it, James Wilsdon of RoRI turned his attention towards meta science in this REvaluation 2024 keynote.

Meta Science, Meta Research, Science of Science, Research of Research are all the same, and not new, but new things are going on. Applying robust methods to understanding, analysing and improving the science system!

He shows that he is not the only one, but more and more RoR is done out of the various discplines and sub-disciplines themselves, in particular, because there is data to do so, and that adds specific domain knowledge to the pool of RoR bottom-up. Top-down, from science funders, are at least in the UK again testing their funding in large-scale experimental settings; and are opening their funding data for the ROR community. Lastly, new institutions contribute to the debate, in Carlifornia funded by the Tech community, trying to "practice meta science". 

Why does this happen? More investment, slow down of disruption and radical change, worries about too much reporting and to little impact, on-going crisis of replication, and rapdily changing landscape of scolar communication (open science, open data), pressure for excellence and AI and research funding. MetaScience is both a contribution to incremental change and a source for radical, systematic change.

Concluding, Wilsdon argues for five priorities to go forward: more and better experimentation for evaluation (e.g. applicants reviewing other applicants, makes it easier to find reviewers, and to speed up the process); investing in infrastructures for doing MetaScience at scale and long-term (e.g. the publish-curate-review-platform MetaROR); Foster cross sector alliances (e.g. MetaScience Alliance); with more global conversations beyond western countries; and demonstrate the worth of MetaScience!

Links:

• RoRI-Research on Research Institute: https://researchonresearch.org/
• Metascience Conference: https://metascience.info/ 
• James' presentation slides

Music: Urbana-Metronica (wooh-yeah mix) (ft. Morusque, Jeris, CSoul, Alex Beroza) by spinningmerkaba http://ccmixter.org/people/jlbrock44; SouljaUnit Remix of DeadRobot Music’s Surfy via https://freesound.org/people/SouljaUnit/sounds/640175/
Photo: Martin Jordan

Thanks to our partners and co-organisers that helped realising the REvaluation Conference 2024: Ministry of Innovation, Mobility and Infrastructure, Ministry of Women, Science and Research, Ministry of Economy, Energy and Tourism, Vienna Science and Technology Fund, COST Action PROFEEDBACK (CA ), Meeting Destination Vienna, FH Campus Vienna, Austrian Science Fund, Austrian Research Promotion Agency, ZSI - Centre for Social Innovation, Fraunhofer ISI, Institut Francilien Recherche, Innovation et Société, Joanneum Research and the whole fteval Community. 

podcast(at)fteval.at - twitter.com/fteval - www.fteval.at

There is momentum in the research community with regard to trandisciplinary research (TDR) and research funding, because societal transformation necessitates more complex solutions; but at the same time, the notion that research happens more and more within disciplines, and not across disciplines or in a transdisciplinary way.

The panel discusses challenges, but also opportunities TDR provides and tackles questions and opportunities on reasons to fund TDR and how to achieve impact, and on related challenges. The panel touches on questions like how new is the approach, what can be done to support it further and how to provide contiunity and manage knowledge within the research system.

The panelists were

• Christian Pohl, ETH Zürich, Co-Director Transdisciplinarity Lab
• Helen Buckley Woods, University College London, Research on Research Institute
• Marianne Penker, BOKU University, Academy of Science 

Moderation: Petra Biberhofer, Austrian Science Fund 

Music: Urbana-Metronica (wooh-yeah mix) (ft. Morusque, Jeris, CSoul, Alex Beroza) by spinningmerkaba http://ccmixter.org/people/jlbrock44; SouljaUnit Remix of DeadRobot Music’s Surfy via https://freesound.org/people/SouljaUnit/sounds/640175/
Photo: Martin Jordan

Thanks to our partners and co-organisers that helped realising the REvaluation Conference 2024: Ministry of Innovation, Mobility and Infrastructure, Ministry of Women, Science and Research, Ministry of Economy, Energy and Tourism, Vienna Science and Technology Fund, COST Action PROFEEDBACK (CA ), Meeting Destination Vienna, FH Campus Vienna, Austrian Science Fund, Austrian Research Promotion Agency, ZSI - Centre for Social Innovation, Fraunhofer ISI, Institut Francilien Recherche, Innovation et Société, Joanneum Research, the whole fteval Community. 

podcast(at)fteval.at - twitter.com/fteval - www.fteval.at

Henriette Spyra explains how the back then Austrian Federal Ministry for Climate Action implements a transformative Innovation Policy since 2021. RTI is one, but not the only one, policy that contributes to transformative change. Climate neutrality, competitiveness and sovereignty as well as resilience, security and well-being need nowadays a broad innovation approach and system view. The Ministry redesigned and focusses its thematic priorities, introduced new instruments and broadened its innovation community. A new approach towards monitoring, evaluation and learning was key. Examples will be presented on (1) what new approaches and elements have been introduced to better monitor and evaluate the entire portfolio of RTI activities and its interlinkages and (2) how especially the contribution of RTI towards green transition is monitored and evaluated. 

The discussion was moderated by Elke Dall.

Links:

• Federal Ministry of Austria - Innovation, Mobility and Infrastructure (BMIMI): https://www.bmimi.gv.at/en.html 
• Henriette Spyra on Linkedin
• Henriette's presentation slides

Music: Urbana-Metronica (wooh-yeah mix) (ft. Morusque, Jeris, CSoul, Alex Beroza) by spinningmerkaba http://ccmixter.org/people/jlbrock44; SouljaUnit Remix of DeadRobot Music’s Surfy via https://freesound.org/people/SouljaUnit/sounds/640175/
Photo: Martin Jordan

Thanks to our partners and co-organisers that helped realising the REvaluation Conference 2024: Ministry of Innovation, Mobility and Infrastructure, Ministry of Women, Science and Research, Ministry of Economy, Energy and Tourism, Vienna Science and Technology Fund, COST Action PROFEEDBACK (CA ), Meeting Destination Vienna, FH Campus Vienna, Austrian Science Fund, Austrian Research Promotion Agency, ZSI - Centre for Social Innovation, Fraunhofer ISI, Institut Francilien Recherche, Innovation et Société, Joanneum Research, the whole fteval Community.

podcast(at)fteval.at - twitter.com/fteval - www.fteval.at

Peer review is a central instrument for the self-governance of the scientific community and for legitimizing e.g. funding decisions made towards both scolars and society. However, it is intransparent and time-consuming, especially so because more and more publications are produced. Therefore, the need to increase its quality and reliability was often recognized in the past years. Today, additional challenges have emerged such as generative AI, the occurance of predatory journals, paper/reviewer mills, etc. Nevertheless, peer review will most likely remain important for the foreseeable future, exemplified also by CoARA. 

On the REvaluation conference 2024,  the editorial board of the academic journal Research Evaluation organised a panel discussion on this topic. The panel focusses on two main questions: 

• How can peer review address the challenges posed by generative AI tools like Chat GPT?
• How should the relationships between peer review and bibliometrics be reviewed under the new CoARA agreement for open science?

The panelists were 

• Jakob Edler, Executive Director of the Fraunhofer Institute for Systems and Innovation Research ISI-Karlsruhe - DE & Prof. of Manchester University 
• Grit Laudel, Research Associate at TU Technical University of Berlin-DE 
• Nicholas Vonortas, Associate Dean for Research Initiatives - GW Elliott School of International Affairs Washington DC-US, Co-Editor Science and Public Policy)

The panel was chaired by Emanuela Reale (IRCrES CNR-Italy). Further contributions are from Julia Melkers (Center for Organization Research & Design, School of Public Affairs, Arizona State University) and Nicolas Robinson-Garcia (Department of Information and Media Studies, University of Granada).

Music: Urbana-Metronica (wooh-yeah mix) (ft. Morusque, Jeris, CSoul, Alex Beroza) by spinningmerkaba http://ccmixter.org/people/jlbrock44; SouljaUnit Remix of DeadRobot Music’s Surfy via https://freesound.org/people/SouljaUnit/sounds/640175/
Photo: Martin Jordan

Thanks to our partners and co-organisers that helped realising the REvaluation Conference 2024: Ministry of Innovation, Mobility and Infrastructure, Ministry of Women, Science and Research, Ministry of Economy, Energy and Tourism, Vienna Science and Technology Fund, COST Action PROFEEDBACK (CA ), Meeting Destination Vienna, FH Campus Vienna, Austrian Science Fund, Austrian Research Promotion Agency, ZSI - Centre for Social Innovation, Fraunhofer ISI, Institut Francilien Recherche, Innovation et Société, Joanneum Research, the whole fteval Community. 

podcast(at)fteval.at - twitter.com/fteval - www.fteval.at