OASIS: Outil d’Analyse de Systèmes de Surveillance
First developed to evaluate a regional surveillance project in the Global South. Then adapted by ANSES, merging it with surveillance network evaluation tool.
Purpose:
To provide a standardized tool for the assessment of surveillance systems on zoonoses and animal diseases in order to provide direct recommendations to coordinators of the system.
Scope:
Designed to be generic and to evaluate structure and activities of surveillance in animal health, food safety and plant health.
Process:
A participatory process is aimed at facilitating endorsement and application. A detailed questionnaire, divided into 10 sections, is used as a checklist in semi-structured interviews, run by internal and external evaluators. Scoring is done with a scoring guide and validated in a one-day workshop with relevant stakeholders.
Output:
Outputs include 10 pie charts representing the 10 sections, an estimation of critical control points for the operation of the system, and an estimation of 10 evaluation attributes as defined by CDC and WHO to evaluate the quality of the surveillance system.
References:
Hendrikx, P., E. Gay, M. Chazel, F. Moutou, C. Danan, C. Richomme, F. Boue, R. Souillard, F. Gauchard, et B. Dufour. « OASIS: An Assessment Tool of Epidemiological Surveillance Systems in Animal Health and Food Safety ». Epidemiology and Infection 139, no 10 (octobre 2011): 1486‑96. https://doi.org/10.1017/S0950268811000161
Links:
ECoSur: Evaluation of collaboration for surveillance
ECoSur (Evaluation of Collaboration for Surveillance) aims at evaluating the organization, functioning, and functionalities of collaboration taking place in a multisectoral surveillance system.
Purpose:
To assess if collaboration as planned and implemented is relevant and functional to produce the expected collaborative outputs.
Scope:
Designed to be generic and applicable to any multisectoral surveillance systems whatever the hazards/events under surveillance and the level of integration achieved.
Process:
Following desktop review and interviews and using a scoring guide, an evaluation team scores 22 attributes and 3 indexes characterizing the organization of collaboration at the governance and operational level and nine attributes referring to core functions of collaboration to ensure the sustainable operation of an effective multisectoral surveillance system. The interpretation of evaluation results supports the identification of strengths and weaknesses of collaboration and the formulation of recommendations for its amelioration. Results and their interpretation discussed and validated with main stakeholders of the surveillance system.
Output:
Three automatically-generated outputs displayed the evaluation results: one individual pie chart for each of the organizational attributes, a single histogram for the three organizational indexes, and a single spider chart for the nine functional attributes. These outputs are intended to be communicated within a report, together with an explanation to justify scores allocated, strengths and weaknesses of collaboration as well as recommendations for its improvement.
References:
Bordier, Marion, Camille Delavenne, Dung Thuy Thi Nguyen, Flavie Luce Goutard, et Pascal Hendrikx. « One Health Surveillance: A Matrix to Evaluate Multisectoral Collaboration ». Frontiers in Veterinary Science 6 (2019). https://doi.org/10.3389/fvets.2019.00109
Links:
SET: FAO Surveillance Evaluation Tool
Developed by FAO, under funding from USAID under the GHSA project. The tool derives from OASIS in its evaluation core competencies of the surveillance system. The final toolkit includes components from other tools used by FAO such as the Laboratory Mapping Tool (LMT), the Epidemiology Mapping Tool (EMT).
Purpose
Aimed to provide countries with a comprehensive and standardized way to evaluate surveillance and accompany a multi-country improvement effort.
Scope:
Animal disease surveillance systems, including zoonoses
Process
SET missions usually last 10-12 days and can be done either externally, led by FAO staff from outside the country, or as a self-assessment done by local staff trained in the evaluation methodology. Evaluation teams typically consist of 3-10 members from FAO (headquarter, regional, local offices) as well as staff from the local veterinary services and/or laboratories. The team may divide into several groups to cover more area of a country.
Output:
Graphical outputs (radar) on core competencies (19) and performance attributes (10). The outcome of SET evaluation missions include an action plan to implement specific, measurable and timely recommendations to better track diseases that affect both animals and people.
Links:
ATLASS: The FAO Assessment Tool for Laboratory and AMR Surveillance Systems
Developed by FAO and first piloted in 2016. During 2016-2019, 28 countries hosted FAO-ATLASS assessment missions. Regarding the surveillance component, the tool derives from the Surveillance Evaluation Tool (SET), hence from OASIS, whereas the laboratory component tool derives from the Laboratory Mapping Tool (LMT).
Purpose:
Assesses and defines targets to improve national AMR surveillance systems in the food and agriculture sectors. Helps countries with prioritizing actions for building reliable national AMR surveillance systems. Progress monitoring is provided through the ATLASS Progressive Improvement Pathway.
Scope:
Assesses the five main areas for effective surveillance: governance, data collection and analysis, laboratories, communication, and sustainability.
Process:
Includes two standardized questionnaires covering the surveillance and laboratory modules, which are completed by the assessors during a one-week mission. FAO-ATLASS missions are carried out by trained assessors, whether external (recommended for baseline assessment) or national (for progress monitoring). The assessment team usually includes two main assessors focusing on laboratory techniques and AMR surveillance, respectively, in addition to FAO staff.
Output:
Graphical outputs are generated automatically for a synthetic view of the system performance. The Tool generates a baseline and classifies a stage for AMR laboratory capacity detection, AMR surveillance, and information dissemination. Moreover, specific steps for improvement are identified.
Links:
- https://reliefweb.int/sites/reliefweb.int/files/resources/CA1476EN.pdf
- http://www.fao.org/antimicrobial-resistance/resources/tools/fao-atlass/en/
- https://www.researchgate.net/publication/328654946_FAO_Assessment_Tool_for_Laboratories_and_AMR_Surveillance_Systems_ATLASS_for_the_food_and_agriculture_sectors
SERVAL: SuRveillance EVALuation framework
SERVAL was developed by staff at the Royal Veterinary College, the Animal Health and Veterinary Laboratories Agency (now Animal and Plant Health Agency), and the Scottish Agricultural College, UK following a technical workshop of international experts followed by a consultation process involving providers and users of surveillance and evaluation data.
Purpose:
To structure the evaluation of animal health surveillance programmes.
Scope:
Aims to be generic and has been designed to assist in the comprehensive evaluation of single surveillance components (activities) or entire surveillance programmes. It is intended to be flexible so it may be used to evaluate surveillance for any health condition in any species (or group of species).
Process:
Organised in five sections each containing detailed guidance notes and examples to assist the evaluator(s): 1) Define the scope of the evaluation, 2) Characterise the surveillance system to be evaluated, 3) Design the evaluation, 4) Conduct the evaluation, 5) Reporting and communication.
Output:
The output of using the framework is expected to be a written evaluation report that includes details of each of the sections listed below under standardised headings. The contents of each section can be tailored to the needs of each surveillance system and so will vary. The evaluation report should be circulated to affected parties including both those implementing the surveillance activities and those using the outputs.
Links:
SurvTool (RiskSur)
The SurvTool was developed by the RISKSUR project, an EU FP7 funded project running from 2012 to 2015.
Purpose:
Aims to support decision for the design of cost-effective risk-based surveillance systems.
Scope:
RISKSUR developed tools and frameworks targeted at the following surveillance objectives associated with livestock diseases: 1)Detection of incursion of exotic, new (emerging) and re-emerging diseases, 2)Declaration of freedom from specified diseases and infections, 3)Monitoring of endemic diseases (case detection, disease frequency estimation)
Process:
The evaluation (EVA) tool helps in the development of an economic and epidemiological evaluation protocol adapted to their needs. It provides step-by-step guidance on defining the evaluation context, selecting a suitable evaluation question, evaluation attributes, criteria, and methods as well as documentation on the process, methodologies and communication of outputs. The EVA tool takes you through a four-step process to develop your evaluation plan, these steps are: 1)describe the evaluation context, 2)select the evaluation question, 3)select the evaluation method, 4) review summary of evaluation protocol. When planning an evaluation, the user is guided through three main steps: defining the evaluation context; defining the evaluation question; selecting the evaluation attributes and economic criteria.
Output:
Expected outputs of the evaluation are reports dependent on the type of evaluation that is chosen and can include: 1) Surveillance system description report, 2) Evaluation context and Evaluation question summary, 3) List of evaluation attributes included in the protocol, 4) Assessment methods selected to measure the evaluation attributes, 5) Economic techniques selected in case of economic evaluation question.
Links:
SurF: Surveillance Evaluation Framework
Developed for internal use by the Ministry for Primary Industries (MPI) of New Zealand. SurF builds upon and adapts previous work, in particular the SERVAL Framework (Drewe et al. 2015), the European Centre for Disease Control guidelines (ECDC, 2014), as well as a preliminary version of the EVA tool (RiskSur Consortium 2013 & 2015).
Purpose
To provide a common umbrella for surveillance evaluation across MPI biosecurity portfolio. To identify and correct problems, sustain and enhance existing strengths of surveillance.
Scope
Surveillance evaluation in the animal, plant, environment and marine sectors.
Process
Follows four components of an evaluation project: 1) Motivation for the evaluation, 2) Scope of the evaluation, 3) Evaluation design and implementation, and 4) Reporting and communication of evaluation outputs. On the basis of 1 and 2, attributes are selected from a master list and methods are decided.
Output
A visual output that allows for comparison of core performance between systems and within individual systems over time.
Links:
- https://www.researchgate.net/publication/325199416_SurF_an_innovative_framework_in_biosecurity_and_animal_health_surveillance_evaluation
- https://www.mpi.govt.nz/dmsdocument/18091-surveillance-evaluation-framework-surf-main-document
- https://www.mpi.govt.nz/dmsdocument/18094-surveillance-evaluation-framework-surf-appendix-1-surf-methods-catalogue
ISSEP: Integrated surveillance system evaluation project
Developed in 2016 by a team of researchers from Canada and UK in collaboration with the Canadian Integrated Program for Antimicrobial Surveillance Resistance with the aim to guide the structuration of evaluation of integrated surveillance systems for AMR.
Purpose:
To provide a conceptual basis for structuring the evaluation of different surveillance outcomes, from the level of integration to the evaluation of the decisions as well as economic efficiency of integrated surveillance systems for AMR.
Scope:
Evaluation of integrated surveillance systems for AMR and AMU.
Process:
Comprises five evaluation levels that target the evaluation of One Health integration in the surveillance system, of its capacity to produce integrated information and expertise, to generate actionable knowledge, to influence decision-making and the evaluation of its health and economic impacts. For each level, a set of evaluation questions are defined, and links are made with existing evaluation tools. A semi-quantitative scale is proposed to show the level of integration of the surveillance system.
Output:
The output of using the framework is expected to be an evaluation protocol for integrated surveillance systems for AMR.
References:
Aenishaenslin, Cécile, Barbara Häsler, André Ravel, Jane Parmley, Katharina Stärk, et David Buckeridge. « Evidence needed for antimicrobial resistance surveillance systems ». Bulletin of the World Health Organization 97, no 4 (1 avril 2019): 283‑89. https://doi.org/10.2471/BLT.18.218917.
PMP-AMR: The FAO Progressive Management Pathway for AMR
The Progressive Management Pathway for AMR was developed by FAO, based on the FAO Action Plan on AMR.
Purpose:
Aims to provide guidance to countries for developing and operationalizing their multi-sector ‘One-Health’ National Action Plans (NAP) from a stepwise approach. Meant to be a complement to other assessment tools such as the Tripartite country self-assessment questionnaire, or the Joint External Evaluation.
Scope:
Focus on the food and agriculture sectors. Covers activities categorized under 4 focus areas: awareness, evidence, governance and practices.
Process:
Self-assessment tool, Excel-based PMP assessment tool, which lists a number of actions to be taken or achievements to be made to make progress with managing AMR risk. Linked to the ATLASS tool for many aspects of surveillance evaluation, which in turn is based on the Surveillance Evaluation tool (derived from OASIS) and Laboratory Mapping Tool.
Output:
Individual scores or stages for each of the 4 focus areas and for each sector. Assessment report template provided with the tool.
Links:
International Health Regulations (IHR) (2005)-IHR Core Capacity Monitoring Framework
The International Health Regulations, or IHR (2005), represent an agreement between 196 countries including all WHO Member States to work together for global health security. The IHR Core Capacity Monitoring Framework was developed by the Secretariat, with a checklist and indicators to monitor progress in the development of the core capacities.
Purpose:
Through IHR, countries have agreed to build their capacities to detect, assess and report public health events. WHO plays the coordinating role in IHR and, together with its partners, helps countries to build capacities.
Scope:
In order to assist the States Parties in their responsibility to report to the Assembly, the IHR Secretariat has developed a data collection tool which will enable each State Party to provide standardized information about the progress of its core capabilities.
Process:
This data collection tool is designed primarily for use by National IHR Focal Points (NFPs) in collaboration with public health professionals, managers and other sectors and stakeholders responsible for implementing the IHR. The questionnaire is divided into thirteen sections, one for each of the eight core capacities, Point of Entry (PoE) and four hazards. Individual questions are grouped by Components and Indicators in the questionnaires. The completed questionnaires are submitted to IHR secretariat.
Output:
The IHR Secretariat prepares an annual report for the World Health Assembly detailing WHO and States Parties progress on IHR implementation.
Links:
JEE: Joint External Evaluation tool (2nd edition)
Developed by WHO, in the application of the IHR (2005) which requires State Parties to develop minimum core public health capacities, with a need for transparency. It draws on the original IHR core capacities and incorporates lessons learnt from various external assessments.
Purpose:
To measure country-specific status and progress in achieving the targets.
To allow countries to prioritize needs and opportunities for enhanced preparedness, response and action, and to engage with donors and partners to target resources effectively.
Scope:
Evaluation of the capacity to prevent, detect, and respond to public health threats defined by IHR (2005).
Process:
On the basis of voluntary country participation, national experts first prepare a self-assessment supplied to the external team prior to the on-site visit. The external team uses the same tool for its independent evaluation, working together with the national team in interactive sessions. Indicators with attributes that reflect levels of capacity with scores of 1 to 5.
Output:
Report identifying status levels for each indicator, as well as an analysis of the country’s capabilities, gaps, opportunities and challenges.
Links:
OH-APP: One Health Assessment for Planning and Performance
The One Health Assessment for Planning and Performance (OH-APP) was developed by USAID. It is a monitoring framework for multisectoral coordination mechanisms (MCM) to self-assess their organizational capacity and performance to inform planning and development assistance.
Purpose:
To assess multisectoral coordination mechanisms (MCM) maturity and provide data for decision-making that would enhance organizational capacity and OH performance
Scope:
Most appropriate for countries who have both: 1) Completed the WHO IHR Joint External Evaluation, and 2) an existing government ministry, department, or agency with a multisectoral coordination mechanism to respond to zoonoses or potential zoonoses, AMR, IHR coordination.
Process:
Administered during a two-day workshop with stakeholders from appropriate One Health ministries. The process for undertaking an OH-APP includes three phases: Phase 1: Pre-Workshop In this phase, a workshop facilitator and participants from the relevant sectors are identified. Relevant documents, strategies and plans are collected to inform the assessment during the workshop. Phase 2: OH-APP Workshop Participants engage in a participatory self-assessment of their MCM’s organizational capacity and performance. Scores and their rationale are entered in real-time into the online OH-APP. Data visualizations produced by the OH-APP are interpreted by participants and used to inform prioritization and planning of key actions to be taken by the MCM over the coming year. Phase 3: Following the Workshop.
Output:
A report produced by the OH-APP, including assessment scores, visualizations and agreed actions, that is used to support MCM annual planning and development assistance.
Links:
PVS: OIE Tool for the Evaluation of Performance of Veterinary Services
Developed by the OIE, it covers the evaluation of the application of standards and guidelines as published and promoted by this organisation among its Member countries. The version issued in 2019 represents the seventh edition.
Purpose:
Provides the methodological basis for the continuous evaluation and improvements of national Veterinary Services, as facilitated by OIE under PVS Pathway in its Member countries.
Scope:
Provides a complete overview of how the national Veterinary Services’ are structures and how they operate in order to protect and improve animal and veterinary public health in the country (e.g. governance mechanisms, disease surveillance programmes, food safety control measures, laboratory capacity, and other regulatory activities in place).
Process:
Only OIE trained PVS experts can carry out OIE PVS Pathway missions. All experts use the abovementioned Tool and an OIE PVS Experts’ manual.
Output:
A narrative report, describing the level of advancement of the Veterinary Services along 45 critical competencies, categorised into 4 fundamental components: I. Human, Physical and Financial Resources, II. Technical Authority and Capability, III. Interaction with Stakeholders, IV. Access to Markets. For each competency, the level of advancement is rated between 1 and 5; corresponding findings, weaknesses, strengths, and recommendations are also highlighted.
Links:
NEOH Evaluation Framework
The NEOH framework was developed by the “Network for Evaluation of One Health” (EU COST Action (TD1404)). It is conceptualised to compare the effort conducted to integrate knowledge (One Health-ness) to the outcomes anticipated by the theory of change and the unexpected outcomes. The tool used in this study only assesses the effort for knowledge integration.
Purpose:
Assesses the extent to which the six aspects of knowledge integration are implemented: systems thinking, planning, transdisciplinary working, sharing, learning and systemic organisation.
Scope:
Conceived to assess initiatives employing integrated approaches to health in general.
Process:
The evaluation framework follows a systems approach to evaluation and captures the stakeholders’ perspectives as a first step. With this knowledge, the tool can be used consisting of approximately 15 questions for each aspect implemented in Excel. The questions are scored between 0 and 1 according to a scoring guide and the justification of the score can be noted. Evaluators can be internal and/or external.
Output:
The median score of each aspect is plotted in a spider diagram. The surface corresponds to the One Health index. The surface covered by operational aspects (systems thinking, planning, transdisciplinary working) is divided by the supporting infrastructure aspects (sharing, learning, systemic organisation) to compute the One Health ratio. The diagram, index and ratio allow identification of gaps and annotations enrich the interpretation.
Links:
OH-EpiCap: One Health Epidemiological surveillance capacities and Capabilities
Purpose:
Scope:
Process:
Output:
Links:
OHE-AMURS: One Health Evaluation of Antimicrobial Use and Resistance Surveillance
Purpose:
Scope:
Process:
Output:
Links: