We used participatory action research to develop and refine the Monitoring & Evaluation (M&E) framework.

Participatory approach
A multi-stakeholder working group consisting of representatives of four European patient organisations, 15 biopharmaceutical companies and two academic institutions was created This working group was tasked with:

• Reviewing the literature on M&E of patient engagement;
• Development and testing of a M&E framework;
• Identification and selection of appropriate metrics for M&E;
•  Clarification of terminology and language to be used.

The working group provided advice on the study design, feedback on documents and versions of the framework, co-analysed case study data, were involved in writing publications and in other dissemination activities. All working group members had an equal say in the framework development process.
The framework development process included three distinct phases: (1) design phase, (2) testing phase, (3) consensus and alignment phase. Each of these involved a different methodological approach described below.

Phase 1: Framework design
The aim of the design phase was to develop early versions of the M&E framework by identifying (1) impacts previously reported from patient engagement initiatives, (2) the conditions needed to achieve these impacts, (3) suggested metrics for M&E of patient engagement. A scoping literature review, three key informant interviews and six test case studies with PARADIGM partners were conducted. Informed by these primary analyses, researchers developed an early version of the framework. This draft framework was then reviewed by PARADIGM partners, adapted and subsequently entered the test phase.

Phase 2: Framework applicability test
The aim of the test phase was to apply the draft framework to real-world patient engagement initiatives in the context of medicines development. We used the M&E framework to validate the identified (sub)components and to select and test suggested metrics in practice. In total, 24 patient engagement initiatives were included as cases. Case study contributors were asked to describe their patient engagement initiative per component of the M&E framework and to select appropriate metrics. Reflection meetings were held between researchers and the case contributors to discuss the framework, metrics and their applicability. A ‘tailored’ M&E framework for each case was developed using an iterative approach. The research team co-analysed and integrated metrics from all cases. Furthermore, the results and applicability of the framework was reviewed by the working group and any changes that derived from the case studies were discussed during the consensus and alignment phase.

Phase 3: Framework consensus and alignment
The aim this phase was to build consensus on the framework and develop agreed ‘sets of metrics’ that align with objectives of patient engagement. An online consensus-building workshop was held with all working group members to develop these sets in multi-stakeholder groups. Next, the researchers identified possible measurement methods from the literature review and those employed in the case studies, which were mapped to the sets of metrics. Remote meetings were held with the multi-stakeholder working group to reach consensus on the final framework, including (sub)components, (sets of) metrics and measurement methods.