RSS conference 2022: How does GPPS stay robust during times of rapid change?

The Royal Statistical Society (RSS), is a leading international organisation advocating for the importance of statistics and data, and working with statisticians and data analysts across the globe. As part of their annual international conference, earlier this year, I presented on the way the GP Patient Survey (GPPS) has managed to maintain the collection of robust and useful data, in a complex and rapidly changing landscape.

Changing context

GPPS was launched in 2006, and there have been drastic changes in the way primary care is delivered, accessed and evaluated since then. In the last five years, some key changes have included:

• the GP Forward view, NHS Long-term plan and new GP contract, which have all moved towards more multi-disciplinary and integrated working, particularly for out-of-hours care
• the COVID-19 pandemic, which has led to a move towards more care being delivered remotely, and new policies of interest (such as shielding)
• the Health and Care Act 2022, which has restructured NHS commissioning, by abolishing Clinical Commissioning Groups (CCGs) and replacing them with Integrated Care Boards or Systems (ICBs/ICSs).

In addition, our world is becoming more online, with even the latest census offered as online first. With increasing cost of living, encouraging participants to take part online makes the survey more cost effective, particularly in a wider context of decreasing response rates across survey research.

All of these have meant that the GP Patient Survey has had to change, in order to keep up-to-date and ensure it can still collect data of relevance to the health and care system. However, as an Official Statistic that is used extensively, including for healthcare regulation and policy measurement, it is vital that changes are made with understanding of their implications for trends, response rates, non-response bias, and data quality.

Changing the questionnaire

When changing the questionnaire, three key stages are followed:

• Consultation: This can include talking to key stakeholders and our steering group, as well as potentially conducting primary research with patients, to inform our questions and ensure they cover areas that are important to people using and delivering primary care services.
• Cognitive testing: This involves testing the questions with eligible users, to understand the reliability and validity of the questions, ensuring they are designed in a way that ensures participants consistently understand the question the way we intend. (For more details, see a detailed summary of cognitive testing on the Ipsos website).
• Analysis: To maintain trends following a change, if appropriate, we can conduct analysis to understand the impact of context effects on trends. This ensures that changes to the surrounding questions do not have an unintentional impact on how participants think about questions that remain unchanged.

Sometimes, this does mean we need to break trends. If this is the case, it’s important to have clear communication, particularly with stakeholders and data users, to ensure they understand as early as possible where trends are broken. We also make sure we clearly document any breaks in trends and what they mean for data analysis within our technical documents and FAQs, to ensure analysts in the future are able to use the data accurately.

Some recent changes to the questionnaire over the last few years have included asking about appointments somewhere other than the patient’s GP practice and with providers other than a GP or nurse, asking about remote appointments and triage, and reviewing our demographics, including a new question on long-COVID.

Changing the methodology

Almost every year, GPPS has tested new methods, to ensure we can try and improve the data quality and impact of our data, while making the survey as cost-effective as possible. In the last five years, the main focus has been on encouraging participants to take part online, with this proportion increasing from 6.0% of respondents in 2017 to 40.7% in 2022, without impacting trends or non-response bias. This has been done through updating materials and replacing a postcard reminder with SMS reminders.

In order to ensure these changes are made robustly, the interventions are informed by a review of the literature, engagement with other surveys, and building on learnings from previous experiments. The intervention is then tested on a subset of the main sample, to ensure we fully understand the implications before a decision is made on whether it is rolled out for everyone.

Changing the reporting

As GPPS is used by a wide variety of data users, with varying priorities, time, and data literacy, the main challenge is ensuring we can provide data in accessible and user friendly formats. Each year, we review the outputs, using feedback from users and the steering group, to understand where we can tweak our outputs to better meet those needs. When thinking about how to share the findings, we think about a wide variety of areas, including:

• What the data is telling us
• Changes to NHS reporting structures
• Accessibility and Government Digital Service requirements
• Ways we can make the data easier to use
• The data people want
• Policy priorities
• Ways we can help the data have impact

This year, we are conducting a more detailed Impact and Output review, which includes three rounds of depth interviews with National stakeholders (such as government departments and arm’s length bodies), regional stakeholders (such as regions, ICSs, and PCNs), and practice stakeholders (such as GPs and PPGs). This will cover how participants currently use the data, how they would like to use the data, and changes in data requirements expected in the coming years.

Next steps

As we move forward, we are currently finalising the questionnaire for 2023 fieldwork, with minimal changes to the questionnaire and no change to the method compared with last year, to ensure we can continue to measure trends overtime. However, a larger change, based on findings to methods experiments over the last two years, and to the questionnaire, to match the changes in the way care is now delivered, is being explored over the next few years.

For reporting, the results of the Impact and Output review will feed into a review of the reporting outputs, to make sure that they meet data user needs.

Feedback

We are always looking for ways to improve the data we share, so if you have any feedback, please get in touch with the GP Patient Survey team at GPPatientSurvey@ipsos.com.

Technical information

The survey received 719,137 responses from patients aged 16 years or over who are registered with a GP practice in England. The survey has a national response rate of 29.1%.

Minor changes were made to the questionnaire in 2022 to ensure that it continued to reflect how primary care services are delivered and how patients experience them. This followed more substantial changes in 2021. Where questions have changed significantly in 2022 or 2021, comparisons have not been made to previous years. Further methodological information is available in the Technical Annex.

Please note that fieldwork for the survey (the period in which surveys are sent and returned) is undertaken from approximately January to March in each respective year. The 2022 survey was completed from 10 January through to 11 April 2022.

More information

For more information, please contact the GP Patient Survey team.

For more information about GPPS, visit the ‘About’ page on the survey website.