Ten questions to ask if you're writing a scrutiny report #4: What's the evidence?

It’s a big word in scrutiny.  We all know that our recommendations should be evidence-based.  But what does that phrase actually mean?

 ■   What is evidence?

Evidence is the facts or sources that support an argument. In a write-up of a scientific experiment, evidence would be the data you collected in your experiment that prove or disprove a thesis. In a scrutiny report, evidence is whatever allows you to prove or disprove your hypothesis. 

 

 

■   Why do we need evidence?

Evidence is central to any argument. Without evidence, an argument is merely an opinion. Evidence is the critical link that helps you prove your points.

■   Can evidence speak for itself?

No. Evidence can only speak in support of a point you’re making. It’s never enough to drop a quotation into a report and expect the reader to be convinced of a point.

■   So how can we use evidence effectively?

When you use evidence, you have to show the reader that the evidence supports your argument.  Explain what the evidence means.  How does the evidence support the larger ideas in the report?  Make the connections clear.

You can’t even begin to look for evidence until you have a hypothesis.  What are you looking for evidence of?  Look at the hypothesis and ask:

■   What would prove or disprove this?

■   Where would we look to find evidence of the proof?

■   How would we argue that the evidence supports our argument?

It’s well known that evidence can be quantitative or qualitative.

■   Quantitative evidence is based on numerical data or data that can be converted into numbers.  Quantitative evidence usually seeks to answer questions like:

‘How many?’, ‘how much?’, ‘How often?’ or ‘To what extent?’  

 

■   Qualitative evidence is based on non-numerical data.  We gather such evidence to see the processes which lie behind patterns of behaviour, people’s emotions or their responses to certain situations.  Qualitative evidence helps us to understand the experiences of service users (or those who don’t use services). 

Typically, it shows us ‘what it’s like’: we can use it to indicate what something means – what impact it has on people – rather than how much it measures.

One way of structuring the search for evidence is to categorise it by source (internal/external) and by type (quantitative/qualitative).  Ideally, scrutiny should be able to draw evidence of all four types:

Internal/quantitative    Internal/qualitative

External/quantitative    External/qualitative

The source of the evidence is of course as critical to your argument as the nature of the evidence.  No evidence is ever truly objective; how could it be, when all evidence by definition is in support of an argument?

Many different individuals and groups will provide evidence for your reviews.  These will include:

■   politicians;

■   council officers;

■   managers;

■   non-executive directors;

■   professionals;

■   members of the public; and

■   academics.

Most of these people are likely to be submitting evidence as part of their professional work.  They will (probably) try to be as factual and objective as possible, while representing the interests of their organisations. 

You’ll sometimes become aware of political or personal motivations in the people offering evidence to you.  A personal or political motivation may be entirely legitimate: a patient may wish to pass on their own experiences to improve the experiences of others; a politician may have made an election commitment to promote the needs of a particular group. 

Ask whether a witness’s position might be inappropriately biasing their evidence.  How might their position influence the presentation of evidence?  Is there any conflict of interest?  A representative of a commercial company, for example,  might be using evidence to support the sales of the company’s products. An opposition politician may deliberately seek out and emphasise negative service users’ experiences in an attempt to embarrass the ruling group on a council. 

Dr Fiona Campbell has written an excellent booklet exploring the questions surrounding evidence.  Download ‘Ten questions to ask if you’re assessing evidence’ to find out more.

Hot tip

Keep asking: “What point are we arguing?”

 Evidence is only evidence if it supports an argument.  It’s up to you to decide whether evidence fairly supports an argument. 

In the next posting, I'll address the question: 'How do we structure the report?'

If you like what you see here, you might like to contact me to discuss working with you.  I am currently working with one of the scrutiny team of a major local assembly in the UK.  I run training courses, and coach individual writers.  I can even give you some feedback on the reports your team is producing, if you want nothing more.  Go to my website to take a look at a sample training programme.