New HLF Evaluation Guidance Aug 17

4 replies [Jump to last post]

I'm commenting on the new evaluation guidance, specifically the recommended budget allocations.  My experience is with LPS but I think my comments would be relevant to other grant streams too. 

The previous guidance (2014) stated that HLF grant of evaluation was “limited to 1% of project costs for grants of £2m or more, and up to 3% for grants below £2m”.

The new guidance states on p19 that we should spend;

“£1m+ projects Spend on evaluation: 7-10% of total project costs and consider use of external supplier.

Use of baseline: Development of baselines & tracking against them

Acceptable Methodologies: Advanced qualitative and quantitative research

Outcomes: Focus on all levels of objectives”

(for £250k - £1m guidance is 5-7%)

NB the guidance is for a %age of total project costs, not total grant.

So the new guidance represents an increase in evaluation budgets by an order of magnitude. For a typical £2m+ LPS this would give an evaluation budget of ~ £200k or even more.  I have no idea how one could possibly spend this much. The evaluation consultant we used on our last LPS, who has done a lot of work for HLF projects, has no idea either (though he’d be very happy to get the cheque!). I'm not even sure there is the consultant capacity to deliver a 10x increase in evaluation work.

Use of an external supplier is only to be “considered”…  without using an external supplier you’d be hard pushed to spend even a fraction of £200k, and anyway for a £1m+ project I’m pretty sure HLF would require external scrutiny, they certainly ought to.

Thinking of the organisations and individuals on a typical Partnership Board, particularly community and Local Authority representatives, I think they’d be very unimpressed indeed at this level of expenditure on evaluation.  They would question whether this was an appropriate use of public money and whether a project spending this much on evaluation was deserving of their support. I know this to be the case because I’ve had informal conversations with a few such people who were on the old TtT Board.

This change adds 10% to the costs of a project at a stroke, and the extra is all overheads.  Evaluation is not something that a project partner is going to explicitly give partnership funding towards. So 10% of the budget going to evaluation will also mean a 10% reduction in the percentage of match funding and a 10% increase in the % of HLF grant.  We know that HLF doesn’t often support projects asking for grant at the maximum possible %, so how will this step change in evaluation costs be considered?

The rest of the new guidance is all helpful, and I hope no-one needs convincing of the importance of evaluating projects (during as well as after) and sharing the results. The guidance does not, however, address a key issue for evaluation which is which is how to disseminate and share the results – even more important if you’re going to spend £0.25m on them. 

Increasing the recommended evaluation budget by an order of magnitude without explanation of either the rationale for doing so, or an idea of how to spend so much money on this, is unhelpful and frankly a bit baffling.

HLF, please explain!  Thank you.

View Bill Jenman's profile Bill Jenman Oct 12 2017 - 12:38pm
  1. Dear Bill,

    Thanks for your comment. Landscape partnerships as a targeted programme has its own guidance here: https://www.hlf.org.uk/landscape-partnerships-evaluation-guidance

    Within that on p.27 we talk about further sources of help on evaluation one source of which we cite as:

    7.1 HLF’s ‘Evaluation: Good-practice guidance’ www.hlf.org.uk/HowToApply/goodpractice/Pages/Evaluation_guidance.aspx - which has been updated. But we also state: This guidance is aimed at Heritage Grants applicants.

    So you if you are working on a LP project you should continue using its guidance.

    I joined HLF in May 2016 and set a new evaluation strategy for 2017-18 a main objective of which is to: Raise the Quality of Project Self-Evaluations. The reason for updating the Heritage Grants guidance simply signals the greater importance we place on evaluation in our main open programme.

    Based on evidence from a review of 200 Heritage Grants projects submitting self-evaluations we rated 25% as poor. See the report here:

    https://www.hlf.org.uk/heritage-grants-programme-evaluation-and-outcomes…

    From the findings we know that budgeting adequately and commissioning an external evaluator are directly correlated with a higher quality self-evaluation. That is why we have updated the HG guidance raising the importance we place on budgeting by increasing recommended amounts and by endorsing the use of external evaluators where feasible.

    Although no government advice or rules of thumb about grant percentage for evaluation are to be found in the guidance from HM Treasury or the NAO, there are plenty of international references which indicate typical evaluation budgets of between 5 and 15% are standard for a wide range of government and private sector funders. For example, The Big Lottery Fund encourages projects to allocate up to 10% of their budget towards monitoring and evaluation.  

    This may include time and resources needed for:   

    ·         planning and managing the evaluation

    ·         attending training to up-skill staff in evaluation methods 

    ·         collecting data and other information about the project and its outcomes 

    ·         communicating the findings.

    Logically projects with more resources should be able to carry out a more extensive piece of work. So for instance a £2m+ project would need dedicated staff planning and managing the evaluation as well as collecting evidence for the duration, which could be a number of years. This data and evidence would need to be shared with an external evaluator and an independent evaluation commissioned etc. In terms of staff costs and commissioning costs this would equate to a reasonable investment in order for a high quality final evaluation to be produced. I randomly reviewed 100 applications to our Heritage Grants programme and 40% had zero budget for evaluation included. If HLF is not emphasising the importance of evaluation then it is fair to assume projects will not see it as important.

    Evaluation is a crucial part of a project and the higher quality the final evaluation is the better source material HLF will have to do a programme evaluation. Our programme evaluations have to robustly demonstrate the value of our grant programmes for heritage, people and communities so in fact it is a very appropriate use of public money.

    I would recommend all self-evaluations are published by our projects. HLF has started sharing examples of excellent evaluation reports for the benefit of all projects.

    Kind regards,

    Kion

  2. View Kion Ahadi's profile Kion Ahadi
    Offline | Last seen: 3 days 20 hours ago
  3. in reply to

    Ok, so it did time out. Sorry for the test message.

    Thanks for the explanation, Kion – now I understand the thinking behind the new guidance and I’m shocked to hear how poor quite a lot of evaluations have been.

    I’m unable to find any reference to the new guidance being for Heritage Grants specifically in the document – please could you point me to where it states this? I presume that if it doesn’t apply to LPS it doesn’t apply to eg Our Heritage either? Most HLF guidance applies across programmes so that’s the default assumption unless it is clearly stated otherwise.

    You also state that the 7-10% figure includes all the staff time for planning and managing the evaluation, collecting data, communicating the findings, and any associated training. With that in mind, the suggested now % seems about right to me; we spent 2.5% of the budget on external evaluation costs (incl dissemination of results) in our recent LPS but we easily spent twice that again in staff time.  As a guide to expected overall effort it’s helpful.

    However the guidance talks about share of budget, not share of effort. In the application form for Heritage Grants (and LPS – the budget tables are identical) Evaluation is clearly a separate budget line from the ones for Staff Costs and Staff Training. The natural interpretation is therefore that when the guidance says that the share of budget for evaluation should be 7-10% of the total project costs that it means that this budget line should be 7-10% of the total project costs.  That’s how everyone I’ve talked to has interpreted the guidance, hence the bafflement.

    I wholeheartedly endorse and applaud HLF’s emphasis on good evaluation, but the guidance as written is extremely confusing when read in conjunction with the application form. Respectfully, I suggest it needs rewriting or a lot of applicants, probably most applicants, will interpret the “recommended share of the budget” for evaluation as being, well, the recommended share of the Budget for evaluation on the application form.

  4. View Bill Jenman's profile Bill Jenman
    Offline | Last seen: 19 hours 24 min ago
  5. Hi Bill,

    Thank you for your feedback which I really value. We have made it clearer on the webpage below and in the introduction of the guidance what parts are relevant to which programmes:

    https://www.hlf.org.uk/evaluation-guidance

    Applicants to HG fill out the application form in different ways. We allow a description box against each cost heading in the form and the stronger applications use this effectively and cross reference it with other sections. For example, as part of a £1m project we could see a cost of £100k in evaluation with a breakdown of what this is being used for. Another application might put £20k under the evaluation heading with the description saying external evaluation cost of consultant, and have other evaluation costs under staff costs where a proportion of that is spent on evaluation etc. Either way the message from the guidance is the same: We want projects to budget more carefully for evaluation. One size won’t fit all, but the amount of budget or share of effort etc. is essentially the same thing.

    We want projects to appreciate the importance we place on high quality self-evaluation and we have tried to empower them in this through the budget increases cited in the guidance and application form for Heritage Grants.

    We appreciate feedback and will always endeavor to incorporate learning into our programmes and guidance.

    Best wishes,

    Kion  

  6. View Kion Ahadi's profile Kion Ahadi
    Offline | Last seen: 3 days 20 hours ago
  7. in reply to

    Thanks for that Kion.

    There does seem to be some confusion about what goes where in the new budget forms, but as long as all the activities you've mentioned are covered somewhere on the form hopefully applicants will be alright! 

     

  8. View Bill Jenman's profile Bill Jenman
    Offline | Last seen: 19 hours 24 min ago
Back to top of page