Evaluation

6 replies [Jump to last post]

Hi all, we're putting together a bid for a Heritage Room in our Victorian building. We hope to have interactive displays, oral histories, etc, telling people our story - how we began as the 'School of Stitchery and Lace' -a pioneer in teaching girls with disabilities a trade -to our work with over 100 people with learning disabilities today.

Has anyone had experience of evalutating a museum type experience like this?

Has anyone received useful comments on their application about evalutation?

thanks

View Gill Caldwell's profile Gill Caldwell Jul 31 2017 - 10:46am
  1. Hi Gill,

    I'm sure you have seen this but just in case you haven't, HLF did a live chat which fouced on Evaluation and some tips from HLF:https://www.hlf.org.uk/community/live-chats/evaluation-13-april-2017  I hope you find this resource useful.

    Michelle

  2. View Michelle O'Neill-Kiddie's profile Michelle O'Neil...
    Offline | Last seen: 8 months 3 weeks ago
  3. in reply to

    Just seen your reply - thanks so much 

  4. View Gill Caldwell's profile Gill Caldwell
    Offline | Last seen: 6 months 2 days ago
  5. Hi Gill, 

    Sounds like an interesting project. What exactly is it you want advice on?

    What is it you want to find out about from the evaluation? Or is it help in defining the aims that you seek? 

    From what you have said, I wonder if you want to know how successful the various interpretation methods have been in communicating the story of your institution? I did something similar when evaluating the Rothschild Room at the Natural History Museum in Tring. In that evaluation, I conducted tracking to find out what exhibits visitors engaged with and did not engage with, what behaviours they exhibited, their dwell time etc. I then conducted exit interviews with visitors leaving the room to explore: what they thought of the experience and what they learnt from it, including some specific questions around the intended learning outcomes we had for the room. 

    Other possible ways of exploring the role of the interpretation methods, would be to conduct observations and interviews at specific exhibits. For example, you could observe people engaging with the oral histories and then ask them a couple of questions about it. 

    Hope this helps. Please let me know if you want more information about this or have any further questions. 

    Best wishes, Marie

     

  6. View Marie Hobson's profile Marie Hobson
    Offline | Last seen: 8 months 2 weeks ago
  7. in reply to

    Thak you Marie - that's really useful practical advice.

     

  8. View Gill Caldwell's profile Gill Caldwell
    Offline | Last seen: 6 months 2 days ago
  9. I'm commenting on the new evaluation guidance, specifically the recommended budget allocations.  My experience is with LPS but I think my comments would be relevant to other grant streams too. 

    The previous guidance (2014) stated that HLF grant of evaluation was “limited to 1% of project costs for grants of £2m or more, and up to 3% for grants below £2m”.

    The new guidance states on p19 that we should spend;

    “£1m+ projects Spend on evaluation: 7-10% of total project costs and consider use of external supplier.

    Use of baseline: Development of baselines & tracking against them

    Acceptable Methodologies: Advanced qualitative and quantitative research

    Outcomes: Focus on all levels of objectives”

    (for £250k - £1m guidance is 5-7%)

    NB the guidance is for a %age of total project costs, not total grant.

    So the new guidance represents an increase in evaluation budgets by an order of magnitude. For a typical £2m+ LPS this would give an evaluation budget of ~ £200k or even more.  I have no idea how one could possibly spend this much. The evaluation consultant we used on our last LPS, who has done a lot of work for HLF projects, has no idea either (though he’d be very happy to get the cheque!). I'm not even sure there is the consultant capacity to deliver a 10x increase in evaluation work.

    Use of an external supplier is only to be “considered”…  without using an external supplier you’d be hard pushed to spend even a fraction of £200k, and anyway for a £1m+ project I’m pretty sure HLF would require external scrutiny, they certainly ought to.

    Thinking of the organisations and individuals on a typical Partnership Board, particularly community and Local Authority representatives, I think they’d be very unimpressed indeed at this level of expenditure on evaluation.  They would question whether this was an appropriate use of public money and whether a project spending this much on evaluation was deserving of their support. I know this to be the case because I’ve had informal conversations with a few such people who were on the old TtT Board.

    This change adds 10% to the costs of a project at a stroke, and the extra is all overheads.  Evaluation is not something that a project partner is going to explicitly give partnership funding towards. So 10% of the budget going to evaluation will also mean a 10% reduction in the percentage of match funding and a 10% increase in the % of HLF grant.  We know that HLF doesn’t often support projects asking for grant at the maximum possible %, so how will this step change in evaluation costs be considered?

    The rest of the new guidance is all helpful, and I hope no-one needs convincing of the importance of evaluating projects (during as well as after) and sharing the results. The guidance does not, however, address a key issue for evaluation which is which is how to disseminate and share the results – even more important if you’re going to spend £0.25m on them. 

    Increasing the recommended evaluation budget by an order of magnitude without explanation of either the rationale for doing so, or an idea of how to spend so much money on this, is unhelpful and frankly a bit baffling.

    HLF, please explain!  Thank you.

  10. View Bill Jenman's profile Bill Jenman
    Offline | Last seen: 2 weeks 6 days ago
  11. in reply to

    For the benefit of others - Bill also posted the above comment in the Landscape Partnerships forum, where it's been answered.

    In short, Landscape Partnerships, as a targeted programme, has its own guidance here: https://www.hlf.org.uk/landscape-partnerships-evaluation-guidance, so anyone working on an LP project you should continue using that guidance.

    Thanks

  12. View Amy Freeborn's profile Amy Freeborn
    Online | Last seen: 7 min 52 sec ago
Back to top of page