Skills for the Future Evaluation

20 replies [Jump to last post]

Skills for the Future evaluation is crucial to the sector’s development.

By evaluation we mean ‘a process of thinking back, in a structured way, on what has worked and why, as your project progresses and reaches completion.’  

All too often it’s left until the end of the project, but good evaluation is built into a project from the beginning and informs the day-to-day delivery as well as the lessons to be passed on to the  sector.  In thinking about their evaluation grantees often ask us for advice on:

  • what to measure;
  • who to work with;
  • how to write an evaluation brief.

Some projects will be evaluated by staff in your own organisation. Depending on the scale and how complicated your project is, you may need to employ somebody to help and assess whether the aims of the project are being successfully met. We have produced general evaluation guidance  covering both scenarios.

Skills for the Future project managers have offered these tips in recent practice-sharing workshops:

  • If you are using outside help procure early so that the contractor can collect information as the project progresses and you can feed the findings into the project.
  • Consider whether you want the contractor to evaluate the project or to set up a system for you to evaluate it. It might be an independent view you are looking for or you might want to bring in new research skills into your team. Be clear about the input you need.
  • Use creative methods to evaluate the impact on trainees – video for example. There can be elements of their development which are difficult to capture, eg soft skills.
  • Ensure your evaluation captures wider impact/incidental impact for your organisation and the impact on the supervisors and other staff affected, including any parners outside your organisation.

Your evaluation should seek to measure the project against the Skills for the Future programme priorities as interpreted in your original application:

  • Increase the range and quality of work-based training to develop skills in the heritage sector;
  • Meet identified skills gaps or shortages in the heritage sector;
  • Increase the capacity of the sector to deliver training and share good practice; and
  • Increase the diversity of the heritage workforce.

What measures have you used to judge the success of your project?

Do you have more tips you can add to our list?

Are you willing to share an example of an effective commissioning brief or an evaluation document? You can upload documents here or link us to another website.

View Nick Randell's profile Nick Randell Nov 4 2014 - 6:13pm
  1. Good advice Nick.  Evaluation is best planned at the START of your project, that way you can ensure you capture all the good (and not so good) things as they happen and you learn from practice as you go along, making the project better and better.   I would only add, don't make getting feedback a life's work - it can easily become too burdensome - questionnaires are such a pain to analyse that they often just sit in filing cabinets.  How about making more use of new technologies so that feedback is fun and immediate?  I know there are lots of good examples of how to go about this in projects I am mentoring - please share your experiences.

  2. View Sharon Goddard's profile Sharon Goddard
    Offline | Last seen: 2 months 2 weeks ago
  3. Yes, evaluation of any project should be ongoing from start to completion. It helps to capture what is happening  at that moment in time. If there is a need to change anything, it can be done there and then. One of the key elements of ongoing evaluation that I use, whenever a trainee has attended a training activity they must complete a feedback. One of the questions is 'how will you use your learning back at your host museum'. This helps to see if the learning activity was useful. 

    When the trainees submit their quarterly reports, it helps me to see if there's been any progression, this helps with the end of project evaluation. The evaluator can see the improvements and skills gained from completed reports and feedback forms. 

    It's important to use the Skills For The Future outcomes as part of the day to day ongoiung evalution, its automatcially running through your project. 

  4. View Paulette Francis-Green's profile Paulette Franci...
    Offline | Last seen: 2 years 2 weeks ago
  5. Good idea Paulette,

    I see the capture of that kind of information as part of my role as project Manager. certianly when out on windy sites, ensurimng that dry stone walling trainees are getting the work place training they need, it has been very useful to have some set questions ready for them to think about and answer.

    It was also very useful to set up a midway evaluation, this gave us some useful pointers to enable us to refine and keep on track. It is being used as a template for the final report and even though we have a new researcher the systme set up 2 years ago is proving very helpful.

    The good news is that most trainees are moving into employment with strong work place training standing them in good stead.

    I should have a report to share soon.

    Lesley

  6. View Lesley Silvera's profile Lesley Silvera
    Offline | Last seen: 1 year 11 months ago
  7. Great to hear the comments and advice about evaluation.  What I've found especially useful for our project was to contact the new employers of our previous trainees for feedback, say a month or two after the trainees have started in their new positions.  We ask 3 very simple questions so its not too much of a burden and this at least helps us see if we are providing what the sector needs.  We've had some great feedback this way and every single manager/employer has responded which is great.  Of course it does rely on the former trainees keeping in touch and providing us with their new manager's contact details (we do it by email) but thankfully this hasn't been a problem so far. 

     

  8. View Lynsey Whitley's profile Lynsey Whitley
    Offline | Last seen: 2 years 2 months ago
  9. I like that method, contacting the trainee's new employer, it would be good to find out how that trainee stood out from the rest of the candidates. As you said, it's good to know what does our current museum/heritage employers want. Did having that trainee, help to bridge a skill gap that hasn't been filled in that institution, which we know is one of the fundamental elements of Skills For the Future programme. 

    I've just pencilled in an area that I'll like to nclude in our evaluation process, talk to employers of previous trainees was a skills gap filled. 

    Paulette 

     

  10. View Paulette Francis-Green's profile Paulette Franci...
    Offline | Last seen: 2 years 2 weeks ago
  11. Hi Everyone,

    The evaluation for Passport to your Future will be presented as an ‘Innovation history’.

    Unlike conventional case study reports, innovation histories gather human stories of what happened during the projects journey, providing a multi-voiced account of the learning, looking at the challenges and the experiences of the participants, the outcomes and the benefits.

    The Innovation History is created by the Project Community.

    Innovation histories encourage key individuals to reflect on their own actions at each milestone in their journey and how they are linked with the actions of others, making it possible, therefore, for external parties to learn from others’ real-life experiences. Innovation histories are presented in a chronological story narrative that juxtaposes with quotes from core participants and other information. 

    You can find out more about Innovation histories from the following links.

    Douthwaite, B. And J. Ashby (2005) “Innovation Histories: A method for learning from experience”, ILAC Brief 5 http://www.cgiar-ilac.org/files/publications/briefs/ILAC_Brief05_Histories.pdf

    http://grassrootsinnovations.files.wordpress.com/2013/02/isle-of-gigha-heritage-trust-ih.pdf

    Throughout the Passport to your Future Programme we have tried to ensure that methodology is accessible to different types of learner and to people with barriers to accessibility.  This innovation history will therefore not just be presented through a written narrative but through a diversity of media types, including film clips, photos, art work etc.., enabling everybody to contribute.

    Claire

  12. View Claire Poulton's profile Claire Poulton
    Offline | Last seen: 2 years 5 months ago
  13. Claire, I really like your idea of using 'innovation histories' to capture the project's impact. I think narrative has to be the way foraward, but I wonder if there is any way of integrating other tools, like videa as Nick suggested, to bring the stories of Skills for the Future to life. 

    Also, while I absolutely believe in the social and regenerative aspects of historic environment conservation, I do think humanities in general need to do more to capture the these benefits quantitatively so that they compared with other oft-cited stats such as private investment levered, jobs created and commercial floor space improved. Others interested in this might like to look at a European project seeking an 'index' for this - www.encatc.org/culturalheritagecountsforeurope/

    Sophie

  14. View Sophie Norton's profile Sophie Norton
    Offline | Last seen: 2 years 5 months ago
  15. Dear All

     As a positive start to the new year I wanted to share our report about our Norfolk and Suffolk based Skills for the Future training scheme. Run in partnership between Norfolk Museums Service and the Museum of East Anglian Life, we have been offering traineeships and apprenticeships in all areas of our museums' work for the past four years. Our trainees have done very well in moving onto their chosen area of work when they move on from us, so this evaluation is really a celebration of their achievements within the scheme.

     Follow the link below to find the report. Scroll  down to the bottom of the webpage to find the links both to the report, and also to our first batch of case studies.

     If any of you want to see more of the trainees, there is also a link to a 15 minute film on the website.

     http://www.museums.norfolk.gov.uk/Visit_Us/Gressenhall_Farm_and_Workhouse/Skills_for_the_Future/index.htm

    Sally

    Images

  16. View Sally Ackroyd's profile Sally Ackroyd
    Offline | Last seen: 1 year 12 months ago
  17. Those of you who were able to attend the Skills for the Future project managers' meeting at NCVO in London today (30 March 2015) will have heard lots of talk of evaluation. We pointed you to Appendix 3 of our application guidance which sets out some questions which might help shape your thinking and planning:

    http://closedprogrammes.hlf.org.uk/HowToApply/programmes/Documents/SFTF_…

    You might also find HLF's general evaluation good practice useful: http://www.hlf.org.uk/evaluation-guidance

    Jo

  18. View Jo Reilly's profile Jo Reilly
    Offline | Last seen: 3 weeks 4 days ago
  19. Thanks to everyone who took part in the discussions on 30 March.  We’ll post all of the work you created at the workshop in due course but specifically relating to this thread on evaluation, we have summarised your collective advice on the subject which is to:

    • Conduct your own trainee surveys at the beginning and end of process – and discuss findings with trainees to check understanding. For example, ask trainees to rate satisfaction on a scale of 1-5
    • Include organisational development – not just trainee development – in your scope
    • Tell the story of your project – see HLF Evaluation guidance for a suggested methodology
    • Include dedicated sections on: Issues; Action; Impact
    • Include video – extracts are easy to share in other contexts too
    • Build an element of evaluation into project celebration events
    • Involve new employers taking on former trainees – send very short questionnaire (3 questions max) to ask if what you skills you helped develop were useful
    • Work with external evaluators closely – check that what they are producing is detailed enough
    • Hold influencing events/conferences based on evaluation results
    • Keeping in touch with trainees beyond the training is not always easy – try to build systems in and use a range of methods to suit them – phone, email, Facebook, Twitter
    • Ask for HLF feedback with a view to improvement, particularly if you have two separate grants.
  20. View Nick Randell's profile Nick Randell
    Offline | Last seen: 6 months 2 weeks ago

    Pages

Back to top of page