Beagrie.com

There was a very successful end of project dissemination workshop and lively discussion last week on implementing the toolkit with funders and other attendees. A full report of the workshop and links to the presentations are provided below. The Benefits Analysis Toolkit will be released on 31 July from the project web site and the KRDS web site.

Tools Background

This is a six month project funded by JISC, testing developing and documenting a toolkit consisting of two evolving tools, the KRDS Benefits Framework and the Value Chain and Benefits Impact tool. The Benefits Framework is the entry level tool and Value Chain and Benefits Impact tool is more advanced with a narrower range of applicable activities. Any benefit from digital curation should fit within the Framework and can be reworded and adapted to fit with the local application. From the funders perspective the easily tailored benefits offer a consistent and powerful way of stimulating thinking. The toolkit’s official release date is July 31^st.

Welcome and Project Background (Liz Lyon UKOLN) [Presentation]

The Toolset (Neil Beagrie, Charles Beagrie Ltd) [Presentation]

Case Studies

Dipak Kalra (Centre for Health Informatics and Multi-professional Education (CHIME) at UCL)

The toolkit was used in an MRC data support service investigation to understand how data sharing takes place. He presented results via a ‘virtual study’ that took all six studies into account to be more comprehensive. Generic benefits were taken from the tool and given a localised expression etc. He summarised that the tool should work for these kinds of studies though some parts are more applicable. Working through a toolkit could be of value for studies and particularly useful for putting forward a case for funding or prioritising resource utilisation within a study. Completing the spreadsheet and working out weightings might be nicely undertaken in a team workshop.

[Presentation]

Catherine Hardman (Archaeology Data Service)

In this case the toolkit was used from the point of view of a repository (more a macro level than micro), for looking in particular at issues of cost in the lifecycle. Archives often have to help justify costs/ effort associated with digital preservation even if they are well established. This can be used to address a range of audiences and with different levels of complexity- in individual projects or within project teams to boost cases for support. The value chain can help with identifying different values for different audiences. Quantification of impact can help in a number of ways: in research bid terms it helps justify resources; in archive preparation terms it helps with selection and retention decisions. The tool can be used as a light touch to help persuade stakeholders of benefits or for deeper insight into project planning decisions.

[Presentation]

Monica Duke (SageCite Project)

Here the tools were used to assess the benefits of data citation, an undertaking with a project perspective based on an organisation whose main business is science. It showed direct benefits as well as indirect ones such as better discovery of network models and better access. The Benefits Framework was easy to apply and helped to articulate benefits, although an intermediary may be required to facilitate the process.

[Presentation]

Matthew Woollard (UK Data Archive (UKDA) at University of Essex)

The tools were put into practice at the UK Data Archive and used to emphasise benefits to stakeholders. They helped to prioritise internal activities, justify costs to stakeholders and give an understanding of the service impact. They showed where value added is needed, where value is added, and who can benefit and when. The framework for activities seems to be where it will be of most use. It is important to note that generic benefits may have impacts to more than one stakeholder.

[Presentation]

Discussion

Q: What is the ongoing support for the tool?

• It will be present on the project website with a persistent web archive copy. There is a commitment to make it continuously available and it may be updated in future in light of future projects and applications. There is extensive documentation and if the need were felt for more support there is the potential for consultancy and assistance from Charles Beagrie Ltd as required.

Q: Do you see it incorporated within an outline data management plan?

• I can definitely see an advantage in the benefits framework. You can also use the value chain in a data heavy project, possibly when sitting down as a project group.

Q: Are we going to get too many statements on value, many of which are blatantly obvious rather than generically just true? If expressions are generic it would be better to cut them out.

• The tool is for focussing the mind and the generic examples only a starting point for what should be customised specific statements of value. In terms of presentation in the user guide we present an alternative version of the completed Framework with more specific examples and level of detail for the benefits. The user should have the ability to select those points of greatest impact for specific stakeholders and develop them ie not presenting a generic benefit.
• We are interested in ensuring researchers can do research and explaining value to the government and other stakeholders. If you’re promoting data sharing benefits then also promoting them to an internal audience is important and powerful for motivation.
• Research Councils can be deluged with metrics- it is better to have a few, simple and powerfully chosen. Case studies are incredibly powerful though not sufficient on their own. You spend little time discussing them compared to the time taken to create them. A case study should actually illustrate a metric.
• The Benefits Framework looks helpful in learning and preparation- in evaluation it should then be less necessary.
• Part of what we are doing is upping the game with studies and funders. Funders will need to respond proactively. There is a need for a more forceful tool but it is premature to deliver it now, such as a planning tool allowing the user to take up to three actions and workup a two or three year action plan. This is a possible direction of travel in the future.

Q: Homogenisation? What does it mean for funders when there is a long checklist of benefits? If established where will it lead us? Funders will have to look closely and make sure they are used carefully to draw out where we want benefits to accrue. Who is this making life easier for?

• Hopefully for researchers. They often have a box to fill in anyway but generally it is not well structured. The framework is fast to use, not a heavyweight commitment. We would hope it could make filling in of a benefits statement richer without much extra work as it provides a more advanced starting point for brainstorming the benefits.
• This has to be looked at against administrative burden. If it enables the user to identify and realise benefits they otherwise would not have realised then it has an advantage. If it isn’t repaid by better realisation of benefits then it deserves to fail.
• The process of using the tool can have valuable results in itself.
• Many benefits feed into other benefits. The funder should ask for requirements, it’s not necessary to show everything. This is a platform that everyone can use in a way that benefits them.
• There is value in prioritisation and communication. If we can work with researchers to highlight the key impacts of what they’re doing then the tool when simply done is of real value.

Q: We are talking about potential benefits- they haven’t actually been achieved. I can see the theoretical value but am worried these benefits could be three or four years ahead of what we can actually achieve.

• The time element of the framework does bring that in. We’re trying to think not just of the long term but how to get there and any benefits along the way. The element is there but you must have some degree of caution with how you apply this tool in the same way as any other.

Q: You mentioned OCLC is a partner in the project. Are they involved because of their interest in cataloguing and metadata?

• Our partner is the research division within OCLC which has a broad range of interests within digital library research. Brian Lavoie who has had an important input to this project and KRDS is a research scientist there. As an economist he has taken a close interest in the economics and benefits of digital preservation and this has been an important theme within OCLC Research – hence their interest and active participation in the project.

Q: I’m not sure who’s going to use the tool. What audience are you promoting it to? Will it mean a generic standard will be adopted?

• Different sectors or disciplines are different. There aren’t homogenous states so there will always be bespoke relevant next steps in working from the tools.
• Examples of all the common benefits listed are not seen in every project. There should always be a degree of selection so you wouldn’t end up with homogenous benefits in every case. Generic benefits should also be customised and expressed in ways specific to a particular project or service.
• What you may have to do is demonstrate your benefits to a wider audience not just a primary beneficiary so sometimes the wider list of benefits is also helpful. It is good to put in front of workers to demonstrate why things are done in a certain way. The audience is not as wide as we would like it to be but the value to those who can use it is great.
• The case studies presented are tailored towards particular projects/services but it can be adapted without too much additional tailoring. There may be elements which need to be tweaked.
• I think there is value beyond the digital preservation community particularly for the Framework and maybe other versions could be needed tailored to those other audiences.

• Arguments for further funding are always made on the science; informatics communities to some extent are disenfranchised. Anything we can do that supports honesty and helps to get discussion going within studies by linking benefits to science and data management must be good. If the Framework can accommodate different perspectives of benefits and allows them to be joined up in the story then we should try it out to more people.
• There can be reciprocal benefits or benefits with clear knock on effects to each other. Actions may give benefits to the user, which give benefits to the creator.
• Could there be eventual development of a web/matrix of benefits? Not one-to-one or even two-way but a network with flow going around it.

Q: Good ideas unless heavily marketed don’t take off. Even if there is a benefit to a tool it wouldn’t be given unless people know to use it. Are there steps funders would advise to encourage researchers and services pro-actively in seizing benefits and using the tools?

• Will people be persuaded to use the tool to compete? You only compete if a competition is created.
• Once certain good policies are floating around everyone uses them to tick the box whether they are applicable or not.

Q: Will presentations from the workshop be available later?

• Yes we intend to make them available later and a short write-up of the day and key areas of discussion.

 

I’m busy preparing with project colleagues for the dissemination workshop for the JISC  Digital Preservation Benefits Analysis Tools project in London on Tuesday.

The Toolkit is nearing its final version and we are adding case studies and worked examples ready for online release at the end of this month.

For those looking for an early taster (and not attending the workshop), here is a quick preview:

The Toolkit consists of two tools: the KRDS Benefits Framework (now in public version 3); and the Value-chain and Benefits Impact tool (now in public version 2). Each tool consists of a more detailed guide and worksheet(s). Both tools have drawn on partner case studies and previous work on benefits and impact for digital curation/preservation. This experience has provided a series of common examples of generic benefits that are employed in both tools for users to modify or add to as required.

It is designed for use by a wide audience including funders, researchers and project staff, and personnel in university central services, data archives and repositories.

I think the project has moved the usability of the KRDS Benefits Framework and the Value-chain and Impact tool on immensely from their early research project roots. Hopefully both existing and new users will find the new Toolkit a big improvement and valuable in their day to day work.

We will announce its release when finalised at the end of July via various email lists and this blog.