Beagrie.com

There was a very successful end of project dissemination workshop and lively discussion last week on implementing the toolkit with funders and other attendees. A full report of the workshop and links to the presentations are provided below. The Benefits Analysis Toolkit will be released on 31 July from the project web site and the KRDS web site.

Tools Background

This is a six month project funded by JISC, testing developing and documenting a toolkit consisting of two evolving tools, the KRDS Benefits Framework and the Value Chain and Benefits Impact tool. The Benefits Framework is the entry level tool and Value Chain and Benefits Impact tool is more advanced with a narrower range of applicable activities. Any benefit from digital curation should fit within the Framework and can be reworded and adapted to fit with the local application. From the funders perspective the easily tailored benefits offer a consistent and powerful way of stimulating thinking. The toolkit’s official release date is July 31^st.

Welcome and Project Background (Liz Lyon UKOLN) [Presentation]

The Toolset (Neil Beagrie, Charles Beagrie Ltd) [Presentation]

Case Studies

Dipak Kalra (Centre for Health Informatics and Multi-professional Education (CHIME) at UCL)

The toolkit was used in an MRC data support service investigation to understand how data sharing takes place. He presented results via a ‘virtual study’ that took all six studies into account to be more comprehensive. Generic benefits were taken from the tool and given a localised expression etc. He summarised that the tool should work for these kinds of studies though some parts are more applicable. Working through a toolkit could be of value for studies and particularly useful for putting forward a case for funding or prioritising resource utilisation within a study. Completing the spreadsheet and working out weightings might be nicely undertaken in a team workshop.

[Presentation]

Catherine Hardman (Archaeology Data Service)

In this case the toolkit was used from the point of view of a repository (more a macro level than micro), for looking in particular at issues of cost in the lifecycle. Archives often have to help justify costs/ effort associated with digital preservation even if they are well established. This can be used to address a range of audiences and with different levels of complexity- in individual projects or within project teams to boost cases for support. The value chain can help with identifying different values for different audiences. Quantification of impact can help in a number of ways: in research bid terms it helps justify resources; in archive preparation terms it helps with selection and retention decisions. The tool can be used as a light touch to help persuade stakeholders of benefits or for deeper insight into project planning decisions.

[Presentation]

Monica Duke (SageCite Project)

Here the tools were used to assess the benefits of data citation, an undertaking with a project perspective based on an organisation whose main business is science. It showed direct benefits as well as indirect ones such as better discovery of network models and better access. The Benefits Framework was easy to apply and helped to articulate benefits, although an intermediary may be required to facilitate the process.

[Presentation]

Matthew Woollard (UK Data Archive (UKDA) at University of Essex)

The tools were put into practice at the UK Data Archive and used to emphasise benefits to stakeholders. They helped to prioritise internal activities, justify costs to stakeholders and give an understanding of the service impact. They showed where value added is needed, where value is added, and who can benefit and when. The framework for activities seems to be where it will be of most use. It is important to note that generic benefits may have impacts to more than one stakeholder.

[Presentation]

Discussion

Q: What is the ongoing support for the tool?

• It will be present on the project website with a persistent web archive copy. There is a commitment to make it continuously available and it may be updated in future in light of future projects and applications. There is extensive documentation and if the need were felt for more support there is the potential for consultancy and assistance from Charles Beagrie Ltd as required.

Q: Do you see it incorporated within an outline data management plan?

• I can definitely see an advantage in the benefits framework. You can also use the value chain in a data heavy project, possibly when sitting down as a project group.

Q: Are we going to get too many statements on value, many of which are blatantly obvious rather than generically just true? If expressions are generic it would be better to cut them out.

• The tool is for focussing the mind and the generic examples only a starting point for what should be customised specific statements of value. In terms of presentation in the user guide we present an alternative version of the completed Framework with more specific examples and level of detail for the benefits. The user should have the ability to select those points of greatest impact for specific stakeholders and develop them ie not presenting a generic benefit.
• We are interested in ensuring researchers can do research and explaining value to the government and other stakeholders. If you’re promoting data sharing benefits then also promoting them to an internal audience is important and powerful for motivation.
• Research Councils can be deluged with metrics- it is better to have a few, simple and powerfully chosen. Case studies are incredibly powerful though not sufficient on their own. You spend little time discussing them compared to the time taken to create them. A case study should actually illustrate a metric.
• The Benefits Framework looks helpful in learning and preparation- in evaluation it should then be less necessary.
• Part of what we are doing is upping the game with studies and funders. Funders will need to respond proactively. There is a need for a more forceful tool but it is premature to deliver it now, such as a planning tool allowing the user to take up to three actions and workup a two or three year action plan. This is a possible direction of travel in the future.

Q: Homogenisation? What does it mean for funders when there is a long checklist of benefits? If established where will it lead us? Funders will have to look closely and make sure they are used carefully to draw out where we want benefits to accrue. Who is this making life easier for?

• Hopefully for researchers. They often have a box to fill in anyway but generally it is not well structured. The framework is fast to use, not a heavyweight commitment. We would hope it could make filling in of a benefits statement richer without much extra work as it provides a more advanced starting point for brainstorming the benefits.
• This has to be looked at against administrative burden. If it enables the user to identify and realise benefits they otherwise would not have realised then it has an advantage. If it isn’t repaid by better realisation of benefits then it deserves to fail.
• The process of using the tool can have valuable results in itself.
• Many benefits feed into other benefits. The funder should ask for requirements, it’s not necessary to show everything. This is a platform that everyone can use in a way that benefits them.
• There is value in prioritisation and communication. If we can work with researchers to highlight the key impacts of what they’re doing then the tool when simply done is of real value.

Q: We are talking about potential benefits- they haven’t actually been achieved. I can see the theoretical value but am worried these benefits could be three or four years ahead of what we can actually achieve.

• The time element of the framework does bring that in. We’re trying to think not just of the long term but how to get there and any benefits along the way. The element is there but you must have some degree of caution with how you apply this tool in the same way as any other.

Q: You mentioned OCLC is a partner in the project. Are they involved because of their interest in cataloguing and metadata?

• Our partner is the research division within OCLC which has a broad range of interests within digital library research. Brian Lavoie who has had an important input to this project and KRDS is a research scientist there. As an economist he has taken a close interest in the economics and benefits of digital preservation and this has been an important theme within OCLC Research – hence their interest and active participation in the project.

Q: I’m not sure who’s going to use the tool. What audience are you promoting it to? Will it mean a generic standard will be adopted?

• Different sectors or disciplines are different. There aren’t homogenous states so there will always be bespoke relevant next steps in working from the tools.
• Examples of all the common benefits listed are not seen in every project. There should always be a degree of selection so you wouldn’t end up with homogenous benefits in every case. Generic benefits should also be customised and expressed in ways specific to a particular project or service.
• What you may have to do is demonstrate your benefits to a wider audience not just a primary beneficiary so sometimes the wider list of benefits is also helpful. It is good to put in front of workers to demonstrate why things are done in a certain way. The audience is not as wide as we would like it to be but the value to those who can use it is great.
• The case studies presented are tailored towards particular projects/services but it can be adapted without too much additional tailoring. There may be elements which need to be tweaked.
• I think there is value beyond the digital preservation community particularly for the Framework and maybe other versions could be needed tailored to those other audiences.

• Arguments for further funding are always made on the science; informatics communities to some extent are disenfranchised. Anything we can do that supports honesty and helps to get discussion going within studies by linking benefits to science and data management must be good. If the Framework can accommodate different perspectives of benefits and allows them to be joined up in the story then we should try it out to more people.
• There can be reciprocal benefits or benefits with clear knock on effects to each other. Actions may give benefits to the user, which give benefits to the creator.
• Could there be eventual development of a web/matrix of benefits? Not one-to-one or even two-way but a network with flow going around it.

Q: Good ideas unless heavily marketed don’t take off. Even if there is a benefit to a tool it wouldn’t be given unless people know to use it. Are there steps funders would advise to encourage researchers and services pro-actively in seizing benefits and using the tools?

• Will people be persuaded to use the tool to compete? You only compete if a competition is created.
• Once certain good policies are floating around everyone uses them to tick the box whether they are applicable or not.

Q: Will presentations from the workshop be available later?

• Yes we intend to make them available later and a short write-up of the day and key areas of discussion.

I’m busy preparing with project colleagues for the dissemination workshop for the JISC  Digital Preservation Benefits Analysis Tools project in London on Tuesday.

The Toolkit is nearing its final version and we are adding case studies and worked examples ready for online release at the end of this month.

For those looking for an early taster (and not attending the workshop), here is a quick preview:

The Toolkit consists of two tools: the KRDS Benefits Framework (now in public version 3); and the Value-chain and Benefits Impact tool (now in public version 2). Each tool consists of a more detailed guide and worksheet(s). Both tools have drawn on partner case studies and previous work on benefits and impact for digital curation/preservation. This experience has provided a series of common examples of generic benefits that are employed in both tools for users to modify or add to as required.

It is designed for use by a wide audience including funders, researchers and project staff, and personnel in university central services, data archives and repositories.

I think the project has moved the usability of the KRDS Benefits Framework and the Value-chain and Impact tool on immensely from their early research project roots. Hopefully both existing and new users will find the new Toolkit a big improvement and valuable in their day to day work.

We will announce its release when finalised at the end of July via various email lists and this blog.

We are pleased to announce that registration is now open for the workshop to disseminate the Digital Preservation Benefit Analysis Toolset and accompanying materials such as user guides and factsheets to the research community.

Workshop registration is free but please note that places are limited and early registration is advised. Further details of the workshop are as follows:

12.30 -16.00 Tuesday 12th July, 2011 South Bank University, Central London

Programme:
12.30 – 13.15 Registration and buffet lunch
13.15 – 13.25 Welcome and Project Background (Liz Lyon UKOLN)
13.25 – 13.55 The Toolset (Neil Beagrie, Charles Beagrie Ltd)
13.55 – 14.50 Disciplinary Test Sites and Applications (chair Manjula Patel UKOLN)

• Dipak Kalra (Centre for Health Informatics and Multi-professional Education at UCL)
• Matthew Woollard (UK Data Archive University of Essex)
• Liz Lyon and Monica Duke (SageCite Project)
• Catherine Hardman (Archaeology Data Service)

14.50 – 15.00 Implications for Funders (Neil Beagrie)
15.00 – 15.20 Break and refreshments
15.20 – 16.00 Plenary Discussion and Questions (chair Liz Lyon, UKOLN)
16.00 Close
—————————
Background

The “Digital Preservation Benefit Analysis Tools” project is funded by the Joint Information Systems Committee (JISC) and runs from 1st February to 31 July 2011.

The project has tested and reviewed the combined use of the Keeping Research Data Safe (KRDS) Benefits Framework and the Value Chain and Impact Analysis tool, which were first applied in the I2S2 project for assessing the benefits and impact of digital preservation of research data. We have extended their utility to, and adoption within, the JISC community by providing user review and guidance for the tools and creating an integrated toolset. The project consortium consists of a mix of user institutions, projects, and disciplinary data services committed to the testing and exploitation of these tools and the lead partners in their original creation.

The project plan is on the project website and the project outputs will be available from the website during the summer.

The project partners are UKOLN and the Digital Curation Centre at the University of Bath,the Centre for Health Informatics and Multi-professional Education (CHIME) at University College London , the UK Data Archive (University of Essex), the Archaeology Data Service (University of York), OCLC Research, and Charles Beagrie Limited.

The Digital Preservation Coalition and Charles Beagrie Limited are delighted to announce a collaboration to produce 5 new DPC Technology Watch Reports. The collaboration follows a DPC call for proposals issued in December last year and selection of Charles Beagrie Limited as the preferred bidder.

The collaboration will produce a series of 5 Technology Watch Reports over the next 12 months under the general supervision of an editorial board and Neil Beagrie as principal investigator and commissioning editor. The 5 proposed reports and their authors are as follows:

• Preserving Email, Chris Prom
• Preserving Moving Picture and Sound, Richard Wright
• Intellectual Property Rights for Preservation, Andrew Charlesworth
• Digital Forensics and Preservation, Jeremy Leighton John
• Trust in Post Cancellation Access Services, Neil Beagrie

The DPC is establishing an editorial board for the series. It will be chaired by William Kilbride, Executive Director of the DPC.

The collaboration represents an exciting new development for the DPC and Charles Beagrie Ltd and the opportunity is being taken to re-vamp the design and layout of the new series. Content outlines for individual reports will be shared with DPC members and shaped by their needs and requirements. DPC members will have a period of privileged advance access to each report prior to wider public release.

The DPC Technology Watch Report series was established in 2002 and has been one of the Coalition’s most enduring contributions to the wider digital preservation community.  They exist to provide authoritative support and foresight to those engaged with digital preservation or having to tackle digital preservation problems for the first time. These publications support members work forces, they identify disseminate and discuss best practices and they lower the barriers to participation in digital preservation.

‘Each ‘Technology Watch Report’ analyses a particular topic in digital preservation, evaluating workable solutions, and investigating new tools and techniques appropriate for different contexts,’ explained William Kilbride of the DPC.  ‘The reports are written by leaders-in-the-field and are peer-reviewed prior to publication.  The intended audience is worldwide, especially in the UK, Europe, Australia, New Zealand, USA, and Canada.’

‘We expect that these reports will have a wide readership,’ explained Neil Beagrie, the commissioning editor.  ‘The audience includes members and non-members of the coalition; staff of commercial and public agencies; repository managers, librarians and archivists charged with managing electronic resources; senior staff and executives of intellectual property organizations in the private and public sectors; those who teach and train information scientists; as well as policy advisors requiring an advanced introduction to specific issues and researchers developing DP solutions.’

Further publicity on each report in the series will be released over the course of the next 12 months to DPC members and the wider community.  The draft outline of contents for the first report – Preserving Email – has already been compiled and will be distributed shortly.

• To find out more about the DPC and its Technology Watch series, including the current ones in the set, see: http://www.dpconline.org/advice/technology-watch-reports
• For more about Charles Beagrie Ltd, see: http://www.beagrie.com/

Projects from the JISC Managing Research Data Programme were involved in a Parallel Session at the annual JISC Conference on Tuesday this week.

Entitled ‘The benefits of more effective research data management in UK Universities’, the session explained how projects have been developing ‘Benefits Case Studies’  with support from Charles Beagrie Ltd to provide evidence of the positive effects of improvements which they have engineered.  The case studies provide significant indications of improved research efficiency through more effective research data management.  The case studies will be synthesised in a report by Neil Beagrie due for release in May.

Presentations from the parallel session are available online at:
http://www.jisc.ac.uk/events/2011/03/jisc11/programme/1researchdata.aspx

They are best perused in the following order:

Simon Hodson, JISCMRD, Introduction
Neil Beagrie, Cost-Benefits and Business Cases Support Role
Manjula Patel and Neil Beagrie, I2S2 Project, UKOLN, University of Bath
June Finch, MaDAM Project, University of Manchester
Jonathan Tedds, HALOGEN Project, University of Leicester

I am pleased to announce the launch of a new project focussing on development of a digital preservation benefits analysis toolset.

The “Digital Preservation Benefit Analysis Tools” project is funded by the Joint Information Systems Committee (JISC) and will run from 1^st February to 31 July 2011.

The project  aims to test, review and promote combined use of the Keeping Research Data Safe (KRDS) Benefits Taxonomy and the Value Chain and Impact Analysis tool first applied in the I2S2 project  for assessing the benefits and impact of digital preservation of research data. We will extend their utility to and adoption within the JISC community by providing user review and guidance for the tools and creating an integrated toolset. The project consortium consists of a mix of user institutions, projects, and disciplinary data services committed to the testing and exploitation of these tools and the lead partners in their original creation. We will demonstrate and critique the tools, and then create and disseminate the toolset and accompanying materials such as User Guides and Factsheets to the wider community.

A project website is at http://beagrie.com/krds-i2s2.php and the project plan and project outputs will be available from the website in due course. A dissemination event to mark the conclusion of the project will be held in central London on 12 July 2011 (further details and registration will be announced in May).

The project partners are UKOLN and the Digital Curation Centre at the University of Bath, the Centre for Health Informatics and Multi-professional Education (CHIME) at University College London , the UK Data Archive (University of Essex), the Archaeology Data Service(University of York),  OCLC Research, and  Charles Beagrie Limited.

The Keeping Research Data Safe (KRDS) project has produced a widely used KRDS Activity Model for costing digital preservation of research data. KRDS has developed from relatively small-scale incremental projects and we recognise that there were still significant areas for future work such as the recently published (Dec 2010) KRDS User Guide. The KRDS2 final report published earlier last year outlined a number of key recommendations for future development including:

• “Examine further development of the pre-archive phase of the KRDS2 activity model and produce versions of the model from a researcher’s perspective.”

This suggested work has now been addressed by one of the outputs from the Infrastructure for Integration in Structural Sciences (I2S2) Project funded under the Research Data Management Infrastructure strand of the JISC’s Managing Research Data Programme.

I2S2 has been using KRDS as a basis for costing and benefits analysis. One of the outputs has been an “Idealised Scientific Research Data Lifecycle Model”, which seeks to extend and adapt from a “researcher perspective”, the Keeping Research Data Safe (KRDS) Activity Model, providing a model which reflects “research data management” or the digital preservation lifecycle in its broadest interpretation. It adapts KRDS from an archive-centric to a researcher-centric view by:

• Defining and emphasising more of the activities in the research (KRDS “Pre-Archive” ) phase where research data is created;
• Adding a “Publication” set of activities;
• Concatenating the KRDS “Archive” phase activities in the centre of the model for simplification and presentational purposes;
• Adding some specific local research administration activities;
• In addition for the purposes of the project, it adds some selective detail of information flows and information objects between the activities.

This is the current version (Dec 2010) of the I2S2 Idealised Model.

Note this is an idealised model and several activities such as peer review or conduct experiment may have multiple instances or repetitions. “Documentation, Metadata, and Storage” may also  be undertaken as researcher activities independent of the archive in other instances and in the KRDS activity model. It also represents a project view as of December 2010 and may be subject to further changes.

A PPT version of the I2S2 model incorporating relevant notes is available on the I2S2 project website.

The I2S2 project aims to understand and identify the requirements for a data-driven research infrastructure in the Structural Sciences.  The work is focused on the exemplar domain of Chemistry, but with a view towards inter-disciplinary application. Current work inter alia includes developing a set of tools and approaches to identify and provide indicators and metrics for the benefits arising from I2S2. This will extend work and the tools available for implementing the KRDS Benefits Taxonomy.

The partners in I2S2 are UKOLN (University of Bath), the Digital Curation Centre, University of Southampton, University of Cambridge, Science & Technology Facilities Council, and Charles Beagrie Ltd.

I am pleased to announce the release of a new User Guide from the Keeping Research Data Safe (KRDS) project on the costs and benefits of digital preservation of research data. This is the second and final work of synthesis from the project. The User Guide is available for download as a PDF from here.

The KRDS User Guide has been developed to support easier assimilation of the combined work of the KRDS1 and KRDS2 projects by those wishing to implement the tools or key findings.

KRDS is a cost framework that can be used to develop and apply local cost models for research data management and long-term preservation. In addition, it includes a Benefits Taxonomy and discussion of benefits which provides a valuable starting point and framework for assessing the impact and benefits of research data management and preservation activities. Finally, KRDS has been a significant research project establishing many key “rules of thumb” for digital preservation costs and approaches to sustaining digital research data. Even those who do not wish to or cannot allocate the resources to develop local models based on KRDS are likely to benefit from its key findings and exemplars, covered in later sections of the Guide.

The Use Guide consists of 39 A4 pages with 15 illustrations (many created specifically for this Guide) and covers the following major areas:

The KRDS Costs Framework;

A Brief “How To” Guide For Life-Cycle Cost Analysis;

KRDS Benefits Analysis;

KRDS Case Studies, Costs Survey, and Factsheet;

Future Development of KRDS.

We hope the User Guide will be of value to the digital preservation and research data communities. In addition to the User Guide we have created the new KRDS webpage which provides a single point of access for the key outputs of both the KRDS1 and KRDS2 projects (including the two recent works of synthesis the KRDS User Guide and the KRDS Factsheet).

The Keeping Research Data Safe studies have been conducted by a partnership of the following institutions: Charles Beagrie Ltd, OCLC Research, the UK Data Archive, the Archaeology Data Service, the University of London Computer Centre, and the universities of Cambridge, King’s College London, Oxford and Southampton. The creation of the Guide has been funded by the JISC Managing Research Data Programme.

We welcome feedback from users of the Guide which will help enhance and update future editions.

The Report from the Research Excellence Framework (REF) institutional pilots of impact assessment were published recently by HEFCE.

There was also some discussion in Times Higher with articles  on the general implications and the other on specific implications for the humanities.

Having read the report, I think the REF Impact assessment is highly relevant and important for many UK research data projects and probably of interest to others internationally.  For me the main points of interest were:

• The introduction of the Impact component of REF can support the business case for research data infrastructure as that infrastructure could help institutions promote/record impact;
• The REF timeline 2011-2014;

• The pilot exercise affirmed the use of case studies as the best approach for REF;
• Use of “reach” and “significance” to assess impact in REF and the initial draft list of impact indicators in Appendix G (draft ‘common menu’ of impact indicators) from the REF draft guidelines;
• The KRDS Benefits Taxonomy has a good fit to a lot of its discussion with dimension 1 (direct/indirect), dimension 2 near-term/long term, and dimension 3 private/public – although for REF only non-academic stakeholders are in scope;
• The “Best Practice” (section 7) and “Bad Practice” (section 8 ) of the report provide good generic guidance on completing impact case studies.

The Pilot report notes that the impact element in the REF has the potential to create a number of positive incentives, including:

• Encouraging collaboration between HE and industry, the public sector and third sector.
• Encouraging institutions to support their researchers in more fully realising the wider benefits of the research they undertake. This should include support for realising the benefits from ‘pure’ or ‘basic’ research, as well as supporting research with more immediate potential application.

On 16 September 2010 a rather unique meeting sponsored by NCDD/DEN/and the KB took place in The Hague: the experts from five past and present projects on cost modelling for digital preservation came together to exchange information and discuss future possibilities for international cooperation.

The conference report by Inge Angevaare of this meeting with photographs and a summary of existing costing models is now available. Those who know Inge well will be aware that a camera is never very far away so the report is beautifully illustrated, concise and well worth reading.

The projects discussed included Keeping Research Data Safe (KRDS, UK), CMDP (Denmark),LIFE3 (UK), DANS (Netherlands), National Archives Testbed (Netherlands). For those wanting to see more detail of individual presentations they are available here.