Volume 21, Number 9/10
Table of Contents
Success Criteria for the Development and Sustainable Operation of Virtual Research Environments
Göttingen State and University Library
Leibniz Institute for Astrophysics Potsdam
Robotics Research Institute, TU Dortmund University
Leibniz Institute for Astrophysics Potsdam
University of Applied Science Potsdam
Robotics Research Institute, TU Dortmund University
In many areas of research, virtual research environments (VREs) have become an essential part of modern research processes. The providers of VREs need to respond to this growing importance with functioning and efficient processes for the development, operation and quality assurance of VREs. We have developed a life-cycle model for VREs, which focuses in particular on the success-critical points for the transition to a VRE's sustainable operation. Furthermore, we discuss a set of success criteria that enables all involved in the VRE (operators, funding bodies, users) to identify which aspects will be relevant to their specific needs prior to the creation of a new VRE. In light of the heterogeneity of VREs, this set of criteria is supplemented in individual cases by discipline-specific criteria.
Virtual research environments (VREs) have become increasingly important in recent years in a variety of research disciplines and belong to the standard repertoire of digital tools and infrastructures being used. However, up to now very few VREs have made the leap from the prototype phase to everyday use . Furthermore, the increased use of VREs in research environments has not resulted in any universally accepted attributes or guidelines for the development and operation of these infrastructures . On the contrary: the different stakeholders particularly from research funding bodies, academia and information infrastructure providers are searching for criteria that could help with the development, operation and quality assurance of VREs and with the optimisation of processes.
For a variety of reasons, these findings pose a challenge for the development of universally applicable success criteria and guidelines for the development of VREs. First, there are significant variations in the research processes, not only between different specialist areas of research but also within the research disciplines themselves. This applies to methodological approaches as well as to the tools that are used. Considering this heterogeneity, a universally applicable set of success criteria is therefore not feasible. On the other hand, it would be time-consuming and costly to plan every VRE individually and from scratch. Therefore it makes sense to identify similarities in the features and processes of VREs. These similarities can then serve as a basis for the development of guidance and support instruments as well as for evaluating the success of VREs.
There are very few classifications and systematic evaluations of VREs in German-speaking countries . The German Research Foundation-funded project "Success criteria for the development and sustainable operation of virtual research environments (DFG-VRE, DFG=German Research Foundation)" has set itself the task of bundling these approaches and turning them into useful recommendations. The project is particularly focused on big, cross-institutional VREs, as these involve specific organisational and legal challenges that can be addressed with pertinent recommendations. Smaller projects, on the other hand, have more straightforward planning and management needs and are therefore given lower priority in the framework of the DFG-VRE project.
For the purposes of our project, we took the definition of the Allianz AG of "Virtual Research Environments" as a basis for all further considerations:
"A Virtual Research Environment, or VRE, is a working platform that enables a cooperative research task to be carried out by numerous researchers in different locations at the same time and without restrictions. As a professional tool, it potentially supports the entire research process from collecting, discussing and processing data, through to the publication of the results while technologically, it is based above all on software services and communication networks. Virtual research environments are important components of modern research infrastructures and play a decisive role in the productivity and competitiveness of the research". 
Our project did not encompass a comprehensive empirical collection of data, for example resulting from surveys or standardised interviews. Based on the preliminary studies of project partners, our approach was focused on developing general approaches and identifying framework conditions that could set the course for the successful and sustainable operation of VREs right from the start. Based on the relevant experiences and contacts of the project partners, we decided to focus on the developers and operators of VREs.
An important tool was to talk to experts, in the form of open, non-standardised interviews. During these discussions, one or two subject areas were discussed on the basis of material gathered earlier in the project. The different subject areas and the dialogue partners are presented in Figure 1.
Figure 1: Overview of the expert interviews conducted between October and December 2013 and the topics discussed.
Apart from institutions that have developed or are operating a VRE, we also conducted interviews with representatives from the European Grid Infrastructure (EGI), the Deutsche Forschungsnetz e.V. (DFN) and the Hochschul-Informationssystem GmbH (HIS). The expert interviews were conducted over a period of three months in the autumn of 2013 and were individually tailored to the respective interviewees. The results of the individual expert interviews influenced the subsequent interviews and led to a modification of the questions as well as of the presented preliminary project results (for example life-cycle-phase model or success criteria). Clearly the subject is connected with a number of requirements for all stakeholders, whether from a user perspective or the perspective of infrastructure developers and operators, or funders of research infrastructures. Focal points of the discussion were defining success criteria for VREs, weighing up individual criteria, and the important constellation of cost structures and financial sustainability.
3 Life-cycle phases of a virtual research environment
The life-cycle of a VRE can be divided into different phases. It should be noted that this kind of schematic illustration always presents an idealised situation. That is why the diagram shown in Figure 2 cannot be directly applied to existing VREs. It should also be noted that the focus of our observations was primarily on the operator's perspective.
Throughout the entire lifespan of a VRE there are five phases. During the prototype phase, the technical and organisational structures of the VRE, as a rule within one project of limited duration, are established and made available to the first users in the form of a prototype. This enables the viability of the system to be put to the test and for it to be evaluated under normal operating conditions. The acceptance of the VRE within its user community can also be assessed during the prototype phase. Building upon this prototype, a series of aspects are then addressed during the development phase that are essential for the sustainable operation of the VRE, but which could not be adequately addressed during the development of the prototype. These aspects include, in particular, an operational concept, in which the services that can be provided by the VRE and the necessary resources to achieve these are presented, as well as a cost and financing plan.
Once this work has been completed, the VRE is a fully developed product ready for the operational phase. In this phase, the finished version of the VRE is available to researchers for their academic work. From the perspective of the operators, the focus is now on assisting users and on the ongoing technical and organisational development of the VRE. If at some point the decision is made to terminate the VRE, the purpose of the transfer phase is to transfer components worth keeping such as research data or programme codes to other (infra)structures. The remaining technical and organisational structures are then dissolved in the dismantling phase.
In terms of the life-cycle phases it is necessary to differentiate between two situations. The described, linearly structured schema applies initially only to VREs that are operated as independent infrastructures. However, if the VRE is one of several modules integrated in a comprehensive infrastructural context (e.g. in a data centre), then each module has its own prototype, development, transfer and dismantling phases, while the operation of the VRE applies across all modules (see Figure 2). In this case, individual components are not necessarily transferred to an external structure during the transfer phase, but possibly also to other phases of a new VRE module.
Figure 2: Life-cycle phases of a modular VRE. A first module A is established analogously to the life-cycle in Figure 2 and moves through the respective phases. However, the operating/product phase is shared with other modules of the same infrastructure. At the end of the lifespan of module A, components can be transferred back into the overall operation or into the prototype and development phase of a new module B.
Naturally, the transition between the different phases is of particular importance, especially in terms of entering the next phase. In this paper, we describe these conditions as milestones. These phase transitions, each with their respective milestones, are included among a VREs success criteria and are therefore discussed below.
3.1 The prototype phase
The prototype phase begins with preparatory work, in which the prototype project itself (e.g. the creation of the internal organisational structures and resource planning) is planned, and the environment of the planned VRE is assessed in the context of a requirements and market analysis. A requirements analysis includes the following aspects:
- An extensive analysis of academic workflows of the potential users of the VRE.
- Positioning the VRE in the respective academic context and evaluating possible collaborations with the stakeholders.
The market analysis, on the other hand, comprises:
- Compiling possible hard- and software components for the technical equipment of the VRE and the expected costs for this.
- Positioning the VRE in the overall infrastructural context. This includes both the existing infrastructures in the respective specialist communities such as e.g. national data centres, but also local, non-disciplinary or cross-disciplinary infrastructures (e.g. other VREs, data processing centres, libraries etc.) at the locations of the partners involved in the prototype phase of the VRE.
Figure 3: The prototype phase of a VRE.
During the technical specification, preceding analyses of the academic workflows are used to define the services of the VRE, and a development plan is made for their implementation with the help of an established software development model, e.g. the waterfall model . Also a decision is made about the standards and formats (for data as well as metadata) the VRE will use and which hard- and software will be used for the services to be provided.
A number of organisational measures also need to be carried out, including establishing access rights and policies, and developing strategies for community work, training sessions and the necessary support facilities. The VRE is also promoted within the relevant academic disciplines with the help of initial community-forming measures. Initial training sessions for users (and later "power users") can already be carried out.
The described developments finally lead to a prototype which is made available to a limited number of users. This serves not only to test the system under operating conditions but also provides an opportunity to adapt the VRE to the requirements of the community with small improvements based on the feedback of the pilot users.
In order to gather reliable information about the requirements of a VRE, particularly with a view to the subsequent operational phase, all described phases are accompanied by intensive monitoring, which is part of the project management.
3.2 The development phase
The pilot phase is important for gaining first impressions of the VRE in a working environment and for evaluating user acceptance in particular. This information is used to make a decision about whether to continue funding the VRE. If a positive decision is reached, further components necessary for the sustainable operation of the VRE are developed in the development phase (see Figure 4).
Figure 4: Components of the development of a VRE into a product during the development phase.
As a rule, the prototype phase of a VRE is financed as a research or infrastructure project. However, long-term operation requires an individually tailored operational concept that specifies the services provided by the VRE and the necessary resources needed to implement them, as well as a corresponding cost and financing plan. Furthermore, it is important to formulate a concept for subsequent usage, which clarifies how and where the components worth keeping (e.g. research data, software) will be transferred after the VRE reaches the end of its regular operational life.
Ideally this work would be carried out at the earliest possible stage, but usually this does not happen because of the specific project character of the prototype phase. In particular, if the pilot phase is judged to be a failure due to lack of user acceptance or for other reasons, the work would have been in vain. These steps are therefore carried out during a second financing stage, after which the VRE can be turned into a product and the operational phase can begin.
3.3 The operational phase
During the operational phase, the services of the VRE are available to the community for research purposes. Besides the technical operation of the VRE, the focus during this phase is also on assisting the user community and on the continued technical and organisational development of the VRE. Involving users in the further configuration of the VRE and acceptance on the part of the targeted specialist communities are particularly important. The operational phase encompasses a series of key aspects which are illustrated in Figure 5.
Figure 5: Central aspects of the operational phase
A help desk, appropriate support as well as record keeping for working with the VRE is at the disposal of users. The community that has formed around the VRE is supported and extended through appropriate events (e.g. workshops, trainings). Publicity work helps to promote the VRE within the academic community and, if applicable, in the public realm.
During the lifespan of the VRE, a variety of development work has to be carried out, which, however, should not impact the stable operation of the VRE. Enhancements to the VREs functionality become necessary particularly based on user feedback and new requirements that inevitably result from the growing circle of users. The coordination of these tasks rests with the requirements management. Major add-ons are carried out in separate projects. Bug fixes and small add-ons are carried out within the existing structure. Besides the production system, a VRE should also operate a development and test system, so that the enhancements or changes to the VRE can be seamlessly transferred to the live environment.
3.4 The transfer phase
If a VRE or part of a modular VRE is to be dismantled, the purpose of the transfer phase is to transfer components of the VRE that are worth keeping to suitable existing or yet to be created structures. This comprises:
- The transfer of data to an archive, research data centre or other VRE
- The further operation of services by partner institutions
- The publication or passing on of software
- The publication of other project results
Figure 6: Transfer phase of a VRE
The transfer may be necessary for legal reasons or due to other specifications (e.g. rules regarding correct academic practice as stipulated by the funder. But there might also be other reasons, for example an existing user base that wants to keep part of the dissolving VRE.
3.5 The dismantling phase
During the dismantling phase, the organisational structures of the VRE are dissolved. In some cases it might be useful to establish an appropriate follow-on organisation. Migration support helps the still-active users to integrate the transfer of components to other infrastructures, described above, into their workflows.
Figure 7: Dismantling phase of a VRE
3.6 Complete life-cycle phases of a VRE
Figure 8: Interviewed VREs mapped on the life-cycle phases
Figure 9: Complete life-cycle phases of a VRE
4 Criteria for the success of a virtual research environment
In this section we will first outline how the success of a VRE can be defined. We will then present the criteria set resulting from our project research. This will lead into a discussion about milestones, which are aligned to the individual life-cycle phases and which are important for the success of the VRE. The chapter ends with an overview of the methodology used for evaluating success.
4.1 Definition of success
Before establishing a set of success criteria, it is necessary to consider what in fact makes a VRE successful. This involves defining the success of a VRE in general and in relation to stakeholders and establishing a coherent definition based on different criteria.
For this purpose we will first define a generic set of success criteria which establishes various aspects of success by means of selected indicators. Important aspects of this set are based on the VRE definition by the Allianz AG . The set is not a fixed catalogue that can be equally applied to every VRE. It is necessary to select the relevant criteria on a case by case basis from this grouping similar to selecting something from a menu and to evaluate it critically at the appropriate time.
Ideally, the goals for a VRE should be defined at the start which is as early as the initial application procedure. The user community strongly influence the service spectrum of the later VRE and who are later catered to with appropriate services by the operator. An optimal alignment of user requirements with the choice of services offered by the VRE lays the foundation for a successful VRE. On the other hand it is necessary to take into account possibly conflicting objectives, for example the desire for a comprehensive choice of functions by the user compared to limited resources on the part of the developer/operator. All of this results in an individual success definition for the VRE.
4.2 Criteria set
The success criteria illustrated in Figure 10 provide a template to be adapted to specific cases. "Case-specific" means that the choice of individual criteria can be varied ("menu"), and can be supplemented with subject-specific criteria, e.g. depending on the disciplinary focus of the VRE. The success criteria are aligned in particular to the use and benefits of the VRE for academic users. They were discussed in the above mentioned expert interviews and to a large part can be traced back to statements from the specific VRE operators.
The diagram groups the criteria around the three most important stakeholder groups. This clearly illustrates how each group judges success by different criteria (weighting of the individual criteria). In addition as also illustrated above there are case-specific criteria.
Figure 10: Success factors and measurable success criteria, weighted by relevance for the different stakeholder categories.
The individual criteria:
- Usage: How intensively and how comprehensively does the target group use the VRE? Are there discrepancies in the amount that particular components of a VRE are used? How many users does the VRE have in relation to the total size of the potential user group? For which part of their academic workflow do the researchers use the VRE?
- Relevance/Community acceptance: How visible is the VRE within the potential user group? How does this group assess the usability of the VRE within its academic field? Is the VRE seen as a potentially useful addition for the group's research purposes? Does the VRE have an impact outside of the specialist community?
- Transfer of knowledge: Aside from concrete research results, to what extent does the VRE contribute to an accumulation of knowledge within the discipline, for example through an exchange of ideas about methods and objectives or publicly available project documentation?
- Collaboration: What kinds of collaboration with other researchers does the infrastructure enable, and to what extent are these put into practice? To what extent are the collaboration possibilities a unique selling proposition compared to other infrastructures?
- Publications/Results: How many and what kind of publications and results are published in the context of the VRE? Can all of the publications and results of the VRE be clearly classified? This criterion is closely connected to the subject-specific publication culture.
- Dissemination of expertise: To what extent does the VRE contribute to a dissemination of expertise within a user community? Typically, this expertise includes academic qualifications as well as post-doctoral publications and degree theses, but it also covers non-academic skills. This criterion covers an area that might otherwise be described as teaching, whereas the other success-oriented criteria cover the area of research.
- Subsequent use of infrastructure: Are infrastructure components of the VRE used in other contexts or after the VRE is dissolved, are selected components transferred to other contexts? This criterion also includes the quality and amount of documentation and the provision of information, findings and results.
- Public perception/Publicity work: How successful is the publicity work of a VRE? Here it is necessary to differentiate between different objectives, e.g. visibility within potential user groups, promoting the VRE to funding bodies, and the extent to which the VRE is known by the general public or within other academic disciplines.
- Interoperability: Do the VRE's research processes adhere to established standards? This is also important in relation to later subsequent usage, as mistakes made at this stage are difficult to correct later on.
- Scalability/Modularity: How adaptable is the infrastructure in terms of expanding the user group or the number of research topics? Has the architecture of the VRE been designed in such a way that at least elements of it can be used subsequently in a different context?
- Incorporation in academic workflows: Which parts of the academic workflows of the user group does the VRE cover? In this regard it is better not to try to be too comprehensive, but to concentrate on particular segments of the workflow.
- Performance/Usability: Does the VRE fulfil users' requirements in terms of learnability and usability? Does it integrate seamlessly into the day-to-day work of researchers? How is the usability of the VRE judged in comparison to other (generic) tools?
The success criteria can be differently weighted on a case-by-case basis. It is not about achieving results that can be quantitatively measured, but rather about defining objectives. The achievement of these objectives enables the success of the VRE to be determined at a later date. An important instrument in the implementation of these success criteria are milestones, which are described in the following chapter.
Apart from this the expert interviews revealed the usefulness of the success criteria although they may not be suitable for quantification or direct comparison with other VREs.
In the course of discussing the success criteria described above with the different VRE stakeholders it quickly became clear that these abstractly formulated criteria are closely connected with concrete milestones. In this sense, the milestones place the success criteria of the individual life-cycle phases in chronological order and connect them with specific tasks for the operator of the VRE. The lack of such milestones, or milestones that are not achieved, can be classified as "failure criteria" (negative success criteria). Assessing these milestones in terms of their structure and content is a further step in the qualitative evaluation.
Some of these milestones are outlined in Figure 11. This list is just a suggestion, as depending on a given case there may be different milestones or the individual milestones may be more strongly structured. As already touched in the discussion about life phases, above, many more milestones are needed in the prototype and development phase than in the subsequent phases.
Figure 11: Milestones relating to the life-cycle phases of a VRE (selection).
The success of a VRE depends to a great extent on integrating the user group right from the start. This takes the form of a user requirements and market analysis, accompanied by project monitoring and is designed to ensure the biggest possible overlap between the requirements of the user group and what the VRE will later offer.
To broker any resulting conflicts of interest, we have created a cost and financing plan, a development plan and a concept for the organisational form. Above all, the integration of all stakeholders in the organisational form enables a continuous exchange of ideas. At the same time, the organisational form and the cost and financing plan set a possible course for the future sustainability of the infrastructure. During the interviews conducted as part of the DFG-VRE project, realistic cost estimation for the establishment of a VRE was consistently described as problematic. It is difficult to make general assumptions, but it is certainly possible to come up with relatively realistic weighted cost structures at the very beginning of planning for the VRE, based on the specific tasks and the size of the user group. These cost structures provide a firm basis for financial planning.
4.4 Methodology for evaluating success
The perceived success of a VRE is determined by the interests of the stakeholders. An academic user community generally has different criteria from a funder or infrastructure provider. These differing perspectives are expressed as described above in a conflict of objectives that must be conceptually addressed.
The process for evaluating success is illustrated in Figure 12:
Figure 12: Evaluation of success in relation to the development phase of the VRE.
For all stakeholders, it begins with the selection of criteria from the overall set of success criteria described above. These are supplemented by subject- or case-specific criteria, which together contribute to a definition of success. The objective of defining success is carried out before the VRE is developed, usually in the context of the application for the prototype phase. The further design and specific development is then carried out in line with these objectives. In this case it is important that the objectives are defined by the stakeholders themselves and that there is process of consensus.
This paper examines different relevant factors for successful development and sustainable operation of a VRE. Structural, functional, organisational and financial aspects are combined. To an extent the different stakeholder perspectives are taken into consideration, although we focus particularly on the developer and operator perspective.
The work is based on experiences gathered in various collaborative projects and was verified in expert interviews and workshop discussions with developers, operators, users and funders. We also took into consideration insights from the development of larger software systems.
In summary, the feedback from the community suggests that:
- The definition of success of a VRE depends strongly on the perspective of the stakeholders and academic discipline. This paper focuses above all on the perspective of the developer and operator.
- The definition of success criteria involves considerable methodological challenges, which means success is not easy to quantify or compare.
- There is no overarching set of criteria. It makes more sense to compile a case-specific set out of a catalogue and to extend this with subject-specific criteria.
- Ideally, the definition of success is established in the planning phase.
- Analysing the VREs in relation to a life-cycle-phase model is useful, as it flags up differences regarding the definition of success not only in terms of the subject area but also in relation to the currently applicable life-cycle phase. The focus may be on different criteria depending on the life-cycle phase of a VRE.
- Connected to this are success-critical milestones over and above the life-cycle phases. The success-critical milestones during the life-cycle phases function as a kind of project management system.
- Definition issues (e.g. virtual or collaborative) are of secondary importance in practice, although there appears to be a development from a comprehensive VRE approach, covering the entire research process, to a modular tool-oriented approach.
- A tool-oriented approach has advantages compared to a comprehensive VRE approach. It concentrates on particular sections of the research process and is better positioned to mediate between community "demand" and infrastructure "offerings". It is also better able to deal with the fact that generic tools (e.g. wiki, cloud services) are generally already available in the researcher's working environment.
- To date, few VREs have carried out a structured analysis of cost structures and business models, which is due not least to the fact that the operational phase of a VRE, when sustainability becomes relevant, has as yet seldom been reached in Germany.
- The success of a VRE depends to a great extent on integrating the target user community as early as possible. This includes the involvement of multiplicators and "development competent" users. They can provide valuable assistance in terms of establishing requirements as well as in the development of the VRE.
- The success also depends on taking into account the different stages of the life cycle right from the start, as this has a decisive effect on the cost structures and potential business models. This point is directed not only at the developers of a VRE but also and in particular at its operators and funders.
The findings made in the course of the DFG-VRE project are just a first step towards an approved instrument for evaluating the success and usability of such infrastructures. The systematic and advanced approach of the pilot study examines the conditions required for the sustainable development of a VRE in relation to today's research infrastructures and funding structures. It would be useful to do an even more in depth study in the form of a bigger survey of the issues examined and developed in this paper. However, the results of the pilot study already allow for current and future projects to reflect on their own criteria for sustainability and success and to orientate themselves using the life-cycle-phase models.
1 This article is a translated and heavily revised edition of a former German language publication. The German edition has been published as DARIAH-DE Working Paper in 2014.
 Arbeitsgruppe Virtuelle Forschungsumgebungen der Allianz-Initiative Digitale Information (2011): Virtuelle Forschungsumgebungen Ein Leitfaden.
 Arbeitsgruppe Virtuelle Forschungsumgebungen der Allianz-Initiative Digitale Information (2012): Momentaufnahme: VREs in 2011/2012.
 Carusi, Annamaria, Reimer, Torsten (2010): JISC Collaborative Landscape Study.
 Fritzsch, B. (2013): Erfolgskriterien für VREs: Naturwissenschaftliche Sicht Erfahrungen aus C3Grid, Workshop: "Erfolgskriterien für Virtuelle Forschungsumgebungen", Leibniz Institut für Astrophysik Potsdam, 19. November 2013.
 Ludwig, Jens, Enke, Harry (2013): Digital Curation of Research Data. Experiences of a Baseline Study in Germany. VWH-Verlag.
 Winston Royce, Proceedings, IEEE WESCON (Ed.) (1970): Managing the Development of Large Software Systems. Institute of Electrical and Electronics Engineers.
About the Authors
Stefan Buddenbohm holds a degree in Political Science and works in the Research and Development Department of the Göttingen State and University Library and in the Max Planck Institute for the Study of Ethnic and Religious Diversity Göttingen. The focus of his work has been with projects in the area of digital research infrastructures (repositories, open access, virtual research environments).
Harry Enke holds a Ph.D. in Physics. He is leading the E-Science group at the Leibniz Institute for Astrophysics in Potsdam (AIP). Main focus is building digital infrastructures for Scientific data and computing in German astrophysics. He was part of the core group for academic use of D-Grid and is member of the German Astrophysics Virtual Observatory (GAVO).
Matthias Hofmann holds a M.Sc. in Informatics. He is currently with the Robotics Research Institute of the Technical University of Dortmund as a Research Associate. His main interests are robotics, artificial intelligence, and scheduling problems. Matthias Hofmann was involved in several Grid-related projects and the RoboCup competition.
Jochen Klar holds a Ph.D. in Astronomy and works in the E-Science group at the Leibnitz Institute for Astrophysics Potsdam (AIP). His work focuses on research data management and research data publication within astronomy as well as on an interdisciplinary level.
Heike Neuroth holds a Ph.D. in Geology and a degree in Library Science. She was head of the Research and Development Department of the Göttingen State and University Library. The focus of her recent work has been with numerous activities in digital preservation, research data management, research infrastructures and digital humanities. Currently she is a professor of Library Science at the University of Applied Sciences Potsdam.
Uwe Schwiegelshohn holds a Dr.-Ing. in Electrical Engineering and the chair of the Computer Engineering Institute and is head of the Robotics Research Institute of the Technical University of Dortmund. He is particular interested in various types of scheduling problems with a focus on online approximation algorithms for job scheduling problems. Further, he addresses resource management and scheduling issues in Grid-Computing and sustainability in large computational Grids.