Key Findings and Top Five Recommendations

The CEF study concluded that transformational progress is required in three distinct, yet synergistic, areas:

1)    Fundamental and broad intellectual advances in the field of experimental methodologies and techniques, with particular focus on complex systems and human-computer interactions.

2)    New approaches to rapid and effective sharing of data and knowledge and information synthesis that accelerate multi-discipline and cross-organizational knowledge generation and community building.

3)    Advanced, accessible experimentation infrastructure capabilities.

The study resulted in roadmap that outlined needs for future cybersecurity experiment infrastructure along three-, five- and ten-year horizons. These are fully documented in the CEF final report. These are the top five recommendations from the study.

Recommendation 1:  Domains of Applicability – Multidisciplinary Experimentation

The creation of specialized experimentation capabilities across multiple domains is already a fast growing area of investment and research. In the near term, adding a focus on multidisciplinary experimentation that includes computer science, engineering, mathematics, modeling, human behavior, sociology, economics, and education will have the greatest impact on accelerating cyber security experimentation. These new capabilities will allow researchers to address open areas around the pervasive nature of cyber security and can provide an avenue to address emerging issues surrounding the Internet of Things (IoT). A focus in this area, along with the other top recommendations, will create the opportunity to coalesce the variety of emerging advances in capturing the human element in other domains.

Recommendation 2: Modeling the Real World for Scientifically Sound Experiments – Human Activity

The ability to accurately represent fully reactionary complex human and group activity in experiments will be instrumental in creating laboratory environments that realistically represent real-world cyber operations. To date most cyber experimentation is conducted in closed environments with minimal synthetic representation of human behavior. In order for this area to provide transformational results it must include the ability to integrate live and synthetic humans without artificialities that may interfere in some experiments, as well as capabilities to help ensure scientific validity when including live humans in experiments. Introducing the human element in experimentation will also open up the door to privacy and ethics issues that must be addressed. Achieving a seamless blending of the cyber and human world experimentation is a high priority mid term activity.

Recommendation 3: Frameworks and Building Blocks for Extensibility – Open Interfaces

Creating open standards and interfaces, for both experimental infrastructure facilities and for experiments themselves, is a high priority mid-term activity. Developing common models of infrastructure and experiment components to open interfaces and standards contributes to the overall goal of fostering a field of the science of cybersecurity experimentation. As a result, communities will be able to conduct, validate, integrate and share experiments, experimental components, and experimental results. This fundamental ability is needed to enable broader research in cybersecurity, as opposed to working in narrow sub disciplines. In addition, this new sharing capability will enable researchers to more easily repeat peer experiments and build upon those results.

Recommendation 4: Experiment Design and Instantiation - Reusable Designs for Science-based Hypothesis Testing

Research, development, and exploration in the area of experiment designs and design patterns for science-based hypothesis testing are required in order to achieve transformational changes in the field of experimental methodologies and techniques for cybersecurity. Researchers across all domains that rely on computational systems must be able to rapidly design meaningful experiments that reflect the real world by reusing and extending existing, validated experiment designs and design components. Experiment designs should be automatically validated and processed akin to the use of software development environments. Advances in key computer science disciplines such as ontologies, meta-data, libraries, and resource discovery are necessary to realize highly automated, extensible, and validated experiment designs.

Recommendation 5: Meta-properties – Usability and Cultural Changes

Cybersecurity research infrastructure must be usable by a wide range of researchers and experts across many different domains of research and not limited to traditional computer science researchers. It is vital that experimental capabilities not be restricted to power users of cybersecurity experimentation infrastructure. Given that the future research infrastructure is envisioned to be dynamic and used in many different ways, it is important to consider not only the usability of the technologies used to administer infrastructure, but also that of the technologies used to create and deploy experimentation infrastructure. In the long term we envision some degree of self-configuring or infrastructure-in-a-box capabilities to ease operational burdens, particularly for prospective researchers who are not from a traditional computer science background.

Along with usability-related properties, the adoption and use of future experimentation infrastructure likely will be characterized by the use of evolvable frameworks that support advances in experimental methods, and multiple models of collaboration, within both the user base of a single infrastructure and collaboration that spans multiple infrastructures. Research activities in several core capability areas discussed in the roadmap are required. Researchers must make a concerted effort to take advantage of community based resources, rather than relying on homegrown approaches. The shift to multi-domain users and the use of shared frameworks will enable both the research infrastructure and cybersecurity researcher communities to co-evolve. Usability and cultural changes are a long-term priority on which work must begin immediately.