Changes between Version 1 and Version 2 of LifeCycle


Ignore:
Timestamp:
May 10, 2011 11:21:49 PM (13 years ago)
Author:
Alefiya Hussain
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • LifeCycle

    v1 v2  
     1
    12== Design Notes ==
    23
    3 This document serves the following purposes:
    4 * It discusses the high-level concepts of an experiment life cycle. This design is preliminary and constantly evolving and this document will be updated periodically.
    5 * Building on the concepts and constructs within the experiment lifecycle, it describes how elm integrates with fedd, seer and cedl.
    6 * It discusses goals for the August review
     4This page documents a collection of preliminary ideas on experiment lifecycle management. It serves the following purposes:
    75
    8 ==Overview==
     6  * It discusses the high-level concepts of an experiment lifecycle and its isomorphism to the software development lifecycle
     7
     8  * It discusses how the experiment lifecycle plug-in will integrate with the DETER testbed and related technologies
     9
     10
     11== Overview ==
    912The current experimental testbed services primarily focus on providing experimenters access to testbed resources with little or no help to configure, correctly execute,
    1013and systematically analyze the experiment data and artifacts. Additionally, while it is well know that experimentation is inherently iterative, there are limited mechanisms to integrate and cumulatively build upon experimentation assets and artifacts during the configure-execute-analyze phases of the experiment lifecyle.
     
    1518
    1619The key capabilities of the ELM plug-in include:
    17 * Mechanisms to record variations and derivations of the experiment assets and artifacts and along with their inter-realtionships for the entire set of tasks over which an experimenter iterates during the study.
    1820
    19 * Inform design and analysis tools to obtain maximum information with the minimum number of experiment trials for a particular study. Every measured value in an experiment is fundamentally a random variable. Hence there are slight variations in the measurements during a trial even when all experimentation factors are kept constant. Hence to be able to characterize such stochastic behavior, it is necessary to execute multiple repetitions and identify confidence levels. Leveraging the tools in the analysis phase, feedback from the analysis phase can be used to control the number of required repetitions for statistically significant results.
     21 
     22 * Mechanisms to record variations and derivations of the experiment assets and artifacts and along with their
     23 inter-realtionships for the entire set of tasks over which an experimenter iterates during the study.
    2024
    21 * Facilitate composition of functional and structural elements of the experiment based on stated and unstated constraints. The ELM workbenches allow creating and linking functional elements of the experiment without specifying the underlying structure and topology. Resolving the constraints to configure a set of realizable and executable experiment trials is  a complex constraint satisfaction problem.
     25 * Inform design and analysis tools to obtain maximum information with the minimum number of experiment trials for a particular study. Every measured value in an experiment is fundamentally a
     26 random variable. Hence there are slight variations in the measurements during a trial even when all experimentation factors are kept constant. Hence to be able to characterize such stochastic behavior,
     27 it is necessary to execute multiple repetitions and identify confidence levels. Leveraging the tools in the analysis phase, feedback from the analysis phase can be used to control the number of
     28 required repetitions for statistically significant results.
    2229
    23 * Facilitate experiment monitoring and analysis for accuracy of results and availability of resources and services. ELM+SEER will enable monitoring the experiment configuration and performance of resources to ensure the experiment is executed correctly. While resource misconfiguration and failures are easier to spot, identifying "incorrect performance" of a resource or service is extremely hard. For stochastic processes as seen typically in networked systems, it is very important to be able to identify such experimentation errors as they can significantly impact results and bias measurements.
     30 * Facilitate composition of functional and structural elements of the experiment based on stated and unstated constraints. The ELM workbenches allow creating and linking functional elements
     31 of the experiment without specifying the underlying structure and topology. Resolving the constraints to configure a set of realizable and executable
     32 experiment trials is  a complex constraint satisfaction problem.
     33 
     34 * Facilitate experiment monitoring and analysis for accuracy of results and availability of resources and services. ELM+SEER will enable monitoring the experiment configuration and performance of 
     35 resources to ensure the experiment is executed correctly. While resource misconfiguration and failures are easier to spot, identifying "incorrect performance" of a resource or service is extremely
     36 hard. For stochastic processes as seen typically in networked systems, it is very important to be able to identify such experimentation errors as they can significantly impact results and bias
     37 measurements.
    2438
    25 * Enable reuse of experiment assets and artifacts. Reuse is driven by the ability to discover the workflows, scenarios, and data. The ELM environment will provide registry and registry views, along with (RDF-based, DAML+OIL) metadata to facilitate the discovery process. ELM will provide tools to index and search semantically rich descriptions and identify experimentation components including models, workflows, services and specialized applications. To promote sharing, ELM will provide annotation workbenches that allow experimenters to add sufficient metadata and dynamically link artifacts based on these annotations.
     39 * Enable reuse of experiment assets and artifacts. Reuse is driven by the ability to discover the workflows, scenarios, and data. The ELM environment will provide registry and registry views, along
     40 with (RDF-based, DAML+OIL) metadata to facilitate the discovery process. ELM will provide tools to index and search semantically rich descriptions and identify experimentation components
     41 including models, workflows, services and specialized applications. To promote sharing, ELM will provide annotation workbenches that allow experimenters to add sufficient metadata and dynamically
     42 link artifacts based on these annotations.
    2643
    27 * Support for multi-party experiments where a particular scenario can be personalized for a team in a ''appropriate'' way by providing restricted views and control over only certain aspects of the experiment. The registry view will allow the team to access only a restricted set of services. The analysis perspectives and views will present relevant animations and graphs to the team. Thus by personalizing a scenario view, the same underlying scenario, can be manipulated and observed in  different ways by multiple teams.
     44 * Support for multi-party experiments where a particular scenario can be personalized for a team in a ''appropriate'' way by providing restricted views and control over only certain aspects of the
     45 experiment. The registry view will allow the team to access only a restricted set of services. The analysis perspectives and views will present relevant animations and graphs to the team. Thus by
     46 personalizing a scenario view, the same underlying scenario, can be manipulated and observed in  different ways by multiple teams.
    2847
    2948
     
    3554Given the above ELM environment, the basic process of creating a scenario consists of the following steps in a spiral:
    3655 
    37 '''Composition Phase '''
    38 *  defining the functional components and functional topology of the study.
    39 * defining the abstractions, models, parameters, and constraints for each functional component
    40 * identifying/defining the experiment workflow and invariants
    41 * identifying/defining the structural physical topology
    42 * Composing the experiment trials by resolving the constraints and exploring the parameter space
     56 '''Composition Phase '''
     57 *  defining the functional components and functional topology of the study.
     58 * defining the abstractions, models, parameters, and constraints for each functional component
     59 * identifying/defining the experiment workflow and invariants
     60 * identifying/defining the structural physical topology
     61 * Composing the experiment trials by resolving the constraints and exploring the parameter space
    4362
    44 '''Execution Phase'''
    45 * sequential or batched execution of experiment trials
    46 * monitoring for error and configuration problems
     63 '''Execution Phase'''
     64 * sequential or batched execution of experiment trials
     65 * monitoring for error and configuration problems
    4766
    48 '''Analysis Phase'''
    49 * analyzing completed trials (some trial may still be executing)
    50 * presenting results to experimenter
    51 * feedback parameters into the composition tools
    52 * annotate data and artifacts and store in the repositories
     67 '''Analysis Phase'''
     68 * analyzing completed trials (some trial may still be executing)
     69 * presenting results to experimenter
     70 * feedback parameters into the composition tools
     71 * annotate data and artifacts and store in the repositories
    5372
    54 == Integration with DETER Technologies ==
    55 The diagram below describes how ELM, fedd, SEER and CEDL interact
    56 
    57   |------------------------|
    58 ELM --> CEDL --> fedd --> SEER
    59                  |                                ^
    60                  ---------------- |
     73== Integration with DETER ==
     74 A scenario construct captures
    6175
    6276(Place holder: need to update)