Changes between Version 16 and Version 17 of NewAPI


Ignore:
Timestamp:
Mar 31, 2014 5:47:00 PM (11 years ago)
Author:
faber
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • NewAPI

    v16 v17  
    11[[TOC]]
    22
    3 = DETER Testbed API =
    4 
    5 The DETER API is really an interface with two sides, an outward facing side (the testbed API) that allows people to manage resources on the testbed and an inward facing side (the containers API)  that coordinates the containers that make up an experiment environment.  This document defines these interfaces and documents how to use them.  It is a living document that will gain detail as the specification grows and implementations are put into service.  At this point (July 2013) it is mostly a roadmap.
     3= DETER Testbed System Programming Interface (SPI) =
     4
     5The DETER ASPI is really an interface with two sides, an outward facing side (the testbed SPI) that allows people to manage resources on the testbed and an inward facing side (the containers SPI)  that coordinates the containers that make up an experiment environment.  This document defines these interfaces and documents how to use them.  It is a living document that will gain detail as the specification grows and implementations are put into service.
    66
    77The testbed API is responsible for managing the following things:
    … …  
    1313 Circles::
    1414   groups of related users used to manage what users can see and do to other testbed resources
    15  Resources::
    16   building blocks for realizing experiments: computers, disk images, external access
    1715 Experiments::
    1816  a research environment that may be stored, edited, and realized on testbed resources
    1917
    20 The testbed API how users ask a DETER testbed to do things for it.  It will generally be called from a more user-friendly front end, tuned to the user's experience and goals, such as the evoloving DETER Beginners Interface.
     18The testbed SPI enables users to ask a DETER testbed to do things for them.  It will generally be called from a more user-friendly front end, tuned to the user's experience and goals, such as the evoloving DETER Beginners Interface.
    2119
    2220The containers API is used by the DETER control system to manage the resources that make up an experimental topology in progress. The goal of this API is to take raw resources inside the testbed that have been allocated using the testbed API, and configure them into a usable environment.  Each logical element of an experiment (a computer, a router, etc.) is represented by a container which must be managed by DETER.
    2321
    2422Managing a container consists of:
    25 
     23 * Allocating resources to it
    2624 * Installing and configuring any virtualization software or configuring hardware
    2725  * This involves translating from a more generic topology/configuration description into the setup for a specific container type
    … …  
    2927 * Exposing and using container-specific features to provide DETER services
    3028
    31 We discuss the testbed API and then the containers API.  Then we discuss some initial implementation details and present a roadmap for development.
    32 
    33 == DETER Testbed API ==
    34 
    35 The testbed API manipulates a few entities to provide the experimentation environment.  This section discusses the key abstractions of the testbed API and how they work together to create an experimental environment.
     29We discuss the testbed SPI and then the containers SPI.  Then we discuss some initial implementation details and present a roadmap for development.
     30
     31== DETER Testbed SPI ==
     32
     33The testbed SPI manipulates a few entities to provide the experimentation environment.  This section discusses the key abstractions of the testbed SPI and how they work together to create an experimental environment.
    3634
    3735=== Users ===
    3836
    39 A '''user''' is a researcher who uses the DETER testbed.  They request testbed services and allocate testbed resources.  Users are the actors that make things happen through test testbed API.
     37A '''user''' is a researcher who uses the DETER testbed.  They request testbed services and allocate testbed resources.  Users are the actors that make things happen through test testbed SPI.
    4038
    4139Each user is identified by a unique string, their '''userid'''.  Userids are assigned when the user is created on the testbed and guaranteed to be unique.
    … …  
    4745 * A password to authenticate the user
    4846 * A valid e-mail address for communication and password resetting
    49  * Resource access information, e.g.,
    50    * ssh public keys
    51    * windows authentication credentials
    5247 * General metadata, e.g,
    5348   * Real name
    … …  
    5651   * Address
    5752
    58 A user identifies themselves to the testbed API by proving that they hold a specific public/private keypair.  An initial such keypair is issued when the user is created, and a user can acquire another valid pair at any time by proving they know their password.  Generally those pairs are short-lived to guard against loss or theft, but the password is administered using local testbed policies.
     53A user identifies themselves to the testbed SPI by proving that they hold a specific public/private keypair.  An initial such keypair is issued when the user is created, and a user can acquire another valid pair at any time by proving they know their password.  Generally those pairs are short-lived to guard against loss or theft, but the password is administered using local testbed policies.
    5954
    6055=== Grouping Users: Projects and Circles ===
    … …  
    8782==== Circles ====
    8883
    89 Circles are groups of users.  They are used to confer rights to groups of users, identify groups of users, and control how resources are configured when experiments are realized on the testbed.  Circles are intended to be lightweight and encourage collaboration inside a project or across projects. This is a more general use of grouping than an Emulab testbed.
     84Circles are groups of users.  They are used to confer rights to groups of users, identify groups of users, and influence how resources are configured when experiments are realized on the testbed.  Circles are intended to be lightweight and encourage collaboration inside a project or across projects. This is a more general use of grouping than an Emulab testbed.
    9085
    9186A circle always has an owner, the user responsible for its creation.  Ownership can be changed, but only by the owner.
    … …  
    10398Each user in an approved project is a member of at least two circles: a circle attached to the project (created and owned by the owner of the project in the project's name space) and a single-user circle owned (and created by) the uid.  A user bob in project newclass will be a member of the circle bob:bob and newclass:newclass.
    10499
    105 Bob will have the right to realize experiments in the bob:bob circle - that is bob cab create experiments that only bob can access.  Bob will also have whatever rights the owner of the newclass project gave bob when he joined. If bob created the newstuff project, bob will have all rights in the newstuff:newstuff project.  No user has the right to add or remove users from their single-user circle.
     100Bob will have the right to realize experiments in the bob:bob circle - that is bob can create experiments that only bob can access.  Bob will also have whatever rights the owner of the newclass project gave bob when he joined. If bob created the newstuff project, bob will have all rights in the newstuff:newstuff project.  No user has the right to add or remove users from their single-user circle.
    106101
    107102Circles are used to control sharing as well.  If bob and alice are made a team in newclass, the owner of that project, e.g., their instructor or teaching assistant, can make them a circle under the project's namespace.  bob and alice may be made members of newclass:team1.  Now experiments realized in that circle will be accessible to bob and alice.
    … …  
    111106The circle linked to a project is manipulated through the project, not as a circle.  Users added or removed to a project are added to or removed from its linked circle.  Permissions are also manipulated through the project interface.  A simple project may never need to create additional circles.
    112107
    113 === Resources ===
    114 
    115 Resources are the physical and conceptual objects managed by the testbed that are used to build experimental environments.  They are the computers, network ports, externally routable addresses, virtual machine images, et al. from which experiments are constructed.
    116 
    117 Resources are a class of objects that are user to build experiments.  Now there are are a few well known resources that are visible to the API, but generic resource objects are also supported by the API and provide a way to integrate new building blocks.  Some of the specialized resources are:
    118 
    119  * computers
    120  * links
    121  * disk images
    122  * ssh keys
    123 
    124 Access to and configuration of resources is affected by the project(s) a user is acting as a member of.  When a user requests resources, thay specify the projects under which they are requesting them.  A user requesting resources as a member of a project representing a university class may have access to different resources than one acting as a member of the testbed administration.  How membership affects the resources a group can claim is set by testbed [wiki:NewAdmin policy].
    125 
    126 In addition, the membership in projects controls how resources will be configured.  A resource in use by a particular project will generally be configured to be accessible to all members of that project.  A student who allocates resources as a member of a small design group while implementing a class project may later allocate resources as a member of the whole class when presenting the work to the class's TA and professor.
    127108
    128109=== Experiments ===
    … …  
    142123 * Storing the experiment specification for repeated use
    143124 * Sharing the experiment specification between users subject to [wiki:NewPolicy policy]
     125 * Constructing an experiment's environment by calling out through teh Containsers SPI
     126
     127We call the process of constructing the experiment's environment ''realizing'' the experiment.
     128
     129In order to support experiments that are making minimal use of experiment control systems, the API allows a user to manipulate a realized experiment, including
     130
     131 * viewing topology and state of containers
     132 * low-level operations on containers (start, shutdown, reconfigure)
     133
     134These interfaces use filtering and graph analysis to present useful views of large experiments.
     135
     136When realizing the experiment, the testbed uses the containers API to configure and control the hardware and software that create the environment.
     137
     138== The Containers API ==
     139
     140The containers API is responsible for allocating resources to an experiment and configuring that hardware in ways appropriate for the user's goals - realizing it.
     141
     142The key abstraction for realizing an experiment is the '''container'''.  A container holds some of a physical resource's computational and networking power and uses it to create part of the experiment at a level of realism appropriate to the researcher's goals.  A researcher who is interested in end-system behavior will not put very much computational power into realizing the routers that forward packets, but a lot into realizing end systems.
     143
     144The containers API coordinates:
     145
     146 * Allocating the resources from the testbed's pool of resources
     147 * Configuring the physcial resources to realize the experiment
     148 * Partitioning the computational and networking power of each resource into appropriate realizations
    144149 * Assigning resources to carry out the experiment
    145150 * Configuring resources using containers so that the experiment can be carried out successfully on limited resources
    146151 * Initializing and supporting an experiment control system like MAGI to carry out the experiment's procedure, police invariants, and gather data.
    147 
    148 These last three bullets construct the environment for experimentation.  We call that process ''realizing'' the experiment.
    149 
    150 In order to support experiments that are making minimal use of experiment control systems, the API allows a user to manipulate a realized experiment, including
    151 
    152  * viewing topology and state of containers
    153  * low-level operations on containers (start, shutdown, reconfigure)
    154 
    155 These interfaces use filtering and graph analysis to present useful views of large experiments.
    156 
    157 When realizing the experiment, the testbed uses the containers API to configure and control the hardware and software that create the environment.
    158 
    159 == The Containers API ==
    160 
    161 The containers API is responsible for allocating resources to an experiment and configuring that hardware in ways appropriate for the user's goals - realizing it.
    162 
    163 The key abstraction for realizing an experiment is the '''container'''.  A container holds some of a physical resource's computational and networking power and uses it to create part of the experiment at a level of realism appropriate to the researcher's goals.  A researcher who is interested in end-system behavior will not put very much computational power into realizing the routers that forward packets, but a lot into realizing end systems.
    164 
    165 The containers API coordinates:
    166 
    167  * Allocating the resources from the testbed's pool of resources
    168  * Configuring the physcial resources to realize the experiment
    169  * Partitioning the computational and networking power of each resource into appropriate realizations
    170152
    171153This API supports communication between the testbed control system and the containers that make up an experiment being realized (and after it is realized).
    … …  
    216198From that point the testbed can issue Clean, Configure, and Start commands to adjust the container state and operation.
    217199
     200=== Resources ===
     201
     202Resources are the physical and conceptual objects managed by the testbed that are used to build experimental environments.  They are the computers, network ports, externally routable addresses, virtual machine images, et al. from which experiments are constructed.
     203
     204Resources are a class of objects that are user to build experiments.  Now there are are a few well known resources that are visible to the SPI, but generic resource objects are also supported by the SPI and provide a way to integrate new building blocks.  Some of the specialized resources are:
     205
     206 * computers
     207 * links
     208 * disk images
     209 * ssh keys
     210
     211Access to and configuration of resources is affected by the project(s) a user is acting as a member of.  When a user requests resources, thay specify the projects under which they are requesting them.  A user requesting resources as a member of a project representing a university class may have access to different resources than one acting as a member of the testbed administration.  How membership affects the resources a group can claim is set by testbed [wiki:NewAdmin policy].
     212
     213In addition, the membership in projects controls how resources will be configured.  A resource in use by a particular project will generally be configured to be accessible to all members of that project.  A student who allocates resources as a member of a small design group while implementing a class project may later allocate resources as a member of the whole class when presenting the work to the class's TA and professor.
     214
     215
    218216=== Configuration and Specialization ===
    219217
    … …  
    229227To understand how the pieces above fit together, here is an example of a researcher using DETER.
    230228
    231 A person has an idea for research they would like to to conduct on DETER.  That researcher accesses a DETER user interface - for example teh DETER web interface - and fills out a form that gathers information about them and their proposed research.  The web interface translates this into a request to create a user and then to create a vetting project for that user.  The information about the research is routed to the DETER administration.
     229A person has an idea for research they would like to to conduct on DETER.  That researcher accesses a DETER user interface - for example the DETER web interface - and fills out a form that gathers information about them and their proposed research.  The web interface translates this into a request to create a user and then to create a vetting project for that user.  The information about the research is routed to the DETER administration.
    232230
    233231After the DETER administration decides that the research is reasonable and meets their criteria, they authorize creation of the vetting project.  The administration sends the user e-mail telling them they can use DETER.  Additionally the testbed may add the new user to a project for new users or other projects for researchers in similar fields.