Overview

This page is a work in progress. Please be sure to read and be familiar with the ​Emulab documentation.

The testbed needs to know what switches are connected to it and what power ports they are plugged into. Right now, we insert these manually into the database.

Testbed nodes need to be setup to boot from the network by default. This is done through the Preboot eXecution Environment, available for most network cards. For onboard network cards, it is typically enabled through the BIOS.

• ​Preboot Execution Environment on Wikipedia
• ​Intel PXE firmware and utilities

Setting up the PXE environment on boss

Setting up the MFS for testbed nodes

These filesystems are PXE booted over the network via TFTP and allow us to perform various parts of node maintenance.

There are three different MFS (memory file system) images that come with DETER/Emulab.

They are:

• The Admin MFS (/tftpboot/freebsd)
  • Primarily used to create new operating system images using imagezip and ssh
• The New Node MFS (/tftpboot/freebsd.newnode)
  • This is the default image for nodes not explicitly listed in dhcpd.conf.
  • Has scripts to try to identify what type of node is being booted based on node_type variables.
  • Runs a process to enable auto-detection of which switch ports the node is wired into.
• The Frisbee MFS (/tftpboot/frisbee)
  • This image is used when loading an operating system image onto

The reason all these tasks are split up among multiple images is to keep the image size down since they are booted over the network. With faster networks, these images will likely be rolled into a single Linux based image in the future.

Each site will have to install root SSH keys from boss into each MFS and change the root password.

This process, along with fetching/unpacking the MFS tarball, has been automated by the script setup_mfs in testbed/install:

[jjh@boss ~/testbed/install]$ sudo ./setup_mfs -h
Usage: setup_mfs
The setup phase is always performed.  Options not required
  -g Get the MFS from web and extract
  -f <filename> Use provided mfs tar.bz2 archive

This script will configure all three MFS images with your boss's root ssh key and a password of your choice.

[jjh@boss ~/testbed/install]$ sudo ./setup_mfs -g

################################################################################
# Fetching http://www.deterlab.net/~jjh/Deter%20OS%20Images/deter-mfs.tar.bz2
# Extracting to /usr/testbed/tftpboot
#
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 37.4M  100 37.4M    0     0  3295k      0  0:00:11  0:00:11 --:--:-- 3287k
Please enter your MFS root password
Password: 
Verifying - Password: 

################################################################################
# Setting up MFS: /usr/testbed/tftpboot/freebsd/boot
#
Created md0...
Created /mnt-md0
Changing the image root password...
Locking toor...
Unmounting /mnt-md0
Removing mount point /mnt-md0
Unconfiguring md0
Running prepare on mfs and kernel
loader.conf: 
loader.rc: 
kernel: 
mfsroot: mfsroot.gz updated...

################################################################################
# Setting up MFS: /usr/testbed/tftpboot/freebsd.newnode/boot
#
Created md0...
Created /mnt-md0
Changing the image root password...
Locking toor...
Unmounting /mnt-md0
Removing mount point /mnt-md0
Unconfiguring md0
Running prepare on mfs and kernel
loader.conf: 
loader.rc: 
kernel: 
mfsroot: mfsroot.gz updated...
acpi.ko: 

################################################################################
# Setting up MFS: /usr/testbed/tftpboot/frisbee/boot
#
Created md0...
Created /mnt-md0
Changing the image root password...
Locking toor...
Unmounting /mnt-md0
Removing mount point /mnt-md0
Unconfiguring md0
Running prepare on mfs and kernel
loader.conf: 
loader.rc: 
kernel: 
mfsroot: mfsroot.gz updated...
[jjh@boss ~/testbed/install]$  

Installing a PXE boot loader

When the PXE boot ROM is loaded during machine boot.

The default bootloader for testbed nodes is /tftpboot/pxeboot.emu. There are four different versions of pxeboot.emu distributed with the tarball.

• pxeboot.emu-null : Does not display pxeloader prompt
• pxeboot.emu-sio : Displays pxeloader prompt via COM1 serial port
• pxeboot.emu-sio2 : Displays pxeloader prompt via COM2 serial port
• pxeboot.emu-vga : Displays pxeloader prompt via video out

Pick a loader that best suits your installation and copy it:

 cp /tftpboot/pxeboot.emu-<null|sio|sio2|vga> /tftpboot/pxeboot.emu

You can also look at modifying /usr/local/etc/dhcpd.conf.template. You will have to generate a new dhcpd.conf configuration using /usr/testbed/sbin/dhcpd_makeconf > /usr/local/etc/dhcpd.conf and restarting dhcpd.

Testbed Nodes

Network connections

Generally each node will have a control network interface and experimental interfaces. The control network interface should be on a switch port that is on the CONTROL (VLAN 2003) network. The experimental interfaces should be on ports that are enabled, but can be in a default VLAN for now.

BIOS settings for testbed nodes

The testbed nodes should be set to boot only off of the network. Disable hard drive boot to prevent failed PXE requests from falling through to booting whatever is on the disk.