This page is a work in progress. Please be sure to read and be familiar with the Emulab documentation.
The testbed needs to know what switches are connected to it and what power ports they are plugged into. Right now, we insert these manually into the database.
Testbed nodes need to be setup to boot from the network by default. This is done through the Preboot eXecution Environment, available for most network cards. For onboard network cards, it is typically enabled through the BIOS.
These filesystems are PXE booted over the network via TFTP and allow us to perform various parts of node maintenance.
There are three different MFS (memory file system) images that come with DETER/Emulab.
They are:
The reason all these tasks are split up among multiple images is to keep the image size down since they are booted over the network. With faster networks, these images will likely be rolled into a single Linux based image in the future.
Each site will have to install root SSH keys from boss into each MFS and change the root password.
This process, along with fetching/unpacking the MFS tarball, has been automated by the script setup_mfs in testbed/install:
[jjh@boss ~/testbed/install]$ sudo ./setup_mfs -h Usage: setup_mfs The setup phase is always performed. Options not required -g Get the MFS from web and extract -f <filename> Use provided mfs tar.bz2 archive
This script will configure all three MFS images with your boss's root ssh key and a password of your choice.
The a copy of the MFS tarball is located in /share/tarballs now.
[deterbuild@boss ~/testbed/install]$ sudo ./setup_mfs -f /share/tarballs/deter-mfs.tar.bz2 ################################################################################ # Extracting /share/tarballs/deter-mfs.tar.bz2 to /usr/testbed/tftpboot # Please enter a MFS root password Password: Verifying - Password: ################################################################################ # Setting up MFS: /usr/testbed/tftpboot/freebsd/boot # Created md0... Created /mnt-md0 ....
You can use the '-g' option to fetch the latest tarball as well:
[jjh@boss ~/testbed/install]$ sudo ./setup_mfs -g
################################################################################
# Fetching http://www.deterlab.net/~jjh/Deter%20OS%20Images/deter-mfs.tar.bz2
# Extracting to /usr/testbed/tftpboot
#
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 37.4M 100 37.4M 0 0 3295k 0 0:00:11 0:00:11 --:--:-- 3287k
Please enter a MFS root password
Password:
Verifying - Password:
################################################################################
# Setting up MFS: /usr/testbed/tftpboot/freebsd/boot
#
Created md0...
Created /mnt-md0
Changing the image root password...
Locking toor...
Unmounting /mnt-md0
Removing mount point /mnt-md0
Unconfiguring md0
Running prepare on mfs and kernel
loader.conf:
loader.rc:
kernel:
mfsroot: mfsroot.gz updated...
################################################################################
# Setting up MFS: /usr/testbed/tftpboot/freebsd.newnode/boot
#
Created md0...
Created /mnt-md0
Changing the image root password...
Locking toor...
Unmounting /mnt-md0
Removing mount point /mnt-md0
Unconfiguring md0
Running prepare on mfs and kernel
loader.conf:
loader.rc:
kernel:
mfsroot: mfsroot.gz updated...
acpi.ko:
################################################################################
# Setting up MFS: /usr/testbed/tftpboot/frisbee/boot
#
Created md0...
Created /mnt-md0
Changing the image root password...
Locking toor...
Unmounting /mnt-md0
Removing mount point /mnt-md0
Unconfiguring md0
Running prepare on mfs and kernel
loader.conf:
loader.rc:
kernel:
mfsroot: mfsroot.gz updated...
[jjh@boss ~/testbed/install]$
By default, the MFS images are set to use the video card as the console device. Before you start, please familiarize yourself with the basics of FreeBSD serial consoles.
A serial console is typically used in conjunction with IPMI. Sometimes IPMI gives access to the COM1 serial interface of a machine (for example HP DL160G6 servers use COM1). Other systems dedicate the COM2 port to the IPMI embedded server. You will have to figure out how your particular hardware is setup. Some of these steps will be handled by the setup_mfs script in the future.
By default, we use 115200 as the baud rate for all MFS images. This is a compile time value, so it is best to use 115200.
When the PXE boot ROM is loaded during machine boot.
The default bootloader for testbed nodes is /tftpboot/pxeboot.emu. There are four different versions of pxeboot.emu distributed with the tarball.
Pick a loader that best suits your installation and copy it (backing up the VGA boot loader first):
cp /tftpboot/pxeboot.emu /tftpboot/pxeboot-vga.emu cp /tftpboot/pxeboot.emu-<null|sio|sio2|vga> /tftpboot/pxeboot.emu
You can also look at modifying /usr/local/etc/dhcpd.conf.template to use different boot loaders for different machines. You will have to generate a new dhcpd.conf configuration using /usr/testbed/sbin/dhcpd_makeconf -i -r.
Edit the file loader.conf.orig for each MFS image:
[jjh@boss /tftpboot]$ find /tftpboot/ -name loader.conf.orig /tftpboot/freebsd/boot/loader.conf.orig /tftpboot/freebsd.newnode/boot/loader.conf.orig /tftpboot/frisbee/boot/loader.conf.orig
At the top of the loader.conf.orig file you will be presented with three choices:
## ## Pick a console: ## ## "comconsole" If you are using a serial port as your console ## "vidconsole" If you are using the VGA as your console on your nodes ## "nullconsole" If you don't have VGA or serial lines, and it appears that ## nodes are not booting, try using this, which gives you a ## non-interactive version of the boot loader. Some sites ## have had problems with "phantom" serial line input from a ## non-existant serial line causing the comconsole version of ## the boot loader to drop into interactive mode. ## console="comconsole" #console="vidconsole" #console="nullconsole"
FreeBSD picks the serial console based on a special flag for the UART you want to use. UART 0 is COM1 and UART 1 is COM2.
[jjh@boss /tftpboot]$ find . -name device.hints -exec grep flags {} \; | grep uart
hint.uart.0.flags="0x10"
hint.uart.0.flags="0x10"
hint.uart.0.flags="0x10"
To use COM2, the second UART, change the hint.uart.0.flags to hint.uart.1.flags
You will have to run the prepare script in each boot directory after changing loader.conf.orig or the device.hints file.
root@boss:/tftpboot/freebsd/boot # ./prepare loader.conf: loader.conf.gz updated... loader.rc: kernel: mfsroot:
Generally each node will have a control network interface and experimental interfaces. The control network interface should be on a switch port that is on the CONTROL (VLAN 2003) network. The experimental interfaces should be on ports that are enabled, but can be in a default VLAN for now.
The testbed nodes should be set to boot only off of the network. Disable hard drive boot to prevent failed PXE requests from falling through to booting whatever is on the disk.
It is important to know what sort of disks your new nodes have. FreeBSD differentiates between SATA and SAS drives by using the device types ad and da, respectively. Also, with older machines, the first ATA disk unit number may be 4. You will need to know what disk type and unit number you are dealing with before attempting to create a node type. It is easiest to probably boot a node to a FreeBSD live USB key and run "geom disk list:
$ geom disk list Geom name: cd0 Providers: 1. Name: cd0 Mediasize: 0 (0B) Sectorsize: 2048 Mode: r0w0e0 descr: NECVMWar VMware IDE CDR10 ident: (null) fwsectors: 0 fwheads: 0 Geom name: da0 Providers: 1. Name: da0 Mediasize: 85899345920 (80G) Sectorsize: 512 Mode: r2w2e3 descr: VMware Virtual disk ident: (null) fwsectors: 63 fwheads: 255
So for this example, our type is da and our unit is 0.
Before creating a node type, you should know which interface is the control network interface on your node. Typically it is the first interface on the system, but sometimes it isn't. You can double check that you have the correct interface by watching a node boot into the "freebsd.newnode" MFS. During the boot, dhcp requests will be sent out of all interfaces in order to determine the control network interface. Here is an example of a machine with 6 interfaces. It turns out that the unit for the control network is 4 instead of 0.
Emulab looking for control net among: igb0 igb1 igb2 igb3 igb4 igb5 ... igb5: link state changed to UP igb3: link state changed to UP igb2: link state changed to UP igb0: link state changed to UP igb4: link state changed to UP Terminated Emulab control net is igb4
You can also log into the MFS as root and run ifconfig -a to see which interface is configured. When new nodes check in, you can also verify that the wiring detected matches which port your control network interface is wired to.
You will have to create a new "Node Type" for each class of node you wish to add to your testbed.
If you are creating a node type for a typical testbed pc, use the class 'pc.' If you are creating a node type for special hardware, choose something else such as 'router' for routers or 'appliance' for appliance type nodes.
See the section on node types. For appliances, two things are important when creating the type:
Connect to the MySQL database on boss (typically tbdb).
mysql> insert into nodes (node_id, type, phys_nodeid, role) values ("juniper1", "junpier", "juniper1", "testnode");
Update the other fields in the node table for this new type of node:
mysql> update nodes set inception=now(), def_boot_path='', def_boot_cmd_line='', next_boot_path='', pxe_boot_path='', rpms='', deltas='', tarballs='', startupcmd='', startstatus='none', bootstatus='unknown', status='', status_timestamp='', failureaction='fatal', eventstate='ISUP', state_timestamp=now(), op_mode='MINIMAL', op_mode_timestamp=now(), allocstate='FREE_DIRTY', allocstate_timestamp=now(), next_op_mode='', ipodhash='' where type="juniper";
Set the node status to up for the newly created nodes:
mysql> insert into node_status values ('juniper1', 'up', NULL);
Card is typically incremented and port is always 1 for testbed nodes (I know, confusing).
mysql> insert into interfaces (node_id, card, port, interface_type, iface, role) values ("juniper1", 1, 1, "fop", "eth0", "expt");
mysql> insert into interfaces (node_id, card, port, interface_type, iface, role) values ("juniper1", 2, 1, "fop", "eth1", "expt");
We must also setup the interface state table:
mysql> insert into interface_state (node_id, card, port, iface) values (\"juniper1", 1, 1, "eth0"); mysql> insert into interface_state (node_id, card, port, iface) values (\"juniper1", 2, 1, "eth1");
Lets say our juniper has two ports on card 2 of our switch on ports 23 and 24:
mysql> insert into wires (node_id1, node_id2, type, card1, port1, card2, port2) values ("juniper1", "hp1", "Node", 1, 1, 2, 23);
mysql> insert into wires (node_id1, node_id2, type, card1, port1, card2, port2) values ("juniper1", "hp1", "Node", 2, 1, 2, 24);
If you have a serial controller, you can add an entry into the tip lines table. The 'tipname' corresponds to the name given to the capture daemon on the serial server:
mysql> insert into tiplines (tipname, node_id, server) values ('juniper1', 'juniper1', 'serial3');
If you have your node hooked up to a power controller, add a line into the outlets table to allow for power cycling:
mysql> insert into outlets (node_id, power_id, outlet) values ('juniper1', 'power7', 3);