Narwhal Cluster

The PDL Narwhal cluster is a "bare metal" computing facility used for system research. Researchers are allocated physical nodes (not virtual machines) for their experiments and have complete remote control of the nodes while the experiment is running. The Narwhal cluster uses a locally customized version of the Emulab testbed software from University of Utah to manage its nodes. Note that Narwhal contains more than one type of node, so users must specify a node type when allocating machines (see table below).

Creating a Narwhal account and joining a Project

Email account creation requests to pdl-gripe@ece.cmu.edu. Be sure to include your Andrew ID and your advisor (if you are a student). Once your account is active, you can login to Narwhal's web interface here: https://narwhal.pdl.cmu.edu/. Once you have an account, you need to join a project in order to allocate nodes. This is done through the "Join Existing Project" section of the "Experimentation" menu (see UserGuide for details).

How to allocate nodes on Narwhal

You must have an account on and be a member of an Emulab Project on Narwhal in order to allocate nodes. Nodes in Emulab experiments can be allocated and managed using either command-line tools or by writing Emulab NS Script files and uploading them to the cluster (normally done through the web). Documentation for either mechanism is linked below:

• Detailed Introductory User Guide --- includes command-line tool usage walkthroughs
• Additional detail about working directly with Emulab NS scripts

Narwhal Node Types

Nodes on Narwhal are allocated by type. The following Node Types are available in Narwhal:

Node Class	Quantity	Node Type	Description
RR	280	rr	IBM Blade Server LS21 - 4 Cores AMD Opteron, 16 GiB RAM, 73.6 GB Disk
BIG	1	big	2013 Intel donation - 80 cores, 1 TiB RAM
QSI	2	qsi	2013 Intel donation - 40 cores, 256 GiB RAM
SUS	34	sus	SUSITNA nodes - 64 cores, 128 GiB RAM, Nvidia Tesla K20c, 40 GbE and FDR10 InfiniBand, 2x 3TB HDDs, 400GB SSD
SUS	2	sus, nofge	Same as above but lack 40 GbE
CI	36	ci	SuperMicro 1U server - 4 Cores Xeon E3-1220 v3, 32 GiB RAM, 2x 400GB Disk, Gigabit Ethernet
WF	616	wf	Cray/Appro GB512X - 32 Threads Xeon E5-2670 @ 2.60GHz, 64 GiB DDR3 RAM, 1x 240GB SSD, Gigabit Ethernet, QLogic QDR Infiniband

Click on a node class for additional hardware specifications.

The node type is used with the makebed script to allocate nodes of a given class. For example, use the /share/testbed/bin/rr-makebed version of the makebed script to allocate RR nodes. You can ls /share/testbed/bin/*makebed to list the current set of node-type scripts available (see UserGuide for more on makebed). If you are using Emulab NS directly (rather than using the makebed script), then use the NS script command add-desire to specify the desired node class.

Logging into allocated nodes

Narwhal has a login node ops.narwhal.pdl.cmu.edu that is on the Narwhal control network. Users can ssh into ops.narwhal and from there access Narwhal nodes.

It is also possible to configure ssh on your local system to use the PDL proxy to access Narwhal nodes. To do this, add the following to your ~/.ssh/config:

Host *.pdl.local.cmu.edu
        # uncomment and customize the following configuration option if your username on the system you are invoking ssh from (e.g., your personal laptop) is different from your PDL username
        #User your_pdl_username
        ProxyCommand ssh proxy.pdl.cmu.edu -W %h:%p

Now ssh (including scp, sftp, rsync, etc.) will work transparently through the proxy server as long as you use the FQDN of your nodes (e.g. something like h0.cdn.disc.pdl.narwhal.local.cmu.edu).

Networking

• Narwhal experiment nodes are on a private PDL network. To access the Internet (e.g. the web, git, apt-get repositories) from a Narwhal node you must go through the PDL Proxy located at http://proxy.pdl.cmu.edu:3128/. This proxy setting has been added to the standard PDL disk images. Prior to March 2020 a proxy server was also available on the ops.narwhal.pdl.cmu.edu node at port 8888/tcp. That server is no longer present, so all references to the ops:8888 or ops.narwhal.pdl.cmu.edu:8888 proxy in existing scripts or experiment configurations should be replaced with the address of the PDL Proxy.
• Nodes may have more than one network interface (e.g. SUS nodes have a gigabit Ethernet for control/login and a 40 GbE network for experiments). Applications that require high performance networking in order to perform well should take care to use the cluster's high speed data network for I/O rather than the control network.
• Internally, node names/FQDN are reported as ending in narwhal.pdl.cmu.edu but these names are only visible within Narwhal. On the CMU campus the Narwhal node names also exist in pdl.local.cmu.edu. For example, a node with the internal Narwhal-only name h0.cdn.disc.narwhal.pdl.cmu.edu will be listed as h0.cdn.disc.narwhal.pdl.local.cmu.edu on the CMU campus network (note the addition of "local" in the domain name). With appropriate routing, this allows narwhal nodes to be referenced by name from other PDL clusters.

Narwhal storage options

In Narwhal, there are multiple types of storage:

Your home directory

/users/$USER - NFS filesystem private to Narwhal. Not intended for storing large data. Emulab manages SSH keys and SSL certificates for you using this directory. Do not directly modify your SSH keys in $HOME/.ssh or your SSL keys in $HOME/.ssl.

Project directories

/proj/ - a directory shared among all users in your project.

Other external NFS mounts

Volumes for large datasets, parallel filesystems, etc.

Node local storage

Different node classes have different local storage options. Please see #Narwhal_Node_Types for more information about different node classes. Common uses for node local storage include scratch space, additional space for installing software, etc. You can create partitions on the disk, but It is recommended that you not modify existing partitions in the partition table.

Your home directory and /proj directories are provided via NFS from an external filer. This is a shared resource and should be treated as such. Please be aware that writes to your home directory from hundreds of nodes could result in a tremendous amount of bandwidth and overhead for the filer. Be cautious and make sure that output, e.g. a core dump, does not unintentionally get written to your home directory. Make sure you set your working directory to a local directory, like /tmp, e.g., cd /tmp && ~/run_experiment

PDL AutoFS Maps

In order to make general PDL automount maps available to Narwhal nodes, AutoFS can be installed and configured to use the PDL LDAP server. A helper script is available to install autofs and perform the minimal configuration:

skitch@ubuntu1:~$ sudo /share/testbed/bin/linux-autofs-ldap
skitch@ubuntu1:~$ ls /n
admin  games      kickstart  oltpbench  scratch  t1disc  vquery
atc    home       media      pana       sid      traces
flow   homestore  music      refdbms    stoat    video 

The linux-autofs-ldap can be run as part of an experiment startup script to enable autofs when an experiment is created.

Additional notes

• PDL Emulab installations do not enable or use Emulab's built in mailing list functionality (some generic Emulab documentation refers to this feature, please ignore it).
• If you attempt to allocate all the available nodes, your experiment is more likely to see a node failure. Node failures are, by default, fatal--the experiment swap-in will fail. If you need a large allocation and are having trouble, contact pdl-gripe@ece.cmu.edu.
• Narwhal monitors all allocated nodes to ensure they are being used. If an experiment sits idle for to long, it will be forcibly swapped out so that others can use the nodes. Narwhal also limits overall duration of node allocations.
• Local data is destroyed on swapout. Intermediate data can (and should) be stored on local disks, but should be copied to persistent storage (e.g. /proj) before the nodes are released or the experiment is swapped out due to max duration or idle-swap. Data on local disks is not preserved after an experiment is swapped out.
• Additional documentation on the PDL lab systems can be found in the PDL GettingStarted Guide.