New Infiniband Network for NIH Biowulf (Biowulf)

The Biowulf staff is pleased to announce the addition of 224 Infiniband-
connected compute nodes to the cluster.  These nodes are to be used for
moderate- to large-size parallel jobs (>16 processors); smaller parallel
jobs should continue using Myrinet or the older Infiniband (InfiniPath)
networks.

The new nodes are 8-processor nodes based on the Intel EMT-64  
architecture running at 2.8 GHz; therefore only code scaling to 
16-processors (2  nodes) or more should be submitted to these nodes.  
To allocate the new  Infiniband nodes, use the "ib" property in the qsub 
command.  To allocate nodes on the previous InfiniPath network, use 
the "ipath" property.  Note that  code compiled to run on one IB 
implementation will not run on the other!

At this time, NAMD is the only systems staff-supported application  available
to run on the new IB nodes.  We hope to announce the availability of  AMBER,
GROMACS and charmm in the near future.  Note that the format of the batch
control script has changed for jobs submitted to the new IB network; see
http://biowulf.nih.gov/apps/namd.html#ib for details.

NAMD benchmarks run by the Biowulf staff indicate reasonable scaling
on standard NAMD molecular systems: 78% efficiency on ApoA1 (64
processors) and 79% on STMV (128 processors).  Users MUST benchmark
their own systems to determine the optimal number of processors on
which to run.  See http://biowulf.nih.gov/apps/namd.html#benchmarks
for details on checking the scalability of your jobs.  Also see
http://www.ks.uiuc.edu/Research/namd/wiki/index.cgi?NamdPerformanceTuning
for tuning your NAMD configuration for best performance.  Keep in mind  
that smaller jobs (16-32 processors) may run better on the older InfiniPath
network.

Users wishing to compile code to run on the new IB nodes should see
http://biowulf.nih.gov/development.html#mpi (section entitled "MPI
over Infiniband") for instructions.

Please send problem reports or questions to staff@biowulf.nih.gov.