-
Datastar
DataStar has 272 (8-way) P655+ and 7
(32-way) P690 compute nodes. The 1.5 GHz 8-way nodes
(176 in number) have 16 GB, the 1.7 GHz 8-way nodes (96
in number) have 32 GB, while the 32-way nodes have 128
GB of memory. There is also one 32-way node with 256 GB
of memory for applications requiring unusually large
memory space. DataStar has a nominal theoretical peak
performance of 15.6 TFlops.
-
Teragrid
SDSC's TeraGrid cluster system is
composed of 262 IBM Itaninum 2 (1.5GHz) nodes with two processors
per each node. Each node is built with SuSe Linux and
interconnected with Myricom's Myrinet network. The
system has a peak performance of 3.1 Teraflops, a total
memory of 1 Terabyte, and a total of 50 Terabytes GPFS
disk through the fiber optic SAN network.
-
Intimidata (Bluegene/L)
SDSC
was the first academic institution to install and offer
allocated time on the IBM BlueGene eServer. Nicknamed
Intimidata, the single-rack server has 2048 processors
offering 5.7 teraflops of compute power and a total
memory of 512 possible gigabytes. The system also offers
the maximum possible ratio of input/output nodes to
compute nodes to support data-intensive computing in a
compact footprint.
The choice of which machine to use is very much a
function of the size and type of job you plan to run and the
allocations you have available. Note, on Datastar and
Teragrid all of AMBER's MD codes are available (sander.MPI,
sander.PIMD.MPI and PMEMD) and so any supported MD job can
be run. On Intimidata (Bluegene/L) the architecture is such
that only PMEMD is available and so only explicit solvent
PME calculations and Generalized Born implicit solvent
simulations can be run. The following tables
provide benchmarks for PMEMD on the various machines. Click
on the machine name in each table to go to a page with
instructions on how to run AMBER simulations on the machine.
All benchmarks are for AMBER 9. All timings are in ps/day
and so large numbers are better. The benchmarks that were
run were Factor IX,
JAC, Cellulose and GB_MB. If you are running
Amber 8 at present it is highly recommended that you switch
to Amber 9 as the performance has been considerably
improved. Bluegene/L needs large scaling for efficient usage
and so only Factor/IX and Cellulose benchmarks are shown.
Factor IX
Large size
explicit water (PME) system. There are 90,906 total
atoms, and PME used with a direct space cutoff of 8.0 Ang.
This benchmark is in amber9/benchmarks/factor_ix. This
uses dt=0.0015, so the conversion is ps-per-day =
129.6/(time-per-step) - Timings are for PMEMD v9.0.
All Bluegene data is for VN mode.
| |
Datastar P655+
1.5GHz (high,normal) |
Datastar P655+
1.7GHz (high, normal) |
Datastar P690
1.7GHz (high32, normal32, highL, normalL) |
TeraGrid (IA64
1.5GHz) |
Intimidata (Bluegene/L
0.7GHz) |
| NCPUS |
Ps/Day |
Speedup |
Ps/Day |
Speedup |
Ps/Day |
Speedup |
Ps/Day |
Speedup |
Ps/Day |
Speedup |
| 1 |
96.67 |
1.00 |
110.71 |
1.00 |
103.65 |
1.00 |
176.11 |
1.00 |
28.73 |
1.00 |
| 2 |
186.15 |
1.93 |
212.71 |
1.92 |
200.83 |
1.94 |
275.74 |
1.57 |
57.32 |
1.99 |
| 4 |
354.11 |
3.66 |
398.11 |
3.60 |
389.63 |
3.76 |
518.00 |
2.94 |
110.17 |
3.83 |
| 8 |
634.73 |
6.57 |
719.46 |
6.50 |
775.96 |
7.49 |
935.50 |
5.31 |
210.81 |
7.34 |
| 16 |
1218.56 |
12.61 |
1380.19 |
12.47 |
1411.46 |
13.62 |
1606.55 |
9.12 |
407.18 |
14.17 |
| 32 |
2292.59 |
23.72 |
2596.15 |
23.45 |
2279.28 |
21.99 |
2652.21 |
15.06 |
739.73 |
25.74 |
| 64 |
4120.17 |
42.62 |
4677.01 |
42.25 |
4041.79 |
38.99 |
3887.81 |
22.08 |
1353.101 |
47.09 |
| 128 |
6822.85 |
70.58 |
7772.11 |
70.20 |
4489.09 |
43.31 |
5737.05 |
32.58 |
2123.55 |
73.91 |
| 192 |
8022.28 |
82.99 |
9784.82 |
88.38 |
|
|
6476.14 |
36.78 |
2495.19 |
86.84 |
| 256 |
9243.94 |
95.63 |
10464.27 |
94.52 |
|
|
|
|
2966.01 |
103.23 |
| 384 |
9123.55 |
94.38 |
10160.72 |
91.78 |
|
|
|
|
3139.16 |
109.25 |
| 512 |
8415.58 |
87.06 |
9452.95 |
85.39 |
|
|
|
|
2997.92 |
104.34 |
| 768 |
6758.80 |
69.92 |
|
|
|
|
|
|
2525.09 |
87.88 |
| 1024 |
6106.01 |
63.17 |
|
|
|
|
|
|
1893.63 |
65.90 |
|
 |
 |
| click
image for larger view |
JAC:
Medium size explicit
water (PME) system. Joint Amber/Charmm DHFR benchmark.
This is the protein DHFR, solvated with TIP3 water, in a
periodic box. There are 23,558 total atoms, and PME
used with a direct space cutoff of 9 Ang. This
benchmark is in amber9/benchmarks/jac. Timings are for
PMEMD v9.0.
| |
Datastar P655+
1.5GHz (high,normal) |
Datastar P655+
1.7GHz (high, normal) |
Datastar P690
1.7GHz (high32, normal32, highL, normalL) |
TeraGrid (IA64
1.5GHz) |
| NCPUS |
Ps/Day |
Speedup |
Ps/Day |
Speedup |
Ps/Day |
Speedup |
Ps/Day |
Speedup |
| 1 |
197.30 |
1.00 |
227.47 |
1.00 |
220.00 |
1.00 |
395.37 |
1.00 |
| 2 |
392.68 |
1.99 |
454.22 |
2.00 |
435.44 |
1.98 |
667.00 |
1.69 |
| 4 |
744.69 |
3.77 |
862.76 |
3.79 |
831.30 |
3.78 |
1200.97 |
3.04 |
| 8 |
1397.29 |
7.08 |
1600.18 |
7.03 |
1682.44 |
7.65 |
2102.80 |
5.32 |
| 16 |
2616.44 |
13.26 |
3008.57 |
13.23 |
3098.11 |
14.08 |
3882.10 |
9.82 |
| 32 |
4851.21 |
24.59 |
5608.93 |
24.65 |
5330.70 |
24.23 |
6115.52 |
15.47 |
| 64 |
8368.85 |
42.42 |
9602.13 |
42.21 |
8803.75 |
40.02 |
7948.48 |
20.10 |
| 128 |
12087.3 |
61.26 |
14154.65 |
62.23 |
11647.34 |
52.94 |
11131.15 |
28.15 |
| 192 |
12910.94 |
65.44 |
14285.71 |
62.80 |
|
|
10611.64 |
26.84 |
| 256 |
13688.21 |
69.38 |
15428.57 |
67.83 |
|
|
10359.71 |
26.20 |
| 384 |
10775.75 |
54.62 |
12206.84 |
53.66 |
|
|
|
|
| 512 |
9538.53 |
48.35 |
6501.13 |
28.58 |
|
|
|
|
| 768 |
7116.97 |
36.07 |
|
|
|
|
|
|
| 1024 |
5566.29 |
28.21 |
|
|
|
|
|
|
|
 |
 |
| click
image for larger view |
Cellulose:
Very Large size explicit
water (PME) system. This us a cellulose fibre of 408K
atoms solvated in TIP3 water, in a periodic box. PME
is
used with a direct space cutoff of 9 Ang. The large size of
this system means that efficient scaling to large numbers of
cpus is possible. Timings are for
PMEMD v9.0.
| |
Datastar P655+
1.5GHz (high,normal) |
TeraGrid (IA64
1.5GHz) |
Intimidata (Bluegene/L
0.7GHz) |
| NCPUS |
Ps/Day |
Speedup |
Ps/Day |
Speedup |
Ps/Day |
Speedup |
| 1 |
14.44 |
1.00 |
28.41 |
1.00 |
4.55 |
1.00 |
| 2 |
28.01 |
1.94 |
45.90 |
1.62 |
9.10 |
2.00 |
| 4 |
53.96 |
3.74 |
87.52 |
3.08 |
17.71 |
3.89 |
| 8 |
103.03 |
7.14 |
162.61 |
5.72 |
34.09 |
7.49 |
| 16 |
197.27 |
13.66 |
298.53 |
10.51 |
64.44 |
14.15 |
| 32 |
371.40 |
25.73 |
530.92 |
18.69 |
124.16 |
27.27 |
| 64 |
633.87 |
43.91 |
792.22 |
27.89 |
215.23 |
47.28 |
| 128 |
945.14 |
65.47 |
915.74 |
32.34 |
283.97 |
62.38 |
| 192 |
1201.50 |
83.23 |
1153.31 |
40.60 |
391.71 |
86.04 |
| 256 |
1342.24 |
92.97 |
1427.86 |
50.26 |
486.79 |
106.93 |
| 384 |
1521.53 |
105.39 |
1545.76 |
54.41 |
660.02 |
144.98 |
| 512 |
1664.42 |
115.29 |
|
|
739.69 |
162.48 |
| 768 |
2622.16 |
181.63 |
|
|
914.77 |
200.94 |
| 1024 |
2558.48 |
177.22 |
|
|
891.59 |
195.85 |
|
 |
 |
| click
image for larger view |
GB_MB:
Medium size Generalized
Born Implicit Solvent system. Generalized Born myoglobin
simulation. This protein has 2492
atoms, and is run with a 20 Ang. cutoff and a salt
concentration of 0.2 M, with nrespa=4 (long range forces
computed every 4 steps.) This benchmark is in
amber9/benchmarks/gb_mb. Timings are for PMEMD v9.0.
| |
Datastar P655+
1.5GHz (high,normal) |
Datastar P690
1.7GHz (high32, normal32
,highL, normalL) |
TeraGrid (IA64
1.5GHz) |
| NCPUS |
Ps/Day |
Speedup |
Ps/Day |
Speedup |
Ps/Day |
Speedup |
| 1 |
220.14 |
1.00 |
249.93 |
1.00 |
191.51 |
1.00 |
| 2 |
430.69 |
1.96 |
488.04 |
1.95 |
358.97 |
1.87 |
| 4 |
855.55 |
3.89 |
955.14 |
3.82 |
789.85 |
4.12 |
| 8 |
1676.76 |
7.62 |
1916.85 |
7.67 |
1524.46 |
7.96 |
| 16 |
3243.97 |
14.74 |
3696.42 |
14.79 |
2839.12 |
14.82 |
| 32 |
5992.51 |
27.22 |
6858.23 |
27.44 |
4830.59 |
25.22 |
| 64 |
9869.77 |
44.83 |
10701.02 |
42.82 |
7651.44 |
39.95 |
| 128 |
14257.43 |
64.77 |
15882.35 |
63.55 |
10695.72 |
55.85 |
|
 |
 |
| click
image for larger view |
Instructions for Running Amber
Simulations at San Diego Supercomputer Center
Each of the links below will take you to a
page that gives details on how to run amber9 simulations on
that machine at SDSC. This includes a re-cap of the
benchmarks, available queues, example Load Leveler and PBS
job scripts and general tips on maximizing the performance.
Executables are pre-installed on all of the machines.
|
Datastar Instructions |
Teragrid
Instructions | Intimidata (Bluegene/L) Instructions |
|
