- Number 310 |
- April 26, 2010
Hot Graphics Cards Fuel Supercomputing
GPU cluster
The hottest video games on the market often have the most realistic
graphics. And the key to such remarkable video is a device called a
graphics processing unit, or GPU. Now, scientists at DOE's Jefferson
Lab are using the power of GPUs to study some of the most fundamental
problems in the universe.
"The reason graphics processors are so powerful is so that they make
your game look realistic. They need to be able to compute and draw
lots of things—at least thirty times a second," said Chip Watson,
manager of Jefferson Lab's High-Performance Computing group in the IT
Division.
This fast computation can also be applied to "drawing things" that are
too small to see directly, such as the sub-atomic particles studied by
nuclear physicists at Jefferson Lab. Interestingly, GPUs owe their
ability to render realistic graphics to physics.
"One of the things game programmers found is that they couldn't
realistically depict something on the screen unless they computed the
equations of motion for things. So, the gamers need lots of physics to
make the games nice," Watson explained.
In gaming systems, those equations of motion describe how a ball arcs
through the air, how raindrops splash and how a roundhouse punch will
fell an enemy troll. Analogously, there are also equations that
describe what a sub-atomic particle will do when given extra energy -
a typical problem that nuclear physicists are interested in rendering
virtually.
In the past, such renderings were made using ordinary central
processing units, or CPUs, wired together to act like one, big
supercomputer. Two such "cluster computers" consisting of several
hundred CPUs serve as supercomputers at Jefferson Lab.
Last fall, when Watson was preparing to purchase the components of
Jefferson Lab's next cluster computer, a new series of GPUs and
associated programming tools came on the market. These new tools made
it possible for experienced computer programmers to convert the
equations that apply to sub-atomic particles into a form that could be
processed by the GPUs.
Watson used a portion of a nearly $5 million grant received as part of
ARRA funding under the auspices of DOE's USQCD collaboration to
purchase 200 GPUs and associated hardware for a new computing cluster
dubbed 9G.
"They're installed in 65 nodes. Collectively, those 65 nodes have
considerably more processing power than the thousands of computers
that our collaboration has deployed at Jefferson Lab, Brookhaven and
Fermilab," Watson said.
A key challenge to making the new cluster work was unlocking its
power. That's where the programming tools were needed. It turns out
that the GPUs won't run on the same computer code that runs JLab's
other two clusters. To run the physics problems on GPUs, it took two
collaborators six months of working with the new programming tools and
additional integration work by Jefferson Lab staff to rewrite the
computer code. The new code optimizes the critical part of the physics
problems to run on the new cluster.
"They're a bear to program, but they are definitely worth the effort,"
Watson said. "A single GPU, in a head-to-head computing match-off,
will match the performance of eight of our older cluster nodes."
The switch to GPUs will provide more than 100 Teraflops of computing
power versus the 17 Teraflops that the laboratory would have generated
had it exclusively purchased ordinary CPUs. That equates to about six
times the computing power for the same investment.
Watson, however, is quick to point out that ordinary CPUs aren't out
of the picture altogether as the new cluster carries out some of its
processing with ordinary CPUs.
"There's a small bit of our software that uses, maybe, 95 percent of
the clock time, so we can push all that into the GPU. The GPU runs it
20 times faster. But you still have the five percent you didn't
accelerate. So even though the GPU is 20 times faster, your program is
only running 10 times faster," Watson said. The remaining five percent
is parsed on the ordinary CPU(s) in the cluster.
"The GPUs themselves are pretty cheap, but I still have to put them
into an expensive box. So, four GPUs cost $1,600. The box into which I
put it is more than twice that," Watson said. The expensive box is a
rack-mounted computer case in which the GPUs are installed.
Watson said the laboratory is still getting more bang for the buck,
since one box with four GPUs in it costs only 50 percent more than a
single computer. "To put it another way, $6,000 ends up giving me the
performance of something over $100K. It's just an enormous gain,"
Watson added.
It will also open up new realms of exploration for computational
nuclear physics, according to Robert Edwards, a senior staff scientist
in Jefferson Lab's Theory group. He and his colleagues aim to use the
new system to compute the excited states of multi-quark particles.
These excited states occur when ordinary particles, such as protons
and neutrons, are given extra energy. Such states form a so-called
spectrum of particles, some of which are studied in experiments at
Jefferson Lab and elsewhere.
"Because of the significant increase in performance provided by this
technology, we are now able to undertake calculations that we would
not otherwise have been able to do," Edwards explains.
In the meantime, Watson and his group are getting ready to install
more GPUs at Jefferson Lab. They expect to purchase another 200-300
GPUs over the next year.
Submitted by DOE's Jefferson Lab