Search  
DOE Pulse
  • Number 310  |
  • April 26, 2010

Hot Graphics Cards Fuel Supercomputing

GPU cluster

GPU cluster

The hottest video games on the market often have the most realistic graphics. And the key to such remarkable video is a device called a graphics processing unit, or GPU. Now, scientists at DOE's Jefferson Lab are using the power of GPUs to study some of the most fundamental problems in the universe.

"The reason graphics processors are so powerful is so that they make your game look realistic. They need to be able to compute and draw lots of things—at least thirty times a second," said Chip Watson, manager of Jefferson Lab's High-Performance Computing group in the IT Division.

This fast computation can also be applied to "drawing things" that are too small to see directly, such as the sub-atomic particles studied by nuclear physicists at Jefferson Lab. Interestingly, GPUs owe their ability to render realistic graphics to physics.

"One of the things game programmers found is that they couldn't realistically depict something on the screen unless they computed the equations of motion for things. So, the gamers need lots of physics to make the games nice," Watson explained.

In gaming systems, those equations of motion describe how a ball arcs through the air, how raindrops splash and how a roundhouse punch will fell an enemy troll. Analogously, there are also equations that describe what a sub-atomic particle will do when given extra energy - a typical problem that nuclear physicists are interested in rendering virtually.

In the past, such renderings were made using ordinary central processing units, or CPUs, wired together to act like one, big supercomputer. Two such "cluster computers" consisting of several hundred CPUs serve as supercomputers at Jefferson Lab.

Last fall, when Watson was preparing to purchase the components of Jefferson Lab's next cluster computer, a new series of GPUs and associated programming tools came on the market. These new tools made it possible for experienced computer programmers to convert the equations that apply to sub-atomic particles into a form that could be processed by the GPUs.

Watson used a portion of a nearly $5 million grant received as part of ARRA funding under the auspices of DOE's USQCD collaboration to purchase 200 GPUs and associated hardware for a new computing cluster dubbed 9G.

"They're installed in 65 nodes. Collectively, those 65 nodes have considerably more processing power than the thousands of computers that our collaboration has deployed at Jefferson Lab, Brookhaven and Fermilab," Watson said.

A key challenge to making the new cluster work was unlocking its power. That's where the programming tools were needed. It turns out that the GPUs won't run on the same computer code that runs JLab's other two clusters. To run the physics problems on GPUs, it took two collaborators six months of working with the new programming tools and additional integration work by Jefferson Lab staff to rewrite the computer code. The new code optimizes the critical part of the physics problems to run on the new cluster.

"They're a bear to program, but they are definitely worth the effort," Watson said. "A single GPU, in a head-to-head computing match-off, will match the performance of eight of our older cluster nodes."
The switch to GPUs will provide more than 100 Teraflops of computing power versus the 17 Teraflops that the laboratory would have generated had it exclusively purchased ordinary CPUs. That equates to about six times the computing power for the same investment.

Watson, however, is quick to point out that ordinary CPUs aren't out of the picture altogether as the new cluster carries out some of its processing with ordinary CPUs.

"There's a small bit of our software that uses, maybe, 95 percent of the clock time, so we can push all that into the GPU. The GPU runs it 20 times faster. But you still have the five percent you didn't accelerate. So even though the GPU is 20 times faster, your program is only running 10 times faster," Watson said. The remaining five percent is parsed on the ordinary CPU(s) in the cluster.

"The GPUs themselves are pretty cheap, but I still have to put them into an expensive box. So, four GPUs cost $1,600. The box into which I put it is more than twice that," Watson said. The expensive box is a rack-mounted computer case in which the GPUs are installed.

Watson said the laboratory is still getting more bang for the buck, since one box with four GPUs in it costs only 50 percent more than a single computer. "To put it another way, $6,000 ends up giving me the performance of something over $100K. It's just an enormous gain," Watson added.

It will also open up new realms of exploration for computational nuclear physics, according to Robert Edwards, a senior staff scientist in Jefferson Lab's Theory group. He and his colleagues aim to use the new system to compute the excited states of multi-quark particles. These excited states occur when ordinary particles, such as protons and neutrons, are given extra energy. Such states form a so-called spectrum of particles, some of which are studied in experiments at Jefferson Lab and elsewhere.

"Because of the significant increase in performance provided by this technology, we are now able to undertake calculations that we would not otherwise have been able to do," Edwards explains.

In the meantime, Watson and his group are getting ready to install more GPUs at Jefferson Lab. They expect to purchase another 200-300 GPUs over the next year.

Submitted by DOE's Jefferson Lab