Xeon Phi and SpF: Pseudospectral Dynamo Models at NASA Goddard

Simulation 5 min read

Contents

My first summer at NASA Goddard Space Flight Center was 2012. I was accepted as an intern under Dan Duffy at the NASA Center for Climate Simulation, but his satellite mapping project kept running into walls, so I moved to work with Tom Clune in the Software Systems Support Office, porting the core of Tom’s DYNAMO code to GPUs with OpenACC. Hamid Oloso and Maharaj Bhat, also in Tom’s group, were just down the hall and a big help whenever I needed it.

In 2013 I wanted to come back and work with Tom again, and he set up a project for me. Tom led SpF (Spectral Framework), a project to make pseudospectral dynamo simulations scale to very large machines. I shared an office with Jess Vriesema and worked closely with Weiyuan Jiang, both also on SpF.

My own project was to see whether a pseudospectral model could run faster on the Intel Xeon Phi (Intel MIC) cards in the newest nodes of NASA’s Discover cluster.

My summer poster

Summer 2013 intern poster: Accelerating Pseudospectral Models With the Xeon Phi
My poster for the Goddard summer intern poster session, 2013. Click for full size.

Each of those Discover nodes had two 8-core Sandy Bridge CPUs (Xeon E5-2670) and one Xeon Phi 5110P, the first-generation Phi (Knights Corner): 60 cores, 240 hardware threads and 8 GB of memory on a PCIe card. That gave each node 16 CPU cores and 60 Phi cores.

I started with an efficient CPU version of the model as a baseline, then tried each way of putting work on the Phi:

  • Manual offload did not work. Explicit offload only moved simple data types, while SpF used type-bound procedures and derived types with pointer and allocatable components, and implicit offload (shared memory between host and card) was only available for C, not Fortran.
  • Automatic offload of Intel MKL calls did not help either. The heavy work in the Legendre transforms is matrix multiplication (DGEMM), but it was split into many calls, none large enough to be worth sending to the card.
  • Native mode, compiling the whole model for the Phi, worked. The card’s 8 GB of memory capped the resolution. Nested parallelism also gave no gain; MKL threading inside OpenMP tasks was no faster than OpenMP alone.
  • Native mode with MPI, splitting the model between the host CPUs and the Phi, was still in progress when the poster was printed, and looked the most promising. In a first small test, giving the Phi a quarter of the kernels and the host the rest reached about 68 GF/s, against about 40 for an even split.

The Legendre transform was the expensive step in both summers. In 2012 I moved it to the GPU with OpenACC; in 2013 I ran it as MKL DGEMM calls on the Xeon Phi.

Some numbers from the poster:

  • With MKL’s DGEMM and Intel’s own example code, I reached about 550 GFlop/s on the Phi, against the 833 GFlop/s Intel advertised.
  • With enough parallel work, one Phi delivered about twice the Flop/s of the two Sandy Bridge CPUs at about the same rated power (225 W for the Phi, 115 W per CPU), so about twice the performance per watt. (The poster used 105 W per CPU; Intel rates the E5-2670 at 115 W.)
  • In SpF’s own runs, one Phi with 120 threads averaged 163 GF/s, against 102 for one Sandy Bridge CPU with 8 threads. That is 1.6 times the speed at about twice the rated power, so in these runs the Phi had lower performance per watt than the CPU.

My conclusion then: the Phi was much easier to program than a GPU, and gave useful speedups once a problem was large enough, but memory size and the offload models’ limits kept the gains for this model below expectations.

Looking back, the per-watt result was the biggest win the Phi showed, even though it did not hold at this model’s problem sizes.

I later posted the poster on NASA Modeling Guru, our group’s site for sharing modeling work, where it had been viewed about 50 times by 2015. That site is gone now, and as far as I can find, this post is the only public copy.

SpF

Dynamo models simulate how moving, electrically conducting fluid generates magnetic fields, for example in Earth’s liquid outer core or the Sun’s convection zone. The AGU poster gives the strengths of pseudospectral methods for dynamo models as efficiency, accuracy and natural boundary conditions.

In a parallel code, each time step alternates numerical transforms with global memory transposes, and existing codes generally hard-coded how the data is split across processes for each transform. Changing that split, or moving to new hardware, means rewriting core data structures, and an improvement made in one model rarely carries over to another.

SpF moves decomposition and communication into the framework, so an application only describes its numerical steps:

  • Kernels: each phase of the calculation is split into independent kernels that run entirely within one processor. A transform is written once, without knowing how the data is distributed.
  • IndexSpace: large, irregular domains are described with a small algebra of outer products and direct sums over 1D axes.
  • Distributor and Permutor: the framework decides the domain decomposition and moves data between layouts. At petascale it uses multi-stage permutations, and it builds the redistribution metadata with a parallel sort in O(N log N) instead of a naive O(N²).
  • Hybrid parallelism: MPI plus OpenMP, optimized vendor libraries for transforms and solvers, and parallel HDF5 for I/O.
  • Testing: test-driven development with pFUnit 2.0, plus nightly regression tests.

The AGU poster lists three targeted NASA applications: ASH, MoSST and DYNAMO.

AGU Fall Meeting 2013

In December Tom presented SpF at the AGU Fall Meeting in San Francisco. The overview above comes from that poster:

T. Clune, W. Jiang, J. Vriesema and G. Gutmann, “SpF: Enabling Petascale Performance for Pseudospectral Dynamo Models,” AGU Fall Meeting 2013, poster GP51A-1072.

AGU 2013 poster: SpF: Enabling Petascale Performance for Pseudospectral Dynamo Models
Click for full size. Original PDF on the NASA Technical Reports Server.

What I took away

Those two summers got me hooked on HPC. Our runs produced very little to look at, and I came away driven to couple simulation and visualization; I wanted to see the science, not just check numbers.