Implementation of 2D domain decomposition in the UCAN gyrokinetic PIC code for non-diffusive transport studies in tokamaks

The massively parallel, nonlinear, 3D, toroidal, electrostatic, gyrokinetic, PIC, Cartesian geometry UCAN code, with particle ions and adiabatic electrons, has been successfully exercised to identify non-diffusive transport characteristics in DIII-D-like tokamak discharges. The limitation in applying UCAN to larger scale discharges is the 1D domain decomposition in the toroidal (or z-) direction for massively parallel implementation using MPI which has restricted the calculations to a few hundred ion Larmor radii per minor radius. To exceed these sizes, we have implemented 2D domain decomposition in UCAN with the addition of the y-direction to the processor mix. This has been facilitated by use of relevant components in the 2D domain decomposed PLIB2 library of field and particle management routines developed for UCLA's UPIC framework of conventional PIC codes. The gyro-averaging in gyrokinetic codes has necessitated the use of replicated arrays for efficient charge accumulation and particle push. The 2D domain-decomposed UCAN2 code reproduces the original 1D domain results within roundoff. Production calculations at large system sizes have been performed with UCAN2 on 131072 processors of the Cray XE6 at NERSC.