Sandia SCEPTRE 1.4 Particle Transport Simulation Software (Boltzmann Transport Equation Solver)

Manual text content

SANDIA REPORT SAND2014-15597 Unlimited Release Printed July 2014 SCEPTRE 1.4 Quick Start Guide William J. Bohnhoff, Clifton R. Drumm, Wesley C. Fan, Shawn D. Pautz and Greg D. Valdez Prepared by Sandia National Laboratories Albuquerque, New Mexico 87185 and Livermore, California 94550 Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy's National Nuclear Security Administration under contract DE -AC04-94AL85000. Approved for public release; further dissemination unlimited. 2 Issued by Sandia National Laboratories, operated for the United States Department of Energy by Sandia Corporation. NOTICE: This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government, nor any agency thereof, nor any of their employees, nor any of their contractors, subcontractors, or their employees, make any warranty, express or implied, or assume any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represent that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government, any agency thereof, or any of their contractors or subcontractors. The views and opinions expressed herein do not necessarily state or reflect those of the United States Government, any agency thereof, or any of their contractors. Printed in the United States of America. This report has been reproduced directly from the best available copy. Available to DOE and DOE contractors from U.S. Department of Energy Office of Scientific and Technical Information P.O. Box 62 Oak Ridge, TN 37831 Telephone: (865) 576-8401 Facsimile: (865) 576-5728 E-Mail: reports@adonis.osti.gov Online ordering: http://www.osti.gov/bridge Available to the public from U.S. Department of Commerce National Technical Information Service 5285 Port Royal Rd. Springfield, VA 22161 Telephone: (800) 553-6847 Facsimile: (703) 605-6900 E-Mail: orders@ntis.fedworld.gov Online order: http://www.ntis.gov/help/ordermethods.asp?loc=7-4-0#online 3 SAND2014-15597 Unlimited Release Printed July 2014 SCEPTRE 1.4 Quick Start Guide William J. Bohnhoff, Clifton R. Drumm, Wesley C. Fan, Shawn D. Pautz and Greg D. Valdez Department 01341 Radiation Effects Theory Sandia National Laboratories P. O. Box 5800 Albuquerque, New Mexico 87185-1179 Abstract This report provides a summary of notes for building and running the Sandia Computational Engine for Particle Transport for Radiation Effects (SCEPTRE) code. SCEPTRE is a general purpose C++ code for solving the Boltzmann transport equation in serial or parallel using unstructured spatial finite elements, multigroup energy treatment, and a variety of angular treatments including discrete ordinates. Either the first-order form of the Boltzmann equation or one of the second-order forms may be solved. SCEPTRE requires a small number of open-source Third Party Libraries (TPL) to be available, and example scripts for building these TPL’s are provided. The TPL’s needed by SCEPTRE are Trilinos, boost, and netcdf. SCEPTRE uses an autoconf build system, and a sample configure script is provided. Running the SCEPTRE code requires that the user provide a spatial finite-elements mesh in Exodus format and a cross section library in a format that will be described. SCEPTRE uses an xml-based input, and several examples will be provided. 4 5 CONTENTS Contents .......................................................................................................................................... 5 1. Introduction ............................................................................................................................. 7 2. Obtaining and installing TPL’s ............................................................................................... 8 2.1. Boost................................................................................................................................. 8 2.2. Netcdf ............................................................................................................................... 8 2.3. Trilinos ............................................................................................................................. 9 3. Installing and testing SCEPTRE ............................................................................................. 9 4. Installing and testing radTools .............................................................................................. 10 5. Input description ................................................................................................................... 11 5.1. Simple serial example .................................................................................................... 11 5.2. Simple parallel test problem ........................................................................................... 14 5.3. Creating a parallel mesh file........................................................................................... 14 5.4. Generating a cross section library .................................................................................. 15 5.5. Options for the angular treatment................................................................................... 15 5.6. Solver options................................................................................................................. 17 5.6.1. Wave front sweeping algorithm keywords ............................................................. 17 5.6.2. Krylov solver keywords .......................................................................................... 19 5.7. Source and initial conditions options ............................................................................. 21 5.8. Time dependent capability ............................................................................................. 22 5.9. Adjoint capability ........................................................................................................... 23 References ..................................................................................................................................... 23 Distribution ................................................................................................................................... 25 6 7 1. INTRODUCTION The Sandia Computational Engine for Particle Transport for Radiation Effects (SCEPTRE) (Pautz, Bohnhoff, Drumm, & Fan, 2009) (Pautz, Drumm, Bohnhoff, & Fan, 2009) is a general purpose C++ code for solving the Boltzmann transport equation in serial or parallel using unstructured spatial finite elements, multigroup energy treatment, and a variety of angular treatments including discrete ordinates. The SCEPTRE code remains under active development, containing some well-tested production capability and also some newer, more experimental capability. SCEPTRE has a number of unique features, partially motivated by the application space for which the code was developed, providing for the transport of both neutral and charged particles (photon/electron/positron). SCEPTRE includes capability for solving the Boltzmann equation using many different numerical and iterative methods and allows for a different transport solver to be used for each energy group, enabling the user to apply the most appropriate methods for accuracy and efficiency for each energy group/particle type in the problem. Either the first-order form of the Boltzmann equation or one of the second-order forms of the Boltzmann equation may be solved (Lewis & Miller, 1984) (Duderstadt & Martin, 1979) (Bell & Glasstone, 1970). SCEPTRE provides a wave front sweeping algorithm for the first-order form of the transport equation using Discontinuous Finite Elements (DFE) (Wareing, McGhee, Morel, & Pautz, 2001). In the wave front sweeps-based solver, the entire source term including the self-scatter source is on the right- hand-side of the equation, the solution for each particle direction is determined independently, and the scattering source term is updated until convergence. In addition to the sweeps-based solver, SCEPTRE has a class of algorithms based on an entirely different solution approach with very different iterative convergence properties (Drumm & Lorenz, 1999). With the alternative algorithm, the self-scatter source term is included with the streaming and removal operators on the left hand side of the equation and the spatial and angular dependences of the solution are solved simultaneously for each energy group. The main drawback of this method is that the memory requirement may be large, but if enough memory is available, this method is a useful alternative to the sweeps solver that generally converges more efficiently for charged particles. In this approach, the linear system is constructed and handed off to Trilinos (Heroux & Willenbring, 2003) for solution using one of the Krylov iterative methods available in the Trilinos package. SCEPTRE has the option of using either Discontinuous Finite Elements (DFE) or Continuous Finite Elements (CFE). Use of DFE tends to be more accurate for certain transport problems but is also more expensive. Under certain conditions the use of CFE results in a Symmetric Positive Definite (SPD) linear system that may be solved using the highly-efficient Conjugate Gradients (CG) algorithm. SCEPTRE requires a small number of open-source Third Party Libraries (TPL) to be available, and example scripts for building these TPL’s are provided. The TPL’s needed by SCEPTRE are Trilinos, boost, and netcdf. SCEPTRE uses an autoconf build system, and a sample configure script is provided. Running the SCEPTRE code requires that the user provide a spatial finite- 8 elements mesh in Exodus format and a cross section library in a format that will be described. SCEPTRE uses an xml-based input, and several examples will be provided. 2. OBTAINING AND INSTALLING TPL’S All of the TPL’s needed to build and run SCEPTRE are freely available and may be installed with minimal modification. The purpose of this document is not to provide detailed installation instructions for the TPL’s, but to provide minimal instructions for building each of them, referring the reader to the specific TPL’s support for help 2.1. Boost The boost software may be obtained from boost.org . SCEPTRE uses only header files from boost, so the boost tarball merely needs to be unzipped somewhere. 2.2. Netcdf The netcdf software may be obtained from unidata.ucar.edu/software/netcdf . The netcdf software does require some modifications for use by SCEPTRE, as described below. These are some brief notes on building and installing Netcdf. Netcdf is available from Unidata: http://www.unidata.ucar.edu/software/netcdf/ First download and untar Netcdf into some directory, which we will call NETCDFDIR. Some uses of Netcdf require larger parameter values than specified in the Netcdf source code. There does not appear to be any way to override these values, so source code modifications are required. The following diffs are required in NETCDFDIR/include/netcdf.h: 228c228 < #define NC_MAX_DIMS 1024 --- > #define NC_MAX_DIMS 65536 230c230 < #define NC_MAX_VARS 8192 --- > #define NC_MAX_VARS 524288 232c232 < #define NC_MAX_VAR_DIMS 1024 /**< max per variable dimensions */ --- > #define NC_MAX_VAR_DIMS 8 /**< max per variable dimensions */ To build Netcdf first create a target directory (NETCDF_TARGET_DIR) and an installation directory (NETCDF_INSTALL_DIR). Then do the following: > cd $NETCDF_TARGET_DIR > $NETCDFDIR/configure --prefix=$NETCDF_INSTALL_DIR CC=$CCOMPILER CXX=$CXXCOMPILER FC= F90= --disable-netcdf-4 --disable-shared --disable-dap Note that you need to specify the compilers explicitly, or the Netcdf build system will try to use whatever compilers it thinks are "vendor" ones - even on Linux it doesn't pick up gcc. Next build, test if desired, and install: > gmake > gmake check 9 > gmake install The above has been successfully tested on several systems for Netcdf version 4.3.1.1 with gcc and Intel. Shawn Pautz 2.3. Trilinos The Trilinos software may be obtained from trilinos.sandia.gov . Trilinos uses a cmake build system, and a sample script is shown here. The user needs to modify this script to provide the locations of MPI, LAPACK, and BLAS libraries, and the locations of the boost and netcdf installations. EXTRA_ARGS=$@ cmake \ -D Trilinos_VERBOSE_CONFIGURE:BOOL=ON \ -D CMAKE_VERBOSE_MAKEFILE:BOOL=TRUE \ -D CMAKE_BUILD_TYPE:STRING=RELEASE \ -D TPL_ENABLE_BinUtils:BOOL=OFF \ -D TPL_ENABLE_MPI:BOOL=ON \ -D MPI_BASE_DIR:PATH=$MPIHOME \ -D LAPACK_LIBRARY_NAMES:STRING="mkl_intel_lp64" \ -D LAPACK_LIBRARY_DIRS:PATH=$MKL_LIB \ -D BLAS_LIBRARY_NAMES:STRING="mkl_intel_lp64;mkl_intel_thread;mkl_core;iomp5;pthread;m" \ -D BLAS_LIBRARY_DIRS:PATH=\ "$MKL_LIB:/opt/ComposerXE/2013/composer_xe_2013.1.117/compiler/lib/intel64" \ -D TPL_ENABLE_Boost:BOOL=ON \ -D Boost_INCLUDE_DIRS:PATH=/apps/boost/1.55.0/boost_1_55_0 \ -D TPL_ENABLE_Netcdf:BOOL=ON \ -D Netcdf_LIBRARY_DIRS:PATH=/apps/netcdf/4.3.1.1/intel-13.0_install/lib \ -D Netcdf_INCLUDE_DIRS:PATH=/apps/netcdf/4.3.1.1/intel-13.0_install/include \ -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF \ -D Trilinos_ENABLE_Amesos:BOOL=ON \ -D Trilinos_ENABEL_Anasazi:BOOL=ON \ -D Trilinos_ENABLE_AztecOO:BOOL=ON \ -D Trilinos_ENABLE_Belos:BOOL=ON \ -D Trilinos_ENABLE_Epetra:BOOL=ON \ -D Trilinos_ENABLE_EpetraExt:BOOL=ON \ -D Trilinos_ENABLE_Tpetra:BOOL=ON \ -D Trilinos_ENABLE_Kokkos:BOOL=ON \ -D Trilinos_ENABLE_Ifpack:BOOL=ON \ -D Trilinos_ENABLE_Ifpack2:BOOL=ON \ -D Trilinos_ENABLE_ML:BOOL=ON \ -D Trilinos_ENABLE_SEACAS:BOOL=ON \ -D Trilinos_ENABLE_Teko:Bool=ON \ -D Trilinos_ENABLE_Teuchos:BOOL=ON \ -D Teuchos_ENABLE_COMPLEX:BOOL=OFF \ -D Trilinos_ENABLE_Triutils:BOOL=ON \ -D Trilinos_ENABLE_Zoltan:BOOL=ON \ -D Trilinos_ENABLE_TESTS:BOOL=OFF \ -D Trilinos_ENABLE_DEBUG:BOOL=OFF \ -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \ -D CMAKE_INSTALL_PREFIX:PATH=/apps/trilinos/11.6.1/intel-13.0-openmpi_install \ /apps/trilinos/11.6.1/trilinos-11.6.1-Source 3. INSTALLING AND TESTING SCEPTRE Release 1.4 consists of two tarballs, one for radlib and the other for radTools. radlib is the package that contains the SCEPTRE code, and radTools is a package that contains various pre- and post-processing tools useful for running the code and analyzing the results. The building and 10 installation of radTools will be covered in the next section, and the building and installation of radlib will be covered in this section. A successful build and installation of radlib will create four executables, 1) sceptre, 2) sceptre- 1gFO, which is test code for simple one energy group first order solves, 3) sceptre-UF, which includes an uncollided-flux solve, and 4) tds which is experimental code for time-dependent transport calculations. radlib uses an autoconf build system, and a sample configuration script is provided here: ../configure \ --prefix=/apps/radlib-1.4/intel-13.0-openmpi_install \ --disable-debug \ --with-unit-testing=partial \ --with-opt=2 \ --with-cxx=/opt/openmpi/1.6/intel-13.0/bin/mpicxx \ --with-cc=/opt/openmpi/1.6/intel-13.0/bin/mpicc \ --with-mpi-type=openMPI \ --with-mpi-basedir=/opt/openmpi/1.6/intel-13.0 \ --with-mpirun-command=/opt/openmpi/1.6/intel-13.0/bin/mpirun \ --with-mpi-procs-flag=-np \ --with-boost-incdir=/apps/boost/1.55.0/boost_1_55_0 \ --with-exodus-basedir=/apps/trilinos/11.6.1/intel-13.0-openmpi_install \ --with-nemesis-basedir=/apps/trilinos/11.6.1/intel-13.0-openmpi_install \ --with-netcdf-basedir=/apps/netcdf/4.3.1.1/intel-13.0_install \ --with-trilinos-basedir=/apps/trilinos/11.6.1/intel-13.0-openmpi_install \ --with-lapack-basedir=/opt/intel/compilers/13.0/current/mkl/lib/intel64 \ --with-lapack-libs=mkl_intel_lp64 \ --with-blas-basedir=/opt/intel/compilers/13.0/current/mkl/lib/intel64:\ /opt/intel/ComposerXE/2013/composer_xe_2013.1.117/compiler/lib/intel64 \ --with-blas-libs=mkl_intel_lp64:mkl_intel_thread:mkl_core:iomp5:pthread:m \ CXX=icpc CC=icc This configuration script should be modified to provide paths to the MPI installation, TPL locations, LAPACK and BLAS libraries, and compiler parameters. The following options may be modified as described below: --prefix=/apps/radlib-1.4/intel-13.0-openmpi_install ! location where SCEPTRE is to be installed --disable-debug ! either enable-debug or disable-debug --with-unit-testing=partial ! either partial or full testing suite --with-opt=2 ! optimization level --with-mpi-type=openMPI ! either mpich, mpich2 or openMPI After successfully executing a configure command, radlib is built and tested using gmake: gmake –j4 install gmake –j4 check In order to fully test the installation, gmake check should be used with a debug version of the code, since the test code contains many assert statements that are checked with a debug version. 4. INSTALLING AND TESTING RADTOOLS 11 The radTools package contains a number of useful utilities for pre- and post-processing, and for cross section reformatting. The building and installation of radTools is very similar to that for radlib. The configure line to configure radTools is almost the same as that for radlib, with the following differences. The location of the installation directory is radTools rather than radlib --prefix=/apps/radTools-1.4/intel-13.0-openmpi_install \ and an additional line included, which contains the location of the radlib installation directory. --with-radlib-basedir=/apps/radlib-1.4/intel-13.0-openmpi_install \ radTools is the built and tested by executing: gmake –j4 install gmake –j4 test 5. INPUT DESCRIPTION Running SCEPTRE requires three input items: 1) a mesh file in Exodus (Schoof & Yarberry, 1994) format, 2) a cross section file, and 3) an xml input file. The Exodus mesh file may be generated either by using CUBIT ( CUBIT user's manual ) or a commercial Ansys product ( ANSYS ICEM CFD ). It is not the purpose of this document to provide detailed guidance in generating a mesh, but some information on preparing an Exodus mesh for a parallel run will be included later in this section. Also included later in this section will be some information on generating a cross section file for SCEPTRE. The next two subsections contain sample xml input files for performing a simple serial test and then a simple parallel test, using mesh and cross section files included with the distribution. Then individual sections of the input file will be described in detail. 5.1. Simple serial example This section contains the complete xml input needed to run a simple serial SCEPTRE run. The xml code below is copied into an xml file, e.g. simpleSerialTest.xml. The mesh and cross section files are contained in the radlib distribution, and the Mesh_File and XS_File lines should be edited to point to the location of the radlib installation. The following command line executes the code. mpirun –np 1 $SCEPTRE_BIN/sceptre simpleSerialTest.xml Where $SCEPTRE_BIN is the file containing the radlib binaries. <?xml version="1.0" encoding="utf-8"?> <SCEPTRE_Input> <Mesh_File>/apps/radlib-1.4/src/driver/test/rg402Tri3.gen</Mesh_File> <XS_File>/apps/radlib-1.4/src/driver/test/rg402_10g.xslib</XS_File> 12 <Output_Prefix>simpleSerialTest.exo</Output_Prefix> <Output_Format>Exodus</Output_Format> <Sn_Options> <Sn_Order>4</Sn_Order> </Sn_Options> <Scattering_Options> <Scattering_Order>1</Scattering_Order> <Delta_Function_Scattering_Correction>false</Delta_Function_Scattering_Correction> </Scattering_Options> <Outer_Iteration_Options> <Maximum_Number_Iterations>2</Maximum_Number_Iterations> <Convergence_Tolerance>1.e-3</Convergence_Tolerance> </Outer_Iteration_Options> <Enable_User_Defined_Solvers>true</Enable_User_Defined_Solvers> <Solvers> <Solver name="1stOrder"> <Solver_Form>First_Order</Solver_Form> <Element_Set_Size>1</Element_Set_Size> <Coarse_Sn_Order>4</Coarse_Sn_Order> <Preconditioner>None</Preconditioner> <Error_Control_Options> <Maximum_Number_Iterations>100</Maximum_Number_Iterations> <Convergence_Tolerance>1.e-4</Convergence_Tolerance> <Error_Metrics> <Error_Metric> <Metric_Type>whole</Metric_Type> <Integration_Policy>discrete</Integration_Policy> <Error_Norm>L</Error_Norm> <Error_Order>2</Error_Order> <Sign_Policy>absolute</Sign_Policy> </Error_Metric> </Error_Metrics> <Boundary_Errors> <Metric_Type>null</Metric_Type> </Boundary_Errors> </Error_Control_Options> </Solver> <Solver name="Sn-SAAF"> <Solver_Form>Krylov</Solver_Form> <Angular_Method>Sn</Angular_Method> <Krylov_Transport_Method>SelfAdjoint</Krylov_Transport_Method> <SpatialFE_Method>SpatialCFE</SpatialFE_Method> <Linear_Solver>Aztec</Linear_Solver> <Iterative_Method>CG</Iterative_Method> <Maximum_Number_Iterations>100</Maximum_Number_Iterations> <Convergence_Tolerance>1.e-2</Convergence_Tolerance> <Preconditioner>None</Preconditioner> <Print_Level>medium</Print_Level> </Solver> </Solvers> <Solver_Assignment explicit="true"> <Solver_By_Group_Range>1stOrder 0 4</Solver_By_Group_Range> <Solver_By_Group_Range>Sn-SAAF 5 9</Solver_By_Group_Range> </Solver_Assignment> <Materials enable="true"> <Material name="copper"> <Density>8.96</Density> <Conductor>true</Conductor> </Material> <Material name="iron"> <Density>7.874</Density> 13 <Conductor>true</Conductor> </Material> <Material name="silver"> <Density>10.5</Density> <Conductor>true</Conductor> </Material> <Material name="teflon"> <Density>2.2</Density> <Conductor>false</Conductor> </Material> </Materials> <Material_Assignment enable="true"> <Material block="1">copper</Material> <Material block="2">iron</Material> <Material block="3">silver</Material> <Material block="4">teflon</Material> <Material block="5">copper</Material> </Material_Assignment> <Boundary_Source_Options enable="true"> <Source> <Energy_Expansion enable="true"> <Group index="1">1</Group> </Energy_Expansion> </Source> </Boundary_Source_Options> </SCEPTRE_Input> Many of the input parameters are self-explanatory, but some of the input blocks will be described here, while some are described in later sections. This example problem defines two solvers, the first is a wave front sweeping first-order transport algorithm, and the second is Self-Adjoint Angular Flux (SAAF) (Morel & McGhee, 1999) second-order transport solver. The names of the solvers are arbitrary and may be defined by the user. The solvers are assigned by energy group range; the first-order solver is assigned to energy groups 0-4 (which are photon groups for this test problem) and the SAAF solver is assigned to groups 5-9 (which are electron groups). More details on the solvers will be provided in later sections. The solver-to-group assignments are 0-based, while some of the other inputs are 1- based, which may cause some confusion. This will be made more consistent in future releases. The reason for this inconsistency is that SCEPTRE is C++ code, which is 0-based, while the Exodus database is 1-based, and typically for radiation transport codes, energy groups have been assigned from high energy to low energy starting with group 1. The cross section file for this test problem contains four materials, and each material is sequentially assigned a name in the Materials section. The Density and Conductor keywords are not currently used but must be assigned anyway. The Density keyword contains the material density (g/cm 3 ), and the Conductor keyword is true for conductors and false for dielectric materials. The mesh file for this test problem contains five material blocks, and materials are assigned to blocks by block number in the Material_Assignment section. A boundary source term is assigned in the Boundary_Source_Options section. Boundary sources may be specified by energy group and S N direction. For this test a boundary source of magnitude 1 is assigned to energy group 1 (1-based). Since an Angle_Expansion is not 14 present, an isotropic angular distribution is used by default. More details on specifying boundary and distributed sources will be provided in a later section. 5.2. Simple parallel test problem Running the previous test problem in parallel is straightforward. The name of the mesh file in the xml file is modified to be the prefix of the parallel files existing in the radlib distribution. <Mesh_File>/apps/radlib-1.4/src/driver/test/rg402Tri3.par</Mesh_File> A four-processor run expects to have four mesh files available: /apps/radlib-1.4/src/driver/test/rg402Tri3.par.4.0 /apps/radlib-1.4/src/driver/test/rg402Tri3.par.4.1 /apps/radlib-1.4/src/driver/test/rg402Tri3.par.4.2 /apps/radlib-1.4/src/driver/test/rg402Tri3.par.4.3 Then the run is executed with four processors by mpirun –np 4 $SCEPTRE_BIN/sceptre simpleParallelTest.xml 5.3. Creating a parallel mesh file Running the CUBIT or ANSYS/ICEM CFD software to create a mesh file is outside of the scope of this document, but creating a parallel mesh file from a serial mesh file is briefly described here. An Exodus file is partitioned by executing the nem_slice and nem_spread executables ( SEACAS documentation ), which are included with the Trilinos distribution. The following assumes that a subdirectory named “1” is included in the directory where the mesh files are located. The first step is to execute nem_slice nem_slice -e -l multikl -o rg402Tri3.nem -m mesh=4 rg402Tri3.gen where the mesh keyword specifies the number of files to split the mesh file into. Then the following lines of code are copied into a file named nem_spread.inp Input FEM file = rg402Tri3.gen LB file = rg402Tri3.nem Parallel Disk Info= number=1 Parallel file location = root=./, subdir=. And then the nem_spread executable is run nem_spread This will write the partitioned mesh files into the “1” subdirectory. 15 5.4. Generating a cross section library Cross sections for SCEPTRE have been typically generated by the CEPXS code, which is distributed with the ITS code package ( ITS/CEPXS RSICC ), but SCEPTRE may be run with other multi-group Legendre cross section sets, provided the cross sections are reformatted into a form that SCEPTRE can read. The first step in generating a cross section library for SCEPTRE is to run the CEPXS code, with the following example lines of code copied to the cepinp file. title 5 photon 5 electron group test xsec *first-order *csda *bfp *csdld second-order energy 0.1 cutoff 0.001 legendre 8 no-lines photon-source partial-coupling photons linear 5 electrons linear 5 material fe material cu material ag material c 0.2402 f 0.7598 density 2.2 print leg 0 print-all Running CEPXS generates a bxslib binary-format multi-group Legendre cross section library. This bxslib file is converted to netcdf format for use by SCEPTRE by using the convertBxslibToNetcdf utility, which is located in the radTools installation /bin directory. convertBxslibToNetcdf bxslib This creates a netcdf file named bxslib.ncd, which can be renamed to something more descriptive mv bxslib.ncd 5photon5electronGroups.xslib which is then ready for SCEPTRE. The file may be converted to a readable format by using the ncdump utility, which is contained in the netcdf distribution. ncdump 5photon5electronGroups.xslib > 5photon5electronGroups.asci 5.5. Options for the angular treatment 16 Parameters for the angular quadrature are specified in the Sn_Options section of the xml input <Sn_Options> <Sn_Order>4</Sn_Order> <Angular_Quadrature_Type>Level_Symmetric</Angular_Quadrature_Type> </Sn_Options> The angular quadrature type can be either Level_Symmetric or Lebedev (Lebedev & Laikov, 1999) for multi-dimensional geometries, or Gauss_Legendre or Gauss_Lobatto for one- dimensional geometry. Level-symmetric quadratures are available for even orders up to order 20, where some of the weights begin to be negative, and Lebedev quadrature are available for many order up to order 130. Available Lebedev quadrature orders: 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 34 40 46 52 58 64 70 76 82 88 94 100 106 112 118 124 130 Parameters for the Legendre scattering are included in the Scattering_Options section of the xml input. <Scattering_Options> <Scattering_Order>1</Scattering_Order> <Delta_Function_Scattering_Correction>true</Delta_Function_Scattering_Correction> <Moment_To_Discrete_Type>STANDARD</Moment_To_Discrete_Type> </Scattering_Options> The cross section library must include Legendre orders at least up to the scattering order specified in the SCEPTRE input file. SCEPTRE has two options for dealing with the highly- forward peaked charged-particle scattering cross sections. One of these is an explicit treatment of  -function downscatter (Drumm C. , 2007), which removes a  -function component from the scattering cross sections, greatly reducing the magnitude of the scattering cross sections. If the  - function scattering correction option is specified, the cross section library must include Legendre orders at least one greater than the scattering order specified in the xml input file. Another method for handling the extreme forward-peaked electron scattering is to use a Galerkin scattering treatment (Morel J. , 1989), which is invoked by setting the Moment_To_Discrete_Type to Galerkin . This is the approach that is used in the one- dimensional CEPXS/ONELD and ADEPT codes. In the Galerkin approach, a square moment-to- discrete matrix is constructed that is invertible, so that the conversion from discrete space to moment space and back again is exact. For one-dimensional geometry, the use of Gauss- Legendre quadrature with the same number of Legendre moments as discrete directions is Galerkin. For multi-dimensional geometries, the Galerkin scattering treatment is not unique, but the procedure is the same. A square, invertible moment-to-discrete matrix is constructed, which is able to exactly handle the  -function down scatter. SCEPTRE contains algorithms for computing Galerkin moment-to-discrete matrices for all available quadrature types, level- symmetric, Lebedev, Gauss-Legendre, and Lobatto. 17 For methods resulting in an SPD linear system, the  -function downscatter method should be used rather than the Galerkin method, since use of the Galerkin scattering treatment results in a non-symmetric matrix. 5.6. Solver options SCEPTRE includes capability for solving the Boltzmann equation using many different numerical and iterative methods and allows for a different transport solver to be used for each energy group, enabling the user to apply the most appropriate methods for accuracy and efficiency for each energy group/particle type in the problem. The user can define as many solvers as desired, giving each solver a unique name. <Solver name="Sn-LS"> <!-- First_Order or Krylov --> <Solver_Form>Krylov</Solver_Form> The Solver_Form keyword can be either First_Order to define a wave front sweeping solver, or Krylov to define a solver using Trilinos to perform a simultaneous space/angle solve. 5.6.1. Wave front sweeping algorithm keywords If First_Order is specified as the solver form, the following options may be specified within that Solver section: The Element_Set_Size option is used to control how the solver aggregates spatial elements during the solution process. The default is 1 if this option is not explicitly specified. This is an experimental option; it may lead to increased number of iterations in order to obtain solution convergence if a value greater than 1 is used. The Coarse_Sn_Order option is used to control how the solver aggregates angular directions during the solution process. The default is the value specified in the Sn_Order option described earlier. Values less than that may reduce the time required per iteration, but they also may increase the number of iterations required for solution convergence; the trade-off will be problem- and machine-dependent. The Preconditioner option is used to control how and whether the solver attempts to speed up the iterative process. Valid options are “none” (the default) or “dsa”. If “dsa” is specified, then additional options may be specified with a Preconditioner_Options keyword. Within that option are the following suboptions:  Solver (valid options are “gmres” (the default) or “cg”)  Maximum_Number_Iterations (default is 1000)  Convergence_Tolerance (default is 1.e-10) The Error_Control_Options section is always required with the sweeping algorithm; some of its suboptions do not have default values. The following suboptions are contained in this section: 18  The Maximum_Number_Iterations suboption specifies the maximum number of source iterations that this solver will use. There is no default value, so this suboption must be explicitly specified.  The Convergence_Tolerance suboption specifies the maximum iterative error allowed; iterations will continue until either the error is smaller or until Maximum_Number_Iterations is reached. There is no default value, so this suboption must be explicitly specified.  The Number_Aggregated_Iterations suboption specifies how many source iterations will be used in between the measurement of errors to determine iterative convergence. The default is 1. Higher numbers can help produce a better estimate of the iterative error, but can also cause more iterations to be used than strictly necessary.  The next two suboptions require some explanation. Within the sweep solver are a variety of tests to determine if iterative convergence has been achieved. Some transport quantities of interest may converge to the desired accuracy more rapidly than others. The suboptions Error_Metrics and Boundary_Errors are provided to allow the user to specify one or more iterative tests that must all pass in order for the iterative process to complete. Each estimates the remaining iterative error in some quantity. We describe each in turn:  The Error_Metrics suboption allows the user to specify one or more metrics that measure some quantity in the interior of the problem. The default is “default” – this creates a single interior error metric with default values for each of the various parameters described below. Alternatively the user may desire to explicitly create some error metrics. To do so the user needs to specify one or more Error_Metric sections within the Error_Metrics suboption. Within each of the sections the following options may appear: o Metric_Type : This specifies the region of interest for the error measurement. The following are valid values:  “whole”: The entire volume of the problem will be examined. This is the default.  “region”: Attention will be restricted to a particular region of the problem. If this value is specified the user will also need to include a Region_Name option that specifies the name of the element block to be examined.  “surface”: Fluxes at a surface will be examined. If this value is specified the user will also need to include a Surface_Name option that specifies the name of the surface (Exodus sideset) to be examined.  “leakage”: This is similar to the “surface” option, except that the net leakage at a surface will be examined. The Surface_Name option will also need to be specified.  “null”: This produces an error metric that does nothing. This can be useful if the user desires to have the solver perform a fixed number of iterations regardless of whether convergence has been achieved. o Integration_Policy : This specifies how the various point values of the fluxes are weighted relative to each other. Valid values are:  “discrete”: The set of fluxes is treated as a simple vector quantity; each value is equally weighted. This is the default. 19  “continuous”: A true integration of the fluxes over the mesh is performed. If the volumes and/or shapes of the elements in the mesh differ from each other the continuous option will yield a different value than the discrete option; it is more accurate but more expensive. o Error_Norm : This specifies whether the angular fluxes themselves will be used or if some operator will be applied first. Valid values are  “L”: This applies the familiar L-norm to the errors in the fluxes. This is the default.  “H”: This applies the H-norm, i.e. the L-norm of |∇𝜓| , the magnitude of the gradient in the errors in the fluxes.  “S”: This applies the streaming norm, which is the L-norm of Ω ⋅ ∇𝜓 . o Error_Order : This specifies the power of the metric. Valid values are “1”, “2” (the default), or “I” (infinity). For example, an Error_Norm of “L” and an Error_Order of “2” produces the L2 norm. o Sign_Policy : This determines whether the absolute value of the integrand is used (a true norm) or whether sign cancellations are allowed (as happens when calculating integral quantities such as reaction rates). Valid values are “absolute” (the default) and “signed”.  The Boundary_Errors suboption has been largely superseded by the Error_Metrics suboption. A single metric may be specified that is applied to the entire external boundary. This suboption has the same sections as the Error_Metrics suboption: Metric_Type , Integration_Policy , Error_Order , and Sign_Policy . The difference is that the default for Metric_Type is “null”, and the keyword “external” is used instead of “surface”. 5.6.2. Krylov solver keywords <Solver name="Sn-LS"> <Solver_Form>Krylov</Solver_Form> <!-- Sn, Pn, AngularCFE or AngularDFE --> <Angular_Method>Sn</Angular_Method> <!-- SelfAdjoint, LeastSquares, EvenOddParity, EvenParity, OddParity or FirstOrder --> <Krylov_Transport_Method>LeastSquares</Krylov_Transport_Method> <!-- SpatialCFE or SpatialDFE --> <SpatialFE_Method>SpatialCFE</SpatialFE_Method> <!-- Aztec or Belos --> <Linear_Solver>Aztec</Linear_Solver> <!-- CG or GMRES --> <Iterative_Method>CG</Iterative_Method> <!-- DiagonalScaling or None --> <Scaling>DiagonalScaling</Scaling> <Maximum_Number_Iterations>100</Maximum_Number_Iterations> <Convergence_Tolerance>1.e-2</Convergence_Tolerance> <Krylov_Subspace_Size>200</Krylov_Subspace_Size> <!-- minimum, medium, or maximum --> <Print_Level>medium</Print_Level> <!-- None, MultiLevel, IncompleteFactorization --> 20 <Preconditioner>BlockDiagonal</Preconditioner> <Block_Diagonal_Preconditioner_Options> <Iterative_Method>GMRES</Iterative_Method> <Maximum_Number_Iterations>200</Maximum_Number_Iterations> <Convergence_Tolerance>1.e-3</Convergence_Tolerance> <Krylov_Subspace_Size>200</Krylov_Subspace_Size> <Print_Level>minimum</Print_Level> <Preconditioner>None</Preconditioner> </Block_Diagonal_Preconditioner_Options> </Solver> The Angular_Method is typically Sn for discrete ordinates, and SCEPTRE contains some experimental code for performing a spherical harmonics model using the Pn keyword, or angular finite elements (Drumm, Fan, & Pautz, 2013), using AngularCFE for continuous angular finite elements or AngularDFE for discontinuous angular finite elements. The Krylov_Transport_Method defines which form of the transport equation to use, either FirstOrder for a solution based on the first-order form of the Boltzmann equation, SelfAdjoint for a SAAF form of the transport equation (Morel & McGhee, 1999), or LeastSquares for a least-squares solution (Drumm, Fan, Bielen, & Chenhall, 2011) of the first- order form of the transport equation. Capability for the Even-Odd Parity (EOP) (Duderstadt & Martin, 1979) form of the transport equation is not included in this release of SCEPTRE but will be included in future releases. The SpatialFE_Method is either SpatialCFE continuous finite elements for a LeastSquares or SelfAdjoint algorithm or SpatialDFE for discontinuous finite elements for a first-order transport algorithm. The Linear_Solver is either Aztec or Belos , which are two of the available solvers in Trilinos (Heroux & Willenbring, 2003). The Iterative_Method is either CG to use a parallel preconditioned CG algorithm for algorithms yielding an SPD linear system ( SelfAdjoint or LeastSquares ), or GMRES for non-SPD linear systems ( FirstOrder ). By using Trilinos to perform the iterative linear solve, it is fairly easy to include various preconditioiners in the algorithm. SCEPTRE includes options for three different preconditioners, 1) an incomplete-factorization method, 2) a multi-level method, and 3) a SCEPTRE-specific preconditioner using the diagonals of the blocks in the system matrix as a preconditioner. It generally takes some experimentation to determine which combination of solvers and preconditioners is optimal for a given application and particle/energy group. The Scaling keyword is either DiagonalScaling for symmetric diagonal scaling for SPD linear systems or diagonal scaling for non-SPD linear systems, or None for no scaling. The Krylov_Subspace_Size is only used by the GMRES solver and is the number of Krylov vectors to store in the restarted GMRES algorithm (Heroux, AztecOO User Guide, 2004). The Preconditioner can be either None for no preconditioner of the linear system, IncompleteFactorization for algebraic preconditioning using IFPACK (Sala & Heroux, 2005), or MultiLevel to use multigrid preconditioning using ML (Gee, Siefert, Hu, Tuminaro, & Sala, 2006). SCEPTRE also has a transport-specific preconditioner that uses the diagonals of the blocks in the system matrix as a preconditioner BlockDiagonal , essentially using the uncollided-flux linear 21 system as a preconditioner (Drumm & Fan, 2003). For this preconditioner, there are a number of options that can be set in the Block_Diagonal_Preconditioner_Options section, governing the iterative solution of the preconditioner linear system. 5.7. Source and initial conditions options Boundary conditions for incoming directions will be set if the Boundary_Source_Options keyword is set to true . Any number of individual boundary sources may be defined, and the boundary sources will be added together to determine the value of the boundary angular fluxes for incoming directions on the external boundary of the mesh. The source terms are assumed to be separable in energy, direction and sideset, where a sideset is an Exodus II descriptor, which consists of a set of edges in the finite-elements mesh. If more complicated boundary terms are required, boundary sources may be added to the Exodus mesh file using preprocessing tools in radTools. If the Angle_Expansion section is absent from a particular source, the angular dependence of the source is assumed to be isotropic. If the Energy_Expansion keyword is absent, the energy distribution is assumed to be uniform over all energy groups. If the SideSet_Expansion keyword is absent, the boundary source term is set for the entire external boundary. <Boundary_Source_Options enable="true"> <Source> </Source> <Source> <Scale_Factor>3.</Scale_Factor> </Source> <Source> <Scale_Factor>2.</Scale_Factor> <Energy_Expansion enable="true"> <Group index="1">0.5</Group> <Group index="2">0.2</Group> <Group index="3">0.1</Group> <Group_Range>0.3 4 6</Group_Range> </Energy_Expansion> </Source> <Source> <Scale_Factor>8.</Scale_Factor> <Energy_Expansion enable="true"> <Group index="1">2.5</Group> <Group index="6">3.5</Group> </Energy_Expansion> <Angle_Expansion enable="true"> <Angle index="0">3.2</Angle> <Angle_Range>3.3 2 3</Angle_Range> </Angle_Expansion> <SideSet_Expansion enable="true"> <SideSet name="sideset5">5.</SideSet> </SideSet_Expansion> </Source> </Boundary_Source_Options> Distributed sources are assigned analogously to boundary sources, except that the distributed sources are specified by element block rather than by sideset. 22 <Fixed_Source_Options enable="true"> <Source> </Source> <Source> <Scale_Factor>3.</Scale_Factor> </Source> <Source> <Scale_Factor>2.</Scale_Factor> <Energy_Expansion enable="true"> <Group index="1">0.5</Group> <Group index="2">0.2</Group> <Group index="3">0.1</Group> <Group_Range>0.3 4 6</Group_Range> </Energy_Expansion> </Source> <Source> <Scale_Factor>8.</Scale_Factor> <Energy_Expansion enable="true"> <Group index="1">2.5</Group> <Group index="6">3.5</Group> </Energy_Expansion> <Angle_Expansion enable="true"> <Angle index="0">3.2</Angle> <Angle_Range>3.3 2 3</Angle_Range> </Angle_Expansion> <Block_Expansion enable="true"> <Block name="block4">5.</Block> </Block_Expansion> </Source> </Fixed_Source_Options> The initial condition for the internal angular flux is set using the Internal_Angular_Flux_Initialization_ Options keyword <Internal_Angular_Flux_Initialization_Options enable="false"> <Scale_Factor>1</Scale_Factor> <Energy_Expansion enable="true"> <Group index="0">1</Group> <Group index="1">1</Group> <Group index="2">1</Group> <Group index="3">1</Group> <Group index="4">1</Group> </Energy_Expansion> <Angle_Expansion enable="false"> <!-- Angle index="0">1</Angle --> </Angle_Expansion> <Block_Expansion enable="true"> <Block index="0">1</Block> <Block index="1">0</Block> <Block index="2">0</Block> <Block index="3">0</Block> <Block index="4">0</Block> </Block_Expansion> <Time_Expansion enable="true"> <Time index="0">1</Time> <Time index="1">1</Time> <Time index="2">1</Time> <Time index="3">1</Time> <Time index="4">1</Time> </Time_Expansion> </Internal_Angular_Flux_Initialization_Options> 5.8. Time dependent capability SCEPTRE contains some experimental code for performing time-dependent transport calculations, but this capability is fairly new and not thoroughly tested. To perform a time- 23 dependent calculation, the Time_Dependent_Options keyword is set to true , and the initial time, final time and number of time steps is specified. The initial condition is set in the Internal_Angular_Flux_Initialization_Options section. <Time_Dependent_Options enable="true"> <Number_Time_Steps>3</Number_Time_Steps> <Initial_Time>0</Initial_Time> <Final_Time>1.e-12</Final_Time> </Time_Dependent_Options> 5.9. Adjoint capability SCEPTRE has partial adjoint capability at this point, only for the Krylov transport solvers. To perform an adjoint calculation, the Transport_Mode keyword is set to Adjoint <Transport_Mode>Adjoint</Transport_Mode> The SAAF and least-squares algorithms are self-adjoint is space and angle, and setting the Transport_Mode keyword to Adjoint triggers a flag that reverses the sign of the streaming term in the first-order transport solver. An adjoint cross section library is obtained by adding an adjoint keyword to the file conversion tool convertBxslibToNetcdf bxslib adjoint SCEPTRE currently does not reverse the group ordering for an adjoint calculation, so the group ordering of the source terms specified in the input file must be reversed manually, and the postprocessing must manually reverse the group ordering of the resulting angular flux. Adjoint capability will be made more automatic in future releases. REFERENCES Bell, G. I., & Glasstone, S. (1970). Nuclear Reactor Theory. New York: Van Nostrand Reinhold. CUBIT user's manual . (n.d.). Retrieved from https://cubit.sandia.gov/public/13.2/help_manual/WebHelp/cubit_users_manual.html Drumm, C. (2007). An Analysis of the Extended-Transport Correction with Application to Electron Beam Transport. Nuclear Science and Engineering , 355-366. Drumm, C. R., & Lorenz, J. (1999). Parallel FE Electron-Photon Transport Analysis on a 2-D Unstructured Mesh. Mathematics and Computation, Reactor Physics and Environmental Analysis in Nuclear Applications (pp. 858-868). Madrid: American Nuclear Society. Drumm, C., & Fan, W. (2003). Uncollided-Flux Preconditioning of the Conjugate Gradients Solution of the Transport Equation. Nuclear Mathematical and Computational Sciences: A Century in Review, A Century Anew. Gatlinburg: American Nuclear Society. Drumm, C., Fan, W., & Pautz, S. (2013). Phase-Space Finite Elements in a Least-Squares Solution of the Transport Equation. Proceedings of the 2013 International Conference on 24 Mathematics and Computational Methods Applied to Nuclear Science and Engineering - M and C 2013 (pp. 877-892). Sun Valley: American Nuclear Society. Drumm, C., Fan, W., Bielen, A., & Chenhall, J. (2011). Least Squares Finite Elements Algorithms in the SCEPTRE Radiation Transport Code. International Conference on Mathematics and Computational Methods Applied to Nuclear Science and Engineering (M&C 2011). Rio de Janeiro: American Nuclear Society. Duderstadt, J. J., & Martin, W. R. (1979). Transport Theory. New York: John Wiley & Sons. Gee, M. W., Siefert, C. M., Hu, J. J., Tuminaro, R. S., & Sala, M. G. (2006). ML 5.0 Smoothed Aggregation User's Guide. Albuquerque: Sandia National Laboratories report, SAND2006-2649. Heroux, M. A. (2004). AztecOO User Guide. Albuquerque: Sandia National Laboratories report, SAND2004-3796. Heroux, M. A., & Willenbring, J. M. (2003). Trilinos Users Guide. Albuquerque: Sandia National Laboratories report, SAND2003-2952. Lebedev, V. I., & Laikov, D. N. (1999). A Quadrature Formula for the Sphere of the 131st Algebraic Order of Accuracy. Doklady Mathematics , 741-745. Lewis, E. E., & Miller, W. F. (1984). Computational Methods of Neutron Transport. New York: John Wiley & Sons. Lorence, L. J., Morel, J. E., & and Valdez, G. D. (1989). Physics Guide to CEPXS: A Multigroup Coupled Electron-Photon Cross-Section Generating Code Version 1.0. Albuquerque, NM 87185: SAND89-1685 Sandia National Laboratories. Morel, J. (1989). A Hybrid Collocation-Galerkin-Sn Method for Solving the Boltzmann Transport Equation. Nuclear Science and Engineering , 72-87. Morel, J. E., & McGhee, J. M. (1999). A Self-Adjoint Angular Flux Equation. Nuclear Science and Engineering , 312-325. Pautz, S., Bohnhoff, W., Drumm, C., & Fan, W. (2009). Parallel Discrete Ordinates Methods in the SCEPTRE Radiation Transport Project. International Conference on Mathematics, Computational Methods & Reactor Physics (M&C 2009). Saratoga Springs, New York: American Nuclear Society. Pautz, S., Drumm, C., Bohnhoff, B., & Fan, W. (2009). Software Engineering in the SCEPTRE Code. International Conference on Mathematics, Computational Methods & Reactor Physics (M&C 2009). Saratoga Springs: American Nuclear Society. Sala, M., & Heroux, M. (2005). Robust Algebraic Preconditioners using IFPACK 3.0. Albuquerque: Sandia National Laboratories report, SAND2005-0662. Schoof, L. A., & Yarberry, V. R. (1994). EXODUS II: A finite element data model. Albuquerque: Sandia National Laboratories report SAND92-2137. Wareing, T. A., McGhee, J. M., Morel, J. E., & Pautz, S. D. (2001). Discontinuous Finite Element SN Methods on Three-Dimensional Unstructured Grids. Nuclear Science and Engineering , 256-268. 25 DISTRIBUTION 1 D. T. Blackfield LLNL P. O. Box 808 Livermore, CA 94551 1 Mark Baird ORNL Reactor and Nuclear Systems Division Oak Ridge, TN 37831 1 MS 0828 F. Pierce 1541 1 MS 0840 J. Tencer 1514 1 MS 1146 J. E. Cash 1384 1 MS 1146 K. R. DePriest 1384 1 MS 1146 P. J. Griffin 1340 1 MS 1152 C. D. Turner 1352 1 MS 1179 W. J. Bohnhoff 1341 1 MS 1179 M. J. Crawford 1341 1 MS 1179 C. R. Drumm 1341 1 MS 1179 W. C. Fan 1341 1 MS 1179 B. C. Franke 1341 1 MS 1179 R. P. Kensek 1341 1 MS 1179 T. W. Laub 1341 1 MS 1179 L. J. Lorence 1341 1 MS 1179 S. D. Pautz 1341 1 MS 1179 G. D. Valdez 1341 1 1 MS 0899 Technical Library 9536 (electronic copy) 26