DESY/TEMF Meeting, Winter 2017

Transkript

1 Advanced space charge tracking methods for high brilliance electron sources S. Schmid, E. Gjonaj, and H. De Gersem Institut für Theorie Elektromagnetischer Felder, TU Darmstadt DESY/TEMF Meeting, Winter 2017 DESY, Hamburg, November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 1

2 Structure Introduction Fast Multipole Methods Numerical Analysis of the Method Preliminary Results for the PITZ Gun Outlook 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 2

3 Introduction Numerical Methods for Photogun Modeling Simulation of electron bunch dynamics Solve equations of motion for N-body interaction problem Particle-Particle Method Brute force approach: Compute N 2 interaction term Open boundaries intrinsically matched Flexible choice of interaction model No aliasing Computational complexity N 2 Runtime limitation for large N Large hardware requirements Particle-Mesh Methods Deposit charges on mesh: Poisson-FFT, FEM, E-M-PIC, etc Faster and more efficient Fully electromagnetic solvers exist Control resolution via mesh parameters Complicated boundary conditions Interpolations, smoothing, adaptive grids, etc. needed Errors from spatial discretization (aliasing) Use a hybrid approach for space charge calculations Fast Multipole Methods 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 3

4 The Concept Aim: Find an efficient way to compute many-body interaction Reduce complexity to less than O(N 2 ) F 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 4

5 The Concept Aim: Find an efficient way to compute many-body interaction Reduce complexity to less than O(N 2 ) Idea: Internal structure of distant agglomerations less important Subdivide particle distribution into boxes 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 5

6 The Concept Idea: Internal structure of distant agglomerations less important Subdivide particle distribution into boxes Parameter θ max : Define minimum spatial resolution θ θ max Use effective force F distant for distant boxes 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 6

7 The Concept Parameter θ max : Define minimum spatial resolution θ θ max Use effective force F distant for distant boxes Evaluation: Compute total interaction force F tot = F distant + F near Exact particle-particle calculations F distant Fnear Effective force from particles in distant box 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 7

8 The Concept Evaluation: Compute total interaction Force F tot = F distant + F near Next Step: Generalize concept for whole bunch 1. Analyze spatial structure of particle distribution (Plot from: J.Kurzak, et al., FMM for particle dynamics, 2006) 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 8

9 1. Spatial Structure: Tree Construction 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 9

10 1. Spatial Structure: Tree Construction 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 10

11 1. Spatial Structure: Tree Construction 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 11

12 1. Spatial Structure: Tree Construction Result: Representation of bunch as a tree Max. #particles per leaf: n crit 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 12

13 2. Effective Force: Multipole Expansion Example: Coulomb potential Φ exact of three point charges (on x-axis) Φ exact x, 0,0 2 x x November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 13

14 2. Effective Force: Multipole Expansion Φ exact x, 0,0 2 x x 1 2 Φ monopole x, 0,0 1 x 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 14

15 2. Effective Force: Multipole Expansion Φ exact x, 0,0 2 x x 1 Φ 2 dipole x, 0,0 1 x xτ x x November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 15

16 2. Effective Force: Multipole Expansion Φ exact x, 0,0 2 x x 1 Φ 2 quadrupole x, 0,0 1 x xτ x x 2 2 x November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 16

17 2. Effective Force: Multipole Expansion General case: Multipole expansion in spherical coordinates truncated at l max N Φ x = 1 4 πε 0 i=0 q i x x i l max l = l=0 m= l L l m r l + M l m r l+1 Y l m θ, φ + O(l > l max ) 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 17

18 3. Evaluate Interaction: Tree Traversal How to get from O(N 2 ) to O(N)? 1. Compute multipole expansion of particles contained in each leaf box. l max (Plot based on: R. Yokota, ExaFMM User s Manual, 2011) 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 18

19 3. Evaluate Interaction: Tree Traversal How to get from O(N 2 ) to O(N)? 1. Compute multipole expansion of particles contained in each leaf box. 2. Express multipoles in parent node. Sum contributions from child nodes. (Plot based on: R. Yokota, ExaFMM User s Manual, 2011) 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 19

20 3. Evaluate Interaction: Tree Traversal How to get from O(N 2 ) to O(N)? 1. Compute multipole expansion of particles contained in each leaf box. 2. Express multipoles in parent node. Sum contributions from child nodes 3. Translate approximation of distant distribution to local parent node. θ max (Plot based on: R. Yokota, ExaFMM User s Manual, 2011) 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 20

21 3. Evaluate Interaction: Tree Traversal How to get from O(N 2 ) to O(N)? 1. Compute multipole expansion of particles contained in each leaf box. 2. Express multipoles in parent node. Sum contributions from child nodes. 3. Translate approximation of distant distribution to local parent node. 4. Express multipole expansion in the local coordinates of the child nodes. (Plot based on: R. Yokota, ExaFMM User s Manual, 2011) 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 21

22 3. Evaluate Interaction: Tree Traversal How to get from O(N 2 ) to O(N)? 1. Compute multipole expansion of particles contained in each leaf box. 2. Express multipoles in parent node. Sum contributions from child nodes. 3. Translate approximation of distant distribution to local parent node. 4. Express multipole expansion in the local coordinates of the child nodes. 5. Evaluate F distant and F near for each particle in the leaf. O N scaling (Plot based on: R. Yokota, ExaFMM User s Manual, 2011) 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 22

23 Numerical Analysis of the Method Implementation: Overview of FMM Codes Strategy: Use an existing FMM code as a basis to extend our in-house particle tracker Overview of available codes: Code Exafmm (C++) (George Washington University & Tokyo Institute of Technology) Tapas (C++) (Tokyo Institute of Technology) Scafacos/PEPC (C++, F90) ( BMBF, DFG Project of German research groups) Comments FMM code developed for exa-scale computing MPI parallelization and GPU acceleration Issues with certain particle distributions Implicitly parallel programming framework Flexible template framework Under development, No mature version available Parallel library to solve electrostatic problems Established code No GPU version 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 23

24 Numerical Analysis of the Method Implementation: Overview of FMM Codes Strategy: Use an existing FMM code as a basis to extend our in-house particle tracker Overview of available codes: Code Exafmm (C++) (George Washington University & Tokyo Institute of Technology) Tapas (C++) (Tokyo Institute of Technology) Scafacos/PEPC (C++, F90) ( BMBF, DFG Project of German research groups) Comments FMM code developed for exa-scale computing MPI parallelization and GPU acceleration Issues with certain particle distributions Implicitly parallel programming framework Flexible template framework Under development, No mature version available Parallel library to solve electrostatic problems Established code No GPU version Our Choice: Exafmm - MPI & GPU parallelization, C++ (LW-code in C), best documentation 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 24

25 Numerical Analysis of the Method Approximation Errors and Speedup Electrostatic interaction Freely propagating bunch 100 processes on cluster Average over ~40 time steps (l max, n crit, θ max ) Scaling: O(N) up to O(N 2 ) (depends on l max, n crit, θ max ) 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 25

26 Numerical Analysis of the Method Approximation Errors and Speedup Electrostatic interaction 500k propagating particles 60 processes on cluster Average over ~40 time steps θ max = 0. 2 n crit = 50 l max 2, 15 Tradeoff: Error vs. speedup 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 26

27 Numerical Analysis of the Method Approximation Errors and Speedup Electrostatic interaction 500k propagating particles 60 processes on cluster Average over ~40 time steps l max = 10 n crit = 50 θ max 0. 1, 0. 6 Tradeoff: Error vs. speedup 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 27

28 Numerical Analysis of the Method Approximation Errors and Speedup Electrostatic interaction Identical compute nodes 10 6 particles particles 10 5 particles Average over ~40 time steps l max = 10 n crit = 50 θ max = 0. 2 MPI scaling of PPM code better than FMM code Best FMM speedup for more particles and less processes 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 28

29 Preliminary Results for the PITZ Gun Emittance study Relativistic rest frame model #particles: N = σ r = 0.4 mm, σ t = 22 ps PPM: 234 processes on cluster Runtime: T PPM 42 h 41 min FMM: 8 processes on desktop Runtime: T FMM 46 min Speedup 55 Max. Deviation % 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 29

30 Preliminary Results for the PITZ Gun Emittance study Relativistic rest frame model #particles: N = σ r = 0.4 mm, σ t = 22 ps PPM: 234 processes on cluster Runtime: T PPM 42 h 41 min FMM: 8 processes on desktop Runtime: T FMM 1 h 37 min Speedup 26 Max. Deviation % 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 30

31 Preliminary Results for the PITZ Gun Emittance study Relativistic rest frame model #particles: N = σ r = 0.4 mm, σ t = 22 ps PPM: 234 processes on cluster Runtime: T PPM 42 h 41 min FMM: 8 processes on desktop Runtime: T FMM 26 h 50 min Speedup 1. 6 Max. Deviation % 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 31

32 Outlook Status quo: ExaFMM merged with in-house particle tracking code Basic models (electrostatic, rel. rest frame, image charges) implemented Numerical studies with FMM: MPI-Scaling, Influence of l max, n crit, θ max First particle tracking simulations for PITZ gun Ongoing Work: Optimization of MPI communication Achieve better scaling on cluster Investigate using GPU parallelization Use tree-algorithms from graphics software for particle tracking Implementation of self-controlled models (e.g. Schottky effect) 20. November 2017 TU Darmstadt Fachbereich 18 Institut Theorie Elektromagnetischer Felder Steffen Schmid 32