R Parallel Random Access Machine (PRAM)

Transkript

1 R Parallel Random Access Machine (PRAM) R-1 R. Hoffmann, FB Informatik, TU-Darmstadt Vorlesung im SS 2010 Inhalt Metriken Optimale Algorithmen Classification and Types Algorithmen Initialize Broadcasting All-to-All Broadcasting Reduction Sum Pointer Jumping (Algorithmen) Naive Sorting Prefix Sum WT-optimaler Algorithmus Simulations between PRAM Models Simulating a Priority CRCW on an EREW PRAM

2 Verwendete Quellen R-2 J. JaJa An Introduction to Parallel Algorithms,1992 Robert van Engelen (Folien) The PRAM Model and Algorithms, Advanced Topics Spring 2008 Behrooz Parhami Introduction to Parallel Processing, Algorithms and Architectures, Plenum Series, Kluwer 2002 (Folien) Arvind Krishnamurthy (Folien) PRAM Algorithms, Fall 2004

3 Data parallel example R-3 Mandelbrot-Menge Global Climate Modeling Problem

4 (Wdh.) Metriken R-4 p max = P = verwendete OTPP T = Zeitschritte Work(p max ) = p(t) Cost(p max ) = p max * T Mehraufwand R(p max ) = Work(p max ) / Work(1) Utilization = Work/Cost Speedup S = T(p=1)/T(p) Effizienz E = S/p Parallelitätsgrad p(t), Darstellung für alle t: Parallelitätsprofil Parallelitätsindex: Mittlerer Parallelitätsgrad I = Work/T Im Zusammenhang mit den PRAM Algorithmen auch verwendete Notation: Cost(n, P(n)) = P(n) * T(n) Work(n, P(n)) = Summe der ausgeführten Operationen in Abh. von der Problemgröße und der Anzahl P(n) der verwendeten OTPP

5 Optimale Algorithmen R-5 Sequentieller Algorithmus Zeit-optimal: Ein sequentieller Algorithmus, dessen (sequentielle) Zeitkomplexität O(T seq (n) nicht verbessert werden kann, heißt zeit-optimal. Paralleler Algorithmus (Work)-optimal: Wenn die Work-Komplexität in der Zeitkomplexität des zugehörigen sequentiellen Algorithmus liegt. (Die Anzahl der Operationen des parallelen Algorithmus ist asymptotisch gleich der Anzahl der Operationen des sequentiellen Algorithmus) Work(n) = O(T seq (n)) [noch strengere Definition: Total number of operations in the parallel algorithm is equal to the number of operations in a sequential algorithm] Work-Time-optimal, WT-optimal: Die Anzahl der parallelen Schritte T par (n) kann nicht durch irgend einen anderen Work-optimalen Algorithmus weiter verringert werden

6 Parallel Random Access Machine (PRAM) Model R-6 PRAM removes algorithmic details concerning synchronization and communication, allowing the algorithm designer to focus on problem properties A PRAM algorithm describes explicitly the operations performed at each time unit PRAM design paradigms have turned out to be robust and have been mapped efficiently onto many other parallel models and even network models

7 The PRAM Model of Parallel Computation R-7 Natural extension of RAM: each processor is a RAM 1) Processors operate synchronously Earliest and best-known model of parallel computation Shared Memory Shared memory with m locations P processors, each with private memory P 1 P 2 P 3 P p All processors operate synchronously, by executing load, store, and operations on data 1) Random Access Machine, Zugriff per Befehl auf eine beliebige Speicherzelle (nicht gemeint ist hier RAM = Random Access Memory)

8 Synchronous PRAM R-8 Synchronous PRAM is an SIMD-style model All processors execute the same program, deshalb heißt das Modell auch SPMD (Single Program Multiple Data) All processors execute the same PRAM step instruction stream in lock-step (im Gleichschritt) Effect of operation depends on individual or common data which processor(i) accesses processor index i local data Instructions can be selectively disabled (if-then-else flow) Asynchronous PRAM Several competing models No lock-step

9 Classification of PRAM Models R-9 A PRAM step ( clock cycle ) consists of three phases 1. Read: each processor may read a value from shared memory 2. Compute: each processor may perform operations on local data 3. Write: each processor may write a value to shared memory Model is refined for concurrent read/write capability Exclusive Read Exclusive Write (EREW) Concurrent Read Exclusive Write (CREW) Concurrent Read Concurrent Write (CRCW)

10 Types of CRCW PRAM R-10 CRCW - Common Arbitrary, Random Priority Undefined Ignore Detecting Max/Min Reduction result all processors must write the same value one of the processors succeeds in writing processor with the lowest index succeeds the value written is undefined no new value is written special collision detection code is written largest / smallest of the values is written SUM, AND, OR, XOR or another function is written

11 Comparison of PRAM Models R-11 A model B is more powerful compared to model A if either The time complexity is asymptotically less in model B for solving a problem compared to A Or the time complexity is the same and the work complexity is asymptotically less in model B compared to A From weakest to strongest: EREW CREW Common CRCW Arbitrary CRCW Priority CRCW

12 Initialize R-12 P processors n elements Initializing an n-vector M (with base address = B) to all 0s with P < n processors P t=0 for t=0 to n/p -1 do // segments parallel [i = 0.. P-1] // processors if (tp + i < n) then M[B + tp + i] 0 endparallel endfor

13 Making P copies of B[0] Data Broadcasting R-13 for t = 0 to log 2 P 1 do parallel [i = t -1] // active B[i+ 2 t ] B[i] // proc(i) writes endparallel endfor for t = 0 to log 2 P 1 do parallel [i = 2 t.. 2 t+1-1] // active B[i] B[i 2 t ] // proc(i) reads endparallel endfor B EREW PRAM data broadcasting without redundant copying. falls Array-Grenze keine Potenz von 2 ist, dann ergänze die Algorithmen um Bedingung

14 Data Broadcasting, Data Parallel Formulation R-14 parallel [i = t -1] B[i+ 2 t ] B[i] endparallel I [0.. 2 t -1] B[I+ 2 t ] B[I] Datenparallele Schreibweise I = Indexvektor B EREW PRAM data broadcasting without redundant copying. entspricht B[2 t.. 2 t+1-1] B[0.. 2 t -1]

15 All-to-All Broadcasting on EREW PRAM R-15 Jeder Prozessor i liest im Schritt t den Wert von seinem Nachbarn i+t parallel [i = 0.. P-1] write own data value into B[i] for t = 1 to P 1 parallel [i = 0.. P-1] Read the data value from B[(i + t) mod P] endparallel endfor This O(P)-step algorithm is time-optimal

16 Reduction on the EREW PRAM R-16 Reduce P values on the P-processor EREW PRAM in O(log P) time Reduction algorithm uses exclusive reads and writes Algorithm is the basis of other EREW algorithms Speicherzellen Prozessoren Datenfluss

17 Sum on the EREW PRAM (1) R-17 Sum of n values using n/2 processors (i) Input: A[1, n], n = 2 k Output: s parallel [i=1.. P] B[i] A[i] endparallel B for t = 1 to log n do parallel [i = 1.. n/2] if i < n/2 t then B[i] B[2i-1] + B[2i]; if i = 1 then s B[i] end B Cost = (n/2) log n Work = n-1 Utilization = U = Work/Cost = [2(n-1)/n] log n= O(1/log n) E = (n-1)/((n/2)log n) = O(1/log n) B B S

18 Sum on the EREW PRAM (2) R-18 Hierbei wird die Anzahl der Prozessoren dynamisch verwaltet (ist nicht PRAM-Standard!) Sum of n values using n/2.. 1 processors (i) Input: A[1, n], n = 2 k Output: s B parallel [i=1.. P] B[i] A[i] endparallel for t = 1 to log n do parallel [i =1.. n/2 t ] // dynamic activity B[i] := B[2i-1] + B[2i] endparallel endfor s := B[i] B B Cost = n -1 Work = n -1 Utilization = 1 B s

19 Pointer Jumping R-19 Finding the roots of a forest using pointer-jumping

20 Pointer Jumping on the CREW PRAM R-20 Input: A forest of trees, each with a self-loop at its root, consisting of arcs (i, P(i)) and nodes i, where 1 < i < n Output: For each node i, the root S[i] of the tree containing i parallel [i = 1.. n] S[i] P[i] endparallel for h = 1 to log n do // while there exists a node such that S[i] S[S[i]] do parallel [i = 1.. n] if S[i] S[S[i]] then S[i] S[S[i]] endparallel endfor T(n) = O(log h) with h the maximum height of trees Cost(n) = O(n log h)

21 Beispiel R Pointer werden transportiert

22 Naive EREW PRAM sorting algorithm (all read from all) R-22 parallel [i= 0.. P-1] R[i] 0 endparallel Werte in S Such-Pointer in R for t = 1 to P 1 do parallel [i= 0.. P-1] neighbor := (i + t) mod P if S[neighbor ] < S[ i ] or S[neighbor ] = S[ i ] and neighbor < i endparallel endfor then R[ i ] := R[ i ] + 1 parallel [i= 0.. P-1] S[R[ i ]] S[ i ] endparallel This O(P)-step sorting algorithm is far from optimal; sorting is possible in O(log P) time

23 Prefix Sum R-23 Applications deleting marked elements from an array (stream compaction), radixsort, solving recurrence equations, solving tri-diagonal linear systems, quicksort i = s i <= Σ k=1..i x i x s s 1 = x 1 s 2 = s 1 + x 2... s i = s i-1 + x i

24 Horn s Algorithm R-24 CREW for t=1 to log 2 n do parallel [i = 2.. n] if i > 2 t-1 then endparallel endfor x[i] x[i-2 t-1 ] + x[i] Problem: Mehrfache Lesezugriffe auf dieselbe Variable kann zu Zugriffsverzögerungen (Leseressourcenkonflikt (Congestion) an einem Speichermodul) führen

25 Prefix Sum EREW Die Prozessoren und die Speicherzellen M[i] enthalten die x-werte P i hat eine lokales Register y i Komplexität wie EREW- Reduktion M Prozessoren R-25 parallel [i=0.. n-1] y[i] M[i] endparallel for t=0 to log n -1 do parallel [i=2 t.. n-1] y[i]:=y[i] + M[i-2 t ] M[i] y[i] endparallel

26 Prefix Sum Algorithmus, Work Optimal R-26 CREW Lese-und Scheibzugriffe 1. 8er1 + 4ew1 2. 2cr2 + 4er1 + 4ew1 3. 1cr4 + 4er1 + 4ew1 log n Schritte n/2 Prozessoren = const. Work(p max =n/2) = n/2 * log n Cost = Work optimal Mehraufwand der parallelen Ausführung R(p max =n/2) = (n/2 * log n)/(n-1) = O(log n)

27 Entwicklung eines WT-optimalen Algorithmus zur Bildung des Maximums von n Zahlen R-27 Use P = n/log n processors instead of n Have a local phase followed by the global phase Local phase: compute maximum over log n values use simple sequential algorithm Time for local phase = O(log n) Global phase: take (n/log n) local maxima and compute global maximum using the reduction tree algorithm Time for global phase = O(log (n/log n)) = O(log n log log n) = O(log n) Work(n) = O(n) Example: n = 16 Number of processors, p = n/log n = 4 Divide 16 elements into four groups of four each Local phase: each processor computes the maximum of its four local elements Global phase: performed amongst the maxima computed by the four processors

28 Simulations between PRAM Models R-28 An algorithm designed for a weaker model can be executed within the same time complexity and work complexity on a stronger model An algorithm designed for a stronger model can be simulated on a weaker model, either with Asymptotically more processors (more work) Or asymptotically more time