Unit 9. Prototype-Based Clustering. Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 176

Transkript

1 Unit 9 Prototype-Based Clustering Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 176

2 Introduction Instead of a complete set description, every cluster C i is represented by a typical value v i, which can usually be interpreted as the center of the cluster. The distance to the nearest prototype determines to which cluster a data point belongs, i.e. x k C i if x k v i = K min j=1 x k v j. Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 177

3 Partition Matrices U = u ik = u 11 u 1n..... u K1 u Kn 1 if x k C i 0 otherwise K i=1 u ik = 1 n u ik > 0 k=1 Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 178

4 The k-means (KM) Model Objective function to be minimized: J KM (U, V ) = K i=1 x k C i x k v i 2 = K i=1 n k=1 u ik x k v i 2 Computation of prototypes: v i = 1 C i x k C i x k = n k=1 u ik x k n k=1 u ik (1) Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 179

5 The k-means Algorithm 1. Given: data set {x 1,..., x n } R p, norm. on R p, predefined number of clusters K, maximum number of iterations t max, distance measure. v, threshold ε. 2. Initialization: V (0) R p 3. For t = 1,..., t max do: Determine U (t)( V (t 1)) (nearest prototype) Determine V (t)( U (t)) (by Eq. (1)) if V (t) V (t 1) v ε, stop 4. Output: partition matrix U, set of prototypes V Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 180

6 The Fuzzy c-means (FCM) Model Objective function to be minimized: J F CM (U, V ) = K n u m ik x k v i 2 i=1 k=1 Computation of prototypes: v i = n k=1 um ik x k n k=1 um ik (2) Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 181

7 The Fuzzy c-means (FCM) Model (cont d) Update of partition matrix: 1 if x k = v i 0 if x k = v l for some v l v i u ik = / ( ) 2 K x 1 k v i m 1 otherwise x k v j j=1 (3) Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 182

8 The Fuzzy c-means Algorithm 1. Given: data set {x 1,..., x n } R p, norm. on R p, predefined number of clusters K, sharpness exponent m, maximum number of iterations t max, distance measure. v, threshold ε. 2. Initialization: V (0) R p 3. For t = 1,..., t max do: Determine U (t)( V (t 1)) (by Eq. (3)) Determine V (t)( U (t)) (by Eq. (2)) if V (t) V (t 1) v ε, stop 4. Output: partition matrix U, set of prototypes V Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 183

9 Adaptive Variants Different distributions and sizes of clusters usually lead to suboptimal results with KM/FCM. In order to adapt to different structures in data, problem-specific distance measures can be used. The Gustafson-Kessel Method is one example of such an adaptive method. The goal/motivation behind Gustafson- Kessel clustering is to tackle non-spherical clusters as well. Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 184

10 Gustafson-Kessel Method: Basic Idea The basic idea is to attach an own potentially non-spherical distance measure to each cluster. These distance measures evolve over time. For cluster i (1 i K), the distance measure d i (., v i ) is defined as d i (x k, v i ) = (x k v i ) T A i (x k v i ), where A i = p det(s i )S 1 i and S i = 1 n k=1 um ik n u m ik(x k v i )(x k v i ) T. k=1 Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 185

11 The Gustafson-Kessel (GK) Model Objective function to be minimized: J GK (U, V ) = K n u m ik d2 i (x k, v i ) i=1 k=1 Computation of prototypes: v i = n k=1 um ik x k n k=1 um ik (4) Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 186

12 The Gustafson-Kessel (cont d) Update of partition matrix: 1 if x k = v i 0 if x k = v l for some v l v i u ik = / ( ) 2 K d 1 i (x k,v i ) m 1 otherwise d j (x k,v j ) j=1 (5) Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 187

13 The Gustafson-Kessel Algorithm 1. Given: data set {x 1,..., x n } R p, predefined number of clusters K, sharpness exponent m, maximum number of iterations t max, distance measure. v, threshold ε. 2. Initialization: V (0) R p 3. For t = 1,..., t max do: Determine U (t)( V (t 1)) (by Eq. (5)) Determine V (t)( U (t)) (by Eq. (4)) if V (t) V (t 1) v ε, stop 4. Output: partition matrix U, set of prototypes V Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 188

14 Non-Convexity of FCM Cluster Memberships In order to overcome the problem of non-convexity, alternative schemes to compute the partition matrix have been proposed. Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 189

15 Sophisticated Variants Alternatively, clustering algorithms may not only work with prototypes in the data space, but also with sophisticated parameter representations in order to extract more complex features. Fuzzy c-lines (FCL): can be used to extract line-shaped clusters Fuzzy c-varieties (FCV): can be used to extract clusters which are arranged along higher dimensional linear manifolds Fuzzy c-shells (FCS): designed for extracting circle/sphere clusters Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 190

16 The Initialization Problem The k-means and the fuzzy c-means algorithm are both sensitive to the initialization of the representatives since they both perform a kind of steepest descent search that is prone to being trapped in a local minimum. Initialization can be based on totally random values or random selection of data points. Both variants are not robust. An agglomerative clustering method for computing initial values leads to better robustness, but is costly in terms of computational effort. Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 191

17 Estimating the Cluster Number Unlike agglomerative methods, prototype-based clustering methods assume prior knowledge about the number of clusters. If Ward clustering is applied for selecting initial values for the representatives, the Ward index can be used to determine an estimation for the number of clusters. Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 192

18 Cluster Validation For complex high-dimensional data sets, it appears often difficult to assess the quality of results of clustering methods. Therefore, methods to measure the plausibility of clusterings have been proposed. Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 193

19 Cluster Validation Indices (1/4) Partition Coefficient: PC(K) = 1 n K n u 2 ik i=1 k=1 Partition Entropy: PE(K) = 1 n K i=1 n k=1 u ik log u ik Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 194

20 Cluster Validation Indices (2/4) Fukuyama-Sugeno Index: FS(K) = K n u m ik (d(x k, v i ) d(v i, v) ) i=1 k=1 with v = 1 K K i=1 v i. Xie-Beni Index: XB(K) = K i=1 n k=1 um ik d2 (x k, v i ) n min i j d 2 (v i, v j ) Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 195

21 Cluster Validation Indices (3/4) Davies-Bouldin Index: DB(K) = 1 K K i=1 max i j S i + S j d(v i, v j ), where S i = ( x X i d(x, v i ) X i ) 1 2 and X i = {x k X u ik > u jk for allj i}. Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 196

22 Cluster Validation Indices (4/4) Partition Separation Index: PS(K) = K i=1 PS i (K), where PS i (K) = n k=1 u2 ik max K i=1 n k=1 u2 ik exp ( min ) k i d(v i, v k ) AVD and AVD = Ki=1 d(v i, v) K with v = 1 K K i=1 v i. Knowledge-Based Methods in Image Processing and Pattern Recognition; Ulrich Bodenhofer 197

23