Data Analysis II (Data Mining Techniques) WS 07/08. Data Analysis II (Data Mining Techniques) WS 07/08. What is Data Mining?

Transkript

1 Contents 0. Introduction 1. Instance Based Learning 2. Naïve Bayes 3. Clustering 4. Decision Trees 5. Association Rules What is Data Mining? Examples Knowledge Discovery in Databases (KD) is a non-trivial process of identifying valid, novel, potentially useful and ultimately understandable (<<interesting>>) patterns in data. Fayyad, 1998 Direct Marketing Cellular Phone Business Reduce cost of mailing by targeting a set of consumers likely to buy a new cellphone-product Approach: use data for a similar product introduced before: which customers decided to buy collect demographic, lifestyle and company interaction facts about these customers (age, salary, house owner,...) use these information to develop a model for decision about current mailing Folie 1 Folie 2 Examples Examples Supermarket Shelf Management Credit Card Fraud Detection Identify items that are brought jointly by sufficiently many customers fraudulent behaviour is somewhat different??? Approach: process the sales data collected with barcode scanners to find dependencies between items but often has no clear pattern classify typical honest behaviour those transactions that do not fit into this picture may be fraudulent Rules found: Milk Coke Diaper, Milk Beer Folie 3 Folie 4

2 Examples Terrorist Investigation Subject: New Poll: NSA data mining of US domestic calls Data mining is again in the news, with the report in USA Today about a controversial NSA activity of collecting and analysis (data mining?) a huge database of nearly all US domestic calls, with the goal of finding potential terrorists. Oct 2006 Usama Bin Laden Folie 5 Folie 6 The KDD (Knowledge Discovery in Databases) Process (again) KDD (Knowledge Discovery in Databases) Usma Fayyad the overall process of discovering useful knowledge from data involves the evaluation and possibly interpretation of the patterns to make the decision of what qualifies as knowledge includes the choice of encoding schemes, pre-processing, sampling, and projections of the data prior to the data mining step DM (Data Mining) the application of algorithms for extracting patterns from data without the additional steps of the KDD process Folie 7 Folie 8

3 The Steps of the KDD Process (I) The Steps of the KDD Process (II) 1. Developing an understanding of : the application domain the relevant prior knowledge the goals of end-users 2.Creating a target data set: selecting a data set (subset of variables, data samples) on which discovery is to be performed 3.Data cleaning and pre-processing Removal of noise or outliers Collecting necessary information to model or account for noise Strategies for handling missing data fields Accounting for time sequence information and known changes Folie 9 4.Data reduction and projection Finding useful features to represent the data depending on the goal of the task Using dimensionality reduction or transformation methods to reduce the effective number of variables under consideration or to find invariant presentations for the data Folie 10 "Man muss die Dinge so tief sehen, dass sie einfach werden." (Konrad Adenauer) The Steps of the KDD Process (III) 5. Choosing the data mining task Deciding whether the goal of the KDD process is classification, regression, clustering, etc. 6.Choosing the data mining algorithm(s) Selecting method(s) to be used for searching for patterns in the data Deciding which models and parameters may be appropriate Matching a particular data mining method with the overall criteria of the KDD process 7. Data mining Searching for patterns of interest in a particular representational form or a set of such representations as classification rules or trees, regression, clustering, and so forth 8. Interpreting mined patterns 9. Consolidating discovered knowledge Folie 11 Folie 12

4 Data Mining as an Intersection of several fields Data Mining - a Discipline at the Intersection of the following fields: Artifical Intelligence - Machine Learning - Statistics - Data Bases Data Mining versus Artifical Intelligence Focus on numerical attributes and quantitative prediction (see below) Non-numerical data processing Seemingly intelligent behavior (autonomous) robots, computer vision board games (checkers, chess, go,...), consulting, disease diagnosis Folie 13 Folie 14 Data Mining versus (statistical) Data Analysis ences/user- 2004/Keynotes/Ripley.pdf The transition AI Data Mining statistical data analysis is a gradual one. Example: linear regression Folie 15 Folie 16

5 Methods in the Focus of Data Mining Some Popular Algorithms Classification Supervised Learning finding a function that maps records into one of several discrete and predefined classes Regression Models: Local Regression Numerical Prediction Rule Induction learning a pattern from examples and using it to predict future values of the target variable Decision Trees Association (~rules) finding patterns, associations, correlations among sets of items or objects Clustering identifying groups records that are similar within each other but different from the rest of data Unsupervised Learning Clustering k-means innovative algorithms Visualisation Polls : Data Mining Techniques (Feb 2005) Self Organised Maps Folie 17 Folie 18 Data Mining Methods Methods show an extremly high degree of overlap w.r.t quality and basic tenet without a clear winner kind of data (ordinal, interval, nominal, mixed,...) Scaling Number of Data Quality of Data Computational Speed Accuracy 1 st 2 nd order errors Understandability Manual Intervention / post-processing Important criteria Data Mining: An Example Hand-written ZIP code recognition (LeCun group at AT&T Bell Labs) ~10,000 digits from the U.S. mail were used to train and test system ZIP codes on envelopes were initially located and segmented by a separate system (difficult task in itself) network input: 16 x 16 pixel array, scaled to standard size three hidden layers, 10 output units (for digits 0-9) first two hidden layers organized into groups of feature detectors weight sharing used to constrain number of free parameters 1256 units links biases, but only 9760 free parameters used an accelerated version of backprop (pseudo-newton rule) trained on 7300 digits, tested on 2000 error rate of ~1% on training set, ~5% on test set if marginal cases were rejected (two or more outputs approximately the same), error reduced to ~1% with 12% rejected used "optimal brain damage" technique to prune unnecessary weights after removing weights and retraining, only ~1/4 as many free parameters as before, but better performance 99% classification accuracy with 9% rejection rate achieved state of the art in digit recognition much problem-specific knowledge was designed into the network architecture preprocessing of input data was crucial to success Folie 19 Folie 20

6 Measure of Interestingness Which of the detected patterns in data mining process are interesting? easily understood by humans valid on new or test data with some degree of certainty potentially useful, novel or validates some hypothesis objective measure (based on statistics / structure of patterns) subjective measure (based on users belief) Folie 21 Folie 22 Usage of Models The Origins of the Data Mining Hype predictive model is a black box that makes predictions about the future based on information from the past and present growing data volume (scanners, www-clickstream) models reach from easy to understand to really difficult model needs to be built once, but it can be used several times Two ways to use a model: qualitative: provide insight data you are working with if city = New York and 30 < age < important age demographic was previously 20 to 25 requires interaction capabilities and good visualisation quantitative: automated process score new gene chip datasets with error model every night at midnight bottom-line orientation quality of data hardware algorithms very often freeware, shareware diminishing returns in stagnant or shrinking markets CRM hype a commercial leverage to data warehousing Folie 23 Folie 24

7 The Origins of the Data Mining Hype TheOriginsoftheDataMiningHype From: Gregory Piatetsky-Shapiro Date: 24 Oct 2005 Subject: Usama Fayyad Interview, 2: on data mining challenges Q2. GPS: What are some of the biggest data mining challenges you face now at Yahoo? Usama Fayyad: Yahoo!'s users, through their use of our network of products, generate over 10 terabytes of data per day. This is the equivalent of the entire text contents of the library of Congress. This is data that describes product usage, and does not include content, , or images, etc. Looking back, we probably underestimated the costs and complexities of the business. While the business is interesting, the fact that Business Intelligence and Data Mining are not well-defined and well-understood by the market, led us into situations where expectations of clients exceeded what technology could deliver. So even though we were deploying solutions that were more successful than what clients tried to build themselves, the lack of maturity in this business required us to build a growing Professional Services (Consulting) arm. While that business is very profitable, it is not scalable. From Folie 25 Folie 26 Further Reading Fayyad, Piatetsky-Shapiro, Smyth and Uthurusamy: Advances in Knowledge Discovery and Data Mining, AAAI /MIT Press Berry, Michael J. A./ Gordon Linoff, Mastering Data Mining: The Art and Science of Customer Relationship Management, New York, Chichester, Weinheim, Brisbane, Singapore, Toronto, On CRISP DM INTERESTINGNESS of a pattern instance measures its quality and has several dimensions. The main dimensions are the validation on the sample set, the reliability on the universe, the degree of redundancy with respect to other already known pattern instances, the generality, thesimplicity, and the usefulness. VALIDATION checks a pattern instance (or component of an instance) referring to the subset of data in the sample set which is connected with the instance. To check an instance, usually a statistical test or some other criteria are validated. Additionally to the decision, whether an instance is valid or not, often also an evidence measure is calculated. EVIDENCE measures the statistical significance or some other kind of conspicuousness of a pattern instance. APPLICATION TEST is a filter which includes some preconditions for a pattern instance to be evaluated as interesting. The filter is used to avoid possibly extensive evaluation efforts, when the interestingness of an instance can be excluded already by the preconditions. RELIABILITY includes some estimation on the validity in the universe of the pattern instance which was discovered in the sample set. Cross-validation methods can be applied to derive some estimation on the correctness of the statement (pattern instance) in the universe. REDUNDANCY relates to several pattern instances or to several knowledge components of a complex pattern instance (e.g. nodes in a tree). Redundancy is given, if one instance or component follows (logically) from another one. Quantification of redundancy can be introduced to measure the conditional probability of one instance or component given the other one. GENERALITY measures the strength of a pattern instance in terms of the size of the subset of objects which are described by the statement. SIMPLICITY measures the syntactical complexity of a statement (pattern instance). USEFULNESS quantifies the possible usefulness of a statement (pattern instance). The usefulness can be related to a task the user of a discovery system has to perform or to a task that a computer system can perform on the basis on the discovered new knowledge. PATTERN TYPES are classes of patterns. Folie 27 Folie 28

8 See also software-mine pdf On Interestingness On Types of knowledge Folie 29