Introduction | Technical Background | Applications | Tools | Websites | Mailing lists

Introduction

Databases can contain vast quantities of data describing decisions, performance and operations. In many cases the database contains critical information concerning past business performance which could be used to predict the future.

Often the sheer volume of the data can make the extraction of this business information impossible by manual methods. Data mining is a set of techniques which allows you to do this.

Data Mining (also known as Knowledge Discovery) technology helps businesses discover hidden data patterns and provides predictive information which can be applied to benefit the business.

The basic approach is to access a database of historical data and to identify relationships which have a bearing on a specific issue, and then extrapolate from these relationships to predict future performance or behaviour. The human analyst plays an important role in that only they can decide whether a pattern, rule or function is interesting, relevant and useful to an enterprise.

Technical Background

Data Mining and Knowledge Discovery in Databases are terms used interchangeably. Other terms often used are data or information harvesting, data archeology, functional dependency analysis, knowledge extraction and data pattern analysis. A high level definition of Data Mining is: the non-trivial process of identifying valid, novel, potentially useful and ultimately understandable patterns in data. Data mining is not a simple process and there is no tool that can do the job automatically. Data mining can be aided by tools, but it requires both human data mining expertise and human domain expertise. Data mining consists of a number of operations, each of which are supported by a variety of technologies, such as rule induction, neural networks, conceptual clustering. In real world applications information extraction requires the cooperative use of several data mining operations and techniques.The basic data mining process is as follows:

1. Define the business objective and expected operational environment of any expected resulting system.
2. Select data. Substantial databases with a meaningful sample of data is required. Selecting data often consists of selecting a time span, geography or product set that you want addressed, and the variables that you want to consider.
3. Transform data. This involves determining how to represent the data for the data mining algorithm, e.g. age, should it be represented as a member of a set (25-35 year olds) or a straight number.
4. Run the data mining algorithm or combination of algorithms. Iteration to step 3 or 2 is often needed here. The AI techniques of rule induction and neural networks are used for this machine learning stage.
5. Analyst examines the output data. Often visualisation plays an important part in helping the analyst, especially if the analyst needs to present their analysis to others.
6. Present results to the business, in order that the insights can be incorporated into business processes (e.g. through producing output data files or installing data mining software).

Data mining is typically not used as a business system delivery technology. Rather it is an extremely powerful and effective set of technologies for analysing and clustering data which can be used to form the basis of a system.

Applications

The key reason why Data Mining is such a buzzword at the moment is that because many organisations recognised the need to better understand their customers. Data mining can deliver real world results. Data mining has been used for the following types of applications:

• Understanding purchasing behaviour of customers
• Detecting credit card or insurance fraud
• Predicting probable changes in financial markets

Tools/Services

The July 1996 Edition of "Intelligent Software Strategies" was devoted to Intelligent Data Mining Tools . They classified the tools by their knowledge discovery techniques. The 7 classifications were: Rule and Decision Tree Discovery; Neural Networks; Conventional Statistical; Advanced Visualisation; Fuzzy Techniques; Knowledge Based; and Multiple Techniques. Over 40 tools are reviewed and addresses of the vendors (some with website addresses) are given.

Websites

• Knowledge Discovery Mine maintained by Gregory Piatetsky-Shapiro at GTE Laboratories. This site serves as a collecting point of information directly relevant to knowledge discovery in databases and data mining, including tools available free on-line, commercial products, pointers to many other relevant groups and organisations.
• ISL produce an advanced data mining tool called Clementine and provide consultancy. Clementine is the leading end-user data mining toolkit, twice winning the UK Governments SMART award for innovation and winner of the 1996 "Computing" newspapers Silver Award for Software Excellence.
• IBM has a Data Mining solutions service as part of its Consulting Service. IBM have extensive expertise in designing, integrating and testing data mining solutions for a wide range of industries, including retail, banking, financial services, healthcare, travel, telecommunications and insurance. IBM also conducts data mining research and markets a data mining tool.
• Unicom organise regular AI related workshops, tutorials and seminars.
• KDD-97. August 14-17th 1997, Newport Beach, California.
• PADD97 - The Practical Application of Knowledge Discovery and Data Mining, Wednesday 23rd to Friday 25th April 1997, London.
• Summary of book Advances in Knowledge Discovery and Data Mining by Usama M. Fayyad; Gregory Piatetsky-Shapiro; Padhraic Smyth, and Ramasamy Uthurusamy. Published by the AAAI Press/MIT Press, 1996. ISBN 0262-56097-6.
• Data Mining and Knowledge Discovery: An International Journal . Published by Kluwers.
• IEEE Expert , Volume 11, Number 5, October 1996. Special Issue on Data Mining.
• AI Magazine , Fall 1996. "From Data Mining to Knowledge Discovery in Databases" by Usama M. Fayyad; Gregory Piatetsky-Shapiro; Padhraic Smyth.

Mailing Lists

• KDD Nuggets electronic newsletter.

Index | About AIAI | People | Projects | Technologies | Business Services | Publications | Info. Services

Last updated 2nd June 1997
by Ian Harrison