Sunday, December 23, 2007

Record Linkage Resources

Download whitepaper on "Approximate Duplicate Detection" which opens with the following:

Detecting database records that are approximate duplicates, but not exact duplicates, is an important task. Databases may contain duplicate records concerning the same real-world entity because of data entry errors, unstandardized abbreviations, or differences in the detailed schemas of records from multiple
databases – such as what happens in data warehousing where records from multiple data sources are integrated into a single source of information – among other reasons. In this paper we review a system to detect approximate duplicate records in a database and provide properties that a pair-wise record matching algorithm must have in order to have a successful duplicate detection system.


Download paper on probabilistic record linkage and an automated procedure to minimize the undecided-matched pair problem. Here is an introduction to this paper:

Probabilistic record linkage allows the assembling of information from different data sources. In this article, we present a procedure in case a one-to-one relationship between records in different files is expected but is not found only by applying the probabilistic record linkage methodology. Our data were births and infant deaths from the 1998-birth cohort whose mother’s place of residence was the City of São Paulo at the time of birth. Our assumption was that pairs for which a one-to-one relationship was obtained, and a best-link was found with the highest combined weight would be considered as uniquevocally matched pairs or gold-standard and should then provide information in order to decide about pairs in which such a relationship could not be established. For example, we observed that the for the unequivocally matched pairs a clear and expected relationship between differences in dates of death and birth registration could be assessed. As a result, such a relationship was used to help solving the remaining pairs for which a one-to-one relationship could not be found. Indeed, we reduced the number of non-uniquely matched records and even though we could not establish a one-to-one relationship for every single death we reduced the number of uncertain. We suggest that future research using record linkage should use combined strategies from results from first record linkage runs before a full clerical review (the standard procedure under uncertainty) in order to most efficiently (and less costly), retrieve record matches.


Applications of Record Linkage Techniques

Here is an excerpt:

Record linkage is simply the bringing together of information from two or more records that is believed to relate to the same entity, e.g., the same individual, the same family or the same business. This may entail the linking of records within a single computer file to identify duplicate records. Alternatively, record linkage may entail the linking of records across two or more files. The challenge lies in bringing together the records for the same individual entities. Such a linkage is known as an exact match. The task is easiest when 1) the files have nearly unique identification numbers (e.g., social security numbers), 2) information is recorded in standardized formats and 3) the files are small. In the absence of nearly unique identifiers, names, addresses, dates-of-birth or other indirect identifiers are frequently used in the matching process.

record linkage description

Here is a good description of "Record Linkage" from the University of Manitoba website:

Record linkage is simply the integration of information from two independent sources. Records from the two sources that are believed to relate to the same individual are matched in such a way that they may then be treated as a single record for that individual. Records brought together in this way are said to be linked. The principles of record linkage may be applied to any field in which it is necessary to bring together information recorded about persons in different places or at different times.

Thursday, December 20, 2007

What is the Perfect Computer?

As a computer programmer, I was intrigued by the opening chapters of Dianetics, the Modern Science of Mental Health, by L. Ron Hubbard claiming that man how the "Analytical Mind" is in fact a perfect computer. It makes calculations based on the data it has. It is ALWAYS RIGHT.

But like a computer program with unfortunate bugs, the calculation may come out with a wrong conclusion, but based on data it has, the program is still "right".

There are two parts to one's mind. The Analytical and the Reactive Mind. Dianetics says that the Analytical Mind IS the Perfect Computer.

"What would you want in a computing machine? The action of the analytical mind—or analyzer—is everything anyone could want from the best computer available. It can and does do all the tricks of a computer. And over and above that, it directs the building of computers. And it is as thoroughly right as any computer ever was. The analytical mind is not just a good computer, it is a perfect computer. It never makes a mistake. It cannot err in any way so long as a human being is reasonably intact (unless something has carried away a piece of his mental equipment).

"The analytical mind is incapable of error, and it is so certain that it is incapable of error that it works out everything on the basis that it cannot make an error. If a person says, “I cannot add,” he either means that he has never been taught to add or that he has an aberration about adding. It does not mean that there is anything wrong with the analytical mind."
- L. Ron Hubbard, Dianetics, The Modern Science of Mental Health

If this were true, how is it possible that people make mistakes, wrong decisions and don't live "perfect" lives? Or more to the point, how does one prevent writing buggy computer software? The answer lies in Dianetics and specifically in the chapter on the Reactive Mind. Dianetics should be required reading for any software developer. It would not only result in less buggy software being developed, it would make for a world of less stressed out programmers!

Anyone interested in Dianetics can get a copy of this book here.