Divergence-from-randomness model

In the field of information retrieval, divergence from randomness (DFR) is a generalization of one of the very first models, Harter's 2-Poisson indexing-model. It is one type of probabilistic model. It is used to measure the amount of information carried in documents. The 2-Poisson model is based on the hypothesis that the level of documents is related to a set of documents that contains words that occur to a relatively greater extent than in the rest of the documents. It is not a model, but a framework for weighting terms using probabilistic methods, and it has a special relationship for term weighting.

Term weights serve as the standard of whether a specific word is in that set or not. Term weights are computed by measuring the divergence between a term distribution produced by a random process and the actual term distribution.

Divergence from randomness models set up by instantiating the three main components of the framework: first selecting a basic randomness model, then applying the first normalization and at last normalizing the term frequencies.