Statistical distributions and entropy considerations in gene codes

In our paper selected linguistic features of genomes to study the statistics of the gene codes are considered. We present the information theory from which it follows that if the system is described by distributions of hyperbolic type it leads to the possibility of entropy loss and stability. We show that the histograms of gene lengths are similar to that of language words. We show the correspondence between presented theory and results for the number of replicated genes and replicated fragments of genes in genomes for Borelia burgdorferi, Escherichia coli and Saccharomyces cerevisiae S288c.


I. INTRODUCTION
In last years there appeared a possibility to provide some knowledge of the genome sequence data in many organisms. The genome was studied intensively by number of different methods [1][2][3][4][5][6][7][8][9][10]. The statistical analysis of DNA is complicated because of its complex structure; it consists of coding and non coding regions, repetitive elements, etc., which have an influence on the local sequence decomposition. Long range correlations in sequence compositions of DNA are much more complex than simple power law; moreover effective exponents of scale regions are varying considerably between different species [1], [5]. In papers [3,4] the Zipf approach to analyzing linguistic texts has been extended to statistical study of DNA base pair sequences.
In our paper we take into account some linguistic features of genomes to study the statistics of gene codes. One of the fundamental observations based on the information theory says that if the system is described by special distribution of hyperbolic type it implies a possibility of entropy loss. Distributions of hyperbolic type describe also property of stability and flexibility which explain that the languages can develop without changing its basic structure (see [11]). In the experimental part of our research we show that a similar situation can occur in a genome sequence data which carries the genetic information.
To demonstrate analogies between coding of genes and representation of words in human languages we generated histograms of the sizes of groups of words of the same length in selected different European languages as well as histograms of the sizes of groups of genes of the same length in a genome. Three genomes were selected for evaluation, that is, two bacteria: Borrelia burgdorferi and Escherischia coli, and a yeast: Saccharomyces cerevisiae S288c. The gene data were obtained from NCBI GeneBank [12]. Each of the histograms was successfully approximated by Asymmetric Normal Inverse Gaussian Distribution.
Then, for each of the three genomes histograms of the sizes of groups of identical genes were generated. We showed that each of them can be modelled by distributions of hyperbolic type defined in the paper.
Additionally, the histograms for sizes of groups of identical fragments of genes of different sizes were calculated. In this case any two genes are regarded as similar when there exist a gene fragment, that is, an ordered sequence of symbols of size n which can be found in both genes. Obtained results confirmed our hypothesis that the basic structure of genome remain stabile and entropy loss is possible.
The paper is organized as follows. In Sect. II we begin with a brief presentation of some common features of linguistics and genetics. Then, in Sect. III we describe Code Length Game, Maximal Entropy Principle and the theorem about hyperbolic type distributions and entropy loss. Sect. IV contains the experimental research concerned analysis of statistical properties of gene strings in three selected genomes and consists of three parts. Histograms of gene lengths are presented in the first part. Evaluation of the frequency of genes in a genome is the subject of the second part, that is, the results of searching for replicated genes in three genomes are presented. In the third, last part we observe frequency of appearance of gene fragments in a genome for different lengths of the fragments. Sect. V concludes the paper.

II. SOME COMMON FEATURES OF LINGUISTICS AND GENETICS
A language is characterized by an alphabet with letters, like for the case of latin languages: a, b, c, etc. The words are denoted as sequences of n letters. In quantum physics the analogy to letters can be attached to pure states, and texts correspond to mixed general states. Languages can have very different alphabets, for example: computers -0,1 (two bits), English language -27 letters with space, or DNA -four nitric bases: G(guanine), A(adenine), C(cytosine), T(thymine). The collection of letters can be ordered or disordered. To quantify the disorder of different collections of the letters we use an entropy where p i denotes probability of occurrence of the i-th letter. If we take the base of logarithm 2 this will lead to the entropy measured in bits. When all letters have the same probability in all states obviously the entropy has maximum value H max . A real entropy has lower value H ef f , because in a real languages the letters have not the same probability of appearance. Redundancy R of language is defined [13] as follows: The quantity H max -H ef f is called an information. Information depends on difference between the maximum entropy and the actual entropy. The bigger actual entropy means the smaller redundancy. Redundancy can be measured by values of the frequencies with which different letters occur in one or more texts. Redundancy R denotes the number that if we remove the part R of the letters determined by redundancy, the meaning of the text will be still understood.
In English some letters occur more frequently than other and similarly in DNA of vertebrates the frequency of nitric bases C and G pairs is usually less than the frequency of A and T pairs. The low value of redundancy allows in easier way to fight transcription errors in a gene code. In the papers [3,4] it is demonstrated that non coding regions of eukaryotes display a smaller entropy and larger redundancy than coding regions.

III. MAXIMAL ENTROPY PRINCIPLE AND CODE LENGTH GAME AND ENTROPY LOSS
In this section we provide some mathematics from the information theory which will be helpful in the quantitative formulation of our approach, for details see [11].
Let A be the alphabet which is a discrete finite or countable infinite set. Let M 1 + and ∼ M 1 + (A) are respectively, the set of probability measures on A and the set of non-negative measures P , such that P (A) ≤ 1. The elements in A can be thought as letters. By K(A) we denote the set of mappings, compact codes, k : A → [0, ∞], which satisfy Kraft's equality [14]: By ∼ K(A) we denote the set of all mappings, general codes, k : A → [0, ∞], which satisfy Kraft's inequality [14]: For k ∈ ∼ K(A) and i ∈ A, k i is the code length, for example the length of the word. For k ∈ ∼ K(A) and P ∈ M 1 + (A) the average code length is defined as There is bijective correspondence between p i and k i k i = − ln p i and p i = exp(−k i ).
For P ∈ M 1 + (A) we also introduce the entropy The entropy can be represented as minimal average code length, (see [11]): < k, P > .
The formula (7) presents the Nash optimal strategies. R min denotes minimum risk, k denotes the Nash equilibrium code, P denotes probability. For example, in a linguistics the listener is a minimizer, speaker is a maximizer. We have words with distributions p i and their codes k i , i = 1, 2, .... The listener chooses codes k i ,the speaker chooses probability distributions p i . We can notice that Zipf argued [15] that in the development of a language vocabulary balance is reached as a result of two opposing forces: unif ication which tends to reduce the vocabulary and corresponds to a principle of least effort, seen from point of view of speaker and diversification connected with the listeners wish to know meaning of speech. The principle (7) is so basic as Maximum Entropy Principle has a sense that search for one type of optimal strategy called as Code Length Game translates directly into a search for distributions with maximum entropy. It is a given a code k ∈ ∼ K(A) and distribution P ∈ M 1 + (A). Optimal strategy according H(P ) = inf k∈ ∼ K < k, P > is represented by entropy H(P ), where actual strategy is represented by < k, P >.
Zipf's law is an empirical observation which relates rank and frequency of words in natural language [15]. This law suggests modeling by distributions of hyperbolic type [11], because no distributions over N have probabilities proportional to 1/i, due to the lack of normalization condition. We consider a class of distributions P = (p 1 , p 2 , ...) over N . If p 1 ≥, p 2 ≥ ..., P is said to be hyperbolic if for any given α > 1, p i ≥ i −α for infinitely many indexes i. As an example we can choose p i ∼ i −1 (log(i)) −c for some constant c > 2.
The Code Length Game for model P ∈ M 1 In such a case a distribution P * is the H max attractor such that P n → P * , if for every sequence (P n ) n>1 ⊆ P for which H(P n ) → H max (P). One expects that H(P * ) = H max (P) but in the case with entropy loss we have H(P * ) < H max (P). Such possibility appears when distributions are hyperbolic. It follows from theorem in [11].
Theorem. Assume that P * ∈ M 1 + (N ) is of finite entropy and it has ordered point probabilities. The necessary and sufficient condition that P * can occur as H max -attractor in a model with entropy loss, is that P * is hyperbolic. If this condition is fulfilled than for every h with H(P * ) < h < ∞, there exists a model P = P h with P * as H max -attractor and H max (P h ) = h. In fact, P h = (P | < k * , P >≤ h) is a largest model. k * denotes the code adopted to P * , that is, k * = − ln(p * ), i > 1.
As an example we can consider "an ideal" language where the frequencies of words are described by hyperbolic distribution P * with a finite entropy. At a certain stage of life one is able to communicate at reasonably height rate about H(P * ) and improve language by introduction of specialized words, which occur seldom in a language as a whole. This process can be continued during the rest of life. One is able to express complicated ideas develop language without changing a basic structure of the language. We can expect similar situation in gene strings, which also carry an information.

A. Part I-Histograms for gene lengths and word lengths
In this part we study the histograms of gene lengths. One can observe, that the histograms are of the same type as histograms for the word lengths. We model histograms of gene lengths by Asymmetric Inverse Gaussian Distributions as in the case of histograms of word lengths. Systems where maximum entropy distributions does not exists can be described by distributions of hyperbolic type. In such systems stability and flexibility is present similarly as in natural languages or in description of genes where redundancy is confirmed. It is interesting to show that the histograms for gene lengths resemble other histograms obtained for word lengths in human languages [16]. In Figure 2 we present the histograms of the sizes of groups of words of the same length for four European languages: Czech, German, Italian and Polish which also the fit the Asymetric Inverse Gaussian distributions. In this case the AIGD has following parameters: for Czech (µ = 4.0883, σ = 2.1168, and γ = 5.4246); for German (µ = −0.5941, σ = 2.2183, and γ = 12.2476); for Italian (µ = 11.0250, σ = 2.4615 and γ = −0.5362); and for Polish (µ = −15.1809, σ = 1.3876, and γ = 27.3526). We investigated asymptotic properties of these distributions. The tails of these distributions appear to be of the power type. We show it for Borrelia and Saccharomyces.

B. Part II -Statistics for replicated genes
In this part we evaluated numbers of identical genes in each of the three genomes. Any two genes are identical iff they are of the same length and have the same content ordered in the same way. Otherwise, the genes are regarded as different, that is, for example, a smallest difference in the order of symbols in any two genes of the same size and content makes them different. The main parameters of the gene "language" for each on the genomes is presented in Table I. One can see, that gene lengths span within the range from 6381 for Borrelia burgdorferi to 14682 -for Saccharomyces cerevisiae S288c. In every case the range is many times wider that the number of genes in the genome. However, in spite of this, some number of duplicated genes can be found in genomes of each of the analysed bacterium and yeast. The lists of identical genes are presented in Tables II, III and IV (the genes are identified with labels originated from GeneBank files). Each table shows groups of genes sorted from the largest groups to the smallest ones. Respectively to the data in the tables two histograms with sorted group sizes for Escherischia coli and Saccharomyces cerevisiae S288c are presented in Figure 6 (the histogram for Borrelia burgdorferi is omitted due to its simplicity -just two groups which consist of only two genes appear in this case).  Figure 6 shows how often the same sequences are repeated in genomes. For example, in the case of Escherischia coli the two mostly repeated sequence with the rank numbers one and two appear in 5 different genes each, the third sequence appears 4 times, etc.
The series with sizes of the groups of identical genes can be interpreted as probabilities of gene appearance in the genome. To obtain this, for every unique sequence we assign its group size, that is the number of appearances in the genome. Then, the group sizes have to be normalized, because probabilities have to sum up to 1. Such probability of appearance of unique sequences is of hyperbolic type in the sense of definition p i ≥ i −α , that is, one can show that the inequality is satisfied for each of the values in the series respectively for: Borrelia burgdorferi -for, e.g., α = 9.2785, Escherischia coli -for, e.g., α = 9.9426, and Saccharomyces cerevisiae S288c -for, e.g., α = 10.528. It is necessary to mention here, that due to the normalization of a series of nonzero values the inequality p i ≥ i −α can never be satisfied for i = 1 because 1 −α = 1 regardless of α (simply, the probability of any option is never equal 1 if there is more that one random option to chose and all of them have a non-zero chance to be selected). Therefore, to show the hyperbolic nature of a single series, we needed to start with comparison of the second element p i=2 and 2 −α , that is, p i=1 is omitted. For clarity, we did also another calculations where the first element p i=1 is compared with (i + 1) −α and so on, and obtained values of α are the same as in the former method.
Graphs of sizes of gene groups from Figure 6 and the observation that probabilities of appearance of unique sequences are of hyperbolic type suggest that the realization of the theorem and that entropy loss is possible. Similarly, as in the case of languages the basic structure of genomes remains stabile.   IV. Groups of identical genes in the genome of Saccharomyces cerevisiae S288c; every group starts with a group size, the length of a unique sequence which is written in brackets and the list of gene labels from GeneBank

C. Part III -statistics for replicated fragments
One can see that the numbers of identical genes in a genome are rather small mostly due to the fact, that gene sizes differ to each other in many cases. In the third part of the experimental research we evaluate numbers of identical fragments of genes in the way where the i-th symbol of one gene sequence being compared is not necessarily aligned against the i-th symbol of the other. Clearly, the size of compared fragment is constant, however, the fragment starting position in the gene may vary. For example, in the case of Borrelia burgdorferi the size of the shortest gene equals 120 symbols whereas the size of the longest one -6501. Assuming that the fragment size equals 120, there is just one fragment in the shortest gene and 6381 fragments in the longest one: a fragment from the first symbol to the 120-th one, from the second to 121-st, from the third to 122-nd, and so on. This way, for the fragment size equal 120 all of the genes are compared to each other many times, that is, each gene can be compared as many times as the number of fragments can be selected inside it.
In this part we started from searching for identical fragments of size equal the size of the shortest gene in the genome, that is, fragments of size 120 for Borrelia burgdorferi, 75 -for Escherischia coli, and 51 for Saccharomyces cerevisiae S288c. The figures with sizes of groups of identical fragments sorted in descending order are presented in Figure 7. In spite of the fact that in this comparisons the fragment starting position in compared genes may vary, both graphs in Figure 6 and graphs in Figure 7 present similar "staircase" type. This confirms the stability and flexibility of gene structures.  14), Escherischia coli (the right graph -fragment sizes vary from 75 to 14) and Saccharomyces cerevisiae S288c (the bottom graph -fragment sizes vary from 51 to 14) It would be interesting to know how the characteristic of identical fragment numbers in a gene varies respectively to different sizes of the fragment size. Therefore, for each of the three genomes we evaluated numbers of identical fragments equal or smaller that the size of the shortest gene. For Borrelia burgdorferi the fragment size varied from 120 to 14, for Escherischia coli -from 75 to 14, and for Saccharomyces cerevisiae S288c -from 51 to 14. In every case the lower limit for the fragment length was set to 14 because the probability of fragment appearance in the gene grow rapidly for shorter fragments due to the limited number of permutations of four symbols in such a short sequence. Aggregated graphs with histograms of sizes of groups of identical fragments sorted in descending order are depicted in Figure 8. The three graphs show a series of histograms obtained for each of the possible fragment sizes for the three genomes. The histograms all together build a surface over the two-dimensional domain where one axis represents a group number in the list sorted by the group size and the other -a fragment size.
These figures present two dimensional surfaces with very regular curves. The graph shape and its regularity confirm stability and flexibility of gene structures.

V. FINAL REMARKS
In this paper we follow the statistical description of genes in three selected genomes aimed at finding arguments supporting thesis that the entropy loss in gene "language" is possible and the basic structure of genomes remain stabile. We show that histograms of gene length and word length are similar and both can be modeled by Asymmetric Inverse Gaussian distributions. Additionally, tails of distributions for gene lengths appear to be of the power type. We test selected genomes and describe the gene code statistics dealing with the number of repetitions and replicated gene fragments in genomes. Considering that distributions of hyperbolic type describe property of stability and flexibility, we show that histograms of identical genes interpreted as probabilities of gene appearance in the genome are of hyperbolic type in the sense of definition p i ≥ i −α , and show example values of α for each of the genomes. We show also regularity and similarity of histograms with numbers of replicated genes and replicated gene fragments. All the obtained results confirm our thesis.