Developing an efficient algorithm for representation and compression of large Bengali text

Efficient coding is one of the challenging aspects of information and communication theory. On the other hand, the natural languages such as Bengali is coded using Unicode technology which requires more space and thus takes more time to transfer the data of that language. In this paper, we have proposed a novel algorithm to represent Bengali text efficiently and then to compress the text offering a better compression ratio. Each Bengali character is represented by a unique 2-digit intermediate decimal value. Indexing and sorting all the word values successive subtraction is performed on the values in hope to reduce the weight of the numbers. The new values of each word can now be encoded with a very few bits. In comparison to other compressors, the compression ratio of the proposed algorithm decreases in a big amount for the large text which may contain more duplicate or redundant words, more words with the same length and more words of the same length with the same prefix called Uposorgo in Bengali.