uniq - it is used to report or filter out repeated lines in a file.

$ cat demo.txt
redhat
debian
ubuntu
ubuntu
centos
fedora
fedora
fedora
fedora

To report or filter out for lines that are adjacent and repeated
$ uniq demo.txt

To only print unique lines
$ uniq -u demo.txt

To prefix lines by the number of occurrences
$ uniq -c demo.txt

To only print duplicate lines, one for each group
$ uniq -d demo.txt

To print all duplicate lines 
$ uniq -D demo.txt

To print all duplicate lines but  allow  separating  groups  with  an  empty line
$ uniq --all-repeated=prepend demo.txt

To ignore differences in case when comparing
$ uniq -i demo.txt

To skip comparing the first X number of characters in each line
$ uniq -s 3 demo.txt

To compare only the first X number of characters
$ uniq -w 4 demo.txt



regards,
T.Dhanasekar