To remove duplicate lines from a file use the command:
uniq in_file out_file
uniq searches the in_file for consecutive duplicate lines - a line that is immediately followed by one or more lines that are identical to it - and removes them. The result is an out_file that contains no duplicate lines.
Use the sort command if you want to remove duplicate lines that occur in different parts of a file.