Find duplicate files: Difference between revisions

For extra points, detect when whole directory sub-trees are identical, or optionally remove or link identical files.

<br><br>

 

=={{header|Elixir}}==

{{trans|Ruby}}

def find_duplicate_files(dir) do

IO.puts "\nDirectory : #{dir}"

end

 

 

{{out}}

=={{header|Go}}==

In theory this should work on any of the operating systems supported by Go (Linux, macOS, Windows, OpenBSD etc.) though only tested on Ubuntu 16.04.

 

import (

fmt.Println()

}

 

{{out}}

- works on Windows as well as Unix Systems (tested with Mint 17 / Windows 7)

</pre>

import Crypto.Hash.MD5 (hash)

import Data.ByteString as BS (readFile, ByteString())

printf "Something went wrong - please use ./%s <dir> <bytes>\n" name

 

 

Example output:

Hard links are indicated by displaying the files on the same line separated by "=".

MD5 checksums are used to detect duplicate files.

import java.nio.*;

import java.nio.file.*;

return Integer.compare(len1, len2);

}

 

{{out}}

file1 = file3

file6

</pre>

 

Should work on Windows, macOS and Linux.

 

 

function find_duplicates(path::String, minsize::Int = 0)

end

 

 

{{out}}

</pre>

 

minSize=Quantity[1,"Megabytes"];

allfiles=Once@Select[FileNames["*","",∞],!Once@DirectoryQ[#]&&Once@FileSize[#]>minSize&];

data={#,Once[FileHash[#,hash,All,"HexString"]]}&/@allfiles[[;;5]];

{{out|Sample output}}

sample directory:

someFile3 bef0039c33277f743b60b0076871110b96e14de34045aafc8e764349de6043b5

directory\someFile eebe4df6d2951e77973b83af039f6565b215f74113028bbc5d8f96b856947abe

sample output:

<pre>35 MB {someFile,directory\someFile}</pre>

 

=={{header|Objeck}}==

Solution works on Windows, macOS and Linux.

use System.Time;

use Collection;

return "{$@name}, {$@size}, {$date_str}";

}

 

{{output}}

/tmp/foo/ee.obe, 20020, 3/29/2019 8:47:33 PM

/tmp/foo/dd.obe, 20020, 3/29/2019 8:47:14 PM

</pre>

 

=={{header|Perl}}==

For supplied directory, compare all files, recursing into sub-directories. By default, showing duplicate files of 1 byte or larger, configurable with command-line option. Using CPAN <code>File</code> modules for enhanced portability.

use File::Compare qw(compare);

use Sort::Naturally;

}

 

{{out}}

<pre> 372 aaa.txt, dir2/aaa.txt

=={{header|Phix}}==

Works on Windows and Linux. No handling of hard (or soft) links.

{{out}}

<pre>

=={{header|PicoLisp}}==

File duplicates in /bin dir on Void Linux. Hash provided by xxhash library via mmap.

(de mmap (L F)

(native "@" "mmap" 'N 0 L 1 2 F 0) )

group

(cadr Lst) )

{{out}}

<pre>

=={{header|Python}}==

 

import os

import hashlib

 

FindDuplicateFiles('/home/tim/Dropbox', 1024*1024)

 

=={{header|Racket}}==

#lang racket

 

 

(show-duplicates (find-system-path 'home-dir) 1024)

 

=={{header|Raku}}==

(formerly Perl 6)

 

 

sub MAIN( $dir = '.', :$minsize = 5, :$recurse = True ) {

for %files.sort( +*.key ).grep( *.value.elems > 1)».kv -> ($size, @list) {

my %dups;

for %dups.grep( *.value.elems > 1)».value -> @dups {

say sprintf("%9s : ", scale $size ), @dups.join(', ');

default { ($bytes / 2**30).round(.1) ~ ' GB' }

}

{{out|Sample output}}

Passing in command line switches: --minsize=0 --recurse=False /home/me/p6

<br>Note that the &nbsp; '''tFID''' &nbsp; (temp) &nbsp; file is hard coded to the &nbsp; '''C:''' &nbsp; drive.

<br>Only minimal error checking is performed.

sep=center(' files are identical in size and content: ',79,"═") /*define the header. */

tFID= 'c:\TEMP\FINDDUP.TMP' /*use this as a temporary FileID. */

 

if lines(tFID)\==0 then 'ERASE' tFID /*do housecleaning (delete temp file).*/

'''output''' &nbsp; when using (checking) with the default root directory:

<pre>

::* &nbsp; uses variables for some command names and command options

::* &nbsp; shows the number of files examined and also the directory name

sep=center(' files are identical in size and content: ',79,"═") /*define the header. */

parse arg !; if !all(arg()) then exit /*boilerplate HELP(?)*/

novalue: syntax: call err 'REXX program' condition("C") 'error',condition("D"),'REXX source statement (line' sigl"):",sourceline(sigl)

p: return word(arg(1),1)

'''output''' &nbsp; when using the DIR (folder): &nbsp; H:\#\REX

<pre>

 

=={{header|Ring}}==

# Project : Find duplicate files

 

next

return alist

Output:

<pre>

=={{header|Ruby}}==

It confirms once by the file size. When the same, it confirms a digest (md5).

 

def find_duplicate_files(dir)

end

 

 

Sample Output:

 

=={{header|Rust}}==

collections::BTreeMap,

fs::{read_dir, File},

 

Ok(())

 

=={{header|Sidef}}==

It uses the portable ''File::Find'' module which means that it should work, virtually, on any platform.

 

require('File::Find')

say "#{files.sort.join(%Q[\n])}\n#{'-'*80}"

}

Section of sample output:

<pre>% sidef fdf.sf 0 /tmp /usr/bin

{{tcllib|fileutil}}

{{tcllib|md5}}

package require md5

 

}

}

Section of sample output:

<pre>

./compat/zlib/contrib/delphi/zlibd32.mak has duplicates

dupe #1: ./compat/zlib/contrib/pascal/zlibd32.mak

</pre>

 

 

File findDupFiles.zkl:

const FLAGS=FILE.GLOB.IGNORE_CASE + FILE.GLOB.NO_DIRS;

var [const] MsgHash=Import("zklMsgHash");

println("Found %d duplicate files:".fmt(files.apply("len").sum(0)));

{{out}}

<pre>