Thermo-Nuclear Tidy-Up of a digital photo collection

Version 7, January 2016

After many many years of archiving pictures (I have 300K+ in my Lightroom archive now), changing software and doubling up on many files when moving to a new archive system, I have a lot of duplicates and junk in my image collection. The purpose of this workflow is to clean that up and start fresh. This is a workflow uses largely using ExifTool to clean up a folder structure of files and XMP sidecar files exported from iPhoto, Aperture and Lightroom and its purpose is to :

1. get rid of duplicates
2. change filenames into a consistent one for files and folders
3. not loose metadata (or as little as possible)
4. create a consistent folder hierarchy (YYYY/YYYYmmdd/YYYYddmm_hhmmss.ext)
5. and filter out (and store separately or give a postfix name) files like screenshots and videos

The workflow is provided as is and is developed for Mac OS X and in particular UNIX-users in mind. Proceed to use at your own risk. Not all steps has been completely tested yet, so perform your own testing on copies of files before committing. Keep backups on hand after cleanup to be able to recover from loss of data. The idea of this workflow is to rename and move images out of a "old" folder and into a "new" folder where renamed/organized files/folders live. That way, the "old" folder should be almost empty when all these commands have been run. Make sure you understand that this script WILL modify your RAW files, both in terms of metadata content and filename. Thanks to Phil Harvey and Hayo Baan for constructive input so far. I would like you to share your experience or modifications to this at http://u88.n24.queensu.ca/exiftool/forum/.

- Kjell Are Refsvik

1. Export all files with XMP sidecar files from Aperture, iPhoto and Lightroom to a folder on a new HD (/Volumes/Bilder3/old):


1.5 Set full access rights to all files in "old" directory recursively
chmod -R 777 /Volumes/Bilder3/old  

2. Remove duplicate images:
Run PhotoSweeper (http://overmacs.com/) to remove duplicate photos in "old" folder (and subfolders) and trash them.
[This will presumably leave a lot of orphaned .XMP sidecar files, but does it matter?]
Also, could be done with fdupes and the command $fdupes -R -d -N (deletes duplicate photos that are identical)

3. Ingest metadata from .xmp files into parent image while overwriting the original file (only to be done if you have a backup of original files)
exiftool -r -X --ext xmp -tagsfromfile %d%f.xmp -xmp -overwrite_original /Volumes/Bilder3/old
Note: A number of "Warning: Error opening files - ..."  appears when running this. Not sure what the reason is or how to fix it.

4. Remove all .xmp sidecar files from disk: 
verify: find /Volumes/Bilder3/old -iname '*.xmp' -type f
delete: find /Volumes/Bilder3/old -iname '*.xmp' -type f -delete 

5. Rename all videofiles with date-based names and move them to a "video" folder:
mkdir /Volumes/Bilder3/video; exiftool '-filename