Replies: 5 comments 11 replies
-
|
@cfc62, See also #1722 and related GitHub discussions:
My goal on this: is to move my ~100K pics Photos Library from an iMac Monterey Referenced Library into Sequoia (once I get a new Mac 😉 ) while preserving as much info as possible -- and yes, having it on an external SSD disk, tag it as a System Library and hopefully get all the Photos analysis done. Library on external disk + System Library + Copy Items to the Photos Library resulted in Persons and Memories being created -- promising! (further testing is underway)
I'm going with the following sequence. Please note I don't have RAW pics and I don't used iCloud storage. example export.zsh command I'm testing with. Check the variables setting at the start. Please note I've edited to be able to share it.. so I did not actually test it. Hope there are no major errors! Say no to step 2 as you don't have the convert_aae.py#!/bin/zsh
#------------------------------------------------------------------------------
# Where to look for autoloaded function definitions
fpath=($fpath ~/.zsh/functions)
# Autoload all shell functions from all directories in $fpath (following
# symlinks) that have the executable bit on (the executable bit is not
# necessary, but gives you an easy way to stop the autoloading of a
# particular shell function). $fpath should not be empty for this to work.
for func in $^fpath/*(N-.x:t); autoload -Uz $func
# Force loading of all the functions in exif_function
# exif_functions
#------------------------------------------------------------------------------
LOGFILE=recreate.export.$$.log
SYNC_EXPORT_LOGFILE=recreate.sync.export.$$.log
SYNC_IMPORT_LOGFILE=recreate.sync.import.$$.log
CONVERT_LOGFILE=recreate.convert.AAE.$$.log
DEST=photos.exported
METADATA=MetaData_exported.db
ORIG_LIBRARY="/Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/database/Photos.sqlite"
LIBRARY="/Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/"
DST_LIBRARY=/Volumes/ExternalDisk/Pictures/Recreate.NonReferenced.photoslibrary
function ShouldIContinue() {
local STEP_NAME=$1
echo "$STEP_NAME: Continue (y/[n])?"
read -r x
[[ $x == "y" ]]
}
# EXPORT -----------------------------------
ShouldIContinue "Step #1: EXPORT"
if [[ $? -eq 0 ]]; then
echo "Ok... Performing Action!"
tail -F ${LOGFILE} & xPID=$(echo $!)
echo Start export $(date) > ${LOGFILE}
mkdir -p ${DEST} >> ${LOGFILE}
caffeinate osxphotos export ${DEST} --load-config recreate.toml --library "${ORIG_LIBRARY}" >> ${LOGFILE} 2>&1
echo End export $(date) >> ${LOGFILE}
kill $xPID
else
echo "... Bypassing Action!"
fi
# CONVERT_AAE -----------------------------------
STEP="Step #2: Convert_AAE"
ShouldIContinue "${STEP}"
if [[ $? -eq 0 ]]; then
echo "Ok... Performing Action!"
tail -F ${CONVERT_LOGFILE} & xPID=$(echo $!)
echo Start "${STEP}" $(date) > ${CONVERT_LOGFILE}
( echo path ; find ${DEST} -type f -name \*.AAE ) | python convert_aae.py --convert-all >> ${CONVERT_LOGFILE}
echo End "${STEP}" $(date) >> ${CONVERT_LOGFILE}
kill $xPID
else
echo "... Bypassing Action!"
fi
# SYNC export -----------------------------------
ShouldIContinue "Step #3: SYNC EXPORT"
if [[ $? -eq 0 ]]; then
echo "Ok... Performing Action!"
tail -F ${SYNC_EXPORT_LOGFILE} & xPID=$(echo $!)
echo Start sync export $(date) > ${SYNC_EXPORT_LOGFILE}
caffeinate -s osxphotos sync --verbose --timestamp --not-hidden --export ${METADATA} --library ${LIBRARY} --report recreate.sync.export.{today.date}.csv >> ${SYNC_EXPORT_LOGFILE} 2>&1
echo End sync export $(date) >> ${SYNC_EXPORT_LOGFILE}
kill $xPIDX
else
echo "... Bypassing Action!"
fi
# SYNC import -----------------------------------
ShouldIContinue "Step #4: SYNC IMPORT"
if [[ $? -eq 0 ]]; then
echo "Ok... Performing Action!"
tail -F ${SYNC_IMPORT_LOGFILE} & xPID=$(echo $!)
echo Open the new library in Photos to run SYNC import
echo Make sure you have imported all pics.
echo Press RETURN to continue...
read ans
echo Start sync import $(date) > ${SYNC_IMPORT_LOGFILE}
caffeinate -s osxphotos sync --verbose --timestamp --import ${METADATA} --unmatched --merge albums,description,favorite,keywords,location,title --library ${DST_LIBRARY} --report 'recreate.sync.import.{today.date}.csv' >> ${SYNC_IMPORT_LOGFILE}
echo End sync import $(date) >> ${SYNC_IMPORT_LOGFILE}
kill $xPID
else
echo "... Bypassing Action!"
fi
Example: recreate.toml... please note I've edited to be able to share it.. so I did not actually test it. Hope there are no major errors![export]
#--- Update and Cleanup
cleanup = true
update = true
#db = "/Users/Shared/Pictures/iPhoto Shared Library.photoslibrary"
#--- Group by Year/Year-Month/Day(01-10,11-20,21-31).
# Filter Albums starting with 1|2... as all my albums have the year YYYY so to filter out other Albums and Slideshows that may exist and not export them
directory = "{album|filter(startswith 1|2)?,}{created.year}/{created.year}-{created.mm}/{created.dd| <= 10?01-10,{created.dd| <= 20?11-20,{created.dd| > 20?21-31,00}}}"
#--- Dry-Run
# dry_run = true
#--- No exiftool in favor of --export_as_hardlink and --sidecar XMP
# exiftool = true
export_as_hardlink = true
exiftool_merge_keywords = true
exiftool_merge_persons = true
sidecar = ["xmp" ] # , "json"]
#--- Extract only from iPhotoEvents
# folder = ["iPhoto Events"]
#--- Edited files. Export AAE options. Double check need to edited_suffix.
export_aae = true
keyword_template = [
"{keyword}",
]
no_progress = true
not_hidden = true
ramdb = true
retry = 2
skip_bursts = false
skip_live = false
#--- From exported Original/AAE: It imports the original and applies AAE. Keeping the original on file. Photos indicates edited file.
#--- see also https://github.com/RhetTbull/osxphotos/issues/1579
skip_edited = true
skip_original_if_edited = false
skip_raw = true
strip = true
timestamp = true
touch_file = true
verbose = true
#--- For Testing you can limit the export.
# from_date = 2010-01-01T00:00:00
# to_date = 2011-01-01T00:00:00
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks for all this, will definitely dive into it. I did a trial export fine... or at least it looked fine. Unfortunately not all was well. On, (re-) importation, I ran into the same "not enough storage space" error that I originally ran into, but this time I ran into it right away and couldn't proceed at all so I've gone back to my backup configuration for the time being. This is odd as I had actually freed up even more space than before. Also, I tried the importation twice, and Photos reported significantly different asset counts each time (once 141k, the other 148k). I also ran into a whole bunch of errors about not being able to create files based on EXIF data which I've not seen before. I was able to see RAW and JPEG pairs were preserved, which is cool - that was something high on the list of things I wanted to have. |
Beta Was this translation helpful? Give feedback.
-
|
Also if keeping everything to the external drive does turn out to be workable, definitely going that route. However, note that it stops running analysis after a few days, or at least that's been my (repeated) experience. Definitely need to keep an eye on it! (Then again, this was on a copied .photoslibrary bundle and not a re-created library). |
Beta Was this translation helpful? Give feedback.
-
Yes, you need space 😉 ! When you do Import in Photos and select the source folder, Photos counts all the assets to be imported.
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks @oPromessa for the detailed response! The ability to fully recreate a library from an export is a goal I have for OSXPhotos. I'm working on a separate project that should help with this as it can use the native Photos API to do the import instead of relying on AppleScript as In the mean time, osxphotos handles most of what you need by exporting Live+movie, RAW+JPEG, etc. automatically. The areas you need to pay attention to are: AAE filesThese hold adjustments for non-destructive edits. Use Edited imagesOSXPhotos normally exports edited images with
This sets the filename to Another way to do this that works for all photos is:
This renames all photos to MetadataTo preserve metadata on import, you can do the following: If using If using Photos to import, you can use If using Other notesNote that one downside of using This is a lot to wade through and eventually I do plan to make a simple
You can use Referenced filesDon't use them! It may be unavoidable given what you're trying to do (circumvent Apple's refusal to perform AI analysis on external drives) but they cause nothing but heartache and grief. This is due to the implementation of referenced files which use security scoped bookmarks to comply with Apple's app sandboxing rules. That is, instead of storing just the path to the referenced files, Photos stored a cryptographically signed bookmark to the file. This bookmark is tied to not just the path but the actual volume (disk). That means if you ever move the photos to a new disk (someday the disk will die) Photos cannot find any of your referenced files. OSXPhotos bypasses this and can find and export the referenced files. Apple may have fixed this in Seqoia (I've not checked) but it's been broken since Catalina when they switched to security scoped bookmarks. I think this is all part of a play by Apple to force uses to buy absurdly expensive SSD upgrades from them but regardless, using a referenced library can cause many problems. Also, |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the details! Even beyond the scoped bookmarks (thanks for that too, I was going to ask for more info on that), there's a bug in Photos where it's mistakenly looking at the wrong volume to determine if there's enough space to import - AND it thinks it's importing into the library even when told not to... For now, this is for a backup version of the library, so even if the disk dies I'm unaffected/there won't be any data loss. |
Beta Was this translation helpful? Give feedback.
-
|
Also FYI I'll admit to completely being ignorant of the importation abilities of osxphotos! :^) |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately, something is amiss. When using Photos to import (referenced) originals from osxphotos, I am again running into a space issue, this one similar to but different to the one I mentioned earlier. Even though I've edited perhaps 5% of my library, when importing Photos wants 2-3x (or more) the storage space. That is to say, when just importing the originals as they are stored in the .photoslibrary bundle (~/*.photoslibrary/originals), the storage space requirements are workable. Here, the number of files is 160k @ 1.38 TB. This yields a library bundle size of 127GB. When importing from the osxphotos export, the number of files is 167K @ 1.44 Tb. Amazingly, trying to import only 1 year out of 12 sees Photos error out, wanting 175GB of free space, which I don't have on this machine. This 175GB is smaller than the size of the source directory, so it's not trying to copy the originals into the bundle; that's the original bug I ran into. Thinking of other ways to tackle this, such as not exporting movies, only exporting edits, etc. |
Beta Was this translation helpful? Give feedback.
-
Try running:
This will print out a count of all the photos of various types in the library which might help figure out why the OSXPhotos count is different than the Photos count. If your edited images exported by OSXPhotos were not in the format |
Beta Was this translation helpful? Give feedback.
-
|
Oh I'm sorry - I understand the difference in photo count - when just looking at the I've narrowed the weirdness down to (I think) edited .mov files. If I remove those things go better. I suspect there's some .mov with some screwed up metadata that Photos is reading and getting all confused over. It will take a while to narrow down though. |
Beta Was this translation helpful? Give feedback.
-
|
@cfc62 Would you mind confirming which approach you're using?
|
Beta Was this translation helpful? Give feedback.
-
|
Hi! I'm trying (it just failed again) to:
This has been nearly always unsuccessful for some unknown reason or reasons. I believe I can manage to get it to work, but it's extremely tedious and likely not worth the effort. More detail below. What was successful was importing from the Importations/failures:
I also see quite a few error messages via I've also seen very odd error messages here and there referring to past .photolibraries that I haven't had for years on this machine. So somewhere MacOS is keeping track of all of the previous libraries (!). I am close to backing up this machine and then resetting it completely, and re-importing with a fresh osxphotos export. I would not doubt that I've careened into some edge case scenario on this machine. Again, the reason for all of this experimentation is to see if I can't convince the Photos suite to work with a transferred library, which it stopped doing in Sequoia. And to see if AI functionality would work with referenced files to avoid the extortionate fees for on-board SSD while still avoiding iCloud. I still haven't tested two things:
|
Beta Was this translation helpful? Give feedback.
-
|
Got it. Let me tell you where I am as my goal is the same... to migrate and move my Photos Library to a new Mac on an external disk.
BTW checking the count of files in osxphotos exported folder and the resulting originals folder the newly created Library I have about 180 diff which I believe to be the edits. $ find photos.hard -iname \*.jpg -o -iname \*.mp4 -o -iname \*.jpeg -o -iname \*.mov -o -iname \*.heic > photos.hard.lst
$ find Recreate.NonReferenced.photoslibrary/originals -iname \*.jpg -o -iname \*.mp4 -o -iname \*.jpeg -o -iname \*.mov -o -iname \*.heic > originals.lst
$ wc -l photos.hard.lst originals.lst
26184 photos.hard.lst
26009 originals.lst
52193 total |
Beta Was this translation helpful? Give feedback.
-
|
OMG look at all the darned work and research we have to do for something that should be dead simple! :^) |
Beta Was this translation helpful? Give feedback.
-
|
I see 15.2 is out, so that I will try today/tomorrow |
Beta Was this translation helpful? Give feedback.
-
|
Yep... some work! FYI... I've updated my post with additional findings. |
Beta Was this translation helpful? Give feedback.
-
Apple currently either has a bug and/or is no longer allowing AI to work on libraries stored on external disks. Hopefully this will be addressed in an upcoming release, but perhaps it won't be.
As documented elsewhere, I've verified that re-importing photos kept (referenced) on an external disk still works, as long as the .photolibrary bundle is on an internal disk.
For insurance purposes, looking to try exporting my master library from one Mac to an external drive, then hanging that external drive on a different (AS) Max and reimporting all assets as referenced files.
Here's where I sheepishly admit I've never used osxphotos for its main purpose of exporting libraries! The doc looks good so I don't think I'll have much issue in constructing the command line options.
What I'm looking for are "gotchas" that might arise. Preference is to preserve edits and the pairing of RAW/jpeg photos, while still exporting the untouched originals. I'm hoping an export in month-year format won't result in too many subdirectories with huge amounts of assets in them which could choke the import process. I also hope to be able to import recursively with Photos, but I've run into issues with that in the past - thinking if subdirectory size is held to a reasonable minimum this won't happen again.
Also, in my non-structured poking around the .photolibrary bundle I do see some interesting information that possibly could be leveraged in the process - mainly, in one of the cache subdirectories there are files which contain Apple's description of the photo contents.
Thanks in advance!
Beta Was this translation helpful? Give feedback.
All reactions