Referenced Photos

[Rott-Apple]

Referenced files were problematic in Photos 3 and 4 for Mac because 1) Moving the original photos didn't update the file paths in the Photos database, 3) Fixing broken links between such moved files and the Photos Library was fiddly, 3) Deleting referenced photos from the Photos library didn’t delete the originals at external location, 4) iCloud Photo Library didn’t import referenced files. It's got worse in Photos 5 and 6, as explained here.

A tool to find all the broken links in Photos, and locate the corresponding original files on the Mac or in connected locations e.g. external HD, thumb drive, Dropbox etc. might be useful. Would need a test library to see what happens on deleting a referenced photo from (a) Photos (b) external location; see what in ZASSET, ZGENERICALBUM, etc. is updated: Presumably, (a) changes ASSET.ZTRASHEDSTATE to 1, while for (b), flag(s) update only when user tries to access the photos.

Wondering if any such tool already exists, and whether you have looked closely at how Photos handles these up to Mojave and for Catalina / Big Sur. Previously, referenced files were identified by RKMASTER.FILEISREFERENCE (Yes = 1, No = 0); not sure what identifies them in newer version (I guess ZASSET.ZCLOUDLOCALSTATE = 1 includes all non-iCloud files i.e. in Photos library as well as referenced?)

[RhetTbull]

osxphotos doesn't currently distinguish between referenced files or imported files but does handle them both transparently. It would be probably be helpful to have an isreference or such property added to osxphotos. This would make it easy to add a "scan for broken links" feature to the command line app.

You're right that referenced files is a mess on Photos 5+. I would recommend no one use this feature as it's completely broken. In osxphotos, I detect referenced files by looking at ZGENERICASSET.ZDIRECTORY -- if this is a fully qualified path, the file is a referenced file. If it's an imported file, ZDIRECTORY is a single letter corresponding to the first letter of the photo's UUID and the photo will be located in Photos Library.photoslibrary/originals/X where X is the letter.

Internally, Photos 5 doesn't use this column though for referenced files. It uses the ZFILESYSTEMBOOKMARK and stores a serialized CFURL blob in the ZBOOKMARKDATA column.

It would be easy to read either ZDIRECTORY or deserialize the CFURL to find where the original file should be. Finding where it moved would be more challenging but likely could be implemented. The problem is fixing this. I tried, unsuccessfully, to write a utility to help someone who stored their Photos library on an external USB drive to move between different Macs. This worked fine on Mojave but on Catalina/BigSur, because Photos uses a CFURL bookmark unique to the machine, it absolutely breaks the library. Photos imported on one Mac can only be found on that Mac and vice versa. Theoretically it would be possible to read the bookmark, find where the photo should be and write a new bookmark to the Photos.sqlite file. I wasn't able to successfully to write a bookmark that Photos would read. However, I didn't spend a great deal of time on it and this is theoretically possible.

[Rott-Apple]

Thanks, a lot to chew on.

I understand "referenced". By "imported" do you mean files imported with Preferences > 'Copy items to the Photos library' checked, so that full resolution are saved in Photos library's Originals folder?

A query using the broken links list could give original file names, titles, sizes etc. A separate program or Applescript could scan for all image files on the Mac and connected/accessible locations e.g. iCloud, external HD, other cloud services etc. Matching these two would tell which of the missing files were found in which location. But user would still have to import them into Photos (as local or referenced files) to recreate fresh Photos database records. And find and delete the thumbnails of the broken links by making a "Referenced" smart album, then clicking them to see which are missing. Messy. Maybe there's a better way. I wouldn't mess with bookmarks or anything else in the Photos.sqlite file.

[RhetTbull]

I've added a --is-reference option to osxphotos that will filter for referenced files in v0.39.20. You can find missing referenced files using:

osxphotos query --is-reference --missing

[RhetTbull]

I understand "referenced". By "imported" do you mean files imported with Preferences > 'Copy items to the Photos library' checked, so that full resolution are saved in Photos library's Originals folder?

Yes

I have a companion project (PhotoScript) used internally by osxphotos that allows full scripting of Photos via AppleScript interface. This could be used to automate the re-import of files with broken links.

[RhetTbull]

In Photos 5, referenced photos appear to be identified by ZGENERICASSET.ZSAVEDASSETTYPE:

3: imported by copying to Photos library
4: shared iCloud photo
6: imported by iCloud (e.g. from iPhone)
10: referenced file (not copied to Photos library)

[Rott-Apple]

Excellent. Thanks!

[bell0q]

I was originally quite interested in this feature to basically "show me the path/filename of a photo or video." The reason was this: I had an older library which had videos encoded in a codec which I no longer had, so they would not play in Photos. But, I found I could copy them out of the library package, re-encoded them with Quicktime (which would Quicktime play the file but not Photos? I have no idea.) and then import them back into Photos.

The problem was, I didn't want to have to go an figure out all the albums and slideshows the original file was part of, so I though if I could sneakily just copy the new file over the original file inside the package, all would be good.

Alas, I used a different method (manually copy/dragging into albums) as it was only 18 files.

[dgleich]

RhetTbull, if you have any of your code that you tried to make the bookmark, could you possibly share? I've been trying to work out how to do this to move my referenced photo library from an old Intel Mac to a new M1 Mac, but also move the photos to a new SSD. In this case, the photos path stays the same, but none of the photos ecosystem works because of the dependency on this darn bookmark data to pass around access information. (This results in absolutely maddening errors... constant Photo app beachballs and all sorts of craziness... ironically, completely deleting the contents of the ZFILESYSTEMBOOKMARK table makes Photos work great in this scenario, but then all sorts of other things don't work, like videos and Live Photos.) [Sidenote, this appears to be all about App Sandboxing, photos needs to tell other things -- like the video player -- about the file data, but because of all the sandboxing, the only way to get them this info securely is through the bookmarks, cool system when it works, and awfully opaque when it fails.]

Since I don't want to give up just yet...

• Likely hopeless ideal scenario... - there's a "straightforward" way to take a file path and generate a bookmarkdata stream that photos will read (which is what I believe you tried above...)

After reading your notes and discovering your photoscript package, it seems like another option would be:

On the new computer...

• setup a simple empty library
• (via photo script) sequentially import each photo / Live Photo / edited photo (need to understand how to handle Live Photos... ) [If needed, you seem to be able to manually resolve the old bookmarks to the right path on the new computer...or have some programatic mapping.]
• grab the bookmark data from that file via the sqlite database...
• insert it into old SQLITE database (or insert it into plj files used in the journal to get photos to rebuild the SQLITE data, which is how I was approaching these tasks)
• then nuke the import-library every X entries.

I fear you might have already tried this, however. As a simple test of (2), I was going to try and move a bookmark data structure between two libraries to see if that might actually work.

[RhetTbull]

@dgleich I found the old bookmark code. In the process of dusting it off, I found and fixed a bug in the bookmark generation which means it might now be working. The following will update the bookmarks in a Photos database (tested only on Catalina). I have done almost no testing so backup your library and create a test library for testing before you use this. This might solve your use case. Let me know what you find! Attached as a zip (and pasted below for future reference) because GitHub doesn't let you attach a python file. You'll need pyobjc and Click installed via pip.

update_photos_bookmarks_py.zip

""" Rewrite bookmarks for photos in a Photos database. WARNING: This will modify your Photos database; make sure you have a backup"""

import os.path
import sqlite3
import sys
import urllib

import click
import CoreFoundation
from Foundation import kCFAllocatorDefault


def resolve_cfdata_bookmark(bookmark: bytes) -> str:
    """Resolve a bookmark stored as a serialized CFData object into a path str"""

    # use CFURLCreateByResolvingBookmarkData to de-serialize bookmark data into a CFURLRef
    url = CoreFoundation.CFURLCreateByResolvingBookmarkData(
        kCFAllocatorDefault, bookmark, 0, None, None, None, None
    )

    # the CFURLRef we got is a sruct that python treats as an array
    # I'd like to pass this to CFURLGetFileSystemRepresentation to get the path but
    # CFURLGetFileSystemRepresentation barfs when it gets an array from python instead of expected struct
    # first element is the path string in form:
    # file:///Users/username/Pictures/Photos%20Library.photoslibrary/
    urlstr = url[0].absoluteString() if url[0] else None

    # now coerce the file URI back into an OS path
    # surely there must be a better way
    if urlstr is not None:
        return os.path.normpath(
            urllib.parse.unquote(urllib.parse.urlparse(urlstr).path)
        )
    else:
        raise ValueError(
            "Could not extract URL String from IPXDefaultLibraryURLBookmark"
        )


def open_sqlite_db(fname: str):
    """Open sqlite database and return connection to the database"""
    try:
        conn = sqlite3.connect(f"{fname}")
        c = conn.cursor()
    except sqlite3.Error as e:
        raise OSError(f"Error opening {fname}: {e}")
    return (conn, c)


def create_cfdata_bookmark(pathstr: str):
    """Create a bookmark (serialized CFData object) from a path string"""

    cfurl = CoreFoundation.CFURLCreateFromFileSystemRepresentation(
        kCFAllocatorDefault,
        pathstr.encode("utf-8"),
        # len(bytes(urlstr, encoding="utf8")),
        # len(urlstr.encode("utf-8")),
        len(pathstr),
        False,
    )

    bookmark, error = CoreFoundation.CFURLCreateBookmarkData(
        kCFAllocatorDefault, cfurl, 0, None, None, None
    )
    if error:
        raise ValueError(f"Error creating bookmark: {error}")

    return bookmark


@click.command()
@click.argument("photos_db_path", type=click.Path(exists=True))
def main(photos_db_path):
    """Update photo bookmarks in a Photos sqlite database"""

    click.confirm(
        "WARNING: this program will rewrite your Photos library database and could result in total data loss.\n"
        "Please ensure you have a backup!\n"
        "Do you want to procced?",
        abort=True,
    )

    (conn, c) = open_sqlite_db(photos_db_path)
    c.execute(
        "SELECT Z_PK, ZPATHRELATIVETOVOLUME, ZBOOKMARKDATA FROM ZFILESYSTEMBOOKMARK"
    )

    # read all the bookmarks and compute new bookmarks
    data_to_update = []
    for row in c:
        pk = row[0]
        pathstr = row[1]
        bookmark_data = row[2]
        if bookmark_data:
            bookmark_path = resolve_cfdata_bookmark(bookmark_data)
            bookmark_new = create_cfdata_bookmark(bookmark_path)
            data_to_update.append((pk, pathstr, bookmark_new))

    # update each bookmark in the database
    for pk, pathstr, bookmark_new in data_to_update:
        click.echo(f"Updating bookmark for {pathstr}")
        c.execute(
            "UPDATE ZFILESYSTEMBOOKMARK SET ZBOOKMARKDATA = ? WHERE Z_PK = ?",
            (bookmark_new, pk),
        )


if __name__ == "__main__":
    main()

[dgleich]

Thanks! I am working on trying this... I needed to insert a conn.commit() line at the end of the loop to get sqlite to persist the changed values. It's... interesting what happens afterwards. Still debugging. It seems like there are some cached values, so lots of restarts of photolibraryd / login/logout tests. I want to make sure I'm not seeing impacts on log files from other libraries I've been testing, so I'm going to try and remove those other libraries.

[RhetTbull]

Doh! Yes, the commit would be helpful! Also should close the connection with conn.close() when done. If you find a way to do this, it could be added as an option to osxphotos. I've got a couple other things I've been toying with that could be added to a an osxphotos repair command, for example osxphotos repair --fix-bookmarks.

[RhetTbull]

I've been playing with this and suspect it's the security scope of the bookmark that my code is creating. Reference CFURLCreateBookmarkData. I did confirm that the bookmark created by my script is not identical to the bookmark created by Photos even if the volume hasn't changed so there's clearly something different but I can't tell yet what it is. Again, I suspect it's the security scope or some other option passed to the call to CFURLCreateBookmarkData.

Also, the volume for a photo is stored in ZINTERNALRESOURCE.ZFILESYSTEMVOLUME which points to ZFILESYSTEMVOLUME.Z_PK which stores the UUID of the file system volume in ZFILESYSTEMVOLUME.ZVOLUMEUUIDSTRING (which can be found with diskutil info /dev/disk2s1 | grep "UUID" where /dev/disk2s1 is the volume mount point for where the photos are stored. I tried updating the volume ID when switching disks but no luck -- that's not the issue. (But likely for a robust solution, we'll need to update or add a new entry in the ZFILESYSTEMVOLUME table for the new mount point and then update ZINTERNALRESOURCE table as well. The volume ID is part of the bookmark as well.

[dgleich]

One thing that helped is giving photos and photolibraryd full disk access. More later :)

…

On Sun, Nov 21, 2021 at 12:27 Rhet Turnbull ***@***.***> wrote: I've been playing with this and suspect it's the security scope of the bookmark that my code is creating. Reference CFURLCreateBookmarkData <https://developer.apple.com/documentation/corefoundation/1542923-cfurlcreatebookmarkdata?language=objc>. I did confirm that the bookmark created by my script is not identical to the bookmark created by Photos even if the volume hasn't changed so there's clearly something different but I can't tell yet what it is. Again, I suspect it's the security scope or some other option passed to the call to CFURLCreateBookmarkData. Also, the volume for a photo is stored in ZINTERNALRESOURCE.ZFILESYSTEMVOLUME which points to ZFILESYSTEMVOLUME.Z_PK which stores the UUID of the file system volume in ZFILESYSTEMVOLUME.ZVOLUMEUUIDSTRING (which can be found with diskutil info /dev/disk2s1 | grep "UUID" where /dev/disk2s1 is the volume mount point for where the photos are stored. I tried updating the volume ID when switching disks but no luck -- that's not the issue. (But likely for a robust solution, we'll need to update or add a new entry in the ZFILESYSTEMVOLUME table for the new mount point and then update ZINTERNALRESOURCE table as well. The volume ID is part of the bookmark as well. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#319 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAIJS3ZB2MY4LIYP636ZM3UNETVHANCNFSM4VRCPSDQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[RhetTbull]

Update: DOH! I realize this is just exactly what @dgleich already suggested above. While it's a kludge, I do think this is the best way to solve the problem and I think it's relatively easy to do with PhotoScript.

I have an idea that may work (correction: @dgleich has an idea that may work)...still need to test it. It would be a bit of a kludge but I think the process could be automated. The basic problem seems to be that, due to App Sandboxing, Photos uses security-scoped CFURL bookmarks for accessing files that were not imported into Photos (and thus managed internally by Photos). Because these bookmarks are security scoped, the bookmarks are only valid if created by Photos itself. So the solution may be to get Photos to create the bookmarks itself. Here's one possible way that may work:

1. Create a temporary, empty Photos library. (This could be saved as a template and automatically copied when needed)
2. For the target Photos Library, read the stored bookmark for each photo and extract the location of the referenced file.
3. Using PhotoScript (my python wrapper around Photos' AppleScript interface), import the referenced file into the temporary Photos library.
4. Read the newly created bookmark from the temporary library and write that bookmark back to the target library.
5. Delete the temporary Photos library

[dgleich]

I agree that idea of important Photos own bookmarks is worth trying -- was later in my planned investigation as it's really annoying.

I was digging into what happens after you "update" the reference to a photo. I think Photos just asks Photolibraryd to edit the asset. So there might be a hidden API for this. I will see if I can get anywhere with disassembling photolibraryd.

[RhetTbull]

One note about this process: It will likely only work if the new POSIX path is exactly the same as the old POSX path (e.g. if stored on an external volume, the new volume must be the same name as the previous volume). This is because CFURLCreateByResolvingBookmarkData will fail to resolve if the target file is not present. In my testing, I've found that, even though the volume UUID is part of the bookmark, the bookmark will still resolve to a URL as long as the POSIX path to the file is the same. This means that this technique cannot be used to move referenced photos from one path to another. However, it may be possible to use Mike Lynn's work to extract the URL directly from the serialized bookmark data. This would add an extra layer of hacky kludginess but would solve a very real use case.

[RhetTbull]

I was digging into what happens after you "update" the reference to a photo.

I tried this many times over the weekend to look at the diff in the bookmark and noticed that at least 50% of the time, Photos outright crashes when updating the reference (e.g. selecting folder after the "fix" dialog opens). But it does appear to update the reference and the file "works" after re-opening Photos.

I will see if I can get anywhere with disassembling photolibraryd

Good hunting!

[RhetTbull]

Also, this might be useful: headers for PhotoLibraryServices

[dgleich]

Here's one that might be helpful... https://github.com/w0lfschild/macOS_headers/blob/master/macOS/PrivateFrameworks/PhotoLibraryServices/101.9.122/PLFileSystemBookmark.h

…

On Mon, Nov 22, 2021 at 1:59 PM Rhet Turnbull ***@***.***> wrote: Also, this might be useful: headers for PhotoLibraryServices <https://github.com/w0lfschild/macOS_headers/tree/master/macOS/PrivateFrameworks/PhotoLibraryServices> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#319 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAIJS2BHVITVNA2UE4NGU3UNKHJZANCNFSM4VRCPSDQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[dgleich]

(I will try later, but I have to run now...)

[RhetTbull]

Good sleuthing! I don't see why I wouldn't be able to call it -- will try tonight.

[dgleich]

Well... Apple seems like it really doesn't want us calling stuff there :) Here's the error I got when trying to alloc the object.

"""Attempting to create a bookmark that Photos can use for accessing referenced files"""
import CoreFoundation
import objc
from CoreData import NSManagedObjectContext

# from CoreFoundation import CFUUIDCreateFromString
from Foundation import NSURL

framework = objc.pathForFramework(
    "/System/Library/PrivateFrameworks/AssetsLibraryServices.framework/"
)
objc.loadBundle("AssetsLibraryServices", globals(), framework)

x = PLLibraryBookmarkManager.alloc()
w = x.init()
# 2021-11-23 21:27:05.772 Python[85846:1383092] *** Assertion failure in -[PLLibraryBookmarkManager _loadFromDefaults], PLLibraryBookmarkManager.m:352
#Traceback (most recent call last):
  #File "<stdin>", line 1, in <module>
#objc.error: NSInternalInconsistencyException - Invalid parameter not satisfying: PLIsReallyAssetsd()

PLIsReallyAssetsd() is a routine to check if the caller really is the assetsd process it seem. So I'll have to keep looking for a calling sequence here.

[RhetTbull]

I got the same result tonight. Bummer.

PLIsReallyAssetsd() is a routine to check if the caller really is the assetsd process it seem

I wonder if it's possible to intercept / override this call to fake the result?

I tried to get clever and inserted the code to create a bookmark inside the completion handler for a PhotoKit call to requestImageDataAndOrientationForAsset:options:resultHandler: thinking that maybe that would cause the bookmark to be created inside the Photos process. I was able to create a bookmark this way but it still fails when inserted back into the database.

[RhetTbull]

Here's what the log shows when trying to access a photo with a rewritten bookmark:

default	21:00:41.772579-0800	Photos	volumeURL for uuid <private> is <private>
default	21:00:41.772624-0800	Photos	Setting PLFileSystemVolume 0x6000019d7a70 URL to <private>
error	21:00:41.773732-0800	kernel	Sandbox: Photos(34637) deny(1) file-read-data /Users/rhet/Downloads/Old Town Philadelphia Map.jpg
default	21:00:41.773257-0800	Photos	Unregistering volume with uuid <private>
error	21:00:41.773766-0800	Photos	Reading image metadata failed for resource at URL: <private>
default	21:00:41.779011-0800	Photos	error loading preview image for <PHAsset: 0x7fb295f550f0> 595767BD-DB20-44A5-9ADE-B77B3E0EF582/L0/001 mediaType=1/0, sourceType=1, (1588x2382), creationDate=2021-11-13 22:19:59 +0000, location=0, hidden=0, favorite=0, adjusted=0 : (null)

It looks like Photos correctly gets the path to the file (in this case, /Users/rhet/Downloads/Old Town Philadelphia Map.jpg) but the sandbox denies access.

[Rott-Apple]

Would the following approach not work? 1) Copy the source folders containing all the photos to the new SSD. On the M1 Mac, create a new empty library, in Preferences uncheck 'Copy items to the Photos library', then drag and drop the folder onto the Photos icon in the dock (This preserves organization). 2) If the Photos library on the old Mac has referenced as well as managed or iCloud-based photos, temporarily delete all the referenced items (Create a smart album containing all the referenced items, and delete); copy the library across to the new Mac, then copy the referenced items folders to SSD and add them to the library on the M1 Mac as in (1) above. On Tue, Nov 23, 2021 at 2:18 AM David Gleich ***@***.***> wrote:

…

Here's one that might be helpful... https://github.com/w0lfschild/macOS_headers/blob/master/macOS/PrivateFrameworks/PhotoLibraryServices/101.9.122/PLFileSystemBookmark.h On Mon, Nov 22, 2021 at 1:59 PM Rhet Turnbull ***@***.***> wrote: > Also, this might be useful: headers for PhotoLibraryServices > < https://github.com/w0lfschild/macOS_headers/tree/master/macOS/PrivateFrameworks/PhotoLibraryServices > > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > < #319 (reply in thread) >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AAAIJS2BHVITVNA2UE4NGU3UNKHJZANCNFSM4VRCPSDQ > > . > Triage notifications on the go with GitHub Mobile for iOS > < https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 > > or Android > < https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub >. > > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#319 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQF3F4VYH36GKEIBYHVBUUDUNKJQDANCNFSM4VRCPSDQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[RhetTbull]

That would import the photos but all associated metadata and other organization (e.g. albums) would be lost (when the files were deleted from the original library). If we can figure out how to update the bookmarks in the database, the library can be moved without losing any of the associated metadata.

[dgleich]

I fear I may have misunderstood this call. This call may be the one that looks up the bookmark given a photos path. (i.e. this is photos own internal interface to the ZFILESYSTEMBOOKMARK table…) but i’m honestly not sure. (e.g. the docs all point to managed object graph data… which definitely seems like this is somem type of coredata table…)

…

On Tue, Nov 23, 2021 at 9:49 AM Rhet Turnbull ***@***.***> wrote: I managed to get the PLFileSystemBookmark class loaded but can't figure out how to use it. I think the second argument to fileSystemBookmarkFromURL:context: is a CoreData NSManagedObjectContext but don't know enough about CoreData (I know nothing, actually) to do anything with that. Can we create a CoreData context and pass it in and it'll just work or does this blow up if it doesn't have Photos' actual context? """Attempting to create a bookmark that Photos can use for accessing referenced files"""import CoreFoundationimport objcfrom CoreData import NSManagedObjectContext # from CoreFoundation import CFUUIDCreateFromStringfrom Foundation import NSURL framework = objc.pathForFramework( "/System/Library/PrivateFrameworks/PhotoLibraryServices.framework/" )objc.loadBundle("PhotoLibraryServices", globals(), framework) # PLFileSystemBookmark.fileSystemBookmarkFromURL_context_# Reference: https://github.com/w0lfschild/macOS_headers/blob/master/macOS/PrivateFrameworks/PhotoLibraryServices/101.9.122/PLFileSystemBookmark.h# second argument is a context, which I think is an NSManagedObjectContext# ...but I don't know anything about CoreData and hence what to do with it. url = NSURL.fileURLWithPath_("/Users/rhet/Desktop/mosaic.jpg")context = NSManagedObjectContext.alloc().init()bkmk = PLFileSystemBookmark.fileSystemBookmarkFromURL_context_(url, context) # fails with:# Traceback (most recent call last):# File "/Users/rhet/Dropbox/Code/tempworking/pls.py", line 27, in <module># bkmk = PLFileSystemBookmark.fileSystemBookmarkFromURL_context_(# ValueError: NSInvalidArgumentException - +entityForName: nil is not a legal NSPersistentStoreCoordinator for searching for entity name 'FileSystemBookmark' — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#319 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAIJS3RSV2VSXKBJ4ZECQLUNOSWTANCNFSM4VRCPSDQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[RhetTbull]

I've done some additional research looking at the original proposal by @dgleich and this appears to be feasible. I've confirmed the following:

1. The bookmark Photos creates for a file in a given location is the same even when the bookmark is created in two different Photos libraries (e.g. the same photo imported to two different files)
2. Thus, copying the bookmark from one library to another works and Photos is able to open the referenced file.

[RhetTbull]

I have a very rough but working implementation as originally proposed by @dgleich in a new repo for easier collaboration and testing. I have tested this on a small library on Catalina and it works but does require several manual steps by the user (prompted by the script). This is a rough proof-of-concept at the moment with no error handling, etc.

[dgleich]

Will try tonight!

…

On Wed, Nov 24, 2021 at 12:36 Rhet Turnbull ***@***.***> wrote: I have a *very rough* but working implementation as originally proposed by @dgleich <https://github.com/dgleich> in a new repo <https://github.com/RhetTbull/RepairPhotosBookmarks> for easier collaboration and testing. I have tested this on a small library on Catalina and it works but does require several manual steps by the user (prompted by the script). This is a rough proof-of-concept at the moment with no error handling, etc. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#319 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAIJS6AZVTVG6MV6O66RXLUNUPDRANCNFSM4VRCPSDQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[dgleich]

Okay, this works great! There was one typo that I fixed that I needed to get things working. ... One of the dictionary keys should have been pathstr (no volume to match ZFILESYSTEMBOOKMARK) instead of bookmark_path (has the volume path...) There were a few things I threw into the script, but it worked for a small test library I made to debug things. Handled a Live Photo, Portrait photo, movie, and mixed volumes... everything worked after running the script.

I had to make a few edits since I'm on big sur, but... it's all downhill from here. I want to try doing this on my 60k photo library to see what happens there. (Tomorrow? Maybe later because of the holiday?)

But... so far so good... ! It's now 'amazing' to see things work when moving between computers.

[RhetTbull]

Thanks for the edits! I made a few more to add sanity checks (verify right library was loaded, Photos running/not running at the right time). I tried to add cleanup to delete the temp library but the shutil.rmtree failed, possibly because photoslibraryd still operating on the folder, so I added a prompt to tell the user to drag the temp library to the trash.

I think this is ready for "beta testing". I keep my files in iCloud so don't have a large library to test this on. Let me know if you run into any issues on your library. One problem that could pop up is AppleScript errors. The scripting interface in Photos is buggy and has been for years. I've noticed that sometimes Photos crashes when being scripted and then sometimes the only solution is a reboot as the AppleScript bridge seems to go into an anomalous state.

One question: You added a --debug-skip-import flag that appears to be unused. I left it in on the chance you were planning to implement something there.

[dgleich]

Oops, sorry, I got a bit busy and behind on this. So I'm trying it on the big 60k library right now. One issue that you may have run into in the past is that after about 1000 imported, Photos will continuously throw errors when getting imported files from the Photoscript library. So you have to restart. I put in some simple crude restart logic. (That was related to the debug-skip-import idea, but I think I've got it setup now where it won't import a file in a restart directory.). I need to check if this is photos having the issue. (In which case things are hopeless and will just require endless tedious restarts...) Or if it's the AppleScript interface (which I might be able to address with multiprocessing and forking out new processes to handle thing...).

It's also curious that there are just some files that Photos seems to refuse to import! I'm trying to track down if this is because they are a part of one of these magic groups (i.e. the first one in the library is a Live Photo with an AAE file for edits.) Will try and report back more soon, but wanted to let you know I'm looking at it again!

(Also, I'll probably use a version of this to rewrite the PLJ files that Photos uses as it's real backup of things... that's one more step once I figure out how to get the new bookmark data... and everything imported!)

[RhetTbull]

I've found the Photos AppleScript interface to be very flaky and crash-prone. One option is to just monitor to see if Photos is running and if not, restart.

[dgleich]

Alright, more info. I have a script I can pretty much leave unattended that will handle about 100k items (60k photos, 5k videos, lots of Live Photos, etc.) One key tool was a small step to move AAE files away so Photos doesn't try and do something that causes the import to fail. (Basically, the script will check for AAE files, if it finds them, it moves them to AAE.bak, does the import, then reverse the rename... I don't like this though as a crash at the wrong point would cause a change to the underlying directories... albeit minor -- as this tool seems destined for expert-users only, this is probably okay. )

Once everything was imported, Photos was very happy with the new library. Memories, etc, worked great. All the facial analysis worked... etc.

One big surprise to me: photos will rewrite the relevant entries in the photo library journal / property list journal files based on the new SQLite entries in Photos.db. (For those who use TimeMachine, the Photos.db file isn't actually backed up, only the files in resoucses/journals... which will allow one to recreate this... more on these files in https://github.com/dgleich/apple-plj-format.) I had thought I would have to do this myself, so this was a nice surprise!

I also tested upgrading from BigSur to Monterey. That... ran into problems with my external disk that I'm still debugging. (I believe everything will work fine once I get the disk issues sorted out, basically Monterey won't mount my external SSD!)

[RhetTbull]

One key tool was a

What a cliff hanger! What was the tool? 😀

[dgleich]

Oh, the key tool/routine/piece was just the code to move the AAE files out of the way for the import (not very sophisticated, but seems to be critical). I'll try and post it to the RepairPhotosBookmarks as soon as I get a chance to clean it up / review again to make sure it isn't totally confusing! (Coming soon, I promise!)

[RhetTbull]

@dgleich do you have code you can share on the fixed you implemented to get RepairPhotosBookmarks working? I'd like to get those incorporated into the tool. Cheers!

[dgleich]

Weird! I thought I had replied! I pushed a light revision of the code I used. I tried to be careful not to introduce any new bugs when I did this, (which is what was holding me up.. I wanted to test it... but there was ANOTHER disaster with the photos library... the SSD it was on failed, so blah... excuses... anyway, it's all working now.) but all the things needed should be there. If there are mistakes, they are in very simple things like the options or very easy things. (I apologize, I gave myself a week to fix them, and about 3 other things came up, )

I put in into a separate file name in the github repo https://github.com/RhetTbull/RepairPhotosBookmarks (repair_photos_bookmarks_dgleich.py)because there are a few smelly code bits in there I'm not overly happy with. But working, is better than not.

A few other notes...

In a few cases, Photos would never import certain files. These were usually mixtures of AAE/MOV/JPG (or HEIC). Setting it to import groups helped in quite a few cases, but still left a few. That said, I was able to import these by importing the entire directory. (It'll detect duplicates, etc..) So there might be some scenario where that would be advantageous. Again, this was an edge case with maybe 50 out of 60,000 files.

[RhetTbull]

Thanks! I plan to eventually roll the RepairPhotosBookmarks functionality into osxphotos. I did this with photos_time_warp in osxphotos v0.48.0.

[dgleich]

Yes — give me about a week or so and I’ll try and get it back to you. It’s mostly cleaned up … just wanted to double check a few things before sending it in, and then… well, everything else intruded :) If it’s more than a week and I haven’t done anything, I’ll just send everything I have right then. David

…

On Apr 25, 2022 at 3:21:45 PM, Rhet Turnbull ***@***.***> wrote: @dgleich <https://github.com/dgleich> do you have code you can share on the fixed you implemented to get RepairPhotosBookmarks working? I'd like to get those incorporated into the tool. Cheers! — Reply to this email directly, view it on GitHub <#319 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAIJS3KDJGWET4UFHX6CZ3VG3WETANCNFSM4VRCPSDQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[balabanovic]

Hi. I am struggling with a similar issue I think. Some time ago I moved a library to a different disk as part of dealing with a broken machine, and since then many photos are disconnected. When I look at RKMASTER I can see a few hundred where volumeID is NULL, whereas others have a volumeID pointing to what looks like the correct disk. The imagePath looks OK. Is it safe to update volumeID (I was thinking to do this direct in the database, not sure if volume info is exposed via osxphotos?), or would this not really fix the problem anyway? Reading this thread it appears that it could be significantly more tangled than that! Related question - which uuid is the main uuid that osxphotos exposes - is it RKVersion.uuid? I am on v4.0 of Photos and Mojave on an older Mini.

[RhetTbull]

Hi @balabanovic this thread relates to referenced photos in Photos 5 (Catalina) and above which completely changed the Photos database structure. Part of this change was moving to security scoped bookmarks which prevent you from moving a library with referenced images. On Mojave and earlier, I do not believe the bookmarks are security scoped but it's been a long time since I've looked at a Photos database that old.

I think you should be able to edit the Photos database to ensure that the volume is correctly in RKVolume. If it's not there, you'll need to add it and you'll need the volume UUID for the diskUuid column. You can get that by selecting the volume in Disk Utility, click the "info" icon and find "File system UUID".

Then for the photos that have the broken link, updated RKMaster.VolumeId column to the value of the modelId value for the volume you added in RKVolume.

osxphotos does not provide a way to do this but if you have any SQL or scripting skills, it shouldn't be too hard (I bet ChatGPT could write the code!)

Make a backup of your photos database (~/Pictures/Photos Library.photoslibrary/database/*) before making any changes!

[RhetTbull]

Related question - which uuid is the main uuid that osxphotos exposes - is it RKVersion.uuid?

Yes, it's:

RKVersion.uuid
WHERE RKVersion.masterUuid = RKMaster.uuid

As this is what Photos uses internally.

[balabanovic]

Thanks! Don't worry I am working off a copy :). Do you know if I will also need to clear the isMissing column in RKMASTER, otherwise will Photos still think the referenced files are not where it thinks they are?

[RhetTbull]

Yes, good point, you probably should clear the isMissing column