Replies: 12 comments 3 replies
-
|
Release v.46.0 was a major rewrite of the export code to make it more reliable and more sustainable. Unfortunately this required the export database used by --update to rewritten which will cause any export using --exiftool to have to re-export those photos. This should be a one time issue and you should get fast updates after then initial migration. Have you tried another update since the first one completed? If it's still running slow there may be a bug but other users have reported fast updates after the initial re-export. It will likely be slightly slower (a few percent) due to changes in the export logic but shouldn't be orders of magnitude slower. |
Beta Was this translation helpful? Give feedback.
-
|
It still seems to have a problem on subsequent runs. However, this can be isolated to using —exiftool. Without this the program runs very fast and as expected. I upgraded to the latest version of exiftool but this didn’t help. Using the —exiftool flag the python memory usage maxes out at around 800MB, but without the flag rises above 1GB. If I change from exiftool to sidecar it also runs perfectly and very fast (I re-exported the complete photo library in around 8mins).
… On Feb 28, 2022, at 10:11, Rhet Turnbull ***@***.***> wrote:
Release v.46.0 was a major rewrite of the export code to make it more reliable and more sustainable. Unfortunately this required the export database used by --update to rewritten which will cause any export using --exiftool to have to re-export those photos. This should be a one time issue and you should get fast updates after then initial migration. Have you tried another update since the first one completed? If it's still running slow there may be a bug but other users have reported fast updates after the initial re-export. It will likely be slightly slower (a few percent) due to changes in the export logic but shouldn't be orders of magnitude slower.
—
Reply to this email directly, view it on GitHub <#649 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKUQTOOBUJHRDKQCIDLPTTU5O3DXANCNFSM5PRVRCRA>.
Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you authored the thread.
|
Beta Was this translation helpful? Give feedback.
-
|
Sounds like perhaps exiftool is reprocessing all the photos (which shouldn't happen with --update). Could you try an export with the following and send me the output (post here or email to rturnbull+git@gmail.com if you'd rather not publicly post (it'll contain file names)
you might try with just a subset of your photos (e.g. use --from-date or just try one album with --album) |
Beta Was this translation helpful? Give feedback.
-
|
Will run it later today and send you the output...
… On Feb 28, 2022, at 14:24, Rhet Turnbull ***@***.***> wrote:
Sounds like perhaps exiftool is reprocessing all the photos (which shouldn't happen with --update). Could you try an export with the following and send me the output (post here or email to ***@***.*** ***@***.***> if you'd rather not publicly post (it'll contain file names)
osxphotos export <all your normal options> --profile --verbose > debug.txt
you might try with just a subset of your photos (e.g. use --from-date or just try one album with --album)
—
Reply to this email directly, view it on GitHub <#649 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKUQTOIOK7TXJMMLZ6PYSDU5PYZVANCNFSM5PRVRCRA>.
Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you authored the thread.
|
Beta Was this translation helpful? Give feedback.
-
|
Great. If you could also run this command and include the output, it would be helpful:
|
Beta Was this translation helpful? Give feedback.
-
|
Will do…
…Sent from my iPhone
On Feb 28, 2022, at 15:59, Rhet Turnbull ***@***.***> wrote:
Great. If you could also run this command and include the output, it would be helpful:
osxphotos exportdb /path/to/your/export/directory --save-config debug.toml
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.
|
Beta Was this translation helpful? Give feedback.
-
|
The problem is likely here: osxphotos/osxphotos/photoexporter.py Lines 734 to 747 in f132e9a Need to stage the files for exiftool only if exiftool needs to be run |
Beta Was this translation helpful? Give feedback.
-
|
And, to your earlier question both the Photos library as well as the export directory are on the same APFS (encrypted) volume, but this is a different physical drive to the separate macOS Monterey System and Data APFS volumes.
It looks like the python temp files are being created in my user tmp directory (for me /var/folders/hr/_yl5ddk149dd_vrxjvhrk02c0000gn/T/) which is on the Monterey Data volume, a different volume to the Photos library / export directory, probably contributing to the problem.
Maybe you could create an osxphotos specific tmp directory in the export directory and pass this to tempfile.TemporaryFile, Then the APFS copy-on-write should work ok and the changes needed would be fairly minimal. Also, probably good to add some comments in the documentation on the use of APFS (although not directly the problem here)…
Chris
… On Feb 28, 2022, at 23:52, Rhet Turnbull ***@***.***> wrote:
The problem is likely here:
https://github.com/RhetTbull/osxphotos/blob/f132e9a8438023e4c69c7fb767d24dea7465db4d/osxphotos/photoexporter.py#L734-L747 <https://github.com/RhetTbull/osxphotos/blob/f132e9a8438023e4c69c7fb767d24dea7465db4d/osxphotos/photoexporter.py#L734-L747>
Need to stage the files for exiftool only if exiftool needs to be run
—
Reply to this email directly, view it on GitHub <#649 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKUQTOPN7THQ77PJVE4EW3U5R3LHANCNFSM5PRVRCRA>.
Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you authored the thread.
|
Beta Was this translation helpful? Give feedback.
-
|
Well, better would be a command line option to allow an informed user to set a custom tmp file directory location if needed...
… On Mar 1, 2022, at 06:14, Chris Forshaw ***@***.***> wrote:
And, to your earlier question both the Photos library as well as the export directory are on the same APFS (encrypted) volume, but this is a different physical drive to the separate macOS Monterey System and Data APFS volumes.
It looks like the python temp files are being created in my user tmp directory (for me /var/folders/hr/_yl5ddk149dd_vrxjvhrk02c0000gn/T/) which is on the Monterey Data volume, a different volume to the Photos library / export directory, probably contributing to the problem.
Maybe you could create an osxphotos specific tmp directory in the export directory and pass this to tempfile.TemporaryFile, Then the APFS copy-on-write should work ok and the changes needed would be fairly minimal. Also, probably good to add some comments in the documentation on the use of APFS (although not directly the problem here)…
Chris
> On Feb 28, 2022, at 23:52, Rhet Turnbull ***@***.*** ***@***.***>> wrote:
>
>
> The problem is likely here:
>
> https://github.com/RhetTbull/osxphotos/blob/f132e9a8438023e4c69c7fb767d24dea7465db4d/osxphotos/photoexporter.py#L734-L747 <https://github.com/RhetTbull/osxphotos/blob/f132e9a8438023e4c69c7fb767d24dea7465db4d/osxphotos/photoexporter.py#L734-L747>
> Need to stage the files for exiftool only if exiftool needs to be run
>
> —
> Reply to this email directly, view it on GitHub <#649 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKUQTOPN7THQ77PJVE4EW3U5R3LHANCNFSM5PRVRCRA>.
> Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
> You are receiving this because you authored the thread.
>
|
Beta Was this translation helpful? Give feedback.
-
Yep, that's the issue. Copying across drives is slow. I've updated the code to copy the file only when exiftool needs to be run. I need to finish testing and add the custom temp directory as you suggested (good idea!) In osxphotos version 0.45 and earlier, the image was copied to the export directory, then exiftool was run. For those exporting to a network attached drive (a fair number of osxphotos users), this use case was very slow because the bytes in the file had to be copied the export drive, copied back to the local mac so exiftool could operate on them then copied back to the network drive. The recent change fixes this use case but slows down another (likely common) use case like yours. If you want to downgrade until this gets fixed, you can do so using:
|
Beta Was this translation helpful? Give feedback.
-
|
Perfect, many thanks. I’m a great fan of osxphotos and your support is fantastic!
All the best,
Chris
… On Mar 1, 2022, at 06:36, Rhet Turnbull ***@***.***> wrote:
And, to your earlier question both the Photos library as well as the export directory are on the same APFS (encrypted) volume, but this is a different physical drive to the separate macOS Monterey System and Data APFS volumes.
Yep, that's the issue. Copying across drives is slow.
I've updated the code to copy the file only when exiftool needs to be run. I need to finish testing and add the custom temp directory as you suggested (good idea!)
In osxphotos version 0.45 and earlier, the image was copied to the export directory, then exiftool was run. For those exporting to a network attached drive (a fair number of osxphotos users), this use case was very slow because the bytes in the file had to be copied the export drive, copied back to the local mac so exiftool could operate on them then copied back to the network drive. The recent change fixes this use case but slows down another (likely common) use case like yours.
If you want to downgrade until this gets fixed, you can do so using:
pipx install osxphotos==0.45.12 --force
—
Reply to this email directly, view it on GitHub <#649 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKUQTI46JLURXH75VUIVZDU5YTOLANCNFSM5PRVRCRA>.
Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you authored the thread.
|
Beta Was this translation helpful? Give feedback.
-
|
Give v0.47.4 a try. It fixes the unnecessary copying of files when using exiftool (copies only when needed) and adds a |
Beta Was this translation helpful? Give feedback.
-
|
Now took 34 mins to update the EXIF data in all 35,000 images/movies (previously a couple of hours):
Processed: 29030 photos, exported: 0, updated: 35035, skipped: 0, updated EXIF data: 34822, missing: 0, error: 1
Elapsed time: 0:34:24
Subsequent runs take only a couple of mins:
Processed: 29030 photos, exported: 0, updated: 15, skipped: 35020, updated EXIF data: 15, missing: 0, error: 0
Elapsed time: 0:01:54
Excellent performance (I set the tmp directory to the volume with the Photos library)!
… On Mar 2, 2022, at 07:13, Rhet Turnbull ***@***.***> wrote:
Give v0.47.4 <https://pypi.org/project/osxphotos/0.47.4/> a try. It fixes the unnecessary copying of files when using exiftool (copies only when needed) and adds a --tmpdir parameter to export. I don't have a similar configuration for testing but you might try it with --tmpdir /path/to/a/folder/on-same-drive-as-your-Photos-library and see if it improves performance.
—
Reply to this email directly, view it on GitHub <#649 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKUQTOCIZENXFNFBZDSADTU56ARRANCNFSM5PRVRCRA>.
Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you authored the thread.
|
Beta Was this translation helpful? Give feedback.
-
|
You might want to try |
Beta Was this translation helpful? Give feedback.
-
|
I will; I did try this earlier before I raised the issue, but the significant time the program was taking made it hard to see any benefit from this option.
Hopefully now it‘ll show an additional perfomance improvement and I‘ll try this tomorrow morning.
All the best,
Chris
…Sent from my iPhone
On Mar 2, 2022, at 20:08, Rhet Turnbull ***@***.***> wrote:
You might want to try --ramdb which will likely speed it up even more
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.
|
Beta Was this translation helpful? Give feedback.
-
I've been running osxphotos for a while exporting a database with 29,000 odd photos using --exiftool --update --cleanup on MacOS Monterey. In the past each run (daily) took say 5 mins only exporting new photos. I updated osxphotos yesterday (I update weekly) and it's now taking over an hour to complete and updating EXIF data on hundreds of photos where there's been no changes (e.g. photos from 2012 and earlier). It seems to start quick, then slows down after processing a thousand plus photos with regular periods where it's almost hanging. I'm not sure if this has been seen by others...
Beta Was this translation helpful? Give feedback.
All reactions