export: select photos with regex on album / folder ?

[garz75]

Hi Rhet and other osxphotos gurus,

Has anyone ever requested such a feature ? I organise my library with folders and albums with dates and places or events in their names, so I would like to be able to export photos with a command like this

• --folder-matching '^200.-.*'
• --album-matching '.My Place.'

I am new at python so my code should be scrutinised ;-), but I would be happy to propose something if you are interested to include it in the project....

[RhetTbull]

@garz75 The --album and --folder options currently match against the entire album or folder name, respectively. Do you need full regex matching or would it be sufficient to use python's str in? For example:

>>> album = "Los Angeles 2020"
>>> "2020" in album
True

This would be easier to implement.

If you wanted to take a stab at implementing this, here are the basic steps. All modifications would be to osxphotos/cli.py.

1. Add the option to query_options which gets used by both osxphotos query and osxphotos export. While the option name has a "-" in it, these get converted to underscore "_" for the associated variable name.

osxphotos/osxphotos/cli.py

Lines 205 to 214 in 3c36b0f

	def query_options(f):
	o = click.option
	options = [
	o(
	"--keyword",
	metavar="KEYWORD",
	default=None,
	multiple=True,
	help="Search for photos with keyword KEYWORD. "
	'If more than one keyword, treated as "OR", e.g. find photos matching any keyword',

2. Add the new argument, e.g. folder_matching to both query() and export() function defs. In export, you'll also need to add it to the config code that saves and restores config from file for --save-config and --load-config. Reference this bit of code:

osxphotos/osxphotos/cli.py

Lines 1030 to 1035 in 3c36b0f

	person = cfg.person
	album = cfg.album
	folder = cfg.folder
	name = cfg.name
	uuid = cfg.uuid
	uuid_from_file = cfg.uuid_from_file

So you'd add folder_matching = cfg.folder_matching etc

3. Modify _query() which does the actual matching for all query options:

osxphotos/osxphotos/cli.py

Lines 1980 to 1985 in 3c36b0f

	def _query(
	photosdb,
	keyword=None,
	person=None,
	album=None,
	folder=None,

You'd add a new block in this method that goes something like this:

if folder_matching:
    # code that matches folder names here....see the section that starts "if folder:"
    photos = new_list_of_photos

In _query, all blocks are progressive --- e.g. they build on the previous block to build up a list of photos to query/export. It's a long but simple function so should be easy to follow.

4. Find all the places _query is called in the cli.py and update the call to include the new options.

5. Write a test! See test_cli.py. Copy one of the test libraries to your Pictures folder to examine it if needed then write a test that tests your new option. Do NOT open the test library from inside the source code folder as if you do that, Photos will constantly be trying to update it (e.g. to do background processing to find faces, etc) and there's no way to turn this off once Photos "finds" the library. See this test as a an example:

osxphotos/tests/test_cli.py

Lines 5703 to 5721 in 3c36b0f

	def test_query_name():
	""" test query --name """
	import json
	import os
	import os.path
	import osxphotos
	from osxphotos.cli import query
	runner = CliRunner()
	cwd = os.getcwd()
	result = runner.invoke(
	query,
	["--json", "--db", os.path.join(cwd, PHOTOS_DB_15_7), "--name", "DSC03584"],
	)
	assert result.exit_code == 0
	json_got = json.loads(result.output)
	assert len(json_got) == 1
	assert json_got[0]["original_filename"] == "DSC03584.dng"

[RhetTbull]

If you want to try to add this I also recommend you look at the changes in this commit where I recently added --name query option.

d51d7a4

[garz75]

Thank you for showing the path for implementing this ! I actually have started implementing —folder-matching, my diff looks a lot like what you propose. I won’t have time to work on this before next weekend, I hope I will be able to push a pull request bien then... À bientôt et merci!

On Mon 5 Apr 2021 at 16:40 Rhet Turnbull ***@***.***> wrote: @garz75 <https://github.com/garz75> The --album and --folder options currently match against the entire album or folder name, respectively. Do you need full regex matching or would it be sufficient to use python's str in? For example: >>> album = "Los Angeles 2020" >>> "2020" in album True This would be easier to implement. If you wanted to take a stab at implementing this, here are the basic steps. All modifications would be to osxphotos/cli.py. 1. Add the option to query_options which gets used by both osxphotos query and osxphotos export. While the option name has a "-" in it, these get converted to underscore "_" for the associated variable name. https://github.com/RhetTbull/osxphotos/blob/3c36b0fb333a47c996b687c704d45b25a1edd95d/osxphotos/cli.py#L205-L214 1. Add the new argument, e.g. folder_matching to both query() and export() function defs. In export, you'll also need to add it to the config code that saves and restores config from file for --save-config and --load-config. Reference this bit of code: https://github.com/RhetTbull/osxphotos/blob/3c36b0fb333a47c996b687c704d45b25a1edd95d/osxphotos/cli.py#L1030-L1035 So you'd add folder_matching = cfg.folder_matching etc 1. Modify _query() which does the actual matching for all query options: https://github.com/RhetTbull/osxphotos/blob/3c36b0fb333a47c996b687c704d45b25a1edd95d/osxphotos/cli.py#L1980-L1985 You'd add a new block in this method that goes something like this: if folder_matching: # code that matches folder names here....see the section that starts "if folder:" photos = new_list_of_photos In _query, all blocks are progressive --- e.g. they build on the previous block to build up a list of photos to query/export. It's a long but simple function so should be easy to follow. 1. Find all the places _query is called in the cli.py and update the call to include the new options. 2. Write a test! See test_cli.py. Copy one of the test libraries to your Pictures folder to examine it if needed then write a test that tests your new option. Do NOT open the test library from inside the source code folder as if you do that, Photos will constantly be trying to update it (e.g. to do background processing to find faces, etc) and there's no way to turn this off once Photos "finds" the library. See this test as a an example: https://github.com/RhetTbull/osxphotos/blob/3c36b0fb333a47c996b687c704d45b25a1edd95d/tests/test_cli.py#L5703-L5721 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#413 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSWNEYFVPB5ZKJJL7JH73TTHHDW7ANCNFSM42MOT64Q> .

-- — Frederic Garzon +33 6 72 87 41 67

[RhetTbull]

@garz75 while not as elegant as a custom option, with v0.42.6, I fixed the --query-eval option so you can use a regular expression to search albums and folders as below. Albums are easier because each photo knows the names of the albums it's contained in. Folders take a bit more work as the photo needs to ask the album for which folders the album is contained in. (photos are contained in albums, not folders).

Albums:
osxphotos query --query-eval "list(filter(re.compile(r'^AlbumRegExHere').match, photo.albums))"

Folders:
osxphotos query --query-eval "list(filter(re.compile(r'^FolderRegExHere').match, [item for sublist in [a.folder_names for a in photo.album_info] for item in sublist]))"

The eval string for folders is a bit more complex because it needs to ask each album for a list of folder names but those are returned as a list of lists in the list comprehension which then needs to be flattened

Of course, you can use this method to use a regex for any search, not just albums or folders.

[garz75]

I just tried your suggestion with --query-eval . It does exactly what I want. Even if it is not as "elegant" and exactly fitting as a dedicated option, I am good with this option.
The osxphotos command-line interface is already very complex, adding yet another option for only one user is adding complexity (in termes of maintenance, documentation, etc.) for not much gain for the community. This --query-eval option helps resolve these kind of specific problems, just like you would use pipes and shell programs to build more complex solutions with simple unix tools like awk / cat / sed / wc / head, etc.

thank you for the advice. Fred

[RhetTbull]

The osxphotos command-line interface is already very complex

Very true. I'm torn between keeping the interface more simple and keeping the tool flexible. While I admire the unix philosophy of "do one thing and do it well" I also value the "Swiss Army Knife" "do a little bit of everything" approach that osxphotos takes. For reference, while unix commands appear simple, they actually have a surprising number of options. For example, ls has 58 options and tar has 139. At last count, osxphotos export had 112 options which can definitely be daunting for the newcomer. But in their simplest form, unix commands are easy use. In it's simplest form, osxphotos export /path/to/export will export all your photos.

So, at the risk of making the interface even more complex, I've prototyped a new --regex options that takes the form --regex REGEX TEMPLATE and applies REGEX against an osxphotos template string. This allows you to search pretty much any metadata, not just folders or albums and it also works with the existing --ignore-case flag. For example, to find albums that contain "beach" and ignore case:
--regex "beach" "{album}" --ignore-case

To find albums that start with Beach:
--regex "^Beach" "{album}"

This will implement #433

[garz75]

Rhet this is a brilliant idea. This one is worth adding a new option IMHO ;-)

[RhetTbull]

Done! Added in 0.42.9 :-)