Support for multiple content folders

[ajtribick]

Currently editing the Celestia content and configuration requires either installing Celestia to a user-editable directory or editing protected areas of the file system, which should be discouraged.

My proposal here is to support multiple content directories, (at least) one of which could then be located in the user's personal directory.

• On Linux, it probably makes sense to follow the XDG base directory specification, support loading content from XDG_DATA_HOME (as a first option), and then falling back to the directories listed in XDG_DATA_DIRS.

• On Windows the user content directory would be located in %APPDATA%, falling back to either a registry key set by the installer, or the %PROGRAMDATA% directory (although we should retrieve these via SHGetKnownFolderPath).

• On Mac this would presumably correspond to the paths listed in Qt's QStandardPaths - presumably there's a way to get this from the system APIs if we need to support non-Qt front-ends on Mac (presumably SDL).

For the Qt front-end, the paths could be retrieved using QStandardPaths (supplemented with reading from the registry on Windows), for the other front-ends this would presumably need a custom implementation. To leave this open for other platforms (Android, iOS, etc.) the paths should be provided to the CelestiaCore by the front-ends, rather than determined by the Celestia core code.

This should be overridable via the command line, e.g. if the option is called -D (Data directory), then passing -Dfirst/data/dir -Dsecond/data/dir should load first from first/data/dir then fall back to -Dsecond/data/dir.

For the celestia.cfg file, XDG would suggest using XDG_CONFIG_HOME and then XDG_CONFIG_DIRS - this suggests that the installer should put this file under /etc/xdg (although I'm not necessarily wedded to that idea, could also use one of the paths in XDG_DATA_DIRS). It would probably be useful to copy the installer-provided celestia.cfg to the user's directory if it isn't found in the user directory at launch.

In the case of files in the data folder, whichever folder comes first wins, i.e. if the user puts an updated stars.dat file in their personal data directory, this would override the one created in the installer. For extras, it possibly makes sense to use reverse order, i.e. load the installed directory (which would provide what's currently in extras-standard) before the user ones. Like config, the various data folders should be created in the user directory on startup if they are not present.

Related issue is #1610 - I'm not sure whether it is worth supporting fallbacks for shaders (and the other resources in this repository), presumably they should be loaded from the installation location, and I would prefer to have this configured with a separate command-line option.

My initial look over how the directories are handled suggests this would require a fair number of changes (especially regarding handling of the locale folders), so would like to gauge support for the general approach before I begin.

[375gnu]

• What means "falback" directory? Should its content override object populated from the "primary" directory? If no, how then correctly distinguish that files in the "primary" can modify content, while in the "falback" they can't?
• Should they be totally independent? I mean if the "falback" directory provides object "Foo" should the "primary" directory contain Foo's duplicate if it has object "Bar" depending on Foo?

Qt & SDL frontends support env var CELESTIA_DATA_DIR which I believe can help with 99% use cases.

[ajtribick]

The primary directory is the first directory searched for the files. The fallback directory is the directory to fall back to if the files aren't found (there might be multiple fallback directories). So we have a priority list, e.g. on Linux

1. XDG_DATA_HOME
2. first directory listed in XDG_DATA_DIRS
3. second directory listed in XDG_DATA_DIRS
4. etc.

On Windows

1. %APPDATA%
2. Directory specified in registry entry created by content installer
3. %PROGRAMDATA%

(Note the actual files would be located in a folder structure underneath these directories, e.g. Celestia Development Team\CelestiaContent\data\solarsys.ssc)

So, e.g. if the folder structure in the XDG_DATA_HOME directory does not contain stars.dat, it would be loaded from the first directory as determined from the list in XDG_DATA_DIRS.

At least for the files under data I would prefer to keep to the simple option of pure replacement, i.e. if solarsys.ssc is found in the XDG_DATA_HOME folder structure, we process it and do not search for the file in the XDG_DATA_DIRS directories.

For extras I think it makes sense to load the files from all the directories, probably in the reverse priority order which would in most cases be the CelestiaContent install, followed by whatever the user has in their directories.

The disadvantage of CELESTIA_DATA_DIR is that it implies only one directory, and it defaults back to CONFIG_DATA_DIR which makes no sense on Windows, where the user can specify a custom location in the installer. I think it makes more sense to stick to platform conventions than inventing our own.

[dave-kaye]

Question: What are you defining as "data directories"? The command line uses --data as an installation directory which can contain subdirectories of data, models, textures and files such as celestia.cfg. But Celestia also has a data directory which contains only *.dat, *.stc, *.dsc and *.ssc files.

As a suggestion, you may wish to review the celestia.cfg file structure. In it there are two entries, ExtrasDirectory and SkipExtras. ExtrasDirectory will allow additional directories to be utilized for models, etc. SkipExtras will remove a catalog file name from the main directory.

If you added something similar to celestia.cfg -- "DataDirectory" and "SkipData" -- you could allow the user to simply modify a couple of configuration lines to send data searches into additional user or system directories. It would be far easier than trying to determine every possible default path for each user interface through Windows, Linux and macOS systems. Also, you would not need to program a hierarchy of file reading. The user would determine it by ordering the directories in the DataDirectory entry. The downside is if the files needed to be read in a specific order, and one of the files in the external directory must be read in the middle of the install directory file chain.

A second suggestion, the configuration file gives a list of the data files it is reading. The list can be edited to place one of the existing data files in a user directory. From there the entries can be modified. I tested this, and was able to move one of the *.ssc files from the initial install directory to a directory in my home directory. The file was read successfully upon Celestia startup. From this location I could make any modifications I desired. The only issue is that texture, mesh, etc files are still looked for in the initial install directory. There would need to be a change in the routine for the file name storage entry if you wanted local copies. Otherwise, you can just load the desired texture and mesh files into the install directory in the proper subdirectories. This suggestion already exists, and gets over the file load sequence issue.

A third suggestion is to do nothing. The present --data command line option will allow differing installation directories to be utilized. These can be in user or program protected space. Here you would only have the additional overhead of having to make copies of the full installation directory for each version of Celestia you wish to create. The downside is that if you need to upgrade the original installation directory, you will have to review and modify any copies that you have made.

A completely different alternative focuses more on the data directory as the holder of the *.dat, *.ssc, *.dsc and *.stc files. Utilize the present ExtrasDirectory entry. According to user literature, there are more capabilities in this entry than simple addons:

Note this change introduced in 1.2.5: 'ExtrasDirectories -- not just directory. Add-on extras directories can contain all types of Celestia files, including those which have traditionally gone into /data/ and /models/.'

Also, *.ssc files have the key words Add, Replace and Modify that affect prior loaded information.

I ran an easy test - adding an .ssc file to a personal ExtrasDirectory with Add and Modify commands which interacted with file data from the system based data/*.ssc. It worked. I can efficiently modify any entry that is found in an .ssc file. Once again, the only issue is that texture, mesh, etc files are still looked for in the initial install directory.

Somewhere I read that users would like to see the Add, Modify and Replace commands expanded to more file types than just *.ssc files. If this is implemented, this would create a capability to do external editing on more celestal objects. This may do everything that you want to do.

---

On the mesh and texture file reading location problem. A simple 3 step logic ladder can be used:

IF the file name is a simple file name, use the default directory.

ELSE IF the file name begins with ./ (Windows version?) the file path is relative to the location of the data file you are presently reading.

ELSE all others the file name is assumed to be an absolute file path. Read the file as if it is an absolute path, log an error and continue on if there is a problem finding the file.

[ajtribick]

By "data directories" I mean data, extras/extras-standard, models, textures, warp, and the associated localisation files, i.e. the stuff in the CelestiaContent repository plus user add-ons. (Probably also the scripts should be included in this)

There are also data files in this repository, e.g. shaders, the splash screen, etc. - these should be handled separately and the ability for users to override them is less critical - these are out-of-scope for what I'm proposing here (and probably should have a separate command-line option).

The problem with just doing it via celestia.cfg is that you still need to edit celestia.cfg, which currently gets installed to a system directory.

As regards finding the directories, at least with Qt there is a QStandardPaths class that does the job for us. For the Windows front-end there are API calls that do a similar thing. The tricky ones here will be GTK and SDL.

[levinli303]

For macOS we can retrieve these paths with objc apis.