diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 0000000000..8d0c336427 --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,8 @@ +**Geochestra/geonetwork checklist** + + +- [ ] PR only involves cherry picked commits from upstream. +- [ ] PR contains custom code which will be soon available in a release on upstream and can be overrided. +- [ ] PR contains custom georchestra code, which need to be verified in future migration. + - [ ] I have properly filled [migration-helper-changelog.md](..%2Fgeorchestra-migration%2Fmigration-helper-changelog.md) file. + diff --git a/georchestra-migration/migration-helper-changelog.md b/georchestra-migration/migration-helper-changelog.md new file mode 100644 index 0000000000..c0991744b4 --- /dev/null +++ b/georchestra-migration/migration-helper-changelog.md @@ -0,0 +1,12 @@ +# Migration changelog. + + + +## Already updated in latest diff-tree \ No newline at end of file diff --git a/georchestra-migration/migration.md b/georchestra-migration/migration.md new file mode 100644 index 0000000000..4d4df40aa2 --- /dev/null +++ b/georchestra-migration/migration.md @@ -0,0 +1,120 @@ +# Geonetwork migration in Georchestra + +This document aims to help simplify future migration for geonetwork in Georchestra. + +## Upgrade Process + +### Simplified guide + +This is an ultra-simplified guide to be able to build an upgraded version of georchestra/geonetwork +- Merge gn-core tag into georchestra/gn wanted branch. Resolve coflicts and get georchestra custom implementions (see below). +- Copy folders `config` and `data` from `web/src/main/webapp/WEB-INF/data/` into the repo [geonetwork_minimal_datadir](https://github.com/georchestra/geonetwork_minimal_datadir) and create associated branch with new version name. +- Build geonetwork (war, docker ...) +- Delete wro4j cache if necessary (use ?debug can be sometimes useful) +- Reindex records if necessary + +A more detailed guide is available in [upgrade_geonetwork.md](upgrade_geonetwork.md). + +## Georchestra custom implementations + +### Georchestra 4.2.7 and Gn 4.2.7 + +All versions in `pom.xml` files must be updated. E.g. for this version `4.2.7-georchestra`. +All italic folder just have the `pom.xml` change. + +- .github + - workflows `linux.yml` `mvn-dep-tree.yml` `sonarcloud.yml` `dependabot.yml` are deleted. Only `georchestra-gn4.yml` is used +- *cachingxslt* +- common + - `ZipUtilTest.java` : In `assertExampleZip` method, assertions which are supposed check folders must check with trailing slash too. +- core + - `JeevesContextLoaderListener.java` : remove java 8 runtime exception as we use java 11. + - `XslUtil.java`: Implement georchestra header specific code + - `config-spring-geonetwork.xml` : Implement `context:property-placeholder` for georchestra's datadir + - `cleanoutdatabase.sql`: Delete from settings-ui at the end of the file +- *csw-server* +- **docker** + - Mandatory, get everything from geOrchestra +- *docs* +- *doi* +- domain + - ExternalGroupLink.java : get file from geOrchestra + - ExternalUserLink.java : get file from geOrchestra +- *es* +- *estest* +- *events* +- **georchestra-integration** + - Mandatory, get everything from geOrchestra +- harvesters + - `Harvester.java`: Use parseJDK11 method + - `HarvesterTest.java` : Ignore and assume true Java 8 +- *healthmonitor* +- *index* +- *inspire-atom* +- *jmeter* +- *listeners* +- *messaging* +- *oaipmh* +- *release* +- schemas + - `src/main/plugin/iso19115-3.2018` + - OGC API features and 3Dtiles added. Differences in files: + - `config/associated-panel/default.json` + - `loc/eng/labels.xml` + - `loc/fre/labels.xml` + - `test/resources/metadata-for-editing.xml` + - `test/resources/metadata-for-editing-light.xml` + - `test/resources/metadata-iso19139-for-editing.xml` + - `src/main/plugin/iso19139/loc` : 3Dtiles added in labels.xml files. +- *schemas-test* +- *sde* +- services + - `config-spring-geonetwork.xml`: Implement `context:property-placeholder` for georchestra's datadir + - `MetadataExtentApiTest.java` : Update image signatures if necessary (tests may fail see [MetadataExtentApiTest-reference](resources%2FMetadataExtentApiTest-reference) for image reference) + - `BatchOpsMetadatReindexerTest.java` : Add PowerMockIgnore + - `pom.xml`: version to update **and to add to gn-services** +- *slave* +- web + - `src/docker`: Mandatory, get everything from geOrchestra + - `main/filters/prod.properties`: Session timeout variable updated + - `main/filters/java/org/fao/geonet/proxy`: Mandatory, get everything from geOrchestra + - `data-db-default.sql` : Some data to retrieve: setting-ui, inspire activated, georchestra name and (backported value) sitemapLinkUrl + - `UpdateMetadataStatus.java`: Some fixes + - `config-security.xml`: Remove config security add start of file, get end of the file from georchestra. + - `config-geonetwork-georchestra.properties`, `config-georchestra-geonetwork-datadirs.xml`, `DKAN-to-ISO19115-3-2018.xsl`, `GeoIDE-services-OGC.xsl`, `udata-to-ISO19115-3-2018.xsl` + - Get files from georchestra + - `base-variables.xsl` and `base-layout.xsl` + - Get header from georchestra + - `defaultJdbcDataSource.xml`: Implement `context:property-placeholder` for georchestra's db + - `config-spring-geonetwork.xml` + - Implement `context:property-placeholder` for georchestra's datadir + - Don't forget to import config resource and logging bean + - `spring-servlet.xml`: Implement `context:property-placeholder` for georchestra's datadir + - `pom.xml`: Update `dockerGnDatadirScmVersion` variable accordingly +- web-ui + - `pom.xml`: version to update **and to add to gn-web-ui** + - `RelatedResourcesService.js`: Add 3DTiles + - `CatController.js`: Menu bar accessible and 3DTiles + - `src/main/resources/catalog/locales/` + - Add OGC API - Features to i18n files + - Get `en-georchestra.json` and `fr-georchestra.json` from georchestra + - `src/main/resources/catalog/style/gn-less`: remove position fixed from `.gn-top-bar` to get gn header after georchestra's header. + - `gn_admin_default.less`: Same (header position) + - `gn_navbar_default.less`: Same (header position) + - `src/main/resources/catalog/views/module.js`: Add 3DTiles + - `src/main/resources/catalog/views/georchestra/`: Get files from georchestra +- *workers* +- *wro4j* +- .gitignore + - add idea and settings to it +- pom.xml + - Implement JDK 11 specific things, add georchestra-integration module, set db-type + +## Process used + +### 4.2.4 -> 4.2.7 + +A branch has been created from core-geonetwork 4.2.7 tag and merged into `georchestra-gn4.2.x`. +All conflicts files have been resolved with taking gn changes and not grochestra's one. +Then changes have been reimplemented. + diff --git a/georchestra-migration/resources/MetadataExtentApiTest-reference/aggregatedWithTwoExtent.png b/georchestra-migration/resources/MetadataExtentApiTest-reference/aggregatedWithTwoExtent.png new file mode 100644 index 0000000000..e2ab9d2524 Binary files /dev/null and b/georchestra-migration/resources/MetadataExtentApiTest-reference/aggregatedWithTwoExtent.png differ diff --git a/georchestra-migration/resources/MetadataExtentApiTest-reference/getOneRecordExtentAsImage.png b/georchestra-migration/resources/MetadataExtentApiTest-reference/getOneRecordExtentAsImage.png new file mode 100644 index 0000000000..212092b829 Binary files /dev/null and b/georchestra-migration/resources/MetadataExtentApiTest-reference/getOneRecordExtentAsImage.png differ diff --git a/georchestra-migration/resources/MetadataExtentApiTest-reference/threeExtentThirdOne-overview.png b/georchestra-migration/resources/MetadataExtentApiTest-reference/threeExtentThirdOne-overview.png new file mode 100644 index 0000000000..17ff0d6f97 Binary files /dev/null and b/georchestra-migration/resources/MetadataExtentApiTest-reference/threeExtentThirdOne-overview.png differ diff --git a/georchestra-migration/resources/MetadataExtentApiTest-reference/threeExtentThirdOne.png b/georchestra-migration/resources/MetadataExtentApiTest-reference/threeExtentThirdOne.png new file mode 100644 index 0000000000..db73bbd703 Binary files /dev/null and b/georchestra-migration/resources/MetadataExtentApiTest-reference/threeExtentThirdOne.png differ diff --git a/georchestra-migration/resources/MetadataExtentApiTest-reference/threeExtentThirdOne115_3-overview.png b/georchestra-migration/resources/MetadataExtentApiTest-reference/threeExtentThirdOne115_3-overview.png new file mode 100644 index 0000000000..7222909f49 Binary files /dev/null and b/georchestra-migration/resources/MetadataExtentApiTest-reference/threeExtentThirdOne115_3-overview.png differ diff --git a/georchestra-migration/resources/MetadataExtentApiTest-reference/threeExtentThirdOne115_3.png b/georchestra-migration/resources/MetadataExtentApiTest-reference/threeExtentThirdOne115_3.png new file mode 100644 index 0000000000..975262f447 Binary files /dev/null and b/georchestra-migration/resources/MetadataExtentApiTest-reference/threeExtentThirdOne115_3.png differ diff --git a/georchestra-migration/resources/MetadataExtentApiTest-reference/threeExtentThirdOneIsABoundingBox.png b/georchestra-migration/resources/MetadataExtentApiTest-reference/threeExtentThirdOneIsABoundingBox.png new file mode 100644 index 0000000000..644e6bdabc Binary files /dev/null and b/georchestra-migration/resources/MetadataExtentApiTest-reference/threeExtentThirdOneIsABoundingBox.png differ diff --git a/georchestra-migration/resources/MetadataExtentApiTest-reference/twoExtentFirstOneWithBothBoundingBoxAndPolygon-1.png b/georchestra-migration/resources/MetadataExtentApiTest-reference/twoExtentFirstOneWithBothBoundingBoxAndPolygon-1.png new file mode 100644 index 0000000000..8cc4684f4f Binary files /dev/null and b/georchestra-migration/resources/MetadataExtentApiTest-reference/twoExtentFirstOneWithBothBoundingBoxAndPolygon-1.png differ diff --git a/georchestra-migration/resources/MetadataExtentApiTest-reference/twoExtentFirstOneWithBothBoundingBoxAndPolygon-overview.png b/georchestra-migration/resources/MetadataExtentApiTest-reference/twoExtentFirstOneWithBothBoundingBoxAndPolygon-overview.png new file mode 100644 index 0000000000..e2ab9d2524 Binary files /dev/null and b/georchestra-migration/resources/MetadataExtentApiTest-reference/twoExtentFirstOneWithBothBoundingBoxAndPolygon-overview.png differ diff --git a/georchestra-migration/upgrade_geonetwork.md b/georchestra-migration/upgrade_geonetwork.md new file mode 100644 index 0000000000..e80ca4ddca --- /dev/null +++ b/georchestra-migration/upgrade_geonetwork.md @@ -0,0 +1,49 @@ +# Upgrading geonetwork + +those are generally the steps to check/remember to do when upgrading +geonetwork, be it for a major or a minor version, eg more or less applies for +3.8->4.0, 4.0->4.2, or 4.2.2->4.2.4. + +## in the geonetwork data directory + +* several files should be updated with the ones from the webapp: + * the configuration of the elasticsearch index + * the schemas + * the formatters used by the complete view +``` +/srv/data/geonetwork# rm -Rf config/schema_plugins data/formatter +/srv/data/geonetwork# cp -r /srv/tomcat/georchestra/webapps/geonetwork/WEB-INF/data/config/schema_plugins config/ +/srv/data/geonetwork# cp -r /srv/tomcat/georchestra/webapps/geonetwork/WEB-INF/data/config/index config/ +/srv/data/geonetwork# cp -r /srv/tomcat/georchestra/webapps/geonetwork/WEB-INF/data/data/formatter data/ +``` +* make sure those files can be overwritten by the user running tomcat/the webapp +``` +/srv/data/geonetwork# chown -R tomcat:tomcat data/formatter config/schema_plugins +``` +* if you use the https://github.com/georchestra/geonetwork_minimal_datadir + repository, generally those changes are already available in the latest + branch of the repository +* drop the `wro4j-cache.mv.db` file (that's equivalent to the 'clear js & css + cache' button in the tools pane of the admin ui, but sometimes this page + might not be accessible when upgrading from a previous version) +* restart/reload the webapp +* check the logs, because if there are database schema changes to apply, those + should be done at the first start of the new version of the webapp + +## in the geonetwork admin UI + +* connected as admin, go to the *Tools* page in the *Admin console* menu +* hit *delete index and reindex* button. This will reset the elasticsearch + index, update its configuration (the `records.json` file in the + `config/index/` subdirectory of the geonetwork data directory) and rebuild + the index +* go in the *user interface* pane of the *Settings* page in the *Admin console* menu +* hit the *Json form* button and *Current configuration*. This will show you + what differs from the default configuration provided by the project. +* make a note of all those changes that were done previously, hit the 'Reset + configuration' button. This will make sure the UI configuration uses the + default provided by the georchestra project for its geonetwork fork. +* then, via the *Configuration form* tab, you can report the modifications you + previously did in the UI config to your own taste. +* make sure that facets in search results do still work, and that you still have + all your metadata :)