Merge pull request #740 from samply/release-v0.17.7

Release v0.17.7

.gitignore

@@ -1,13 +1,5 @@
 target
-/.lein-*
-.nrepl-port
-/queries
-/generate-docs.sh
-/fill-store.sh
-/load-test.sh
-/start-db.sh
-/kube*
-/nginx.conf
 .cpcache
 .cache
-.env
+blaze-load-tests/node_modules
+.nrepl-port

CHANGELOG.md

@@ -2,6 +2,14 @@
 
 ## v0.17.6
 
+### Other
+
+* Introduce Database Versioning ([#738](https://github.com/samply/blaze/pull/738))
+
+The full changelog can be found [here](https://github.com/samply/blaze/milestone/45?closed=1).
+
+## v0.17.6
+
 ### Performance
 
 * Improve Interning of Complex Types ([#725](https://github.com/samply/blaze/issues/725))

README.md

@@ -14,7 +14,7 @@ The goal of this project is to provide a FHIR® Store with an internal CQL Evalu
 
 Blaze passes all [Touchstone FHIR 4.0.1 Basic Tests][12] and almost all [CQL Tests][3]. Please refer to the [Conformance](docs/conformance.md) section and report any issues you encounter during evaluation.
 
-Latest release: [v0.17.6][5]
+Latest release: [v0.17.7][5]
 
 ## Quick Start
 
@@ -72,7 +72,7 @@ Unless required by applicable law or agreed to in writing, software distributed
 
 [3]: <https://cql.hl7.org/tests.html>
 [4]: <https://alexanderkiel.gitbook.io/blaze/deployment>
-[5]: <https://github.com/samply/blaze/releases/tag/v0.17.6>
+[5]: <https://github.com/samply/blaze/releases/tag/v0.17.7>
 [6]: <https://www.yourkit.com/java/profiler/>
 [7]: <https://www.yourkit.com/.net/profiler/>
 [8]: <https://www.yourkit.com/youmonitor/>

deps.edn

@@ -80,7 +80,7 @@
   :outdated
   {:replace-deps
    {com.github.liquidz/antq
-    {:mvn/version "1.6.774"}
+    {:mvn/version "1.7.798"}
 
     org.slf4j/slf4j-nop
     {:mvn/version "1.7.36"}}

docs/database/migration.md

@@ -0,0 +1,61 @@
+# Database Migration
+
+If Blaze starts up with the following error message:
+
+```
+Incompatible index store version <x> found. This version of Blaze needs
+version <y>.
+
+Either use an older version of Blaze which is compatible with index store
+version <x> or do a database migration described here:
+
+  https://github.com/samply/blaze/tree/master/docs/database/migration.md
+```
+
+you need to do a migration of the index store.
+
+Blaze will do that migration automatically at startup if you delete the index store. So all you need to do is to make a
+backup of all the data Blaze has written to disk, **plan for a downtime**, delete the index store and start Blaze normally.
+
+## Deleting the Index Store
+
+Please start Blaze with a shell assuming that you use the volume `blaze-data`:
+
+```sh
+docker run -it -v blaze-data:/app/data samply/blaze:0.17 sh
+```
+
+in that shell, go into `/app/data` and list all directories:
+
+```sh
+cd /app/data
+ls
+```
+
+you should see the three directories `index`,  `resource` and `transaction`.
+
+If you have enough disk space, just rename the index directory into `index-old`. If not, delete it assuming you have a
+backup!
+
+Exit the shell und start Blaze normally.
+
+## Index Store Migration at Start
+
+If you start Blaze without an index store, it will use the transaction log and the resource store to recreate the index
+store. During that process Blaze will not be available for reads and writes. Reading from Blaze will result in
+a `503 Service Unavailable` with the following `OperationOutcome`:
+
+```json
+{
+  "resourceType": "OperationOutcome",
+  "issue": [
+    {
+      "severity": "error",
+      "code": "timeout",
+      "diagnostics": "Timeout while trying to acquire the latest known database state. At least one known transaction hasn't been completed yet. Please try to lower the transaction load or increase the timeout of 10000 ms by setting DB_SYNC_TIMEOUT to a higher value if you see this often."
+    }
+  ]
+}
+```
+
+The time needed for the rebuild of the index store is roughly the time all transactions ever happened in this instance of Blaze took.

docs/deployment/docker-deployment.md

@@ -27,7 +27,7 @@ Blaze should log something like this:
 2021-06-27T11:02:37.834Z ee086ef908c1 main INFO [blaze.core:64] - JVM version: 16.0.2
 2021-06-27T11:02:37.834Z ee086ef908c1 main INFO [blaze.core:65] - Maximum available memory: 1738 MiB
 2021-06-27T11:02:37.835Z ee086ef908c1 main INFO [blaze.core:66] - Number of available processors: 8
-2021-06-27T11:02:37.836Z ee086ef908c1 main INFO [blaze.core:67] - Successfully started Blaze version 0.17.6 in 8.2 seconds
+2021-06-27T11:02:37.836Z ee086ef908c1 main INFO [blaze.core:67] - Successfully started Blaze version 0.17.7 in 8.2 seconds
 ```
 
 In order to test connectivity, query the health endpoint:
@@ -47,7 +47,7 @@ that should return:
 ```json
 {
   "name": "Blaze",
-  "version": "0.17.6"
+  "version": "0.17.7"
 }
 ```
 

docs/deployment/manual-deployment.md

@@ -2,12 +2,12 @@
 
 The installation works under Windows, Linux and macOS. The only dependency is an installed OpenJDK 11. Blaze is tested with [AdoptOpenJDK][1].
 
-Blaze runs on the JVM and comes as single JAR file. Download the most recent version [here](https://github.com/samply/blaze/releases/tag/v0.17.6). Look for `blaze-0.17.6-standalone.jar`.
+Blaze runs on the JVM and comes as single JAR file. Download the most recent version [here](https://github.com/samply/blaze/releases/tag/v0.17.7). Look for `blaze-0.17.7-standalone.jar`.
 
 After the download, you can start blaze with the following command (Linux, macOS):
 
 ```sh
-java -jar blaze-0.17.6-standalone.jar -m blaze.core
+java -jar blaze-0.17.7-standalone.jar -m blaze.core
 ```
 
 Blaze will run with an in-memory, volatile database for testing and demo purposes.
@@ -17,14 +17,14 @@ Blaze can be run with durable storage by setting the environment variables `STOR
 Under Linux/macOS:
 
 ```sh
-STORAGE=standalone java -jar blaze-0.17.6-standalone.jar -m blaze.core
+STORAGE=standalone java -jar blaze-0.17.7-standalone.jar -m blaze.core
 ```
 
 Under Windows, you need to set the Environment variables in the PowerShell before starting Blaze:
 
 ```powershell
 $Env:STORAGE="standalone"
-java -jar blaze-0.17.6-standalone.jar -m blaze.core
+java -jar blaze-0.17.7-standalone.jar -m blaze.core
 ```
 
 This will create three directories called `index`, `transaction` and `resource` inside the current working directory, one for each database part used.
@@ -42,7 +42,7 @@ The output should look like this:
 2021-06-27T11:02:37.834Z ee086ef908c1 main INFO [blaze.core:64] - JVM version: 16.0.2
 2021-06-27T11:02:37.834Z ee086ef908c1 main INFO [blaze.core:65] - Maximum available memory: 1738 MiB
 2021-06-27T11:02:37.835Z ee086ef908c1 main INFO [blaze.core:66] - Number of available processors: 8
-2021-06-27T11:02:37.836Z ee086ef908c1 main INFO [blaze.core:67] - Successfully started Blaze version 0.17.6 in 8.2 seconds
+2021-06-27T11:02:37.836Z ee086ef908c1 main INFO [blaze.core:67] - Successfully started Blaze version 0.17.7 in 8.2 seconds
 ```
 
 In order to test connectivity, query the health endpoint:
@@ -62,7 +62,7 @@ that should return:
 ```json
 {
   "name": "Blaze",
-  "version": "0.17.6"
+  "version": "0.17.7"
 }
 ```
 

modules/cql/Makefile

@@ -2,7 +2,8 @@ lint:
 	clj-kondo --lint src test deps.edn
 
 cql-test:
-	wget https://cql.hl7.org/tests.zip
+	wget http://cql.hl7.org/tests.zip
+	echo "0d48a7441c43b6ee46e71d73decfa0cf4ea81e2ce70951f20e9163c3bebfc49a  tests.zip" | sha256sum --check --status
 	unzip -jd cql-test tests.zip
 	rm tests.zip
 	# See: https://github.com/HL7/cql/pull/69

modules/db/src/blaze/db/api.clj

@@ -104,7 +104,7 @@
 
   Please use `pull` to obtain the full resource."
   [db type id]
-  (log/trace "fetch resource handle of " type "" id)
+  (log/trace "fetch resource handle of" (str type "/" id))
   (p/-resource-handle db (codec/tid type) (codec/id-byte-string id)))
 
 

modules/db/src/blaze/db/node.clj

@@ -22,6 +22,7 @@
     [blaze.db.node.tx-indexer :as tx-indexer]
     [blaze.db.node.tx-indexer.verify :as tx-indexer-verify]
     [blaze.db.node.validation :as validation]
+    [blaze.db.node.version :as version]
     [blaze.db.resource-store :as rs]
     [blaze.db.search-param-registry.spec]
     [blaze.db.tx-log :as tx-log]
@@ -387,12 +388,46 @@
     [:blaze.db/enforce-referential-integrity]))
 
 
+(def ^:private expected-kv-store-version 0)
+
+
+(defn- kv-store-version [kv-store]
+  (or (some-> (kv/get kv-store version/key) version/decode-value) 0))
+
+
+(def ^:private incompatible-kv-store-version-msg
+  "Incompatible index store version %1$d found. This version of Blaze needs
+  version %2$d.
+
+  Either use an older version of Blaze which is compatible with index store
+  version %1$d or do a database migration described here:
+
+    https://github.com/samply/blaze/tree/master/docs/database/migration.md
+
+  ")
+
+
+(defn- incompatible-kv-store-version-ex [actual-version expected-version]
+  (ex-info (format incompatible-kv-store-version-msg actual-version expected-version)
+           {:actual-version actual-version :expected-version expected-version}))
+
+
+(defn- check-version! [kv-store]
+  (when (tx-success/last-t kv-store)
+    (let [actual-kv-store-version (kv-store-version kv-store)]
+      (if (= actual-kv-store-version expected-kv-store-version)
+        (log/info "Index store version is" actual-kv-store-version)
+        (throw (incompatible-kv-store-version-ex actual-kv-store-version
+                                                 expected-kv-store-version))))))
+
+
 (defmethod ig/init-key :blaze.db/node
   [_ {:keys [tx-log resource-handle-cache tx-cache indexer-executor kv-store
              resource-indexer resource-store search-param-registry poll-timeout]
       :or {poll-timeout (time/seconds 1)}
       :as config}]
   (init-msg config)
+  (check-version! kv-store)
   (let [node (->Node (ctx config) tx-log resource-handle-cache tx-cache kv-store
                      resource-store search-param-registry resource-indexer
                      (atom (initial-state kv-store))

modules/db/src/blaze/db/node/version.clj

@@ -0,0 +1,23 @@
+(ns blaze.db.node.version
+  (:refer-clojure :exclude [key])
+  (:require
+    [blaze.byte-buffer :as bb])
+  (:import
+    [java.nio.charset StandardCharsets]))
+
+
+(set! *warn-on-reflection* true)
+
+
+(def key
+  (.getBytes "version" StandardCharsets/ISO_8859_1))
+
+
+(defn encode-value [version]
+  (-> (bb/allocate Integer/BYTES)
+      (bb/put-int! version)
+      (bb/array)))
+
+
+(defn decode-value [bytes]
+  (bb/get-int! (bb/wrap bytes)))

modules/db/test/blaze/db/node_test.clj

@@ -6,11 +6,14 @@
     [blaze.db.api :as d]
     [blaze.db.api-spec]
     [blaze.db.impl.db-spec]
+    [blaze.db.impl.index.tx-success :as tx-success]
+    [blaze.db.kv :as kv]
     [blaze.db.kv.mem-spec]
     [blaze.db.node :as node]
     [blaze.db.node-spec]
     [blaze.db.node.resource-indexer :as resource-indexer]
     [blaze.db.node.tx-indexer :as-alias tx-indexer]
+    [blaze.db.node.version :as version]
     [blaze.db.resource-handle-cache]
     [blaze.db.resource-store :as rs]
     [blaze.db.search-param-registry]
@@ -29,7 +32,8 @@
     [cognitect.anomalies :as anom]
     [integrant.core :as ig]
     [juxt.iota :refer [given]]
-    [taoensso.timbre :as log]))
+    [taoensso.timbre :as log])
+  (:import [java.time Instant]))
 
 
 (set! *warn-on-reflection* true)
@@ -82,6 +86,12 @@
              {:resource-store (ig/ref ::rs/kv)})))
 
 
+(defn- with-index-store-version [system version]
+  (assoc-in system [[::kv/mem :blaze.db/index-kv-store] :init-data]
+            [[version/key (version/encode-value version)]
+             (tx-success/index-entry 1 Instant/EPOCH)]))
+
+
 (deftest init-test
   (testing "nil config"
     (given-thrown (ig/init {:blaze.db/node nil})
@@ -129,7 +139,14 @@
       [:explain ::s/problems 6 :pred] := `(fn ~'[%] (contains? ~'% :resource-store))
       [:explain ::s/problems 7 :pred] := `(fn ~'[%] (contains? ~'% :search-param-registry))
       [:explain ::s/problems 8 :pred] := `boolean?
-      [:explain ::s/problems 8 :val] := ::invalid)))
+      [:explain ::s/problems 8 :val] := ::invalid))
+
+  (testing "incompatible version"
+    (given-thrown (ig/init (with-index-store-version system -1))
+      :key := :blaze.db/node
+      :reason := ::ig/build-threw-exception
+      [:cause-data :expected-version] := 0
+      [:cause-data :actual-version] := -1)))
 
 
 (deftest duration-seconds-collector-init-test
@@ -220,3 +237,8 @@
 
     ;; but it isn't terminated yet
     (is (not (ex/terminated? indexer-executor)))))
+
+
+(deftest existing-data-with-compatible-version
+  (with-system [{:blaze.db/keys [node]} (with-index-store-version system 0)]
+    (is node)))

modules/fhir-structure/deps.edn

@@ -29,7 +29,7 @@
   {:mvn/version "2022.03.27-397"}
 
   metosin/jsonista
-  {:mvn/version "0.3.5"}
+  {:mvn/version "0.3.6"}
 
   org.clojure/alpha.spec
   {:git/url "https://github.com/alexanderkiel/spec-alpha2.git"

modules/rest-api/src/blaze/rest_api/capabilities.clj

@@ -117,7 +117,7 @@
          :copyright
          #fhir/markdown"Copyright 2019 - 2022 The Samply Community\n\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at\n\nhttp://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
          :kind #fhir/code"instance"
-         :date #fhir/dateTime"2022-06-01"
+         :date #fhir/dateTime"2022-06-05"
          :software
          {:name "Blaze"
           :version version}

modules/rest-util/src/blaze/middleware/fhir/db.clj

@@ -17,7 +17,7 @@
 
 
 (defn- timeout-msg [timeout]
-  (format "Timeout while trying to acquire the latest known database state. At least one known transaction hasen't been completed yet. Please try to lower the transaction load or increase the timeout of %d ms by setting DB_SYNC_TIMEOUT to a higher value if you see this often.", timeout))
+  (format "Timeout while trying to acquire the latest known database state. At least one known transaction hasn't been completed yet. Please try to lower the transaction load or increase the timeout of %d ms by setting DB_SYNC_TIMEOUT to a higher value if you see this often.", timeout))
 
 
 (defn- db [node timeout {:keys [query-params] :as request}]

modules/rest-util/test/blaze/middleware/fhir/db_test.clj

@@ -88,7 +88,7 @@
 
         (given-failed-future ((db/wrap-db handler ::node 2000) {})
           ::anom/category := ::anom/busy
-          ::anom/message := "Timeout while trying to acquire the latest known database state. At least one known transaction hasen't been completed yet. Please try to lower the transaction load or increase the timeout of 2000 ms by setting DB_SYNC_TIMEOUT to a higher value if you see this often.")))
+          ::anom/message := "Timeout while trying to acquire the latest known database state. At least one known transaction hasn't been completed yet. Please try to lower the transaction load or increase the timeout of 2000 ms by setting DB_SYNC_TIMEOUT to a higher value if you see this often.")))
 
     (testing "default timeout are 10 seconds"
       (with-redefs
@@ -99,4 +99,4 @@
 
         (given-failed-future ((db/wrap-db handler ::node) {})
           ::anom/category := ::anom/busy
-          ::anom/message := "Timeout while trying to acquire the latest known database state. At least one known transaction hasen't been completed yet. Please try to lower the transaction load or increase the timeout of 10000 ms by setting DB_SYNC_TIMEOUT to a higher value if you see this often.")))))
+          ::anom/message := "Timeout while trying to acquire the latest known database state. At least one known transaction hasn't been completed yet. Please try to lower the transaction load or increase the timeout of 10000 ms by setting DB_SYNC_TIMEOUT to a higher value if you see this often.")))))