Nippy 3.3.0
Released under the Eclipse Public License - v 1.0
The fastest serialization library for Clojure.
Installation
To install, add the following dependency to your project or build file:
[com.taoensso/nippy "3.3.0"]
Namespaces
taoensso.nippy
High-performance serialization library for Clojure
Public variables and functions:
- *auto-freeze-compressor*
- *custom-readers*
- *freeze-fallback*
- *freeze-serializable-allowlist*
- *incl-metadata?*
- *thaw-serializable-allowlist*
- *thaw-xform*
- -cache-proxy
- aes128-cbc-encryptor
- aes128-encryptor
- aes128-gcm-encryptor
- allow-and-record-any-serializable-class-unsafe
- cache
- compress
- decompress
- decrypt
- default-freeze-serializable-allowlist
- default-thaw-serializable-allowlist
- encrypt
- extend-freeze
- extend-thaw
- fast-freeze
- fast-thaw
- freezable?
- freeze
- freeze-to-file
- freeze-to-out!
- freeze-to-string
- get-recorded-serializable-classes
- IFreezable1
- IFreezable2
- inspect-ba
- lz4-compressor
- lz4hc-compressor
- lzma2-compressor
- public-types-spec
- read-quarantined-serializable-object-unsafe!
- snappy-compressor
- stress-data
- stress-data-benchable
- stress-data-comparable
- thaw
- thaw-from-file
- thaw-from-in!
- thaw-from-resource
- thaw-from-string
- throw-unfreezable
- try-write-readable
- try-write-serializable
- write-id
- write-unfreezable
taoensso.nippy.tools
Utils for community tools that want to add user-configurable Nippy support. +Nippy 3.4.0-RC1 Generated by Codox
Nippy 3.4.0-RC1
\ No newline at end of file diff --git a/taoensso.nippy.html b/taoensso.nippy.html index 8abf5c6b..2e8df093 100644 --- a/taoensso.nippy.html +++ b/taoensso.nippy.html @@ -1,6 +1,6 @@ -Nippy 3.4.0-RC1
Released under the Eclipse Public License - v 1.0
The fastest serialization library for Clojure.
Installation
To install, add the following dependency to your project or build file:
[com.taoensso/nippy "3.4.0-RC1"]Namespaces
taoensso.nippy
High-performance serialization library for Clojure.Public variables and functions:
- *auto-freeze-compressor*
- *custom-readers*
- *freeze-fallback*
- *freeze-serializable-allowlist*
- *incl-metadata?*
- *thaw-serializable-allowlist*
- *thaw-xform*
- -cache-proxy
- aes128-cbc-encryptor
- aes128-encryptor
- aes128-gcm-encryptor
- allow-and-record-any-serializable-class-unsafe
- cache
- compress
- decompress
- decrypt
- default-freeze-serializable-allowlist
- default-thaw-serializable-allowlist
- encrypt
- extend-freeze
- extend-thaw
- fast-freeze
- fast-thaw
- freezable?
- freeze
- freeze-to-file
- freeze-to-out!
- freeze-to-string
- get-recorded-serializable-classes
- IFreezable
- IFreezableWithMeta
- inspect-ba
- lz4-compressor
- lz4hc-compressor
- lzma2-compressor
- public-types-spec
- read-quarantined-serializable-object-unsafe!
- stress-data
- thaw
- thaw-from-file
- thaw-from-in!
- thaw-from-resource
- thaw-from-string
- throw-unfreezable
- try-write-readable
- try-write-serializable
- write-id
- write-unfreezable
- zstd-compressor
taoensso.nippy.tools
Utils for community tools that want to add user-configurable Nippy support. Used by Carmine, Faraday, etc.Public variables and functions:
taoensso.nippy documentation Generated by Codox
Nippy 3.3.0
taoensso.nippy
High-performance serialization library for Clojure +taoensso.nippy documentation Generated by Codox
Nippy 3.4.0-RC1
\ No newline at end of file +See also `freeze-to-string`.taoensso.nippy
High-performance serialization library for Clojure.*auto-freeze-compressor*
dynamic
(fn [byte-array])->compressor used by `(freeze <x> {:compressor :auto}), nil => default*custom-readers*
dynamic
{<hash-or-byte-id> (fn [data-input])->read}*freeze-fallback*
dynamic
Controls Nippy's behaviour when trying to freeze an item for which Nippy @@ -22,17 +22,17 @@ 3. [Advanced] Custom (fn [^java.io.DataOutput out item]) that must write exactly one value to the given `DataOutput` stream*freeze-serializable-allowlist*
dynamic
Used when attempting to <freeze/thaw> an object that: - - Does NOT implement Nippy's Freezable protocol. - - DOES implement Java's Serializable interface. + - Does NOT implement Nippy's `Freezable` protocol. + - DOES implement Java's `Serializable` interface. -In this case, the allowlist will be checked to see if Java's -Serializable interface may be used. +In this case, an allowlist will be checked to see if Java's +`Serializable` interface may be used. This is a security measure to prevent possible Remote Code Execution (RCE) when thawing malicious payloads. See [1] for details. -If `freeze` encounters a disallowed Serialized class, it will throw. -If `thaw` encounters a disallowed Serialized class, it will: +If `freeze` encounters a disallowed `Serializable` class, it will throw. +If `thaw` encounters a disallowed `Serializable` class, it will: - Throw if it's not possible to safely quarantine the object (object was frozen with Nippy < v2.15.0-final). @@ -59,24 +59,24 @@ Allowlist values may be overridden with `binding`, `alter-var-root`, or: - - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-base` JVM property - - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-add` JVM property + - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-base` JVM property value + - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-add` JVM property value - - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_BASE` env var - - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_ADD` env var + - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_BASE` Environment variable value + - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_ADD` Environment variable value If present, these will be read as comma-separated lists of class names and formed into sets. Each initial allowlist value will then be: (into (or <?base> <default>) <?additions>). I.e. you can use: - - The "base" property/var to replace Nippy's default allowlists. - - The "add" property/var to add to Nippy's default allowlists. + - The "base" property/var to REPLACE Nippy's default allowlists. + - The "add" property/var to ADD TO Nippy's default allowlists. The special `"allow-and-record"` value is also possible, see [2]. Upgrading from an older version of Nippy and unsure whether you've been -using Nippy's Serializable support, or which classes to allow? See [2]. +using Nippy's `Serializable` support, or which classes to allow? See [2]. See also `taoensso.encore/name-filter` for a util to help easily build more advanced predicate functions. @@ -87,17 +87,17 @@ [1] https://github.com/ptaoussanis/nippy/issues/130 [2] See `allow-and-record-any-serializable-class-unsafe`.*incl-metadata?*
dynamic
Include metadata when freezing/thawing?*thaw-serializable-allowlist*
dynamic
Used when attempting to <freeze/thaw> an object that: - - Does NOT implement Nippy's Freezable protocol. - - DOES implement Java's Serializable interface. + - Does NOT implement Nippy's `Freezable` protocol. + - DOES implement Java's `Serializable` interface. -In this case, the allowlist will be checked to see if Java's -Serializable interface may be used. +In this case, an allowlist will be checked to see if Java's +`Serializable` interface may be used. This is a security measure to prevent possible Remote Code Execution (RCE) when thawing malicious payloads. See [1] for details. -If `freeze` encounters a disallowed Serialized class, it will throw. -If `thaw` encounters a disallowed Serialized class, it will: +If `freeze` encounters a disallowed `Serializable` class, it will throw. +If `thaw` encounters a disallowed `Serializable` class, it will: - Throw if it's not possible to safely quarantine the object (object was frozen with Nippy < v2.15.0-final). @@ -124,24 +124,24 @@ Allowlist values may be overridden with `binding`, `alter-var-root`, or: - - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-base` JVM property - - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-add` JVM property + - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-base` JVM property value + - `taoensso.nippy.<freeze/thaw>-serializable-allowlist-add` JVM property value - - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_BASE` env var - - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_ADD` env var + - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_BASE` Environment variable value + - `TAOENSSO_NIPPY_<FREEZE/THAW>_SERIALIZABLE_ALLOWLIST_ADD` Environment variable value If present, these will be read as comma-separated lists of class names and formed into sets. Each initial allowlist value will then be: (into (or <?base> <default>) <?additions>). I.e. you can use: - - The "base" property/var to replace Nippy's default allowlists. - - The "add" property/var to add to Nippy's default allowlists. + - The "base" property/var to REPLACE Nippy's default allowlists. + - The "add" property/var to ADD TO Nippy's default allowlists. The special `"allow-and-record"` value is also possible, see [2]. Upgrading from an older version of Nippy and unsure whether you've been -using Nippy's Serializable support, or which classes to allow? See [2]. +using Nippy's `Serializable` support, or which classes to allow? See [2]. See also `taoensso.encore/name-filter` for a util to help easily build more advanced predicate functions. @@ -150,7 +150,7 @@ identifying this vulnerability. [1] https://github.com/ptaoussanis/nippy/issues/130 -[2] See `allow-and-record-any-serializable-class-unsafe`.*thaw-xform*
dynamic
added in v3.3.0-RC1 (2023-08-02)
Experimental, subject to change. Feedback welcome. +[2] See `allow-and-record-any-serializable-class-unsafe`.*thaw-xform*
dynamic
added in v3.3.0-RC1 (2023-08-02)
Experimental, subject to change. Feedback welcome! Transducer to use when thawing standard Clojure collection types (vectors, maps, sets, lists, etc.). @@ -248,43 +248,43 @@ to `*freeze-serializable-allowlist*` and/or `*thaw-serializable-allowlist*` that: - - Will allow ANY class to use Nippy's Serializable support (unsafe). + - Will allow ANY class to use Nippy's `Serializable` support (unsafe). - And will record {<class-name> <frequency-allowed>} for the <=1000 classes that ~most frequently made use of this support. `get-recorded-serializable-classes` returns the recorded state. This predicate is provided as a convenience for users upgrading from -previous versions of Nippy that allowed the use of Serializable for all +previous versions of Nippy that allowed the use of `Serializable` for all classes by default. While transitioning from an unsafe->safe configuration, you can use this predicate (unsafe) to record information about which classes have -been using Nippy's Serializable support in your environment. +been using Nippy's `Serializable` support in your environment. Once some time has passed, you can check the recorded state. If you're -satisfied that all recorded classes are safely Serializable, you can +satisfied that all recorded classes are safely `Serializable`, you can then merge the recorded classes into Nippy's default allowlist/s, e.g.: (alter-var-root #'thaw-serializable-allowlist* (fn [_] (into default-thaw-serializable-allowlist - (keys (get-recorded-serializable-classes)))))cache
(cache x)Experimental, subject to change. Feedback welcome. + (keys (get-recorded-serializable-classes)))))cache
(cache x)Experimental, subject to change. Feedback welcome! Wraps value so that future writes of the same wrapped value with same metadata will be efficiently encoded as references to this one. (freeze [(cache "foo") (cache "foo") (cache "foo")]) - will incl. a single "foo", plus 2x single-byte references to "foo".compress
(compress compressor ba)decompress
(decompress compressor ba)decrypt
(decrypt encryptor pwd ba)default-freeze-serializable-allowlist
Allows *any* class-name to be frozen using Java's Serializable interface. + will incl. a single "foo", plus 2x single-byte references to "foo".compress
(compress compressor ba)decompress
(decompress compressor ba)decrypt
(decrypt encryptor pwd ba)default-freeze-serializable-allowlist
Allows *any* class-name to be frozen using Java's `Serializable` interface. This is generally safe since RCE risk is present only when thawing. See also `*freeze-serializable-allowlist*`.default-thaw-serializable-allowlist
A set of common safe class-names to allow to be frozen using Java's -Serializable interface. PRs welcome for additions. +`Serializable` interface. PRs welcome for additions. See also `*thaw-serializable-allowlist*`.encrypt
(encrypt encryptor pwd ba)extend-freeze
macro
(extend-freeze type custom-type-id [x out] & body)Extends Nippy to support freezing of a custom type (ideally concrete) with given id of form: * Keyword - 2 byte overhead, keywords hashed to 16 bit id * ℕ∈[1, 128] - 0 byte overhead -NB: be careful about extending to interfaces, Ref. http://goo.gl/6gGRlU. +NB: be careful about extending to interfaces, Ref. <http://goo.gl/6gGRlU>;. (defrecord MyRec [data]) (extend-freeze MyRec :foo/my-type [x data-output] ; Keyword id @@ -311,38 +311,43 @@ Equivalent to (but a little faster than) `thaw` with opts: - :compressor nil - :encryptor nil - - :no-header? truefreezable?
(freezable? x)(freezable? x {:keys [allow-clojure-reader? allow-java-serializable?]})Alpha - subject to change. -Returns truthy iff Nippy *appears* to support freezing the given argument. - -`:allow-clojure-reader?` and `:allow-java-serializable?` options may be -used to enable the relevant roundtrip fallback test(s). These tests are -only **moderately reliable** since they're cached by arg type and don't -test for pre/post serialization value equality (there's no good general -way of doing so).freeze
(freeze x)(freeze x {:as opts, :keys [compressor encryptor password serializable-allowlist incl-metadata?], :or {compressor :auto, encryptor aes128-gcm-encryptor}})Serializes arg (any Clojure data type) to a byte array. + - :no-header? truefreezable?
(freezable? x)(freezable? x {:as opts, :keys [recursive? allow-clojure-reader? allow-java-serializable?], :or {recursive? true}})Alpha, subject to change. +Returns ∈ #{:native :maybe-clojure-reader :maybe-java-serializable nil}, +truthy iff Nippy seems to support freezing for the given argument. + +Important: result can be inaccurate in some cases. To be completely sure you +unfortunately need to try freeze then thaw the argument, and check the thawed +value. + +Options include: + `recursive?` - Check recursively into given arg? + `allow-clojure-reader?` - Allow freezing with Clojure's reader? + `allow-java-serializable?` - Allow freezing with Java's `Serializable`?freeze
(freeze x)(freeze x {:as opts, :keys [compressor encryptor password serializable-allowlist incl-metadata?], :or {compressor :auto, encryptor aes128-gcm-encryptor}})Serializes arg (any Clojure data type) to a byte array. To freeze custom types, extend the Clojure reader or see `extend-freeze`.freeze-to-file
(freeze-to-file file x)(freeze-to-file file x freeze-opts)Convenience util: like `freeze`, but writes to `(clojure.java.io/file <file>)`.freeze-to-out!
(freeze-to-out! data-output x)Serializes arg (any Clojure data type) to a DataOutput. This is a low-level util: in most cases you'll want `freeze` instead.freeze-to-string
(freeze-to-string x)(freeze-to-string x freeze-opts)Convenience util: like `freeze`, but returns a Base64-encoded string. See also `thaw-from-string`.get-recorded-serializable-classes
(get-recorded-serializable-classes)Returns {<class-name> <frequency>} of the <=1000 classes that ~most -frequently made use of Nippy's Serializable support via +frequently made use of Nippy's `Serializable` support via `allow-and-record-any-serializable-class-unsafe`. -See that function's docstring for more info.IFreezable1
protocol
members
-freeze-without-meta!
(-freeze-without-meta! x data-output)IFreezable2
protocol
members
-freeze-with-meta!
(-freeze-with-meta! x data-output)inspect-ba
(inspect-ba ba)(inspect-ba ba thaw-opts)Experimental, subject to change. Feedback welcome. -lz4-compressor
Default net.jpountz.lz4 compressor: - Ratio: low. -Write speed: very high. - Read speed: very high. - -A good general-purpose compressor, competitive with Snappy. - -Thanks to Max Penet (@mpenet) for our first implementation, -Ref. https://github.com/mpenet/nippy-lz4lz4hc-compressor
Like `lz4-compressor` but trades some write speed for ratio. -lzma2-compressor
Default org.tukaani.xz.LZMA2 compressor: - Ratio: high. -Write speed: _very_ slow (also currently single-threaded). - Read speed: slow. - -A specialized compressor for large, low-write data in space-sensitive -environments.public-types-spec
Public representation of Nippy's internal type schema. +See that function's docstring for more info.IFreezable
protocol
Private implementation detail. +Protocol that types must implement to support native freezing by Nippy. +Don't use this directly, instead see `extend-freeze`.members
-freezable?
(-freezable? _)-freeze-without-meta!
(-freeze-without-meta! _ data-output)IFreezableWithMeta
protocol
Private implementation detail. +Wrapper protocol around `IFreezable` used to handle possible metadata.members
-freeze-with-meta!
(-freeze-with-meta! _ data-output)inspect-ba
(inspect-ba ba)(inspect-ba ba thaw-opts)Experimental, subject to change. Feedback welcome! +lz4-compressor
Default `LZ4` compressor: + - Compression ratio: `C` (0.58 on reference benchmark). + - Compression speed: `A` (238 msecs on reference benchmark). + - Decompression speed: `A+` (31 msecs on reference benchmark). + +Good general-purpose compressor, favours speed. +See `taoensso.nippy-benchmarks` for detailed comparative benchmarks.lz4hc-compressor
deprecated in v3.4.0-RC1 (2024-02-06)
Different LZ4 modes no longer supported, prefer `lz4-compressor`. +lzma2-compressor
Default `LZMA2` compressor: + - Compression ratio: `A+` (0.4 on reference benchmark). + - Compression speed: `E` (18.5 secs on reference benchmark). + - Decompression speed: `D` (11.8 secs on reference benchmark). + +Specialized compressor, strongly favours ratio. +See `taoensso.nippy-benchmarks` for detailed comparative benchmarks.public-types-spec
Public representation of Nippy's internal type schema. For use by tooling and advanced users. **HIGHLY EXPERIMENTAL!** @@ -385,14 +390,7 @@ **MAY BE UNSAFE!** Don't call this unless you absolutely trust the payload to not contain any malicious code. -See `*thaw-serializable-allowlist*` for more info.snappy-compressor
Default org.iq80.snappy.Snappy compressor: - Ratio: low. -Write speed: very high. - Read speed: very high. - -A good general-purpose compressor.stress-data
Reference data used for tests & benchmarks -stress-data-benchable
Reference data with stuff removed that breaks reader or other utils we'll -be benching with.stress-data-comparable
Reference data with stuff removed that breaks roundtrip equality. +See `*thaw-serializable-allowlist*` for more info.stress-data
(stress-data {:keys [comparable?], :as opts})Returns map of reference stress data for use by tests, benchmarks, etc.thaw
(thaw ba)(thaw ba {:as opts, :keys [v1-compatibility? compressor encryptor password serializable-allowlist incl-metadata? thaw-xform], :or {compressor :auto, encryptor :auto}})Deserializes a frozen Nippy byte array to its original Clojure data type. To thaw custom types, extend the Clojure reader or see `extend-thaw`. @@ -409,4 +407,10 @@ This is a low-level util: in most cases you'll want `thaw` instead.thaw-from-resource
(thaw-from-resource res)(thaw-from-resource res thaw-opts)Convenience util: like `thaw`, but reads from `(clojure.java.io/resource <res>)`.thaw-from-string
(thaw-from-string s)(thaw-from-string s thaw-opts)Convenience util: like `thaw`, but takes a Base64-encoded string. -See also `freeze-to-string`.throw-unfreezable
(throw-unfreezable x)try-write-readable
(try-write-readable out x)try-write-serializable
(try-write-serializable out x)write-id
macro
(write-id out id)write-unfreezable
(write-unfreezable out x)throw-unfreezable
(throw-unfreezable x)try-write-readable
(try-write-readable out x)try-write-serializable
(try-write-serializable out x)write-id
macro
(write-id out id)write-unfreezable
(write-unfreezable out x)zstd-compressor
Default `Zstd` (`Zstandard`) compressor: + - Compression ratio: `B` (0.53 on reference benchmark). + - Compression speed: `C` (1300 msecs on reference benchmark). + - Decompression speed: `B` (400 msecs on reference benchmark). + +Good general-purpose compressor, balances ratio & speed. +See `taoensso.nippy-benchmarks` for detailed comparative benchmarks.
Generated by Codox
Nippy 3.3.0
taoensso.nippy.tools
Utils for community tools that want to add user-configurable Nippy support. +taoensso.nippy.tools documentation Generated by Codox
Nippy 3.4.0-RC1
taoensso.nippy.tools
Utils for community tools that want to add user-configurable Nippy support. Used by Carmine, Faraday, etc.*freeze-opts*
dynamic
*thaw-opts*
dynamic
freeze
(freeze x)(freeze x default-opts)Like `nippy/freeze` but uses as options the following, merged in order of ascending preference: