diff --git a/book/_toc.yml b/book/_toc.yml index d4ca5f2..aac1d29 100644 --- a/book/_toc.yml +++ b/book/_toc.yml @@ -72,6 +72,8 @@ parts: - file: framework/explanations/types - file: framework/explanations/formats - file: framework/explanations/provenance + - file: framework/explanations/garbage-collection + - file: framework/explanations/metaprogramming - file: framework/references/intro sections: - file: framework/references/archive-versions diff --git a/book/framework/explanations/garbage-collection.md b/book/framework/explanations/garbage-collection.md new file mode 100644 index 0000000..7450511 --- /dev/null +++ b/book/framework/explanations/garbage-collection.md @@ -0,0 +1,21 @@ +(garbage-collection-explanation)= +# Garbage Collection + +This page does not offer any particularly deep or useful answers, but seeks to serve as a warning that this part of QIIME 2 is *difficult* to understand. +If you think you have a better or more robust way to manage resource allocation, we would love to hear from you! + +Because QIIME 2 {term}`Artifacts` are really directory structures, synchronizing filesystem state with memory can be difficult. +It is further excacerbated by the lifetimes of the objects being unkown. +For example viewing an artifact can produce anything from a in-memory object to an entirely new directory structure. + +Currently, QIIME 2 chooses to tie the filesystem state directly with an objects lifetime in memory. +This means that instead of allocating a location and automatically destroying it with a static lifetime (e.g. RAII/context manager) it is handled dynamically. +It is up to the Python garbage collector to invoke a destructor that cleans the filesystem when appropriate. +This pushes the issue off from managing lifetimes, to ensuring that destruction occurs and that the data *should* be destroyed (e.g. it is not user-input). + +The objects responsible for managing filesystem paths (and destroying them) are located in `qiime2.core.path`. + +Additionally it is not always the case that the garbage collecter *will* call a destructor. +For example in a multiprocessing context, `sys._exit` is called instead of `sys.exit`, meaning that any filesystem objects created in the child-process will not be cleaned up. +Exceptions are another way in which normal cleanup can become confounded. +Fortunately there is an additional object `qiime2.sdk.context:Context` which can be used to juggle this information and can provide a context-manager for when a static lifetime is known. diff --git a/book/framework/explanations/metaprogramming.md b/book/framework/explanations/metaprogramming.md new file mode 100644 index 0000000..e9a808e --- /dev/null +++ b/book/framework/explanations/metaprogramming.md @@ -0,0 +1,11 @@ +(metaprogramming-explanation)= +# Metaprogramming + +The framework uses a significant amount of runtime metaprogramming in which properties and methods are swapped out or constructed at the last minute based on plugin information. +This list shows some of the aspects of the Python language that are used to achieve this: + +- decorators (used everywhere, `qiime2.sdk.action:Action` is really just a decorator-object) +- descriptor protocol (used in `qiime2.core.util:LateBindingAttribute`) +- import hooks (used by Artifact API) +- metaclasses (used by `qiime2.plugin.model.directory_format`) +- eval (`qiime2.skd.util:parse_type` and `decorator` package for signature rewriting)