diff --git a/docsrc/index.rst b/docsrc/index.rst index 357f8ca..8a30092 100644 --- a/docsrc/index.rst +++ b/docsrc/index.rst @@ -21,6 +21,7 @@ If you want to develop the code, there are some additional dependencies which ar /examples /phase_space_format /supported_particles + /limitations /code_docs Indices and tables diff --git a/docsrc/limitations.md b/docsrc/limitations.md new file mode 100644 index 0000000..968a098 --- /dev/null +++ b/docsrc/limitations.md @@ -0,0 +1,7 @@ +# Limitations + +The major limitation of this code at the time of writing is that it can only easily handle data that fits inside memory. This is partially a result of the choice of pandas as the backend - [this page](https://pandas.pydata.org/docs/user_guide/scale.html) describes some difficulties and solutions of handling large data with pandas. + +As is discussed in the above link: if your data is too big to fit inside RAM, it should be possible to read and process your data in 'chunks' where each chunk can fit inside memory. This is not supported in most data loaders, but should be possible with minimal extensions - open an issue and we can talk about it! + +Beyond this, libraries such as [DASK](https://www.dask.org/) may enable using this library of distributed resources. This is discussed a little bit in [this issue](https://github.com/bwheelz36/ParticlePhaseSpace/issues/158), with an example of utilising DASK on an OpenPMD dataset. \ No newline at end of file