High_Level_Introduction_to_ASDF.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The What and Why of ASDF (Advanced Scientific Data Format)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a brief summary of the ASDF format. Other tutorials will go into greater detail about the rationales for it, and the details of the internal format. This notebook will only summarize these items briefly. This will also give a brief listing of some of the features of ASDF, particularly with regard to FITS. Finally we will list some of the tutorial resources available (as they become available). **Note the list of tutorials at the end of this notebook.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the reasons we developed ASDF instead of using FITS\n",
    "\n",
    "- 8 character limitation on keyword sizes\n",
    "- Keyword, value, comment restriction to 80 characters\n",
    "- Lack of any widely accepted mechanism for hierarchical organization of metadata or data\n",
    "- Poor support for widely used binary types (particularly unsigned 16 bit integers)\n",
    "- Lack of versioning of the standard\n",
    "- Lack of good validation tools\n",
    "- Allowance of many nonstandard FITS files that libraries must support\n",
    "- Lack of streaming support\n",
    "- Inflexible data models for cloud use\n",
    "- Very poor support for complex World Coordinate System transforms (a very large headache for HST data)\n",
    "\n",
    "The last item is a topic in its own right and probably the single biggest factor in developing ASDF.\n",
    "\n",
    "Other existing formats were looked at and all had serious flaws. (See xxx for more details)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What to retain from FITS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Readable metadata\n",
    "- Support for binary data, including arrays and tables\n",
    "- Useful as an archival format (i.e., not just a software library)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## High Level ASDF File Structure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An ASDF file consists of a single YAML header, followed by zero or more binary blocks.\n",
    "\n",
    "YAML (Yet Another Markup Language) is a standard text format that is easier to read than XML and more concise and readable than JSON. It is generally intended that the YAML header will hold all the metadata for the data contained within the file. It also is used to indicate the organization of all the data and how it is associated with relevant metadata."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Added Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- No limit on the length of attribute names (analogous to FITS keyword names).\n",
    "- No limit on attribute values, which may theselves be complex values. This allows arbitrarily nested structures.\n",
    "- Ability to \"tag\" values as having a special type that software may know how to specially handle.\n",
    "- Ability to have different attributes reference the same data or object without making copies, so that data or object can be shared by many things.\n",
    "- Full support for all standard numerical binary values.\n",
    "- Supports mode of putting some or all data \"in-line\", i.e., as text values.\n",
    "- Supports very flexible WCS model.\n",
    "- Mechanism for extensions to the standard format, both in the standard itself, and for local purposes.\n",
    "- Versioned globally, and for specific extensions.\n",
    "- Support for schema files that can be used to validate the proper structure of the file, both for reading and writing.\n",
    "- Support for streamed data.\n",
    "- Language agnostic, but currently only fully supported in Python\n",
    "\n",
    "Many of these items value may not yet be obvious to the reader, but tutorials will illustrate  the value with examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FITS Features not yet Supported"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Variable length records in tables (using heap storage)\n",
    "- Random Groups"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites to Tutorials\n",
    "\n",
    "### Installations\n",
    "\n",
    "Of course Python 3.6 or later must be installed, along with ipython, jupyter, numpy, astropy, gwcs, and asdf. We generally recommend installing Python using miniconda since conda allows easy cloning of different Python environments ([miniconda download](https://docs.conda.io/en/latest/miniconda.html) and using pip install for the rest after activating the miniconda environment. For example, at the shell level:\n",
    "\n",
    "`\n",
    "pip install ipython\n",
    "pip install jupyter\n",
    "pip install numpy\n",
    "pip install astropy\n",
    "pip install asdf\n",
    "pip install gwcs\n",
    "pip install matplotlib\n",
    "`\n",
    "### Data Files\n",
    "\n",
    "The tutorials generally require accessing sample ASDF files. The tutorials usually do this by referring to files on the network that will be downloaded if not already present on your system. The needed downloads will be part of the tutorials."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tutorials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Anatomy of an ASDF file](Anatomy_of_an_ASDF_file.ipynb): Basic explantion of ASDF file structure. Recommended if starting from the beginning with no particular ASDF data to deal with.\n",
    "- [Reading a JWST ASDF file](Reading_a_JWST_ASDF_file.ipynb): Addresses an example JWST ASDF file. Most useful place to start if given a JWST file and want to quickly be able to examine its contents. More useful examples of looking at a more complex file than in the above tutorial.\n",
    "- [ASDF and World Coordinate Systems](ASDF_and_World_Coordinate_Systems.ipynb): Examples of using Generalized World Coordinate Systems saved and read from ASDF files. Should start with one of the two above tutorials.\n",
    "- [Validation and dealing with errors](Validation_and_Dealing_with_Errors.ipynb) How to deal with errors that may arise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References and Documentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Original ASDF Paper](https://www.sciencedirect.com/science/article/pii/S2213133715000645)\n",
    "- [Python ASDF Library Documenation](https://asdf.readthedocs.io/en/2.7.1/)\n",
    "- [ASDF Standard Documentation](https://asdf-standard.readthedocs.io/en/1.5.0/)\n",
    "- [asdf-users mailing list](https://groups.google.com/g/asdf-users)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}