docs/examples/otelib.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use OTEAPI-OPTIMADE with OTElib\n",
    "\n",
    "[OTElib](https://github.com/EMMC-ASBL/otelib) is a Python client library for the OTEAPI system.\n",
    "It has two usable backends:\n",
    "\n",
    "1. Python-based.\n",
    "   This uses the [OTEAPI Core](https://github.com/EMMC-ASBL/oteapi-core) library directly to exectue pipelines.\n",
    "   This backend should mainly be used for _development_ purposes.\n",
    "2. HTTP requests-based.\n",
    "   This uses the [OTEAPI Services](https://github.com/EMMC-ASBL/oteapi-services) library to execute pipelines via HTTP requests to a running OTEAPI server.\n",
    "   This backend should be used for _production_ purposes.\n",
    "\n",
    "This notebook demonstrates how to use OTEAPI-OPTIMADE with OTElib.\n",
    "\n",
    "## Example\n",
    "\n",
    "Using OTElib we will do different OPTIMADE queries.\n",
    "First, we will search through [the Materials Project](https://materialsproject.org) database for all structures with the formula `Al2O3`.\n",
    "Then, we will search through the Materials Project database for all structures that include `Al` and `O` in their chemical formula.\n",
    "Finally, we will search through [the Materials Cloud](https://materialscloud.org) [MC3D - Materials Cloud three-dimensional crystals database](https://mc3d.materialscloud.org/) for all structures that include `Al` and `O` in their chemical formula."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HTTP requests-based backend\n",
    "\n",
    "This backend is equivalent to running in production.\n",
    "It requires a running OTEAPI server.\n",
    "\n",
    "### Setup\n",
    "\n",
    "First, we need to have a running OTEAPI server.\n",
    "If one is not available, we can start one using [Docker](https://docker.com).\n",
    "See the [overview](/oteapi-optimade/examples/#setup-for-examples) for instructions on how to start a server.\n",
    "\n",
    "Further documentation about the OTEAPI Service is available as [a README on GitHub](https://github.com/EMMC-ASBL/oteapi-services#run-in-docker).\n",
    "\n",
    "<!-- The mkdocs-jupyter plugin does not support admonitions and similar stuff,\n",
    "so we need to write in the HTML directly here. -->\n",
    "<div class=\"admonition note\">\n",
    "  <p class=\"admonition-title\">Note</p>\n",
    "  <p>\n",
    "    It is advisable to run the OTEAPI server in a separate terminal window, so that we can see the logs from the server.\n",
    "    Furthermore, we can stop the server by pressing <code>Ctrl+C</code> in the terminal window.\n",
    "  </p>\n",
    "</div>\n",
    "\n",
    "After following the instructions, we should have a running OTEAPI server at [localhost:80](http://localhost:80/docs).\n",
    "\n",
    "### Create a client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from otelib import OTEClient\n",
    "\n",
    "client = OTEClient(\"http://localhost:80\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Search through Materials Project for all structures with the formula `Al2O3`\n",
    "\n",
    "To search through Materials Project for all structures with the formula `Al2O3`, we need to create the OTE strategies that we want to use for creating the OTE pipeline:\n",
    "\n",
    "<!-- The mkdocs-jupyter plugin creates a folder 'otelib' wherein it puts the generated HTML\n",
    "Therefore, we need to go up one more folder level to reference the image.\n",
    "One could use an \"absolute\" link, however, in production, mike will append the version to the path,\n",
    "which this link could not dynamically include. -->\n",
    "![Pipeline example 1](../../img/chem_formula_pipeline.png)\n",
    "\n",
    "To create the strategies, we need to know how to configure them.\n",
    "This information is available in the [OTEAPI Core documentation](https://emmc-asbl.github.io/oteapi-core), specifically for [Data Resource strategies](https://emmc-asbl.github.io/oteapi-core/latest/api_reference/models/resourceconfig/#oteapi.models.resourceconfig.ResourceConfig) and for [Filter strategies](https://emmc-asbl.github.io/oteapi-core/latest/api_reference/models/filterconfig/#oteapi.models.filterconfig.FilterConfig).\n",
    "\n",
    "For the data resource strategy, i.e., the OPTIMADE DB strategy, we need to know the base URL of the OPTIMADE API for Materials Project.\n",
    "OPTIMADE has a useful [providers dashboard](https://www.optimade.org/providers-dashboard/) that lists all known OPTIMADE providers and their (sub-)databases.\n",
    "Here we can find the base URL for [the Materials Project](https://www.optimade.org/providers-dashboard/providers/mp.html): `https://optimade.materialsproject.org`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_resource_strategy = client.create_dataresource(\n",
    "    accessService=\"OPTIMADE\",\n",
    "    accessUrl=\"https://optimade.materialsproject.org/\",\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the filter strategy, we need to know the OPTIMADE query that we want to execute.\n",
    "\n",
    "For retrieving all structures with the formula `Al2O3`, we can use the following OPTIMADE filter query:\n",
    "\n",
    "```\n",
    "chemical_formula_descriptive = \"Al2O3\" OR chemical_formula_reduced = \"Al2O3\" OR chemical_formula_hill = \"Al2O3\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter_strategy = client.create_filter(\n",
    "    filterType=\"OPTIMADE\",\n",
    "    query='chemical_formula_descriptive = \"Al2O3\" OR chemical_formula_reduced = \"Al2O3\" OR chemical_formula_hill = \"Al2O3\"',\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can create the OTE pipeline shown above and execute it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = filter_strategy >> data_resource_strategy\n",
    "session = pipeline.get()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The returned session is a JSON object we can parse and investigate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['optimade_resources', 'optimade_config', 'optimade_resource_model'])\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "parsed_session: dict = json.loads(session)\n",
    "print(parsed_session.keys())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As can be seen, there are three keys in the returned session.\n",
    "`optimade_config` summarizes the query that has been performed to Materials Project.\n",
    "The OPTIMADE structures are listed under the `optimade_resources`. It is named as such due to there being different OPTIMADE resources, e.g., `structures`, `references`, `links`, etc.\n",
    "The [OPTIMADE Python tools](https://github.com/Materials-Consortia/optimate-python-tools) has a useful [OPTIMADE Structure model class](http://www.optimade.org/optimade-python-tools/latest/api_reference/adapters/structures/adapter/#optimade.adapters.structures.adapter.Structure) that can be used to parse the OPTIMADE structures into Python objects as well as validating them according to the [OPTIMADE specification](https://github.com/Materials-Consortia/OPTIMADE/blob/master/optimade.rst).\n",
    "Again, since one can query for different OPTIMADE resources, the specific Python class to use is given in `optimade_resource_model`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The query resulted in 20 structures found (on page 1) of the returned data.\n",
      "Their Materials Project IDs are: ['mp-1228448', 'mp-755483', 'mp-1245081', 'mp-1244878', 'mp-1105018', 'mp-754401', 'mp-1245063', 'mp-1245265', 'mp-684591', 'mp-1244898', 'mp-1245211', 'mp-1245056', 'mp-642363', 'mp-1244930', 'mp-1244937', 'mp-1244967', 'mp-1244954', 'mp-1244874', 'mp-1245008', 'mp-1245023']\n"
     ]
    }
   ],
   "source": [
    "from importlib import import_module\n",
    "\n",
    "import_path, class_name = parsed_session[\"optimade_resource_model\"].split(\":\", maxsplit=1)\n",
    "ResourceClass = getattr(import_module(import_path), class_name)\n",
    "\n",
    "parsed_structures = [ResourceClass(structure) for structure in parsed_session[\"optimade_resources\"]]\n",
    "print(f\"The query resulted in {len(parsed_structures)} structures found (on page 1) of the returned data.\")\n",
    "print(f\"Their Materials Project IDs are: {[structure.id for structure in parsed_structures]}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find them on the Materials Project website, go to `materialsproject.org/materials/<ID>`, for example: [materialsproject.org/materials/mp-1228448](https://materialsproject.org/materials/mp-1228448).\n",
    "\n",
    "What is more, we can investigate the structure according to the [well-defined OPTIMADE structure model attributes](https://github.com/Materials-Consortia/OPTIMADE/blob/master/optimade.rst#82structures-entries).\n",
    "For example, so assert the chemical formula is what we expected, we can check the different chemical formula attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mp-1228448\n",
      "chemical_formula_descriptive: Al2O3\n",
      "chemical_formula_reduced: Al2O3\n",
      "chemical_formula_hill: Al2O3\n",
      "chemical_formula_anonymous: A3B2\n"
     ]
    }
   ],
   "source": [
    "structure = parsed_structures[0]\n",
    "print(structure.id)\n",
    "for attribute in (\"descriptive\", \"reduced\", \"hill\", \"anonymous\"):\n",
    "    print(f\"chemical_formula_{attribute}: {getattr(structure, f'chemical_formula_{attribute}', '(not defined)')}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Search through Materials Project for all structures that include `Al` and `O` in their chemical formula\n",
    "\n",
    "We must use a different OPTIMADE filter query for this search:\n",
    "\n",
    "```\n",
    "elements HAS ALL \"Al\",\"O\"\n",
    "```\n",
    "\n",
    "`elements` is a structure attribute that lists all elements in the structure.\n",
    "The `HAS ALL` operator matches if, for each value, there is at least one element in `elements` equal to that value. (This implements the set operator `>=`.)\n",
    "\n",
    "To do this search, we can reuse the OTE pipeline from the previous search, but change the filter strategy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter_strategy = client.create_filter(\n",
    "    filterType=\"OPTIMADE\",\n",
    "    query='elements HAS ALL \"Al\",\"O\"',\n",
    "    limit=5,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this query, we have added the `limit` parameter to the filter configuration, which will pass a `page_limit` query parameter to the OPTIMADE API ensuring that we only retrieve the first 5 structures (it limits each result's page to 5 resources).\n",
    "\n",
    "Let us investigate the result again, checking the list of elements and the chemical formula attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The query resulted in 5 structures found (on page 1) of the returned data.\n",
      "Their Materials Project IDs are: ['mp-1038042', 'mp-1182891', 'mp-1208627', 'mp-1247835', 'mp-1521059']\n"
     ]
    }
   ],
   "source": [
    "pipeline = filter_strategy >> data_resource_strategy\n",
    "parsed_session = json.loads(pipeline.get())\n",
    "print(f\"The query resulted in {len(parsed_session['optimade_resources'])} structures found (on page 1) of the returned data.\")\n",
    "\n",
    "import_path, class_name = parsed_session[\"optimade_resource_model\"].split(\":\", maxsplit=1)\n",
    "ResourceClass = getattr(import_module(import_path), class_name)\n",
    "structures = [ResourceClass(structure) for structure in parsed_session[\"optimade_resources\"]]\n",
    "print(f\"Their Materials Project IDs are: {[structure.id for structure in structures]}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us also check the query parameters used for the request to the OPTIMADE API to ensure that the `page_limit` query parameter was passed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'query_parameters': {'filter': 'elements HAS ALL \"Al\",\"O\"', 'page_limit': 5}}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parsed_session[\"optimade_config\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mp-1038042\n",
      "elements: ['Al', 'Cr', 'Mg', 'O']\n",
      "chemical_formula_descriptive: AlCrMg30O32\n",
      "chemical_formula_reduced: AlCrMg30O32\n",
      "chemical_formula_hill: AlCrMg30O32\n",
      "chemical_formula_anonymous: A32B30CD\n"
     ]
    }
   ],
   "source": [
    "structure = structures[0]\n",
    "print(structure.id)\n",
    "print(f\"elements: {structure.elements}\")\n",
    "for attribute in (\"descriptive\", \"reduced\", \"hill\", \"anonymous\"):\n",
    "    print(f\"chemical_formula_{attribute}: {getattr(structure, f'chemical_formula_{attribute}', '(not defined)')}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Search through MC3D database in Materials Cloud for all structures that include `Al` and `O` in their chemical formula\n",
    "\n",
    "Finally, let us reuse the same filter strategy we used for the previous search, but change the data resource strategy to point to the MC3D database in Materials Cloud.\n",
    "\n",
    "The base URL for the MC3D database is `https://aiida.materialscloud.org/mc3d/optimade` as found in the [providers dashboard](https://www.optimade.org/providers-dashboard/providers/mcloud.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_resource_strategy = client.create_dataresource(\n",
    "    accessService=\"OPTIMADE\",\n",
    "    accessUrl=\"https://aiida.materialscloud.org/mc3d/optimade\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The query resulted in 5 structures found (on page 1) of the returned data.\n",
      "Their Materials Cloud (AiiDA) IDs are: ['13952', '43703', '43800', '56101', '56270']\n"
     ]
    }
   ],
   "source": [
    "pipeline = filter_strategy >> data_resource_strategy\n",
    "parsed_session = json.loads(pipeline.get())\n",
    "print(f\"The query resulted in {len(parsed_session['optimade_resources'])} structures found (on page 1) of the returned data.\")\n",
    "\n",
    "import_path, class_name = parsed_session[\"optimade_resource_model\"].split(\":\", maxsplit=1)\n",
    "ResourceClass = getattr(import_module(import_path), class_name)\n",
    "structures = [ResourceClass(structure) for structure in parsed_session[\"optimade_resources\"]]\n",
    "print(f\"Their Materials Cloud (AiiDA) IDs are: {[structure.id for structure in structures]}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, let's check the query parameters used for the request to the OPTIMADE API to ensure it is equivalent to the previous search:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'query_parameters': {'filter': 'elements HAS ALL \"Al\",\"O\"', 'page_limit': 5}}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parsed_session[\"optimade_config\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13952\n",
      "elements: ['Al', 'O']\n",
      "chemical_formula_descriptive: Al2O6\n",
      "chemical_formula_reduced: AlO3\n",
      "chemical_formula_hill: Al2O6\n",
      "chemical_formula_anonymous: A3B\n"
     ]
    }
   ],
   "source": [
    "structure = structures[0]\n",
    "print(structure.id)\n",
    "print(f\"elements: {structure.elements}\")\n",
    "for attribute in (\"descriptive\", \"reduced\", \"hill\", \"anonymous\"):\n",
    "    print(f\"chemical_formula_{attribute}: {getattr(structure, f'chemical_formula_{attribute}', '(not defined)')}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MC3D structures can, unfortunately, not be found easily in the Materials Cloud website, as the ID in the DISCOVER section does not match the ID in the OPTIMADE structure.\n",
    "However, the full OPTIMADE structure can be found at `<OPTIMADE_BASE_URL>/structures/<ID>`, for example: [aiida.materialscloud.org/mc3d/optimade/structures/13952](https://aiida.materialscloud.org/mc3d/optimade/structures/13952).\n",
    "\n",
    "<!-- The mkdocs-jupyter plugin does not support admonitions and similar stuff,\n",
    "so we need to write in the HTML directly here. -->\n",
    "<div class=\"admonition note\">\n",
    "  <p class=\"admonition-title\">Note</p>\n",
    "  <p>\n",
    "    The structure with the OPTIMADE ID 13952 is found with the Materials Cloud DISCOVER ID <code>mc3d-76896</code> and can be found <a href=\"https://mc3d.materialscloud.org/#/details/Al2O6/mc3d-76896/pbe\">here</a>.\n",
    "  </p>\n",
    "</div>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python-based backend\n",
    "\n",
    "With this backend, we do not need a running OTEAPI server.\n",
    "\n",
    "### Create a client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = OTEClient(\"python\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can go through the same searches as we did with the HTTP requests-based backend.\n",
    "The result should not change.\n",
    "\n",
    "### Search through Materials Project for all structures with the formula `Al2O3`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setting filter from query.\n",
      "Setting filter from query.\n",
      "resource_config: OPTIMADEResourceConfig(user=None, password=None, token=None, client_id=None, client_secret=None, configuration=OPTIMADEConfig(version='v1', endpoint='structures', query_parameters=OPTIMADEQueryParameters(filter='chemical_formula_descriptive = \"Al2O3\" OR chemical_formula_reduced = \"Al2O3\" OR chemical_formula_hill = \"Al2O3\"', response_format='json', email_address='', response_fields='', sort='', page_limit=20, page_offset=0, page_number=None, page_cursor=0, page_above=None, page_below=None, include='references', api_hint=''), datacache_config=DataCacheConfig(cacheDir=PosixPath('oteapi'), accessKey=None, hashType='md5', expireTime=86400, tag='optimade'), return_object=False, use_dlite=False), description='Resource Strategy Data Configuration.\\n\\n    Important:\\n        Either of the pairs of attributes `downloadUrl`/`mediaType` or\\n        `accessUrl`/`accessService` MUST be specified.\\n\\n    ', downloadUrl=None, mediaType=None, accessUrl=OPTIMADEUrl('https://optimade.materialsproject.org/', base_url='https://optimade.materialsproject.org', scheme='https', tld='org', host_type='domain'), accessService='OPTIMADE', license=None, accessRights=None, publisher=None)\n",
      "resource_config: OPTIMADEResourceConfig(user=None, password=None, token=None, client_id=None, client_secret=None, configuration=OPTIMADEConfig(version='v1', endpoint='structures', query_parameters=OPTIMADEQueryParameters(filter='chemical_formula_descriptive = \"Al2O3\" OR chemical_formula_reduced = \"Al2O3\" OR chemical_formula_hill = \"Al2O3\"', response_format='json', email_address='', response_fields='', sort='', page_limit=20, page_offset=0, page_number=None, page_cursor=0, page_above=None, page_below=None, include='references', api_hint=''), datacache_config=DataCacheConfig(cacheDir=PosixPath('oteapi'), accessKey=None, hashType='md5', expireTime=86400, tag='optimade'), return_object=False, use_dlite=False), description='Resource Strategy Data Configuration.\\n\\n    Important:\\n        Either of the pairs of attributes `downloadUrl`/`mediaType` or\\n        `accessUrl`/`accessService` MUST be specified.\\n\\n    ', downloadUrl=None, mediaType=None, accessUrl=OPTIMADEUrl('https://optimade.materialsproject.org/', base_url='https://optimade.materialsproject.org', scheme='https', tld='org', host_type='domain'), accessService='OPTIMADE', license=None, accessRights=None, publisher=None)\n",
      "optimade_query after update: OPTIMADEQueryParameters(filter='chemical_formula_descriptive = \"Al2O3\" OR chemical_formula_reduced = \"Al2O3\" OR chemical_formula_hill = \"Al2O3\"', response_format='json', email_address='', response_fields='', sort='', page_limit=20, page_offset=0, page_number=None, page_cursor=0, page_above=None, page_below=None, include='references', api_hint='')\n",
      "optimade_query after update: OPTIMADEQueryParameters(filter='chemical_formula_descriptive = \"Al2O3\" OR chemical_formula_reduced = \"Al2O3\" OR chemical_formula_hill = \"Al2O3\"', response_format='json', email_address='', response_fields='', sort='', page_limit=20, page_offset=0, page_number=None, page_cursor=0, page_above=None, page_below=None, include='references', api_hint='')\n",
      "OPTIMADE URL to be requested: https://optimade.materialsproject.org/v1/structures?filter=chemical_formula_descriptive%20%3D%20%22Al2O3%22%20OR%20chemical_formula_reduced%20%3D%20%22Al2O3%22%20OR%20chemical_formula_hill%20%3D%20%22Al2O3%22&response_format=json&page_limit=20&include=references\n",
      "OPTIMADE URL to be requested: https://optimade.materialsproject.org/v1/structures?filter=chemical_formula_descriptive%20%3D%20%22Al2O3%22%20OR%20chemical_formula_reduced%20%3D%20%22Al2O3%22%20OR%20chemical_formula_hill%20%3D%20%22Al2O3%22&response_format=json&page_limit=20&include=references\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/cwa/.venvs/oteapi-optimade/lib/python3.9/site-packages/optimade/server/config.py:113: UserWarning: Unable to find config file at /home/cwa/.optimade.json, using the default settings instead.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "data_resource_strategy = client.create_dataresource(\n",
    "    accessService=\"OPTIMADE\",\n",
    "    accessUrl=\"https://optimade.materialsproject.org/\",\n",
    ")\n",
    "\n",
    "filter_strategy = client.create_filter(\n",
    "    filterType=\"OPTIMADE\",\n",
    "    query='chemical_formula_descriptive = \"Al2O3\" OR chemical_formula_reduced = \"Al2O3\" OR chemical_formula_hill = \"Al2O3\"',\n",
    ")\n",
    "\n",
    "pipeline = filter_strategy >> data_resource_strategy\n",
    "session = pipeline.get()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As one can see, this backend runs locally within the same Python environment as the notebook. This is useful for development purposes, where the logging messages are shown directly in the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['optimade_config', 'optimade_resources', 'optimade_resource_model'])\n"
     ]
    }
   ],
   "source": [
    "parsed_session: dict = json.loads(session)\n",
    "print(parsed_session.keys())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get the same keys in the returned session as we did with the HTTP requests-based backend.\n",
    "If we again import the OPTIMADE Structure model class from the OPTIMADE Python tools, we can parse the OPTIMADE structures into Python objects as well as validating them according to the OPTIMADE specification, just as we did with the HTTP requests-based backend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The query resulted in 20 structures found (on page 1) of the returned data.\n",
      "Their Materials Project IDs are: ['mp-1228448', 'mp-755483', 'mp-1245081', 'mp-1244878', 'mp-1105018', 'mp-754401', 'mp-1245063', 'mp-1245265', 'mp-684591', 'mp-1244898', 'mp-1245211', 'mp-1245056', 'mp-642363', 'mp-1244930', 'mp-1244937', 'mp-1244967', 'mp-1244954', 'mp-1244874', 'mp-1245008', 'mp-1245023']\n",
      "mp-1228448\n",
      "chemical_formula_descriptive: Al2O3\n",
      "chemical_formula_reduced: Al2O3\n",
      "chemical_formula_hill: Al2O3\n",
      "chemical_formula_anonymous: A3B2\n"
     ]
    }
   ],
   "source": [
    "import_path, class_name = parsed_session[\"optimade_resource_model\"].split(\":\", maxsplit=1)\n",
    "ResourceClass = getattr(import_module(import_path), class_name)\n",
    "\n",
    "parsed_structures = [ResourceClass(structure) for structure in parsed_session[\"optimade_resources\"]]\n",
    "print(f\"The query resulted in {len(parsed_structures)} structures found (on page 1) of the returned data.\")\n",
    "print(f\"Their Materials Project IDs are: {[structure.id for structure in parsed_structures]}\")\n",
    "\n",
    "structure = parsed_structures[0]\n",
    "print(structure.id)\n",
    "for attribute in (\"descriptive\", \"reduced\", \"hill\", \"anonymous\"):\n",
    "    print(f\"chemical_formula_{attribute}: {getattr(structure, f'chemical_formula_{attribute}', '(not defined)')}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Search through Materials Project for all structures that include `Al` and `O` in their chemical formula"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setting filter from query.\n",
      "Setting filter from query.\n",
      "Setting filter from query.\n",
      "Setting page_limit from limit.\n",
      "Setting page_limit from limit.\n",
      "Setting page_limit from limit.\n",
      "resource_config: OPTIMADEResourceConfig(user=None, password=None, token=None, client_id=None, client_secret=None, configuration=OPTIMADEConfig(version='v1', endpoint='structures', query_parameters=OPTIMADEQueryParameters(filter='elements HAS ALL \"Al\",\"O\"', response_format='json', email_address='', response_fields='', sort='', page_limit=5, page_offset=0, page_number=None, page_cursor=0, page_above=None, page_below=None, include='references', api_hint=''), datacache_config=DataCacheConfig(cacheDir=PosixPath('oteapi'), accessKey=None, hashType='md5', expireTime=86400, tag='optimade'), return_object=False, use_dlite=False), description='Resource Strategy Data Configuration.\\n\\n    Important:\\n        Either of the pairs of attributes `downloadUrl`/`mediaType` or\\n        `accessUrl`/`accessService` MUST be specified.\\n\\n    ', downloadUrl=None, mediaType=None, accessUrl=OPTIMADEUrl('https://optimade.materialsproject.org/', base_url='https://optimade.materialsproject.org', scheme='https', tld='org', host_type='domain'), accessService='OPTIMADE', license=None, accessRights=None, publisher=None)\n",
      "resource_config: OPTIMADEResourceConfig(user=None, password=None, token=None, client_id=None, client_secret=None, configuration=OPTIMADEConfig(version='v1', endpoint='structures', query_parameters=OPTIMADEQueryParameters(filter='elements HAS ALL \"Al\",\"O\"', response_format='json', email_address='', response_fields='', sort='', page_limit=5, page_offset=0, page_number=None, page_cursor=0, page_above=None, page_below=None, include='references', api_hint=''), datacache_config=DataCacheConfig(cacheDir=PosixPath('oteapi'), accessKey=None, hashType='md5', expireTime=86400, tag='optimade'), return_object=False, use_dlite=False), description='Resource Strategy Data Configuration.\\n\\n    Important:\\n        Either of the pairs of attributes `downloadUrl`/`mediaType` or\\n        `accessUrl`/`accessService` MUST be specified.\\n\\n    ', downloadUrl=None, mediaType=None, accessUrl=OPTIMADEUrl('https://optimade.materialsproject.org/', base_url='https://optimade.materialsproject.org', scheme='https', tld='org', host_type='domain'), accessService='OPTIMADE', license=None, accessRights=None, publisher=None)\n",
      "resource_config: OPTIMADEResourceConfig(user=None, password=None, token=None, client_id=None, client_secret=None, configuration=OPTIMADEConfig(version='v1', endpoint='structures', query_parameters=OPTIMADEQueryParameters(filter='elements HAS ALL \"Al\",\"O\"', response_format='json', email_address='', response_fields='', sort='', page_limit=5, page_offset=0, page_number=None, page_cursor=0, page_above=None, page_below=None, include='references', api_hint=''), datacache_config=DataCacheConfig(cacheDir=PosixPath('oteapi'), accessKey=None, hashType='md5', expireTime=86400, tag='optimade'), return_object=False, use_dlite=False), description='Resource Strategy Data Configuration.\\n\\n    Important:\\n        Either of the pairs of attributes `downloadUrl`/`mediaType` or\\n        `accessUrl`/`accessService` MUST be specified.\\n\\n    ', downloadUrl=None, mediaType=None, accessUrl=OPTIMADEUrl('https://optimade.materialsproject.org/', base_url='https://optimade.materialsproject.org', scheme='https', tld='org', host_type='domain'), accessService='OPTIMADE', license=None, accessRights=None, publisher=None)\n",
      "optimade_query after update: OPTIMADEQueryParameters(filter='elements HAS ALL \"Al\",\"O\"', response_format='json', email_address='', response_fields='', sort='', page_limit=5, page_offset=0, page_number=None, page_cursor=0, page_above=None, page_below=None, include='references', api_hint='')\n",
      "optimade_query after update: OPTIMADEQueryParameters(filter='elements HAS ALL \"Al\",\"O\"', response_format='json', email_address='', response_fields='', sort='', page_limit=5, page_offset=0, page_number=None, page_cursor=0, page_above=None, page_below=None, include='references', api_hint='')\n",
      "optimade_query after update: OPTIMADEQueryParameters(filter='elements HAS ALL \"Al\",\"O\"', response_format='json', email_address='', response_fields='', sort='', page_limit=5, page_offset=0, page_number=None, page_cursor=0, page_above=None, page_below=None, include='references', api_hint='')\n",
      "OPTIMADE URL to be requested: https://optimade.materialsproject.org/v1/structures?filter=elements%20HAS%20ALL%20%22Al%22%2C%22O%22&response_format=json&page_limit=5&include=references\n",
      "OPTIMADE URL to be requested: https://optimade.materialsproject.org/v1/structures?filter=elements%20HAS%20ALL%20%22Al%22%2C%22O%22&response_format=json&page_limit=5&include=references\n",
      "OPTIMADE URL to be requested: https://optimade.materialsproject.org/v1/structures?filter=elements%20HAS%20ALL%20%22Al%22%2C%22O%22&response_format=json&page_limit=5&include=references\n",
      "The query resulted in 5 structures found (on page 1) of the returned data.\n",
      "Their Materials Project IDs are: ['mp-1038042', 'mp-1182891', 'mp-1208627', 'mp-1247835', 'mp-1521059']\n"
     ]
    }
   ],
   "source": [
    "filter_strategy = client.create_filter(\n",
    "    filterType=\"OPTIMADE\",\n",
    "    query='elements HAS ALL \"Al\",\"O\"',\n",
    "    limit=5,\n",
    ")\n",
    "\n",
    "pipeline = filter_strategy >> data_resource_strategy\n",
    "parsed_session = json.loads(pipeline.get())\n",
    "print(f\"The query resulted in {len(parsed_session['optimade_resources'])} structures found (on page 1) of the returned data.\")\n",
    "\n",
    "import_path, class_name = parsed_session[\"optimade_resource_model\"].split(\":\", maxsplit=1)\n",
    "ResourceClass = getattr(import_module(import_path), class_name)\n",
    "structures = [ResourceClass(structure) for structure in parsed_session[\"optimade_resources\"]]\n",
    "print(f\"Their Materials Project IDs are: {[structure.id for structure in structures]}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The configuration in `optimade_config` is quite more extensive than with the HTTP requests-based backend, as it includes more information, automatically setting the default values so they are shown in the config:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'version': 'v1',\n",
       " 'endpoint': 'structures',\n",
       " 'query_parameters': {'filter': 'elements HAS ALL \"Al\",\"O\"', 'page_limit': 5},\n",
       " 'datacache_config': {'cacheDir': 'oteapi',\n",
       "  'accessKey': None,\n",
       "  'hashType': 'md5',\n",
       "  'expireTime': 86400,\n",
       "  'tag': 'optimade'},\n",
       " 'return_object': False,\n",
       " 'use_dlite': False}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parsed_session[\"optimade_config\"]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the first structure again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mp-1038042\n",
      "elements: ['Al', 'Cr', 'Mg', 'O']\n",
      "chemical_formula_descriptive: AlCrMg30O32\n",
      "chemical_formula_reduced: AlCrMg30O32\n",
      "chemical_formula_hill: AlCrMg30O32\n",
      "chemical_formula_anonymous: A32B30CD\n"
     ]
    }
   ],
   "source": [
    "structure = structures[0]\n",
    "print(structure.id)\n",
    "print(f\"elements: {structure.elements}\")\n",
    "for attribute in (\"descriptive\", \"reduced\", \"hill\", \"anonymous\"):\n",
    "    print(f\"chemical_formula_{attribute}: {getattr(structure, f'chemical_formula_{attribute}', '(not defined)')}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Search through MC3D database in Materials Cloud for all structures that include `Al` and `O` in their chemical formula"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setting filter from query.\n",
      "Setting filter from query.\n",
      "Setting filter from query.\n",
      "Setting page_limit from limit.\n",
      "Setting page_limit from limit.\n",
      "Setting page_limit from limit.\n",
      "resource_config: OPTIMADEResourceConfig(user=None, password=None, token=None, client_id=None, client_secret=None, configuration=OPTIMADEConfig(version='v1', endpoint='structures', query_parameters=OPTIMADEQueryParameters(filter='elements HAS ALL \"Al\",\"O\"', response_format='json', email_address='', response_fields='', sort='', page_limit=5, page_offset=0, page_number=None, page_cursor=0, page_above=None, page_below=None, include='references', api_hint=''), datacache_config=DataCacheConfig(cacheDir=PosixPath('oteapi'), accessKey=None, hashType='md5', expireTime=86400, tag='optimade'), return_object=False, use_dlite=False), description='Resource Strategy Data Configuration.\\n\\n    Important:\\n        Either of the pairs of attributes `downloadUrl`/`mediaType` or\\n        `accessUrl`/`accessService` MUST be specified.\\n\\n    ', downloadUrl=None, mediaType=None, accessUrl=OPTIMADEUrl('https://aiida.materialscloud.org/mc3d/optimade', base_url='https://aiida.materialscloud.org/mc3d/optimade', scheme='https', tld='org', host_type='domain'), accessService='OPTIMADE', license=None, accessRights=None, publisher=None)\n",
      "resource_config: OPTIMADEResourceConfig(user=None, password=None, token=None, client_id=None, client_secret=None, configuration=OPTIMADEConfig(version='v1', endpoint='structures', query_parameters=OPTIMADEQueryParameters(filter='elements HAS ALL \"Al\",\"O\"', response_format='json', email_address='', response_fields='', sort='', page_limit=5, page_offset=0, page_number=None, page_cursor=0, page_above=None, page_below=None, include='references', api_hint=''), datacache_config=DataCacheConfig(cacheDir=PosixPath('oteapi'), accessKey=None, hashType='md5', expireTime=86400, tag='optimade'), return_object=False, use_dlite=False), description='Resource Strategy Data Configuration.\\n\\n    Important:\\n        Either of the pairs of attributes `downloadUrl`/`mediaType` or\\n        `accessUrl`/`accessService` MUST be specified.\\n\\n    ', downloadUrl=None, mediaType=None, accessUrl=OPTIMADEUrl('https://aiida.materialscloud.org/mc3d/optimade', base_url='https://aiida.materialscloud.org/mc3d/optimade', scheme='https', tld='org', host_type='domain'), accessService='OPTIMADE', license=None, accessRights=None, publisher=None)\n",
      "resource_config: OPTIMADEResourceConfig(user=None, password=None, token=None, client_id=None, client_secret=None, configuration=OPTIMADEConfig(version='v1', endpoint='structures', query_parameters=OPTIMADEQueryParameters(filter='elements HAS ALL \"Al\",\"O\"', response_format='json', email_address='', response_fields='', sort='', page_limit=5, page_offset=0, page_number=None, page_cursor=0, page_above=None, page_below=None, include='references', api_hint=''), datacache_config=DataCacheConfig(cacheDir=PosixPath('oteapi'), accessKey=None, hashType='md5', expireTime=86400, tag='optimade'), return_object=False, use_dlite=False), description='Resource Strategy Data Configuration.\\n\\n    Important:\\n        Either of the pairs of attributes `downloadUrl`/`mediaType` or\\n        `accessUrl`/`accessService` MUST be specified.\\n\\n    ', downloadUrl=None, mediaType=None, accessUrl=OPTIMADEUrl('https://aiida.materialscloud.org/mc3d/optimade', base_url='https://aiida.materialscloud.org/mc3d/optimade', scheme='https', tld='org', host_type='domain'), accessService='OPTIMADE', license=None, accessRights=None, publisher=None)\n",
      "optimade_query after update: OPTIMADEQueryParameters(filter='elements HAS ALL \"Al\",\"O\"', response_format='json', email_address='', response_fields='', sort='', page_limit=5, page_offset=0, page_number=None, page_cursor=0, page_above=None, page_below=None, include='references', api_hint='')\n",
      "optimade_query after update: OPTIMADEQueryParameters(filter='elements HAS ALL \"Al\",\"O\"', response_format='json', email_address='', response_fields='', sort='', page_limit=5, page_offset=0, page_number=None, page_cursor=0, page_above=None, page_below=None, include='references', api_hint='')\n",
      "optimade_query after update: OPTIMADEQueryParameters(filter='elements HAS ALL \"Al\",\"O\"', response_format='json', email_address='', response_fields='', sort='', page_limit=5, page_offset=0, page_number=None, page_cursor=0, page_above=None, page_below=None, include='references', api_hint='')\n",
      "OPTIMADE URL to be requested: https://aiida.materialscloud.org/mc3d/optimade/v1/structures?filter=elements%20HAS%20ALL%20%22Al%22%2C%22O%22&response_format=json&page_limit=5&include=references\n",
      "OPTIMADE URL to be requested: https://aiida.materialscloud.org/mc3d/optimade/v1/structures?filter=elements%20HAS%20ALL%20%22Al%22%2C%22O%22&response_format=json&page_limit=5&include=references\n",
      "OPTIMADE URL to be requested: https://aiida.materialscloud.org/mc3d/optimade/v1/structures?filter=elements%20HAS%20ALL%20%22Al%22%2C%22O%22&response_format=json&page_limit=5&include=references\n",
      "The query resulted in 5 structures found (on page 1) of the returned data.\n",
      "Their Materials Cloud (AiiDA) IDs are: ['13952', '43703', '43800', '56101', '56270']\n",
      "{'version': 'v1', 'endpoint': 'structures', 'query_parameters': {'filter': 'elements HAS ALL \"Al\",\"O\"', 'page_limit': 5}, 'datacache_config': {'cacheDir': 'oteapi', 'accessKey': None, 'hashType': 'md5', 'expireTime': 86400, 'tag': 'optimade'}, 'return_object': False, 'use_dlite': False}\n",
      "13952\n",
      "elements: ['Al', 'O']\n",
      "chemical_formula_descriptive: Al2O6\n",
      "chemical_formula_reduced: AlO3\n",
      "chemical_formula_hill: Al2O6\n",
      "chemical_formula_anonymous: A3B\n"
     ]
    }
   ],
   "source": [
    "data_resource_strategy = client.create_dataresource(\n",
    "    accessService=\"OPTIMADE\",\n",
    "    accessUrl=\"https://aiida.materialscloud.org/mc3d/optimade\",\n",
    ")\n",
    "\n",
    "pipeline = filter_strategy >> data_resource_strategy\n",
    "parsed_session = json.loads(pipeline.get())\n",
    "print(f\"The query resulted in {len(parsed_session['optimade_resources'])} structures found (on page 1) of the returned data.\")\n",
    "\n",
    "import_path, class_name = parsed_session[\"optimade_resource_model\"].split(\":\", maxsplit=1)\n",
    "ResourceClass = getattr(import_module(import_path), class_name)\n",
    "structures = [ResourceClass(structure) for structure in parsed_session[\"optimade_resources\"]]\n",
    "print(f\"Their Materials Cloud (AiiDA) IDs are: {[structure.id for structure in structures]}\")\n",
    "\n",
    "print(parsed_session[\"optimade_config\"])\n",
    "\n",
    "structure = structures[0]\n",
    "print(structure.id)\n",
    "print(f\"elements: {structure.elements}\")\n",
    "for attribute in (\"descriptive\", \"reduced\", \"hill\", \"anonymous\"):\n",
    "    print(f\"chemical_formula_{attribute}: {getattr(structure, f'chemical_formula_{attribute}', '(not defined)')}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "oteapi-optimade",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}