This awesome list is dedicated to resources for using Large Language Models (LLMs) to generate JSON or other structured outputs.
- Terminology
- Hosted Models
- Local Models
- Python Libraries
- Blog Articles
- Videos
- Jupyter Notebooks
- Leaderboards
Unfortunately, generating JSON goes by a few different names that roughly mean the same thing:
- Structured Outputs: Using an LLM to generate any structured output including JSON, XML, or YAML regardless of technique (e.g. function calling, guided generation).
- Function Calling: Providing an LLM a hypothetical (or actual) function definition for it to "call" in it's chat or completion response. The LLM doesn't actually call the function, it just provides an indication that one should be called via a JSON message.
- JSON Mode: Specifying that an LLM must generate valid JSON. Depending on the provider, a schema may or may not be specified and the LLM may create an unexpected schema.
- Tool Usage: Giving an LLM a choice of tools such as image generation, web search, and "function calling". The functional calling parameter in the API request is now called "tools".
- Guided Generation: For constraining an LLM model to generate text that follows a prescribed specification such as a Context-Free Grammar.
- GPT Actions: ChatGPT invokes actions (i.e. API calls) based on the endpoints and parameters specified in an OpenAPI specification. Unlike the capability called "Function Calling", this capability will indeed call your function hosted by an API server.
None of these names are great, that's why I named this list just "Awesome LLM JSON".
| Provider | Models | Links |
|---|---|---|
| Anthropic | claude-3-opus-20240229 claude-3-sonnet-20240229 claude-3-haiku-20240307 |
API Docs Pricing |
| AnyScale | Mistral-7B-Instruct-v0.1 Mixtral-8x7B-Instruct-v0.1 |
Function Calling JSON Mode Pricing Announcement (2023) |
| Azure | gpt-4 gpt-4-turbo gpt-35-turbo mistral-large-latest mistral-large-2402 |
Function Calling OpenAI Pricing Mistral Pricing |
| Cohere | Command-R Command-R+ |
Function Calling Pricing Command-R (2024-03-11) Command-R+ (2024-04-04) |
| Fireworks.ai | firefunction-v1 | Function Calling JSON Mode Grammar mode Pricing Announcement (2023-12-20) |
| gemini-1.0-pro | Function Calling Pricing |
|
| Groq | llama2-70b mixtral-8x7b gemma-7b-it |
Function Calling Pricing |
| Hugging Face TGI | many open-source models | Grammars, JSON mode, Function Calling and Tools For free locally, or via dedicated or serverless endpoints. |
| Mistral | mistral-large-latest | Function Calling Pricing |
| OpenAI | gpt-4 gpt-4-turbo gpt-35-turbo |
Function Calling JSON Mode Pricing Announcement (2023-06-13) |
| Rysana | inversion-sm | API Docs Pricing Announcement (2024-03-18) |
| Together AI | Mixtral-8x7B-Instruct-v0.1 Mistral-7B-Instruct-v0.1 CodeLlama-34b-Instruct |
Function Calling JSON Mode Pricing Announcement 2024-01-31 |
Parallel Function Calling
Below is a list of hosted API models that support multiple parallel function calls. This could include checking the weather in multiple cities or first finding the location of a hotel and then checking the weather at it's location.
- anthropic
- claude-3-opus-20240229
- claude-3-sonnet-20240229
- claude-3-haiku-20240307
- azure/openai
- gpt-4-turbo-preview
- gpt-4-1106-preview
- gpt-4-0125-preview
- gpt-3.5-turbo-1106
- gpt-3.5-turbo-0125
- cohere
- command-r
- together_ai
- Mixtral-8x7B-Instruct-v0.1
- Mistral-7B-Instruct-v0.1
- CodeLlama-34b-Instruct
Mistral 7B Instruct v0.3 (2024-05-22, Apache 2.0) an instruct fine-tuned version of Mistral with added function calling support.
C4AI Command R+ (2024-03-20, CC-BY-NC, Cohere) is a 104B parameter multilingual model with advanced Retrieval Augmented Generation (RAG) and tool use capabilities, optimized for reasoning, summarization, and question answering across 10 languages. Supports quantization for efficient use and demonstrates unique multi-step tool integration for complex task execution.
Hermes 2 Pro - Mistral 7B (2024-03-13, Nous Research) is a 7B parameter model that excels at function calling, JSON structured outputs, and general tasks. Trained on an updated OpenHermes 2.5 Dataset and a new function calling dataset, it uses a special system prompt and multi-turn structure. Achieves 91% on function calling and 84% on JSON mode evaluations.
Gorilla OpenFunctions v2 (2024-02-27, Apache 2.0 license, Charlie Cheng-Jie Ji et al.) interprets and executes functions based on JSON Schema Objects, supporting multiple languages and detecting function relevance.
NexusRaven-V2 (2023-12-05, Nexusflow) is a 13B model outperforming GPT-4 in zero-shot function calling by up to 7%, enabling effective use of software tools. Further instruction-tuned on CodeLlama-13B-instruct.
Functionary (2023-08-04, MeetKai) interprets and executes functions based on JSON Schema Objects, supporting various compute requirements and call types. Compatible with OpenAI-python and llama-cpp-python for efficient function execution in JSON generation tasks.
Hugging Face TGI enables JSON outputs and function calling for a variety of local models.
DSPy (MIT) is a framework for algorithmically optimizing LM prompts and weights. DSPy introduced typed predictor and signatures to leverage Pydantic for enforcing type constraints on inputs and outputs, improving upon string-based fields.
FuzzTypes (MIT) extends Pydantic with autocorrecting annotation types for enhanced data normalization and handling of complex types like emails, dates, and custom entities.
guidance (Apache-2.0) enables constrained generation, interleaving Python logic with LLM calls, reusable functions, and calling external tools. Optimizes prompts for faster generation.
Instructor (MIT) simplifies generating structured data from LLMs using Function Calling, Tool Calling, and constrained sampling modes. Built on Pydantic for validation and supports various LLMs.
LangChain (MIT) provides an interface for chains, integrations with other tools, and chains for applications. LangChain offers chains for structured outputs and function calling across models.
LiteLLM (MIT) simplifies calling 100+ LLMs in the OpenAI format, supporting function calling, tool calling, and JSON mode.
LlamaIndex (MIT) provides modules for structured outputs at different levels of abstraction, including output parsers for text completion endpoints, Pydantic programs for mapping prompts to structured outputs using function calling or output parsing, and pre-defined Pydantic programs for specific output types.
Marvin (Apache-2.0) is a lightweight toolkit for building reliable natural language interfaces with self-documenting tools for tasks like entity extraction and multi-modal support.
Outlines (Apache-2.0) facilitates structured text generation using multiple models, Jinja templating, and support for regex patterns, JSON schemas, Pydantic models, and context-free grammars.
Pydantic (MIT) simplifies working with data structures and JSON through data model definition, validation, JSON schema generation, and seamless parsing and serialization.
SGLang (MPL-2.0) allows specifying JSON schemas using regular expressions or Pydantic models for constrained decoding. Its high-performance runtime accelerates JSON decoding.
SynCode (MIT) is a framework for the grammar-guided generation of Large Language Models (LLMs). It supports CFG for Python, Go, Java, JSON, YAML, and many more.
Mirascope (MIT) is an LLM toolkit that supports structured extraction with an intuitive python API.
Magnetic (MIT) call LLMs from Python using 3 lines of code. Simply use the @prompt decorator to create functions that return structured output from the LLM, powered by Pydantic.
Formatron (MIT) is an efficient and scalable constrained decoding library that enables controlling over language model output format using f-string templates that support regular expressions, context-free grammars, JSON schemas, and Pydantic models. Formatron integrates seamlessly with various model inference libraries.
Transformers-cfg (MIT) extends Hugging Face Transformers with context-free grammar (CFG) support via an EBNF interface. It enables grammar-constrained generation with minimal changes to existing code of transformers and supports JSON mode and JSON Schema.
How fast can grammar-structured generation be? (2024-04-12, .txt Engineering) demonstrates an almost cost-free method to generate text that follows a grammar. It is shown to outperform llama.cpp by a factor of 50x on the C grammar.
Structured Generation Improves LLM performance: GSM8K Benchmark (2024-03-15, .txt Engineering) demonstrates consistent improvements across 8 models, highlighting benefits like "prompt consistency" and "thought-control."
LoRAX + Outlines: Better JSON Extraction with Structured Generation and LoRA (2024-03-03, Predibase Blog) combines Outlines with LoRAX v0.8 to enhance extraction accuracy and schema fidelity through structured generation, fine-tuning, and LoRA adapters.
FU, Show Me The Prompt. Quickly understand inscrutable LLM frameworks by intercepting API calls (2023-02-14, Hamel Husain) provides a practical guide to intercepting API calls using mitmproxy, gaining insights into tool functionality, and assessing necessity. Emphasizes minimizing complexity and maintaining close connection with underlying LLMs.
Coalescence: making LLM inference 5x faster (2024-02-02, .txt Engineering) shows how structured generation can be made faster than unstructured generation using a technique called "coalescence", with a caveat regarding how it may affect the quality of the generation.
Why Pydantic became indispensable for LLMs (2024-01-19, Adam Azzam) explains Pydantic's emergence as a critical tool, enabling sharing data models via JSON schemas and reasoning between unstructured and structured data. Highlights the importance of quantizing the decision space and potential issues with LLMs overfitting to older schema versions.
Getting Started with Function Calling (2024-01-11, Elvis Saravia) introduces function calling for connecting LLMs with external tools and APIs, providing an example using OpenAI's API and highlighting potential applications.
Pushing ChatGPT's Structured Data Support To Its Limits (2023-12-21, Max Woolf) delves into leveraging ChatGPT's capabilities using paid API, JSON schemas, and Pydantic. Highlights techniques for improving output quality and the benefits of structured data support.
Why use Instructor? (2023-11-18, Jason Liu) explains the benefits of the library, offering a readable approach, support for partial extraction and various types, and a self-correcting mechanism. Recommends additional resources on the Instructor website.
Using grammars to constrain llama.cpp output (2023-09-06, Ian Maurer) integrates context-free grammars with llama.cpp for more accurate and schema-compliant responses, particularly for biomedical data.
Using OpenAI functions and their Python library for data extraction (2023-07-09, Simon Willison) demonstrates extracting structured data using OpenAI Python library and function calling in a single API call, with a code example and suggestions for handling streaming limitations.
GPT Extracting Unstructured Data with Datasette and GPT-4 Turbo (2024-04-09, Simon Willison) showcases the datasette-extract plugin's ability to populate database tables from unstructured text and images, leveraging GPT-4 Turbo's API for data extraction.
LLM Structured Output for Function Calling with Ollama (2024-03-25, Andrej Baranovskij) demonstrates function calling-based data extraction using Ollama, Instructor and Sparrow agent.
Hermes 2 Pro Overview (2024-03-18, Prompt Engineer) introduces Hermes 2 Pro, a 7B parameter model excelling at function calling and structured JSON output. Demonstrates 90% accuracy in function calling and 84% in JSON mode, outperforming other models.
Mistral AI Function Calling (2024-02-24, Sophia Yang) demonstrates connecting LLMs to external tools, generating function arguments, and executing functions. Could be extended to generate or manipulate JSON data.
Function Calling in Ollama vs OpenAI (2024-02-13, Matt Williams) clarifies that models generate structured output for parsing and invoking functions. Compares implementations, highlighting Ollama's simpler approach and using few-shot prompts for consistency.
LLM Engineering: Structured Outputs (2024-02-12, Jason Liu, Weights & Biases Course) offers a concise course on handling structured JSON output, function calling, and validations using Pydantic, covering essentials for robust pipelines and efficient production integration.
Pydantic is all you need (2023-10-10, Jason Liu, AI Engineer Conference) discusses the importance of Pydantic for structured prompting and output validation, introducing the Instructor library and showcasing advanced applications for reliable and maintainable LLM-powered applications.
Function Calling with llama-cpp-python and OpenAI Python Client demonstrates integration, including setup using the Instructor library, with examples of retrieving weather information and extracting user details.
Function Calling with Mistral Models demonstrates connecting Mistral models with external tools through a simple example involving a payment transactions dataframe.
chatgpt-structured-data by Max Woolf provides demos showcasing ChatGPT's function calling and structured data support, covering various use cases and schemas.## Leaderboards
Berkeley Function-Calling Leaderboard (BFCL) is an evaluation framework for LLMs' function-calling capabilities including over 2k question-function-answer pairs across languages like Python, Java, JavaScript, SQL, and REST API, focusing on simple, multiple, and parallel function calls, as well as function relevance detection.