Reading Next.js turbopack

[hi-ogawa]

todo

• how is turbopack integrated to next.js code base?
  • via git-based crate https://github.com/vercel/next.js/blob/6604c187ece3021e9b429ecf1207f34d41efbe0a/Cargo.toml#L40-L45
  • napi-based @next/swc npm package exposes turbopack-binding
• check packages/next/src/server/dev/hot-reloader-turbopack.ts
• check react-server-dom-turbopack
• notable PRs?
  • Move next-* crates from Turbopack vercel/next.js#47019, Remove the next-* crates vercel/turborepo#4176
  • Turbopack: App Router build POC vercel/next.js#52036
  • Turobpack: Next.rs API (part 1) vercel/next.js#52259
• https://nextjs.org/docs/app/api-reference/next-config-js/turbo
• https://turbo.build/pack/docs
• https://github.com/vercel/turbo/blob/8ef95da2a4c8a0ca009a43f6dc82f04c4c9360cc/crates/turbopack/examples/turbopack.rs
• how to dev ssr?
  • still cjs output + require.cache invalidation?

turbopack basics

• ModuleAssetContext
  • layer (like webpack?)
  • transitions (see "client reference discovery" below for example)

    Some kind of operation that is executed during reference processing. e.g. you can transition to a different environment on a specific import (reference).

  • CompileTimeInfo
    • environment (e.g. nodejs, edge, browser, ...)
    • defines
  • ModuleOptionsContext
    • transform options (e.g. ts, tsx, mdx, ...)
    • rules (like webpack?)
  • ResolveOptionsContext
    • conditions, extensions, etc...

next-build-test (mini example)

https://github.com/vercel/next.js/tree/83a085acc74d431f48a223f9b5f4c133b379be46/packages/next-swc/crates/next-build-test

main ->
  TurboTasks::new
  main_inner ->
    ProjectContainer::new
    ProjectContainer::entrypoints -> Project::entrypoints ->
      AppProject::routes ->
        app_entrypoints -> get_entrypoints -> directory_tree_to_entrypoints (crawling for fs route)
        app_entry_point_to_route -> ...
      middleware_endpoint -> middleware_context -> ModuleAssetContext::new (layer "middleware")
      Entrypoints (routes, middleware, ...)
    render_routes -> Endpoint::write_to_disk for each route e.g.
      AppEndpoint::write_to_disk ->
        output ->
          app_page_entry ->
            rsc_module_context ->
               ModuleAssetContext::new with (layer "app-rsc" and client_reference_transition, ssr_transition, etc...)
            get_app_page_entry (next-core crate) -> 
              LoaderTreeModule::build -> ... ModuleAssetContext::process ??
          client_reference_graph (from rsc entry) ->
            collect client references using module graph Visitor utility from turbopack
          ClientReferenceManifest::build_output
          create_font_manifest
          create_server_actions_manifest
        emit_all_output_assets
    hmr -> ??

client reference discovery (transition example)

• ClientDirectiveTransformer
• NextEcmascriptClientReferenceTransition
• EcmascriptClientReferenceProxyModule
• EcmascriptClientReferenceModule

//// page.js (server component processed in `rsc_module_context`)
import Counter from "./counter.js";

//// counter.js (client component original source)
"use client"
export default function Counter() { ... }

//// counter.js (ClientDirectiveTransformer::transform in `rsc_module_context`)
import * as clientProxy from "./couter.js" with { TURBOPACK_HELPER: true, ANNOTATION_TRANSITION: "client_reference_transition" }
__turbopack_export_namespace__(clientProxy)

// then 
// custom `import with` is analyzed by turbopack (namely crates/turbopack-ecmascript/src/analyzer/imports.rs)
// and eventually client_reference_transition.process (i.e. NextEcmascriptClientReferenceTransition.process)
// will process "./counter.js" module into two different contexts
// - `client_module_context` (for browser) and 
// - "app-ssr" layer (for ssr)

AppProject::rsc_module_context ->
  ModuleAssetContext::new with
     layer "app-rsc"
     client_reference_transition
     ssr_transition
     rsc_module_options_context -> get_server_module_options_context ->
       get_next_server_transforms_rules (basic shared rules)
       then if ServerContextType::AppRSC
         add ClientDirectiveTransformer::new(ecmascript_client_reference_transition_name)
         ...

(note that this transformer is provided as turbopack-ecmascript-plugins)
ClientDirectiveTransformer::transform ->         
  if is_client_module (i.e. "use client"), then create_proxy_module ->
    generate code like
      import * as clientProxy from "..." with { TURBOPACK_HELPER: true, ANNOTATION_TRANSITION: client transition }
      __turbopack_export_namespace__(clientProxy)

AppProject::client_reference_transition -> 
  NextEcmascriptClientReferenceTransition::new with
    client_transition -> FullContextTransition::new(client_module_context)
    ssr_transition -> ContextTransition::new (with "app-ssr" layer)

NextEcmascriptClientReferenceTransition.process ->
   EcmascriptClientReferenceProxyModule::new with modules from 
     client_transition.process
     ssr_transition.process

EcmascriptClientReferenceProxyModule::references ->
  proxy_module -> `registerClientReference` transform for client_module.get_exports
  EcmascriptClientReferenceModule::new for ssr_module

dev

 ... -> setupDevBundler -> startWatcher ->
   createHotReloaderTurbopack -> 
     loadBindings -> ??
     bindings.turbo.createProject -> binding.projectNew ->
       (packages/next-swc/crates/napi/src/next_api/project.rs) project_new -> ??
     project.entrypointsSubscribe -> (napi) project_entrypoints_subscribe -> ??
     handleEntrypointsSubscription -> handleEntrypoints -> ??
     handleProjectUpdates -> ??
     also implement NextJsHotReloaderInterface e.g. ensurePage