Code

Fixed memory issues during forward#5

Code

Merged

ischlag merged 41 commits intoswiss-ai:mainswiss-ai/nanotron:mainfrom AleHD:fix_tp_mem_cacheAleHD/nanotron:fix_tp_mem_cacheCopy head branch name to clipboardSep 2, 2024

+317-57

Commits

Commits on Apr 22, 2024

• readme

  zzhhjjj

  committed
  33fd672

Commits on May 14, 2024

• wip

  C-TC

  committed
  e484d99

• layer recompute

  C-TC

  committed
  7e15516

Commits on May 16, 2024

• fix row parallel

  C-TC

  committed
  7dd5beb

Commits on May 27, 2024

• Add 1-sqrt function for the cooldown phase.

  eliebak

  committed
  e4d3010

• fix typo

  eliebak

  committed
  180faf4

Commits on May 29, 2024

• Merge pull request huggingface#185 from eliebak/add-decay-function

  Show description for c721f4f
  3outeille

  authored
  c721f4f

• add rope_theta to hf conversion script

  jquesnelle

  authored
  97c9780

Commits on May 30, 2024

• Merge pull request huggingface#188 from jquesnelle/patch-1

  Show description for 6771639
  NouamaneTazi

  authored
  6771639

Commits on Jun 10, 2024

• feat(ci): add trufflehog secrets detection

  McPatate

  committed
  1753921

• fix(ci): remove unnecessary permissions

  McPatate

  committed
  1db85f3

• Merge pull request huggingface#193 from huggingface/feat/add_trufflehog_ci

  Show description for ee785d6
  xrsrke

  authored
  ee785d6

Commits on Jun 27, 2024

• Implemented global memory buffer to reduce activation memory of differentiable distributed operations

  AleHD

  committed
  bcf405d

• GLU fusion

  AleHD

  committed
  ed1ca7d

• precommit

  AleHD

  committed
  9b0de5b

Commits on Jul 8, 2024

• Update README.md

  xrsrke

  authored
  ed5a11c

• Merge pull request huggingface#205 from huggingface/xrsrke-patch-2

  Show description for cb51ed8
  xrsrke

  authored
  cb51ed8

• Update README.md

  xrsrke

  authored
  d5cf7c4

• Merge pull request huggingface#206 from huggingface/xrsrke-patch-3

  Show description for f1adf52
  xrsrke

  authored
  f1adf52

Commits on Jul 11, 2024

• Merge branch 'main' into fix_tp_mem_cache

  AleHD

  committed
  bbc259f

Commits on Jul 14, 2024

• Merge pull request huggingface#207 from C-TC/recompute

  Show description for 4c23ed0
  3outeille

  authored
  4c23ed0

Commits on Jul 16, 2024

• Wrong backward fixed

  AleHD

  committed
  803b6da

• Removed useless prints

  AleHD

  committed
  59bfb6b

Commits on Jul 17, 2024

• Minor fixes

  AleHD

  committed
  2c69e9a

• precommit

  AleHD

  committed
  30439fd

• Added tp_recompute_allgather option

  AleHD

  committed
  1e02a9c

• Changed recompute default

  AleHD

  committed
  9cc81bb

• Changed recompute default

  AleHD

  committed
  956fbfd

• Moved ColumnLinearNoAsync module for consistency

  AleHD

  committed
  b9e9201

Commits on Jul 21, 2024

• Merge pull request huggingface#145 from zzhhjjj/readme

  Show description for 5f82f7a
  zzhhjjj

  authored
  5f82f7a

Commits on Jul 23, 2024

• Merge branch 'main' into fix_tp_mem_cache

  AleHD

  committed
  49633df

• Fixed List not found

  AleHD

  committed
  2afd007

• Fixed tp=1 case

  AleHD

  committed
  7e758db

Commits on Jul 29, 2024

• fix row parallel

  

  C-TC

  authored and

  3outeille

  committed
  41f11f0

• Merge branch 'fix-row-parallel' of https://github.com/c-tc/nanotron into fix-row-parallel

  3outeille

  committed
  b713f7b

Commits on Jul 30, 2024

• Merge pull request huggingface#172 from C-TC/fix-row-parallel

  Show description for 2e48d66
  3outeille

  authored
  2e48d66

• Merge branch 'main' into fix_tp_mem_cache

  AleHD

  committed
  ce2a96b

• Fixed column parallel

  AleHD

  committed
  cd84d4f

• Added tp_recompute_allgather test

  AleHD

  committed
  d3db06a

Commits on Aug 2, 2024

• Minor restyling

  AleHD

  committed
  7daa186

• Fixed names

  AleHD

  committed
  31c3c5a