-
Not sure if it is error? or limitation of my hardware who does have more than logs(venv) C:\LLaMA-Factory>set CUDA_VISIBLE_DEVICES=0 && llamafactory-cli train --stage sft --do_train True --model_name_or_path ./checkpoints/Meta-Llama-3.1-8B-Instruct --preprocessing_num_workers 16 --finetuning_type lora --template alpaca --rope_scaling linear --flash_attn fa2 --dataset_dir data --dataset alpaca-gpt4_de,dolly_15k_de --cutoff_len 8192 --learning_rate 5e-05 --num_train_epochs 3.0 --max_samples 100000 --per_device_train_batch_size 1 --gradient_accumulation_steps 1 --lr_scheduler_type cosine --max_grad_norm 1.0 --logging_steps 100 --save_steps 1000 --warmup_steps 1000 --neftune_noise_alpha 5 --optim adamw_8bit --packing True --neat_packing True --report_to none --output_dir saves\LLaMA3.1-8B-Chat\lora\Llama-3.1-8B-German --bf16 True --plot_loss True --ddp_timeout 180000000 --include_num_input_tokens_seen True --quantization_bit 4 --quantization_method bitsandbytes --lora_rank 32 --lora_alpha 64 --lora_dropout 0.15 --lora_target all --use_adam_mini True --create_new_adapter True
[2024-09-02 17:14:53,212] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:14:53.310000 44360 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
09/02/2024 17:14:54 - WARNING - llamafactory.hparams.parser - We recommend enable `upcast_layernorm` in quantized training.
09/02/2024 17:14:54 - INFO - llamafactory.hparams.parser - Process rank: 0, device: cuda:0, n_gpu: 1, distributed training: False, compute dtype: torch.bfloat16
[INFO|tokenization_utils_base.py:2287] 2024-09-02 17:14:54,745 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2287] 2024-09-02 17:14:54,745 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2287] 2024-09-02 17:14:54,745 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2287] 2024-09-02 17:14:54,745 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2533] 2024-09-02 17:14:54,908 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
09/02/2024 17:14:54 - INFO - llamafactory.data.template - Add pad token: <|eot_id|>
09/02/2024 17:14:54 - INFO - llamafactory.data.loader - Loading dataset mayflowergmbh/alpaca-gpt4_de...
Converting format of dataset (num_proc=16): 0%| | 0/49969 [00:00<?, ? examples/s][2024-09-02 17:15:01,474] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:01.624000 37672 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:01,653] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:01,680] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:01,714] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:01.805000 25584 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:01,823] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:01,823] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:01.838000 36424 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:01,868] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:01.874000 43136 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:01,895] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:01,907] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:01,932] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:01.979000 42220 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:01.980000 36228 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:02.028000 20600 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:02.062000 40316 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:02,072] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:02,072] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:02,077] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:02.079000 38524 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:02,089] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:02.096000 44716 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:02,099] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:02,122] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:02.243000 43264 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:02.245000 38144 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:02.256000 43156 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:02.264000 45892 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:02.278000 45000 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:02.307000 28624 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
Converting format of dataset (num_proc=16): 100%|███████████████████████| 49969/49969 [00:08<00:00, 6126.38 examples/s]
09/02/2024 17:15:04 - INFO - llamafactory.data.loader - Loading dataset mayflowergmbh/dolly-15k_de...
Converting format of dataset (num_proc=16): 0%| | 0/15011 [00:00<?, ? examples/s][2024-09-02 17:15:11,430] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:11,559] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:11,565] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:11.589000 41356 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:11,591] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:11,640] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:11.720000 28812 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:11.723000 39296 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:11.753000 42288 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:11.803000 41844 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:11,913] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:11,916] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:11,965] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:12,063] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:12,069] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:12.082000 20664 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:12.082000 26616 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:12,096] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:12,102] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:12,118] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:12,118] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:12,122] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:12.137000 18232 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:12,156] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:12.245000 41692 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:12.255000 25836 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:12.287000 38360 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:12.298000 18908 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:12.313000 41644 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:12.316000 44620 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:12.325000 45684 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:12.348000 42368 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
Converting format of dataset (num_proc=16): 100%|███████████████████████| 15011/15011 [00:08<00:00, 1836.04 examples/s]
Running tokenizer on dataset (num_proc=16): 0%| | 0/64980 [00:00<?, ? examples/s][2024-09-02 17:15:19,425] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:19.582000 34144 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:19,632] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:19.795000 37672 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:19,915] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:20.086000 45776 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:20,107] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:20,272] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:20.275000 35644 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:20,342] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:20.431000 39224 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:20.497000 43732 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:20,575] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:20.749000 45648 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:20,828] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:20.996000 42452 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:21,146] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:21.308000 44460 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:21,343] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:21,357] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:21.513000 46024 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:21.527000 40524 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:21,555] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:21.710000 42244 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:21,719] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:21,817] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0902 17:15:21.884000 7128 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:21.987000 41088 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
[2024-09-02 17:15:22,204] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-02 17:15:22,237] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Running tokenizer on dataset (num_proc=16): 2%|▍ | 1000/64980 [00:07<07:45, 137.49 examples/s]W0902 17:15:22.368000 34296 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
W0902 17:15:22.404000 1380 torch\distributed\elastic\multiprocessing\redirects.py:27] NOTE: Redirects are currently not supported in Windows or MacOs.
Running tokenizer on dataset (num_proc=16): 84%|███████████████████▎ | 54674/64980 [00:12<00:02, 4303.89 examples/s]
multiprocess.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "C:\LLaMA-Factory\venv\lib\site-packages\multiprocess\pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "C:\LLaMA-Factory\venv\lib\site-packages\datasets\utils\py_utils.py", line 678, in _write_generator_to_queue
for i, result in enumerate(func(**kwargs)):
File "C:\LLaMA-Factory\venv\lib\site-packages\datasets\arrow_dataset.py", line 3575, in _map_single
writer.write_batch(batch)
File "C:\LLaMA-Factory\venv\lib\site-packages\datasets\arrow_writer.py", line 568, in write_batch
arrays.append(pa.array(typed_sequence))
File "pyarrow\\array.pxi", line 248, in pyarrow.lib.array
File "pyarrow\\array.pxi", line 112, in pyarrow.lib._handle_arrow_array_protocol
File "C:\LLaMA-Factory\venv\lib\site-packages\datasets\arrow_writer.py", line 208, in __arrow_array__
out = cast_array_to_feature(
File "C:\LLaMA-Factory\venv\lib\site-packages\datasets\table.py", line 1804, in wrapper
return func(array, *args, **kwargs)
File "C:\LLaMA-Factory\venv\lib\site-packages\datasets\table.py", line 2075, in cast_array_to_feature
casted_array_values = _c(array.values, feature.feature)
File "C:\LLaMA-Factory\venv\lib\site-packages\datasets\table.py", line 1804, in wrapper
return func(array, *args, **kwargs)
File "C:\LLaMA-Factory\venv\lib\site-packages\datasets\table.py", line 2116, in cast_array_to_feature
return array_cast(
File "C:\LLaMA-Factory\venv\lib\site-packages\datasets\table.py", line 1804, in wrapper
return func(array, *args, **kwargs)
File "C:\LLaMA-Factory\venv\lib\site-packages\datasets\table.py", line 1963, in array_cast
return array.cast(pa_type)
File "pyarrow\\array.pxi", line 1000, in pyarrow.lib.Array.cast
File "C:\LLaMA-Factory\venv\lib\site-packages\pyarrow\compute.py", line 405, in cast
return call_function("cast", [arr], options, memory_pool)
File "pyarrow\\_compute.pyx", line 590, in pyarrow._compute.call_function
File "pyarrow\\_compute.pyx", line 385, in pyarrow._compute.Function.call
File "pyarrow\\error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow\\error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Integer value 128 not in range: -128 to 127
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\LLaMA-Factory\venv\Scripts\llamafactory-cli.exe\__main__.py", line 7, in <module>
sys.exit(main())
File "C:\LLaMA-Factory\src\llamafactory\cli.py", line 111, in main
run_exp()
File "C:\LLaMA-Factory\src\llamafactory\train\tuner.py", line 50, in run_exp
run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
File "C:\LLaMA-Factory\src\llamafactory\train\sft\workflow.py", line 46, in run_sft
dataset_module = get_dataset(model_args, data_args, training_args, stage="sft", **tokenizer_module)
File "C:\LLaMA-Factory\src\llamafactory\data\loader.py", line 237, in get_dataset
dataset = _get_preprocessed_dataset(
File "C:\LLaMA-Factory\src\llamafactory\data\loader.py", line 183, in _get_preprocessed_dataset
dataset = dataset.map(preprocess_func, batched=True, remove_columns=column_names, **kwargs)
File "C:\LLaMA-Factory\venv\lib\site-packages\datasets\arrow_dataset.py", line 602, in wrapper
out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
File "C:\LLaMA-Factory\venv\lib\site-packages\datasets\arrow_dataset.py", line 567, in wrapper
out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
File "C:\LLaMA-Factory\venv\lib\site-packages\datasets\arrow_dataset.py", line 3253, in map
for rank, done, content in iflatmap_unordered(
File "C:\LLaMA-Factory\venv\lib\site-packages\datasets\utils\py_utils.py", line 718, in iflatmap_unordered
[async_result.get(timeout=0.05) for async_result in async_results]
File "C:\LLaMA-Factory\venv\lib\site-packages\datasets\utils\py_utils.py", line 718, in <listcomp>
[async_result.get(timeout=0.05) for async_result in async_results]
File "C:\LLaMA-Factory\venv\lib\site-packages\multiprocess\pool.py", line 774, in get
raise self._value
pyarrow.lib.ArrowInvalid: Integer value 128 not in range: -128 to 127
My solution was to change |
Beta Was this translation helpful? Give feedback.
Answered by
Katehuuh
Sep 16, 2024
Replies: 1 comment 2 replies
-
which dataset? |
Beta Was this translation helpful? Give feedback.
2 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Stale issue - Solution I provide is ok.