在裸金属服务器中部署CANN环境,包括包:
Ascend-hdk-910-npu-driver_24.1.rc1_linux-aarch64.run
Ascend-hdk-910-npu-firmware_7.1.0.6.220.run
Ascend-cann-kernels-910_8.0.RC1_linux.run
Ascend-cann-toolkit_8.0.RC1_linux-aarch64.run
在昇腾社区找到这四个包后传入服务器开始安装。
在安装之前需要给到四个文件权限: chmod +x Ascend*
安装顺序: npu-driver -> npu-firmware -> cann-toolkit -> cann-kernels
其中npu使用--full安装 cann使用--install安装
./Ascend-hdk-910-npu-driver_24.1.rc1_linux-aarch64.run --full
./Ascend-hdk-910-npu-firmware_7.1.0.6.220.run --full
#openEuler系统的第三方依赖
yum install -y gcc gcc-c++ make cmake unzip zlib-devel libffi-devel openssl-devel pciutils net-tools sqlite-devel lapack-devel gcc-gfortran
#在保证Python版本在3.10后安装python库。
pip3 install attrs numpy decorator sympy cffi pyyaml pathlib2 psutil protobuf scipy requests absl-py wheel typing_extensions
./Ascend-cann-toolkit_8.0.RC1_linux-aarch64.run --install
./Ascend-cann-kernels-910_8.0.RC1_linux.run --install
软件准备阶段
#拉取Llama-factory
git clone -b v0.7.0 https://github.com/hiyouga/LLaMA-Factory.git
#安装依赖
cd LLaMA-Factorycd
pip install -e .[metrics] -i https://pypi.tuna.tsinghua.edu.cn/simple/
pip install transformers==4.40.2 -i https://pypi.tuna.tsinghua.edu.cn/simple/
#拉取安装pytorch及torch_npu
Wget https://download.pytorch.org/whl/cpu/torch-2.2.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
pip3 install torch-2.2.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
wget https://gitee.com/ascend/pytorch/releases/download/v6.0.rc1-pytorch2.2.0/torch_npu-2.2.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
pip3 install torch_npu-2.2.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
修改transformers utils文件
pip show transformers
vim /root/miniconda3/lib/python3.11/site-packages/transformers/generation/utils.py
如上图添加next_token_scores = next_token_scores.cpu()和next_token_scores = next_token_scores.npu()后保存退出
修改llm参数
vim LLaMA-Factory/src/cli_demo.py进入文件添加torch库
同一文件内在main中添加npu加速指令:torch.npu.set_compile_mode(jit_compile=False)
保存退出
模型推理
cd /root/ljq/LLaMA-Factory/examples/inference\
vim cli_demo.sh 具体修改如下
保存退出后运行。
bash cli_demo.sh