昇思25天学习打卡营第12天 | LLM原理和实践：MindNLP ChatGLM-6B StreamChat

1. MindNLP ChatGLM-6B StreamChat

本案例基于MindNLP和ChatGLM-6B实现一个聊天应用。

ChatGLM-6B应该是国内第一个发布的可以在消费级显卡上进行推理部署的国产开源大模型，2023年3月就发布了。我在23年6月份的时候就在自己的笔记本电脑上部署测试过，当时的1代6B模型已经能解鸡兔同笼的数学问题，感觉上是真正“理解”了人类语言的语义。我认为和chatgpt相比，也并没有非常明显的差距。当然存在的问题也有不少。

1.1 环境配置

安装mindnlp

!pip install mindnlp

安装过程:

Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting mindnlp
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/72/37/ef313c23fd587c3d1f46b0741c98235aecdfd93b4d6d446376f3db6a552c/mindnlp-0.3.1-py3-none-any.whl (5.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/5.7 MB 16.7 MB/s eta 0:00:00a 0:00:01
Requirement already satisfied: mindspore in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindnlp) (2.2.14)
Requirement already satisfied: tqdm in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindnlp) (4.66.4)
Requirement already satisfied: requests in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindnlp) (2.32.3)
Collecting datasets (from mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/60/2d/963b266bb8f88492d5ab4232d74292af8beb5b6fdae97902df9e284d4c32/datasets-2.20.0-py3-none-any.whl (547 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 547.8/547.8 kB 16.4 MB/s eta 0:00:00
Collecting evaluate (from mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c2/d6/ff9baefc8fc679dcd9eb21b29da3ef10c81aa36be630a7ae78e4611588e1/evaluate-0.4.2-py3-none-any.whl (84 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.1/84.1 kB 25.4 MB/s eta 0:00:00
Collecting tokenizers (from mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ba/26/139bd2371228a0e203da7b3e3eddcb02f45b2b7edd91df00e342e4b55e13/tokenizers-0.19.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 19.6 MB/s eta 0:00:0000:0100:01
Collecting safetensors (from mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c6/02/28e6280ed0f1bde89eed644b80f2ece4e5ae212dc9ee70d7f56fadc93602/safetensors-0.4.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 20.2 MB/s eta 0:00:00a 0:00:01
Collecting sentencepiece (from mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a3/69/e96ef68261fa5b82379fdedb325ceaf1d353c6e839ec346d8244e0da5f2f/sentencepiece-0.2.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 15.5 MB/s eta 0:00:00a 0:00:01
Collecting regex (from mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/70/70/fea4865c89a841432497d1abbfd53878513b55c6543245fabe31cf8df0b8/regex-2024.5.15-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (774 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 774.7/774.7 kB 17.0 MB/s eta 0:00:00a 0:00:01
Collecting addict (from mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/6a/00/b08f23b7d7e1e14ce01419a467b583edbb93c6cdb8654e54a9cc579cd61f/addict-2.4.0-py3-none-any.whl (3.8 kB)
Collecting ml-dtypes (from mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/50/96/13d7c3cc82d5ef597279216cf56ff461f8b57e7096a3ef10246a83ca80c0/ml_dtypes-0.4.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 12.6 MB/s eta 0:00:00a 0:00:01
Collecting pyctcdecode (from mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a5/8a/93e2118411ae5e861d4f4ce65578c62e85d0f1d9cb389bd63bd57130604e/pyctcdecode-0.5.0-py2.py3-none-any.whl (39 kB)
Collecting jieba (from mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c6/cb/18eeb235f833b726522d7ebed54f2278ce28ba9438e3135ab0278d9792a2/jieba-0.42.1.tar.gz (19.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.2/19.2 MB 19.3 MB/s eta 0:00:0000:0100:01
  Preparing metadata (setup.py) ... done
Collecting pytest==7.2.0 (from mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/67/68/a5eb36c3a8540594b6035e6cdae40c1ef1b6a2bfacbecc3d1a544583c078/pytest-7.2.0-py3-none-any.whl (316 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 316.8/316.8 kB 17.8 MB/s eta 0:00:00
Requirement already satisfied: attrs>=19.2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pytest==7.2.0->mindnlp) (23.2.0)
Requirement already satisfied: iniconfig in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pytest==7.2.0->mindnlp) (2.0.0)
Requirement already satisfied: packaging in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pytest==7.2.0->mindnlp) (23.2)
Requirement already satisfied: pluggy<2.0,>=0.12 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pytest==7.2.0->mindnlp) (1.5.0)
Requirement already satisfied: exceptiongroup>=1.0.0rc8 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pytest==7.2.0->mindnlp) (1.2.0)
Requirement already satisfied: tomli>=1.0.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pytest==7.2.0->mindnlp) (2.0.1)
Requirement already satisfied: filelock in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from datasets->mindnlp) (3.15.3)
Requirement already satisfied: numpy>=1.17 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from datasets->mindnlp) (1.26.4)
Collecting pyarrow>=15.0.0 (from datasets->mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/87/60/cc0645eb4ef73f88847e40a7f9d238bae6b7409d6c1f6a5d200d8ade1f09/pyarrow-16.1.0-cp39-cp39-manylinux_2_28_aarch64.whl (38.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 38.1/38.1 MB 18.3 MB/s eta 0:00:0000:0100:01
Collecting pyarrow-hotfix (from datasets->mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e4/f4/9ec2222f5f5f8ea04f66f184caafd991a39c8782e31f5b0266f101cb68ca/pyarrow_hotfix-0.6-py3-none-any.whl (7.9 kB)
Requirement already satisfied: dill<0.3.9,>=0.3.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from datasets->mindnlp) (0.3.8)
Requirement already satisfied: pandas in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from datasets->mindnlp) (2.2.2)
Collecting xxhash (from datasets->mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/7c/b9/93f860969093d5d1c4fa60c75ca351b212560de68f33dc0da04c89b7dc1b/xxhash-3.4.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (220 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 220.6/220.6 kB 17.8 MB/s eta 0:00:00
Collecting multiprocess (from datasets->mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/da/d9/f7f9379981e39b8c2511c9e0326d212accacb82f12fbfdc1aa2ce2a7b2b6/multiprocess-0.70.16-py39-none-any.whl (133 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.4/133.4 kB 17.5 MB/s eta 0:00:00
Collecting fsspec<=2024.5.0,>=2023.1.0 (from fsspec[http]<=2024.5.0,>=2023.1.0->datasets->mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ba/a3/16e9fe32187e9c8bc7f9b7bcd9728529faa725231a0c96f2f98714ff2fc5/fsspec-2024.5.0-py3-none-any.whl (316 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 316.1/316.1 kB 19.4 MB/s eta 0:00:00
Collecting aiohttp (from datasets->mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/eb/45/eebe8d2215328434f33ccb44a05d2741ff7ed4b96b56ca507e2ecf598b73/aiohttp-3.9.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 18.2 MB/s eta 0:00:00a 0:00:01
Requirement already satisfied: huggingface-hub>=0.21.2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from datasets->mindnlp) (0.23.4)
Requirement already satisfied: pyyaml>=5.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from datasets->mindnlp) (6.0.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from requests->mindnlp) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from requests->mindnlp) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from requests->mindnlp) (2.2.2)
Requirement already satisfied: certifi>=2017.4.17 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from requests->mindnlp) (2024.6.2)
Requirement already satisfied: protobuf>=3.13.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindspore->mindnlp) (5.27.1)
Requirement already satisfied: asttokens>=2.0.4 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindspore->mindnlp) (2.0.5)
Requirement already satisfied: pillow>=6.2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindspore->mindnlp) (10.3.0)
Requirement already satisfied: scipy>=1.5.4 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindspore->mindnlp) (1.13.1)
Requirement already satisfied: psutil>=5.6.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindspore->mindnlp) (5.9.0)
Requirement already satisfied: astunparse>=1.6.3 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindspore->mindnlp) (1.6.3)
Collecting pygtrie<3.0,>=2.1 (from pyctcdecode->mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ec/cd/bd196b2cf014afb1009de8b0f05ecd54011d881944e62763f3c1b1e8ef37/pygtrie-2.5.0-py3-none-any.whl (25 kB)
Collecting hypothesis<7,>=6.14 (from pyctcdecode->mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/58/14/a4c621cb713f6053f37afa78ab3809f9d879182422071ca9d4af61c6d1d9/hypothesis-6.105.0-py3-none-any.whl (462 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 462.2/462.2 kB 21.2 MB/s eta 0:00:00
Requirement already satisfied: six in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from asttokens>=2.0.4->mindspore->mindnlp) (1.16.0)
Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from astunparse>=1.6.3->mindspore->mindnlp) (0.43.0)
Collecting aiosignal>=1.1.2 (from aiohttp->datasets->mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/76/ac/a7305707cb852b7e16ff80eaf5692309bde30e2b1100a1fcacdc8f731d97/aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting frozenlist>=1.1.1 (from aiohttp->datasets->mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/57/15/172af60c7e150a1d88ecc832f2590721166ae41eab582172fe1e9844eab4/frozenlist-1.4.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (239 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 239.4/239.4 kB 19.6 MB/s eta 0:00:00
Collecting multidict<7.0,>=4.5 (from aiohttp->datasets->mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d0/10/2ff646c471e84af25fe8111985ffb8ec85a3f6e1ade8643bfcfcc0f4d2b1/multidict-6.0.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (125 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 125.9/125.9 kB 16.9 MB/s eta 0:00:00
Collecting yarl<2.0,>=1.0 (from aiohttp->datasets->mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c6/d6/5b30ae1d8a13104ee2ceb649f28f2db5ad42afbd5697fd0fc61528bb112c/yarl-1.9.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (300 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 300.9/300.9 kB 14.5 MB/s eta 0:00:00
Collecting async-timeout<5.0,>=4.0 (from aiohttp->datasets->mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a7/fa/e01228c2938de91d47b307831c62ab9e4001e747789d0b05baf779a6488c/async_timeout-4.0.3-py3-none-any.whl (5.7 kB)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from huggingface-hub>=0.21.2->datasets->mindnlp) (4.11.0)
Collecting sortedcontainers<3.0.0,>=2.1.0 (from hypothesis<7,>=6.14->pyctcdecode->mindnlp)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
Requirement already satisfied: python-dateutil>=2.8.2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pandas->datasets->mindnlp) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pandas->datasets->mindnlp) (2024.1)
Requirement already satisfied: tzdata>=2022.7 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pandas->datasets->mindnlp) (2024.1)
Building wheels for collected packages: jieba
  Building wheel for jieba (setup.py) ... done
  Created wheel for jieba: filename=jieba-0.42.1-py3-none-any.whl size=19314459 sha256=b34d50dd74d300723e8f42cfea23a680ef40525b2f94488be07da842e838a41c
  Stored in directory: /home/nginx/.cache/pip/wheels/1a/76/68/b6d79c4db704bb18d54f6a73ab551185f4711f9730c0c15d97
Successfully built jieba
Installing collected packages: sortedcontainers, sentencepiece, pygtrie, jieba, addict, xxhash, safetensors, regex, pytest, pyarrow-hotfix, pyarrow, multiprocess, multidict, ml-dtypes, hypothesis, fsspec, frozenlist, async-timeout, yarl, pyctcdecode, aiosignal, tokenizers, aiohttp, datasets, evaluate, mindnlp
  Attempting uninstall: pytest
    Found existing installation: pytest 8.0.0
    Uninstalling pytest-8.0.0:
      Successfully uninstalled pytest-8.0.0
  Attempting uninstall: fsspec
    Found existing installation: fsspec 2024.6.0
    Uninstalling fsspec-2024.6.0:
      Successfully uninstalled fsspec-2024.6.0
Successfully installed addict-2.4.0 aiohttp-3.9.5 aiosignal-1.3.1 async-timeout-4.0.3 datasets-2.20.0 evaluate-0.4.2 frozenlist-1.4.1 fsspec-2024.5.0 hypothesis-6.105.0 jieba-0.42.1 mindnlp-0.3.1 ml-dtypes-0.4.0 multidict-6.0.5 multiprocess-0.70.16 pyarrow-16.1.0 pyarrow-hotfix-0.6 pyctcdecode-0.5.0 pygtrie-2.5.0 pytest-7.2.0 regex-2024.5.15 safetensors-0.4.3 sentencepiece-0.2.0 sortedcontainers-2.4.0 tokenizers-0.19.1 xxhash-3.4.1 yarl-1.9.4

[notice] A new release of pip is available: 24.1 -> 24.1.1
[notice] To update, run: python -m pip install --upgrade pip

安装mdtex2html

!pip install mdtex2html

Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting mdtex2html
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ff/e8/c5fab9aa5d9254ad7c7e37d33a3c32fd49d82b4c6b54da337bbca378eb5c/mdtex2html-1.3.0-py3-none-any.whl (13 kB)
Requirement already satisfied: gradio in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (4.26.0)
Collecting markdown (from mdtex2html)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/fc/b3/0c0c994fe49cd661084f8d5dc06562af53818cc0abefaca35bdc894577c3/Markdown-3.6-py3-none-any.whl (105 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 105.4/105.4 kB 11.6 MB/s eta 0:00:00
Collecting latex2mathml (from mdtex2html)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f2/0a/181ed55562ce90179aedf33b09fcd79db31c868a5d480f3cb71a31d19692/latex2mathml-3.77.0-py3-none-any.whl (73 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 73.7/73.7 kB 22.7 MB/s eta 0:00:00
Requirement already satisfied: aiofiles<24.0,>=22.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (22.1.0)
Requirement already satisfied: altair<6.0,>=4.2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (5.3.0)
Requirement already satisfied: fastapi in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.111.0)
Requirement already satisfied: ffmpy in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.3.2)
Requirement already satisfied: gradio-client==0.15.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.15.1)
Requirement already satisfied: httpx>=0.24.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.27.0)
Requirement already satisfied: huggingface-hub>=0.19.3 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.23.4)
Requirement already satisfied: importlib-resources<7.0,>=1.3 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (6.4.0)
Requirement already satisfied: jinja2<4.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (3.1.4)
Requirement already satisfied: markupsafe~=2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (2.1.5)
Requirement already satisfied: matplotlib~=3.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (3.9.0)
Requirement already satisfied: numpy~=1.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (1.26.4)
Requirement already satisfied: orjson~=3.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (3.10.5)
Requirement already satisfied: packaging in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (23.2)
Requirement already satisfied: pandas<3.0,>=1.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (2.2.2)
Requirement already satisfied: pillow<11.0,>=8.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (10.3.0)
Requirement already satisfied: pydantic>=2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (2.7.4)
Requirement already satisfied: pydub in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.25.1)
Requirement already satisfied: python-multipart>=0.0.9 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.0.9)
Requirement already satisfied: pyyaml<7.0,>=5.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (6.0.1)
Requirement already satisfied: ruff>=0.2.2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.4.10)
Requirement already satisfied: semantic-version~=2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (2.10.0)
Requirement already satisfied: tomlkit==0.12.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.12.0)
Requirement already satisfied: typer<1.0,>=0.9 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from typer[all]<1.0,>=0.9; sys_platform != "emscripten"->gradio) (0.12.3)
Requirement already satisfied: typing-extensions~=4.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (4.11.0)
Requirement already satisfied: uvicorn>=0.14.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.30.1)
Requirement already satisfied: fsspec in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio-client==0.15.1->gradio) (2024.5.0)
Requirement already satisfied: websockets<12.0,>=10.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio-client==0.15.1->gradio) (11.0.3)
Requirement already satisfied: jsonschema>=3.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from altair<6.0,>=4.2.0->gradio) (4.22.0)
Requirement already satisfied: toolz in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from altair<6.0,>=4.2.0->gradio) (0.12.1)
Requirement already satisfied: anyio in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from httpx>=0.24.1->gradio) (4.4.0)
Requirement already satisfied: certifi in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from httpx>=0.24.1->gradio) (2024.6.2)
Requirement already satisfied: httpcore==1.* in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from httpx>=0.24.1->gradio) (1.0.5)
Requirement already satisfied: idna in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from httpx>=0.24.1->gradio) (3.7)
Requirement already satisfied: sniffio in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from httpx>=0.24.1->gradio) (1.3.1)
Requirement already satisfied: h11<0.15,>=0.13 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from httpcore==1.*->httpx>=0.24.1->gradio) (0.14.0)
Requirement already satisfied: filelock in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from huggingface-hub>=0.19.3->gradio) (3.15.3)
Requirement already satisfied: requests in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from huggingface-hub>=0.19.3->gradio) (2.32.3)
Requirement already satisfied: tqdm>=4.42.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from huggingface-hub>=0.19.3->gradio) (4.66.4)
Requirement already satisfied: zipp>=3.1.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from importlib-resources<7.0,>=1.3->gradio) (3.17.0)
Requirement already satisfied: contourpy>=1.0.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from matplotlib~=3.0->gradio) (1.2.1)
Requirement already satisfied: cycler>=0.10 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from matplotlib~=3.0->gradio) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from matplotlib~=3.0->gradio) (4.53.0)
Requirement already satisfied: kiwisolver>=1.3.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from matplotlib~=3.0->gradio) (1.4.5)
Requirement already satisfied: pyparsing>=2.3.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from matplotlib~=3.0->gradio) (3.1.2)
Requirement already satisfied: python-dateutil>=2.7 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from matplotlib~=3.0->gradio) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pandas<3.0,>=1.0->gradio) (2024.1)
Requirement already satisfied: tzdata>=2022.7 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pandas<3.0,>=1.0->gradio) (2024.1)
Requirement already satisfied: annotated-types>=0.4.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pydantic>=2.0->gradio) (0.7.0)
Requirement already satisfied: pydantic-core==2.18.4 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pydantic>=2.0->gradio) (2.18.4)
Requirement already satisfied: click>=8.0.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != "emscripten"->gradio) (8.1.7)
Requirement already satisfied: shellingham>=1.3.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != "emscripten"->gradio) (1.5.4)
Requirement already satisfied: rich>=10.11.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != "emscripten"->gradio) (13.7.1)
WARNING: typer 0.12.3 does not provide the extra 'all'
Requirement already satisfied: starlette<0.38.0,>=0.37.2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from fastapi->gradio) (0.37.2)
Requirement already satisfied: fastapi-cli>=0.0.2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from fastapi->gradio) (0.0.4)
Requirement already satisfied: ujson!=4.0.2,!=4.1.0,!=4.2.0,!=4.3.0,!=5.0.0,!=5.1.0,>=4.0.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from fastapi->gradio) (5.10.0)
Requirement already satisfied: email_validator>=2.0.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from fastapi->gradio) (2.2.0)
Requirement already satisfied: importlib-metadata>=4.4 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from markdown->mdtex2html) (7.0.1)
Requirement already satisfied: dnspython>=2.0.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from email_validator>=2.0.0->fastapi->gradio) (2.6.1)
Requirement already satisfied: attrs>=22.2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (23.2.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (2023.12.1)
Requirement already satisfied: referencing>=0.28.4 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (0.35.1)
Requirement already satisfied: rpds-py>=0.7.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (0.18.1)
Requirement already satisfied: six>=1.5 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib~=3.0->gradio) (1.16.0)
Requirement already satisfied: markdown-it-py>=2.2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from rich>=10.11.0->typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != "emscripten"->gradio) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from rich>=10.11.0->typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != "emscripten"->gradio) (2.15.1)
Requirement already satisfied: exceptiongroup>=1.0.2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from anyio->httpx>=0.24.1->gradio) (1.2.0)
Requirement already satisfied: httptools>=0.5.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from uvicorn[standard]>=0.12.0->fastapi->gradio) (0.6.1)
Requirement already satisfied: python-dotenv>=0.13 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from uvicorn[standard]>=0.12.0->fastapi->gradio) (1.0.1)
Requirement already satisfied: uvloop!=0.15.0,!=0.15.1,>=0.14.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from uvicorn[standard]>=0.12.0->fastapi->gradio) (0.19.0)
Requirement already satisfied: watchfiles>=0.13 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from uvicorn[standard]>=0.12.0->fastapi->gradio) (0.22.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from requests->huggingface-hub>=0.19.3->gradio) (3.3.2)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from requests->huggingface-hub>=0.19.3->gradio) (2.2.2)
Requirement already satisfied: mdurl~=0.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != "emscripten"->gradio) (0.1.2)
Installing collected packages: latex2mathml, markdown, mdtex2html
Successfully installed latex2mathml-3.77.0 markdown-3.6 mdtex2html-1.3.0

[notice] A new release of pip is available: 24.1 -> 24.1.1
[notice] To update, run: python -m pip install --upgrade pip

安装gradio

!pip install gradio

配置环境变量

# 设置环境变量 HF_ENDPOINT，其值为 https://hf-mirror.com
# 这个环境变量通常用于指定 Hugging Face Transformers 模型位于国内的镜像站点
# 以便从该镜像站点下载模型和相关资源，提高下载速度和稳定性,不需要代理
export HF_ENDPOINT=https://hf-mirror.com

1.2 代码开发

模型参数量为6B, 磁盘空间大小占用约12G, 下载权重产加载大约需要20分钟

# 导入 MindNLP 库中的 AutoModelForSeq2SeqLM 类和 AutoTokenizer 类
# MindNLP 是一个基于 PyTorch 的自然语言处理库，提供了许多预训练模型和工具
from mindnlp.transformers import AutoModelForSeq2SeqLM, AutoTokenizer

# 导入 Gradio 库，用于创建一个交互式的 Web 界面
import gradio as gr

# 导入 mdtex2html 库，用于将 Markdown 格式的文本转换为 HTML 格式
import mdtex2html

# 使用 AutoModelForSeq2SeqLM 类从预训练模型 'ZhipuAI/ChatGLM-6B' 创建一个模型实例
# 'ZhipuAI/ChatGLM-6B' 是一个序列到序列的语言模型，用于文本生成任务
# mirror="modelscope" 参数指定了模型的镜像站点为 "modelscope"，以提高下载速度和稳定性.half()是将float32转为float16.
model = AutoModelForSeq2SeqLM.from_pretrained('ZhipuAI/ChatGLM-6B', mirror="modelscope").half()

# 将模型设置为评估模式，即不进行训练
model.set_train(False)

# 使用 AutoTokenizer 类从预训练模型 'ZhipuAI/ChatGLM-6B' 创建一个分词器实例
# 分词器用于将输入文本转换为模型可以理解的 tokens
tokenizer = AutoTokenizer.from_pretrained('ZhipuAI/ChatGLM-6B', mirror="modelscope")

输出:

100%
 773/773 [00:00<00:00, 50.6kB/s]
100%
 32.6k/32.6k [00:00<00:00, 2.78MB/s]
Downloading shards: 100%
 8/8 [16:43<00:00, 107.69s/it]
100%
 1.62G/1.62G [02:14<00:00, 14.2MB/s]
100%
 1.75G/1.75G [02:19<00:00, 16.8MB/s]
100%
 1.84G/1.84G [02:27<00:00, 17.9MB/s]
100%
 1.78G/1.78G [02:21<00:00, 17.5MB/s]
100%
 1.75G/1.75G [02:20<00:00, 14.6MB/s]
100%
 1.75G/1.75G [02:20<00:00, 18.7MB/s]
100%
 1.00G/1.00G [01:18<00:00, 16.7MB/s]
100%
 1.00G/1.00G [01:19<00:00, 8.88MB/s]
Loading checkpoint shards: 100%
 8/8 [00:51<00:00,  5.84s/it]
100%
 441/441 [00:00<00:00, 38.4kB/s]
100%
 2.58M/2.58M [00:00<00:00, 6.99MB/s]

1.3 进行推理

# 定义一个字符串变量 prompt，内容为 "你好"，表示用户的提问或对话输入
prompt = '你好'

# 定义一个列表变量 history，用于存储对话历史
# 在这个例子中，对话历史为空，因为这是新的对话
history = []

# 调用 model 的 chat 方法，传入 tokenizer、prompt、history 和 max_length 参数
# chat 方法是模型的一个函数，用于生成对话响应
# tokenizer 是之前加载的分词器，用于处理输入文本
# prompt 是用户的输入文本
# history 是对话历史，用于提供上下文信息
# max_length 是生成响应的最大长度
response, _ = model.chat(tokenizer, prompt, history=history, max_length=20)

# 打印出模型的对话响应
print(response)

输出:

\
The dtype of attention mask (Float32) is not bool
|
'你好👋！我是人工智能助手 ChatGLM-6B'

响应非常地慢, 不知是什么原因.
根据npu监控信息来看, 一开始完全没有利用到ai core, 而是一直在跑内存. 直到开始调用ai core之后,很快就给出了响应.

prompt = '房间里有7只鸡和兔子,共20只脚.问有几只鸡,几只兔子?'
history = []
response, _ = model.chat(tokenizer, prompt, history=history, max_length=512)
response

输出:
鸡兔同笼

可以看到,他有正确的解题思路. 虽然数学稀烂,二元一次方程组都能解错.
耗时约200s, 输出250多个字符.如果算token的话,可能就是每秒1个token, 这个性能是真的拉垮…
当然,这是免费的虚拟资源，并且很大概率是共享的，可能并不能和实体的昇腾芯片相提并论。

2. 小结

本文主要介绍了使用mindnlp下载chatglm-6B预训练模型，并基于此模型，通过输入提示词完成回答文本生成的模型推理任务，从而实现了一个简单的聊天应用。

昇思25天学习打卡营第12天 | LLM原理和实践：MindNLP ChatGLM-6B StreamChat

1. MindNLP ChatGLM-6B StreamChat

1.1 环境配置

1.2 代码开发

1.3 进行推理

2. 小结

相关文章

2024年江苏省研究生数学建模科研创新实践大赛C题气象数据高精度融合技术研究论文和代码分析

绝区壹--LLM的构建模块

Qt（MSVC）下报“语法错误缺少“}““语法错误缺少“常数“ 的解决办法

Xilinx FPGA:vivado关于fifo的一些零碎知识

第6章选课学习：需求分析，添加选课，支付，支付通知，在线学习

以黑盒与白盒的角度分析和通关xss-labs(XSS漏洞类型与总结)

C++初级——C++入门（2）：函数重载

java自旋锁

Spring Cloud Alibaba - Sentinel 分布式系统流量哨兵

ECharts在最新版本中使用getInstanceByDom报错处理

【Docker系列】Docker 镜像构建中的跨设备移动问题及解决方案

Spring中的事件监听器使用学习

vue3+electron项目搭建,遇到的坑

DNS正向解析与反向解析实验

AI绘画Stable Diffusion【图生图教程】：图片高清修复的三种方案详解，你一定能用上！（附资料）

Linux运维：mysql主从复制原理及实验

Flutter-实现双向PK进度条

仪器校准后出了校准证书后，是不是就代表仪器合格了？

苍穹外卖--sky-take-out（五）前端

基于单片机的多功能电子时钟的设计