"Large Language Model: Principles and Engineering Practice" - Basic Technology of Large Model

rtyu789

"Large Language Model: Principles and Engineering Practice" - Basic Technology of Large Model [Copy link]

I would like to express my sincere gratitude to EEWorld and the Publishing House of Electronics Industry for providing this opportunity to read and share this book. # 1. Book Overview The data uses rich diagrams to show the model, which makes it easier for readers to understand! [836522](/data/attachment/forum/202408/28/003002k9elo71414ae8rbq.jpg.thumb.jpg?rand=4268.273136761895) 
<ignore_js_op>

<div class="tip tip_4 aimg_tip" id="aimg_836527_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>图示2.jpg</strong> <em class="xg1">(267.89 KB, downloads: 0)</em></p>
<p>
<a href="javascript:;" class="loginf" >download attach</a>

<a href="javascript:;" onclick="showWindow(this.id, this.getAttribute('url'), 'get', 0);" id="savephoto_836527" url="home.php?mod=spacecp&ac=album&op=saveforumphoto&aid=836527&handlekey=savephoto_836527">save to album</a>

</p>

</div>
<div class="tip_horn"></div>
</div>

</ignore_js_op>
 References are added after each chapter to facilitate readers' reading 
<ignore_js_op>

<div class="tip tip_4 aimg_tip" id="aimg_836525_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>参考文献.jpg</strong> <em class="xg1">(379.7 KB, downloads: 0)</em></p>
<p>
<a href="javascript:;" class="loginf" >download attach</a>

<a href="javascript:;" onclick="showWindow(this.id, this.getAttribute('url'), 'get', 0);" id="savephoto_836525" url="home.php?mod=spacecp&ac=album&op=saveforumphoto&aid=836525&handlekey=savephoto_836525">save to album</a>

</p>

</div>
<div class="tip_horn"></div>
</div>

</ignore_js_op>
 # 2. Basic technologies of large models Usually, we know more about the reasoning of large models, but we know less about the technology behind training large models. This chapter mainly introduces the basic technologies of large models from four aspects 1. Basic representation methods of natural language 1. Word representation technology 1. Unique hot representation of words 2. Distributed representation of words 3. Word embedded representation based on pre-training 2. Word segmentation technology 1. Word granularity 2. Character granularity 3. Subword granularity 2. Classic structure Transformer 3. Transformer pre-trained language model 1. Encoder-Only pre-trained language model - BERT 2. Decoder-Only pre-trained language model - GPT 3. Encoder-Decoder pre-trained language model - T5, BART 4. Introduction to open source and closed source large models respectively 1. InstructGPT 2. LLaMA This book describes the chapters in part 2.3.4 in detail, so I mainly supplement the content of the first part, which I am more interested in# 3. Basic representation methods of natural language 1. Word representation technology The unique hot representation of a word is the simplest and most direct way to vectorize a word. The main steps include two steps: 1) Encode all the words in the text that need to be used (assuming that there are N words in common), and each word has a unique subscript (0~N) 2) Generate a vector of length N according to the word subscript i. Except for the i-th bit which is 1, all other bits are 0. Let’s take a simple example to illustrate: ```shell I like deep learning I like NLP I enjoy flying ``` Ignore uppercase and lowercase letters and encode different words: ```shell word_dict = {0: 'i', 1: 'like', 2: 'deep', 3: 'learning', 4: 'NLP', 5: 'enjoy', 6: 'flying'} ``` Obviously, according to the number of different words (7), the length of the One-Hot vector we need to construct is 7: | Word | Encoding | One-hot representation | | -------- | ---- | ------------- | | i | 0 | [1,0,0,0,0,0,0] | | like | 1 | [0,1,0,0,0,0,0] | | deep | 2 | [0,0,1,0,0,0,0] | | learning | 3 | [0,0,0,1,0,0,0] | | NLP | 4 | [0,0,0,0,1,0,0] | | enjoy | 5 | [0,0,0,0,0,1,0] | | flying | 6 | [0,0,0,0,0,0,1] | 2. Static word embedding language model based on pre-trained word embedding, Word2vec, GloVe Dynamic word embedding language model, ELMo, GPT, BEERT This paper mainly introduces Word2vec, which can be called word embedding. Although the one-hot vector of the words introduced above is very simple, it cannot accurately express the similarity between each word, so in 2013, Google scientists proposed the Word2vec method. In the word embedding mode, there are mainly two models: the Skip-Gram model and the Continuous Bag of Words (CBOW) model. 1) Skip-Gram model Main idea: Assume that a word can be used to generate a diagram of the words around it in a text sequence: 
<ignore_js_op>

<div class="tip tip_4 aimg_tip" id="aimg_836524_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>word2vec2.drawio.png</strong> <em class="xg1">(36.11 KB, downloads: 0)</em></p>
<p>
<a href="javascript:;" class="loginf" >download attach</a>

<a href="javascript:;" onclick="showWindow(this.id, this.getAttribute('url'), 'get', 0);" id="savephoto_836524" url="home.php?mod=spacecp&ac=album&op=saveforumphoto&aid=836524&handlekey=savephoto_836524">save to album</a>

</p>

</div>
<div class="tip_horn"></div>
</div>

</ignore_js_op>
 For the following text sequence, given the central word NLP ```shell I like NLP and deep learning ``` The skip-gram model considers the conditional probability of generating the context words I, like, and, deep, learning ```shell P("I","like","and","deep","learning") ``` Assume that the context words are independently generated given the central word (that is, conditional independence). In this case, the above conditional probability can be rewritten as ```shell P("NLP"|"I"),P("NLP"|"like"),P("NLP"|"and"),P("NLP"|"deep"),P("NLP"|"learning") ``` The following formula can be referred to reference 14.1. Word Embedding (word2vec) 1) Continuous Bag of Words CBOW Main idea: Assume that the central word is generated based on its surrounding context words in the text sequence: 
<ignore_js_op>

<div class="tip tip_4 aimg_tip" id="aimg_836523_menu" style="position: absolute; display: none" disautofocus="true">
<div class="xs0">
<p><strong>word2vec1.drawio.png</strong> <em class="xg1">(32.16 KB, downloads: 0)</em></p>
<p>
<a href="javascript:;" class="loginf" >download attach</a>

<a href="javascript:;" onclick="showWindow(this.id, this.getAttribute('url'), 'get', 0);" id="savephoto_836523" url="home.php?mod=spacecp&ac=album&op=saveforumphoto&aid=836523&handlekey=savephoto_836523">save to album</a>

</p>

</div>
<div class="tip_horn"></div>
</div>

</ignore_js_op>
 The content written here is just for reference. Readers can learn more NLP content by themselves. Language processing is also a deep subject that requires continuous learning# Reference [One-hot Representation of Words](https://blog.csdn.net/weixin_38659482/article/details/123316173) [One-hot in simple terms](https://zhuanlan.zhihu.com/p/634296763) [How to understand Word2Vec in simple terms (23 years revised edition)](https://blog.csdn.net/v_JULY_v/article/details/102708459) [14.1. Word embedding (word2vec)](https://zh.d2l.ai/chapter_natural-language-processing-pretraining/word2vec.html) [Distributed Representations of Words and Phrases and their Compositionality](https://arxiv.org/abs/1310.4546) [Efficient Estimation of Word Representations in Vector Space](https://arxiv.org/abs/1301.3781)

oceansky

"Large Language Model: Principles and Engineering Practice" - Basic Technology of Large Models, is there a free PDF download?

rtyu789

oceansky posted on 2024-8-28 14:00 "Large Language Model: Principles and Engineering Practice" - Basic Technology of Large Models, is there a free PDF download?

This is an evaluation activity on the EEWorld website. No electronic version is available for download. This is the activity page

【《Big Language Model: Principles and Engineering Practice》】Review | 【《Big Language Model: Principles and Engineering Practice》】Free Trial_Electronic Engineering World (eeworld.com.cn)

You can buy it yourself if you need it

freebsder

The price of color printing should be over 100

rtyu789

freebsder posted on 2024-8-29 16:06 The price of color printing should be over 100

The price on the back of the book is 119

"Large Language Model: Principles and Engineering Practice" - Basic Technology of Large Model [Copy link]

Latest reply

Comments

Comments