Prompt Engineering for LLMs by John Berryman
Author:John Berryman
Language: eng
Format: epub
Publisher: O'Reilly Media
Published: 2023-09-28T00:00:00+00:00
More on Inertness
The last bit, inertness, depends on your tokenizer, which might use different tokens to tokenize a composite string A + B than to tokenize each string individually. That can easily increase or decrease the number of tokens needed to tokenize a composite string (see Table 6-4).
Pasting strings together doesnât mean the arrays of tokens just get concatenated. Token IDs have been obtained for OpenAIâs GPT-3.5-and-later tokenizer, but both examples also work for the GPT-3-and-before tokenizer used in many non-OpenAI LLMs.
Table 6-4. Token count isnât additive Example 1 Example 2
Strings âbeâ + âamâ â âbeamâ âcatâ + âtailâ â âcattailâ
Tokens [be] + [am] â [beam] [cat] + [tail] â [c], [att], [ail]
Token ids 1395 + 309 â 54971 4719 + 14928 â 66, 1617, 607
Token count 1 + 1 â 1 1 + 1 â 3
Download
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.
Algorithms of the Intelligent Web by Haralambos Marmanis;Dmitry Babenko(8310)
Test-Driven Development with Java by Alan Mellor(6815)
Data Augmentation with Python by Duc Haba(6728)
Principles of Data Fabric by Sonia Mezzetta(6473)
Learn Blender Simulations the Right Way by Stephen Pearson(6381)
Microservices with Spring Boot 3 and Spring Cloud by Magnus Larsson(6245)
Hadoop in Practice by Alex Holmes(5966)
Jquery UI in Action : Master the concepts Of Jquery UI: A Step By Step Approach by ANMOL GOYAL(5816)
RPA Solution Architect's Handbook by Sachin Sahgal(5645)
Big Data Analysis with Python by Ivan Marin(5404)
The Infinite Retina by Robert Scoble Irena Cronin(5337)
Life 3.0: Being Human in the Age of Artificial Intelligence by Tegmark Max(5160)
Pretrain Vision and Large Language Models in Python by Emily Webber(4370)
Infrastructure as Code for Beginners by Russ McKendrick(4138)
Functional Programming in JavaScript by Mantyla Dan(4044)
The Age of Surveillance Capitalism by Shoshana Zuboff(3964)
WordPress Plugin Development Cookbook by Yannick Lefebvre(3851)
Embracing Microservices Design by Ovais Mehboob Ahmed Khan Nabil Siddiqui and Timothy Oleson(3654)
Applied Machine Learning for Healthcare and Life Sciences Using AWS by Ujjwal Ratan(3630)
