Generative Pre-Trained Transformer - 2

—

Today's Amazon Deals - https://amzn.to/3FeoGyg

–

Generative Pre-Trained Transformer - 2

Launch date: 14th Feb, 2019

The next version of the GPT model was introduced in 2019, GPT-2 which was trained on a larger dataset and enriched with more parameters to make this model better. In this second version and on the typical improvisation on GPT - 1, it is basically built to tackle multiple tasks together such as question answering, machine translation, reading comprehension, and summarization; and trying to achieve more closer tasks to human-abilities. It was scaled to have more than 10x the number of parameters than GPT - 1 (or the small GPT - 2).

Base Framework

The base model is similar to the initial GPT model, which is a transformer based architecture with decoder blocks only. To perform the task, the learning goal is needed to be adjusted to P (output|input, task). Task conditioning alludes to this modification, in which different outputs for the same input for different tasks are expected from the model. Some models give the model both the task and the input at the architectural level, using task conditioning. For language models, the job, input, and output are all linguistic stanzas. As a result, task conditioning for language models is carried out by giving the model examples or instructions in natural language. The foundation for zero-shot task transfer, mentioned in GPT-2, is task conditioning.

GPT 2’s capacity to transfer zero shot tasks is intriguing. As a special case of zero shot task transfer, zero shot learning occurs when no examples are given at all, and the model is instructed to perform the task. For fine-tuning, input to GPT-2 was presented in a format that anticipated the model to comprehend the nature of the assignment and provide answers rather than altering the sequences as was done for GPT-1. To mimic zero-shot task transfer behavior, this was done. For instance, the model was given an English sentence, followed by the word France, and a prompt for the English to French translation assignment. The model was expected to comprehend that the task involved translation and provide the French equivalent of the English statement. These tasks are expected to be executed in an unsupervised manner.

In order to create a substantial and excellent dataset, the authors scraped the Reddit site( posts which at least had minimum 3 karma) and gathered data from outbound links of highly upvoted posts. The final product, called WebText, had 40GB of text data from over 8 million publications. This dataset, which was huge, was used to train the GPT-2 model as opposed to the Book Corpus dataset, which was used to train the GPT-1 model. Due to the prevalence of Wikipedia material in test sets, WebText lacks Wikipedia content. The encoding is done in a unicode mechanism which increased the vocabulary base from 256 to 130,000.

Model Specifications

1.5 billion parameters were in GPT-2 which is ten times the amount of GPT-1 (117M parameters). There are some major elements in the model which are similar to GPT - 1 though there are few significant variations from GPT-1 included as well:

For word embedding, GPT-2 ( for GPT large) used 1600 dimensional vectors across 48 layers and a total 50,257 tokens from a larger vocabulary were used.

Larger batch size of 512 and larger context window from 512 to 1024 tokens were used.

Layer normalization was moved to the input of each sub-block and an additional layer normalization was added after the final self-attention block.

At initialisation, the weight of residual layers was scaled by 1/√N, where N was the number of residual layers.

There have been around 117M (GPT-1), 345M, 762M, and 1.5B (GPT-2) parameters to train four language models with 12,24,36,48 layers respectively along with 768, 1024, 1280, 1600 dimensional layers respectively. Every successive model was less perplexing than the one before it. This shows that as the number of parameters increases, the complexity of language models on the same dataset reduces. Also, every downstream task was completed better by the model with the most parameters.

Evaluation

Many datasets of downstream tasks, such as reading comprehension, summarization, translation, and question-answering, were used to evaluate GPT-2. The GPT-2 model has gone through many different kinds of objectives and database testing:

In zero shot settings, GPT-2 improved the then-current state-of-the-art for 7 of the 8 language modeling datasets across domains and datasets. Though it lacked a lot with One Billion Word Benchmark from performance perspective, most likely due to it being the most data samples and having the most destructive pre-processing.

The Children’s Book Dataset assesses how well language models perform when applied to various word categories, including nouns, prepositions, and named entities; basically to estimate the correct omitted word out of 10 possible choices. GPT-2 achieved a steady growth in accuracy with both CBT-named entity and CBT-common as the model parameter grows; with new state of the art accuracy results of 93.3% and 89.1% respectively for common nouns and named entities.

The LAMBADA dataset evaluates how well models do at finding far-off dependencies and guessing the sentence’s last word. GPT-2 enhanced the state of the art accuracy by Language models(LMs) from 19% to 52.66% and cut down perplexity from 99.8 to 8.6. It worked better with valid continuations of the sentence but not with valid final words. By adding, a stop-filter, it worked better with an improvement by 4%

By assessing a system’s capacity to resolve ambiguities in the text, the Winograd Schema challenge seeks to gauge its capacity for commonsense thinking. GPT–2 got a better rate of accuracy of 70.70% with an increment of 7%.

The CoQA dataset comprises papers from several fields that naturally exchange questions and answers. The exercise measures one’s capacity for reading comprehension as well as their capacity to respond to inquiries based on prior conversations. GPT-2 matched or exceeded the results from 3 of 4 baselines on zero shot tasks involving reading comprehension, which were trained on the 127,000+ question-answer pairs of the training data.

On an overview, The language model’s ability to grasp tasks and outperform the state-of-the-art on numerous tasks in zero shot scenarios was improved, according to GPT-2, by training on a larger dataset and employing more parameters. The essay claims that as model capacity increased, performance increased in a log-linear manner.

Figure 9.2: Performance of GPT-2 in CBT dataset

[Source: GPT -2 paper]

Also, when the number of parameters increased, the drop in language model perplexity did not approach a point of saturation. The WebText dataset really underfit GPT-2, and perhaps lengthier training sessions further reduced perplexity. According to research, the GPT-2 model size was not the maximum and that a larger language model will help people grasp natural language by reducing confusion.

Figure 9.3: Performance of Winograd Schema Challenge of GPT -2

[Source: GPT -2 paper]

To learn more technical aspect of GPT - 2, you can refer to - Language Models are Unsupervised Multitask Learners - https://tinyurl.com/3x7b74n9

Generative Pre-Trained Transformer - 1

—

Today's Amazon Deals - https://amzn.to/3FeoGyg

–

Generative Pre-Trained Transformer - 1

Launch date: 11th June, 2018

In 2018, the first GPT model, GPT-1, was released which was trained with a diverse level of unlabeled textual corpus data to get a strong Natural language understanding( NLU) base with fine-tuning and generative pre-training.

Basic Framework

The GPT-1 model really trained the language model using a transformer structure with about 12 layers of decoders and disguised self-attention. It was trained using data from the BookCorpus dataset, which contained over 7000 unpublished books to get the idea of working that model under unrecognized and unseen data with long stretched data which makes the model get better and longer contexts.

Model Training Stages

GPT - 1 model has 3 stages training:

Pre-training the model on the high corpus textual data where texts are getting tokenized and fed into likelihood function to optimize.

In this stage, the fine-tuning is being engaged to get the model accustomed with discriminative task with labeled data - which was passed through a transformer’s block and forwarded into L2 maximization and finally infused the a final linear optimization objective function

Task-specific Input Transformations contain organized inputs like triplets of documents, ordered sentence pairs, questions, and replies for particular tasks like question answering or textual entailment. The tokens of each input sequence are reinforced into an order with start and end tokens as well as delimiter tokens to maintain the order.

Figure 9.1: Picture defines the normal transformer architecture and input patterns for different information for different tasks for fine-tuning

[Source: GPT -1 paper]

Model Implementation Specifications

Model used a 768-dimensional state for encoding tokens into word embeddings and for position wise feed forward layer 3072-dimensional state was used with 12 attention heads. The adam optimiser was used with a learning rate 2.5 x 10 -4 and this learning rate is increased with 0 to 2000 updates with a cosinusoidal schedule. Attention, residual, byte pair encoding (BPE) vocabulary with 40,000 merges and embedding dropout rates with 0.1 were used for regularization and the Gaussian Error Linear Unit (GELU) was used as activation function. The model was trained for 100 epochs on mini-batches of size 64 and sequence length of 512. The model had 117M parameters in total.

For the fine-tuning part, the same hyperparameters settings have been observed from pretraining. The dropout rate was 0.1, with a learning rate 6.25e-5 and a batch size of 32. The fine-tune was made very prompt with 3 steps of epochs and Warmup occurs over 0.2% of training and is scheduled using a linear learning rate decay schedule.

Evaluation

The study showed how pre-training improved the model’s zero shot performance on a variety of NLP tasks, including sentiment analysis, question answering, and schema resolution. The architecture was capable of performing a range of NLP tasks with comparatively little fine-tuning and enabled transfer learning. This model demonstrated the efficacy of generative pre-training and created opportunities for future models to better realize this efficacy using larger datasets and additional parameters. GPT-1 performed better than specifically trained supervised state-of-the-art models in 9 out of 12 tasks the models were compared on.

They’ve made use of the just recently made available RACE dataset, which consists of English texts and the corresponding questions from middle and high school exams. It has been demonstrated that this corpus contains more questions of the reasoning variety than other datasets like CNN or SQuaD, making it the ideal testing ground for the model, which was trained to handle long-range contexts. Also, they assessed using the Narrative Cloze Test, which requires choosing the right conclusion from two possibilities for stories with several sentences. The GPT -1 model once again performed significantly better on these tasks than the prior best results, with gains of up to 8.9% on Story Cloze and 5.7% overall on RACE.

To learn more technical aspect of GPT - 1, you can refer to- Improving Language Understanding by Generative Pre-Training - https://tinyurl.com/3fu53mrd

Chapter 9: Historical Flow and Development of GPT Series

—

Today's Amazon Deals - https://amzn.to/3FeoGyg

–

Chapter 9: Historical Flow and Development of GPT Series

Introduction

The Generative Pre-trained Transformer (GPT) models are among the most popular natural language processing models used today. This chapter delves into the intricacies of the GPT-1 and GPT-2 models, discussing their architectures, training stages, implementation specifications, and evaluation. The GPT-1 was first introduced in June 2018 and was designed to develop a strong natural language understanding base through fine-tuning and generative pre-training. It was trained with diverse levels of unlabeled textual corpus data, enabling it to learn patterns and relationships between words and phrases. The model was able to generate coherent text and complete sentences, making it useful in a wide range of applications such as chatbots, language translation, and summarization.

In February 2019, the GPT-2 was released, boasting a larger dataset and more parameters than its predecessor. The GPT-2 was able to generate longer and more coherent sentences, and it was also able to tackle multiple tasks simultaneously. Overall, this chapter provides a detailed overview of the technical aspects of the GPT-1 and GPT-2 models. It highlights their strengths and limitations and discusses their potential applications in various fields. Understanding the workings of these models is essential for anyone interested in natural language processing and machine learning.

Chapter 8: Brief of Top Other NLP Models

—

Today's Amazon Deals - https://amzn.to/3FeoGyg

–

Chapter 8: Brief of Top Other NLP Models

Introduction

This table describes some other existing high-level models in the NLP domain exploring their structure and capabilities and technically tested performance.

Table 8.1: Various NLP models

Name

Details

BERT

Model:

The model represents a bi-directional training with random masking of input tokens in transformer

The model has around 24 Transformer blocks, 1024-hidden, 340M parameters, and training with 3.3 billions of word corpus

Performance:

GLUE benchmark score ~ 80.4%, more than 7.6% from the previous best result;

Has an accuracy of 93.2 % on SQuAD 1.1 benchmark outperforming the human interpretation by 2%

Capabilities:

BERT gives more better angle in building sentiment analysis tools & more efficiency towards providing better customer experience using chatbots

XLnet

Model:

This model represents a combination of core concepts of TransformerXL and BERT, autoregressive technology of TransformerXL and bidirectional nature of BERT for tackle the limitations of both

Performance:

XLnet successfully performed 18 different NLP tasks with state-of-the-art achievements and outperformed BERT on 20 tasks

Capabilities:

XLnet is better at question answering, sentiment analysis, priority ranking with similar conversational business application can be done here

RoBERTa

Model:

This model is trained with higher number of dataset than the original BERT, almost 10x with longer iteration of training

Training batch number has been also increased up to 8000

Byte-pair-encoding vocabulary with more than 50k subwords units

Performance:

Almost outperformed BERT in every aspects as expected

Capabilities:

RoBERTa can be applied with the similar use cases like BERT and XLnet with better performance expectations

ALBERT

Model:

With the objective of reducing the unnecessary length parameters in the large NLP models and break the Moore’s law in NLP model building, ALBERT is introduced with parameter reduction mechanisms like factorized embedding parameterization and cross-layer parameter sharing

Performance:

Without observing a significant downfall in performance, ALBERT solved the issue of bulkiness of the models by reducing 18x fewer parameters and 1.7x faster training rate

Achieved an F1 score of 92.2 with SQuAD benchmark and GLUE benchmark of 89.4

Capabilities:

ALBERT can be applied with the similar use cases like BERT and XLnet with better performance expectations

PaLM

Model:

In this model around 540B training parameter existed and to accommodate that during the training phase, the help of data parallelization was taken across two cloud TPU v4 pods and finally achieved a training utilization of 57.8% effectively of hardware.

Performance:

It outperformed many large models on 28 out of 29 major NLP tasks. It surpassed many benchmark tasks such as SuperGLUE, BIG-bench with significant margin than others

PaLM outperforms the improved Codex 12B despite requiring 50 times less Python code for training, demonstrating that big language models are more efficient at transferring knowledge from other computer languages and natural language data.

Capabilities:

PaLM may be used for a variety of downstream activities, including conversational AI, question answering, machine translation, document categorization, ad copy production, code issue correction, and more. This is similar to other newly announced pre-trained language models.

MegaTron

Model:

This model has 530 billion parameters with 105 layers, 20480 hidden dimensions, and 128 attention heads. In this, 8-way tensor and 35-way pipeline parallelism, with 2048 sequence length, batch size 1920. It’s trained with 15 datasets consisting of a total of 339 billion tokens. During training, we opted to blend the datasets into heterogeneous batches according to variable sampling weights given in Figure 2, with an emphasis on higher-quality datasets. We trained the model on 270 billion tokens.

Performance:

It performed significantly well against reputed benchmarks like LAMBADA, RACE-h, BoolQ, PiQA, HellaSwag, WinoGrand, ANLI-R2, HANS, WiC with few-shots, zero-shots and one-shots. It performed well especially against Lambada, PiQA, HellaSwag and shows it’s performance over last word prediction, question answering, logical reasoning

Capabilities:

MT can be used for a variety of downstream activities, including conversational AI, question answering, machine translation, document categorization, ad copy production, code issue correction, and more. This is similar to other newly announced pre-trained language models. It also performed well against the mathematical interferences as well

Join Our Book's Discord Space

Join the book's Discord Workspace for Latest updates, Offers, Tech happenings around the world, New Release and Sessions with the Authors:

https://discord.bpbonline.com

Points to Remember

—

Today's Amazon Deals - https://amzn.to/3FeoGyg

–

Points to Remember

Natural language processing (NLP) has been one of the subdomains in the arena of Artificial Intelligence which only captures almost 1/5th market share and number of solutions, focusing on the interaction between computers and human language.

NLP uses computational techniques to enable computers to understand, interpret, and generate human language.

NLP has made significant advancements in recent years, thanks to the availability of large datasets, powerful computing resources, and advanced machine learning algorithms.

With its ability to process and understand human language, NLP is helping to bridge the gap between humans and machines and making our interactions with technology more intuitive and natural.

In the 2000s and 2010s, NLP made significant advancements with the development of deep learning algorithms and the availability of large datasets, such as Wikipedia and social media data.

In the latter part of the last decade, Natural Language Processing (NLP) has continued to advance, with researchers making significant progress in areas such as deep learning, transfer learning, and pre-training.

These models are trained on massive amounts of text data and can perform a wide range of NLP tasks, including text classification, question answering, and language generation.

In addition to these advancements, researchers have also focused on improving the robustness and fairness of NLP models.

This includes developing methods to detect and mitigate bias in language data and models and to ensure that NLP applications are accessible to people from diverse linguistic and cultural backgrounds.

Overall, these advancements in NLP have opened up new possibilities for developing more sophisticated and accurate language-based applications, from chatbots to virtual assistants, and are likely to have far-reaching implications for many industries in the years to come.

Join Our Book's Discord Space

Join the book's Discord Workspace for Latest updates, Offers, Tech happenings around the world, New Release and Sessions with the Authors:

https://discord.bpbonline.com

GPT and ChatGPT

—

Today's Amazon Deals - https://amzn.to/3FeoGyg

–

GPT and ChatGPT

Talking about the Generative Pre-trained Transformer (GPT), it is a sophisticated neural network architecture that underpins ChatGPT with their version 3.5 of the GPT series( known as InstructGPT), being their most recent development. The Transformers model, created by Google in 2017, is the foundation and the preliminary element for this GPT model. It is based on the intuition of the attention-based model that was first presented in the paper “Attention is all you need.”

GPT Series by OpenAI

Between 2019 and 2022, the whole GPT series had numerous technical model and hyper-parameter adjustments by openAI and they have been improvising on many micro-level changes. The entire GPT-3 consists of approximately 175B parameters in its entire model which is around 50x higher than the language model that Google introduced in 2018, BERT; though there are some heavily loaded language models available in the research of NLP - like Megatron-NLG, by NVIDIA, with 530B parameters which is composed of 560 DGX A100 servers, each containing eight A100 80GB GPUs, capable of auto-completing phrases and statements. Google’s PaLM scaled to 540B parameters is another example of such a highly multi-tasking NLP model, trained on the largest TPU of the world with 6144 chips. Google also introduced LaMDA; in contrast to the task-based replies that conventional models frequently provide, the model may produce conversational chat in a free-form manner, which also has around 137B parameters. The following bubble chart by Dr Alan D. Thompson blog series explains about the estimation on recent developments of heavy load models with large parameters in language model:

Figure 7.1: Leading NLP models with large parameters

[Source: Lifearchitect.ai]

Evolution of NLP

—

Today's Amazon Deals - https://amzn.to/3FeoGyg

–

Evolution of NLP

According to Stanford university, the first need towards NLP began during World war II where urgency translation was reflected. Back to the 1950s when researchers began to explore the possibility of using computers to understand and generate human language. In 1950, Alan Turing proposed the “Turing Test,” a benchmark for machine intelligence that involved a computer’s ability to carry on a conversation that was indistinguishable from a human. This led to the development of early NLP systems, such as the “ELIZA” program developed in the 1960s, which simulated a conversation between a computer and a human therapist.

In the 1970s, researchers began to develop more advanced NLP algorithms, such as the “SHRDLU” program, which could understand natural language commands and manipulate virtual objects in a simulated environment. In the 1980s and 1990s, researchers focused on developing statistical models for language processing, which allowed computers to learn from large datasets of human language.

In the 2000s and 2010s, NLP made significant advancements with the development of deep learning algorithms and the availability of large datasets, such as Wikipedia and social media data. These advancements have led to the development of more sophisticated NLP applications, such as voice assistants, chatbots, and machine translation.

In the latter part of the last decade, Natural Language Processing (NLP) has continued to advance, with researchers making significant progress in areas such as deep learning, transfer learning, and pre-training.

One of the most significant developments in NLP has been the emergence of large pre-trained language models such as BERT (Bidirectional Encoder Representations from Transformers), GPT-2 (Generative Pre-trained Transformer 2), and GPT-3. These models are trained on massive amounts of text data and can perform a wide range of NLP tasks, including text classification, question answering, and language generation. They have enabled researchers to achieve state-of-the-art results on a variety of NLP benchmarks.

Another important development in NLP has been the use of transfer learning, where models are first pre-trained on a large dataset and then fine-tuned for a specific task. This approach has been used to achieve high performance on a variety of NLP tasks, including sentiment analysis, named entity recognition, and text classification.

In addition to these advancements, researchers have also focused on improving the robustness and fairness of NLP models. This includes developing methods to detect and mitigate bias in language data and models and to ensure that NLP applications are accessible to people from diverse linguistic and cultural backgrounds.

Overall, these advancements in NLP have opened up new possibilities for developing more sophisticated and accurate language-based applications, from chatbots to virtual assistants, and are likely to have far-reaching implications for many industries in the years to come. From then, LUNAR- scientific qualitative data, ELIZA - the first chatbot, from the complex models and use cases of today’s date such as smart Alexa, conversational bots is Siri with high-level complex neural networks at backend. In the context of ChatGPT, it’s one of the modern advanced NLP architectures developed, which is able to perform very high level tasks with more quantitative and qualitative accuracy and precision, closer to human perceptions and interpretations. In between, there has been a gradual yet constant development of the process of improvement from Word2Vec model to today’s ChatGPT through neural networks, LSTM models, encoder-decoder, Attention models, Transformer model, Google’ BERT, imageBERT.

Introduction to Natural Language Processing

—

Today's Amazon Deals - https://amzn.to/3FeoGyg

–

Introduction to Natural Language Processing

NLP uses computational techniques to enable computers to understand, interpret, and generate human language. It is one of crucial segments of AI which deals with the linguistic tasks and automates the process of analyzing and getting meaningful context out of any phrase. The tasks involve sentiment analysis, context-mapping, chatbots, content predictions, captioning, answer generation, machine translation, content classification etc and are used across different industries like banking, finance, customer service, health and medical, educational and almost in every other entity. NLP has made significant advancements in recent years, thanks to the availability of large datasets, powerful computing resources, and advanced machine learning algorithms. With its ability to process and understand human language, NLP is helping to bridge the gap between humans and machines and making our interactions with technology more intuitive and natural.

Chapter 7: ChatGPT Technical Overview—Introduction

—

Today's Amazon Deals - https://amzn.to/3FeoGyg

–

Chapter 7: ChatGPT Technical Overview—Introduction

Introduction

Artificial Intelligence or Machine learning, provides automated both supervised and unsupervised learning across many modalities, be it textual, or imagery, or vocal, maybe across different types such as numerical data, contextual data, feature-based data, pattern-based data. Natural language processing (NLP) has been one of the subdomains in the arena of Artificial Intelligence which only captures almost 1/5th market share and number of solutions, focusing on the interaction between computers and human language.

Points to Remember

—

Today's Amazon Deals - https://amzn.to/3FeoGyg

–

Points to Remember

The level of authenticity and validity of content created by Generative AI depends on the specific application and the quality of the training data used to develop the AI model.

There are limitations to the accuracy and reliability of Generative AI-generated content.

Another challenge is the difficulty of capturing nuances and context in the generated content.

Despite these limitations and challenges, there are still many applications where Generative AI-generated content can be useful and effective.

While Generative AI has many potential benefits and use cases, there are also several dangers and potential negative consequences associated with its use.

Some of the main dangers of Generative AI include - Spread of Misinformation: Generative AI can be used to create fake news, fake reviews, and other forms of misinformation.

In order to mitigate these dangers, it is important to develop ethical guidelines and best practices for the development and use of Generative AI.

ChatGPT is a variant of the GPT (Generative Pre-training Transformer) language model, which uses a transformer architecture.

ChatGPT may also use other types of AI models and techniques, such as language understanding models, to perform tasks such as named entity recognition and sentiment analysis.

Join Our Book's Discord Space

Join the book's Discord Workspace for Latest updates, Offers, Tech happenings around the world, New Release and Sessions with the Authors:

https://discord.bpbonline.com