GPT and ChatGPT

—

Today's Amazon Deals - https://amzn.to/3FeoGyg

–

GPT and ChatGPT

Talking about the Generative Pre-trained Transformer (GPT), it is a sophisticated neural network architecture that underpins ChatGPT with their version 3.5 of the GPT series( known as InstructGPT), being their most recent development. The Transformers model, created by Google in 2017, is the foundation and the preliminary element for this GPT model. It is based on the intuition of the attention-based model that was first presented in the paper “Attention is all you need.”

GPT Series by OpenAI

Between 2019 and 2022, the whole GPT series had numerous technical model and hyper-parameter adjustments by openAI and they have been improvising on many micro-level changes. The entire GPT-3 consists of approximately 175B parameters in its entire model which is around 50x higher than the language model that Google introduced in 2018, BERT; though there are some heavily loaded language models available in the research of NLP - like Megatron-NLG, by NVIDIA, with 530B parameters which is composed of 560 DGX A100 servers, each containing eight A100 80GB GPUs, capable of auto-completing phrases and statements. Google’s PaLM scaled to 540B parameters is another example of such a highly multi-tasking NLP model, trained on the largest TPU of the world with 6144 chips. Google also introduced LaMDA; in contrast to the task-based replies that conventional models frequently provide, the model may produce conversational chat in a free-form manner, which also has around 137B parameters. The following bubble chart by Dr Alan D. Thompson blog series explains about the estimation on recent developments of heavy load models with large parameters in language model:

Figure 7.1: Leading NLP models with large parameters

[Source: Lifearchitect.ai]