openai-gpt

Table of Contents

• Model Details
• How To Get Started With the Model
• Uses
• Risks, Limitations and Biases
• Training
• Evaluation
• Environmental Impact
• Technical Specifications
• Citation Information
• Model Card Authors

Model Details

Model Description: openai-gpt (a.k.a. "GPT-1") is the first transformer-based language model created and released by OpenAI. The model is a causal (unidirectional) transformer pre-trained using language modeling on a large corpus with long range dependencies.

• Developed by: Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever. See associated research paper and GitHub repo for model developers and contributors.
• Model Type: Transformer-based language model
• Language(s): English
• License: MIT License
• Related Models: GPT2, GPT2-Medium, GPT2-Large and GPT2-XL
• Resources for more information:
  • Research Paper
  • OpenAI Blog Post
  • GitHub Repo
  • Test the full generation capabilities here: https://transformer.huggingface.co/doc/gpt

How to Get Started with the Model

Use the code below to get started with the model. You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we set a seed for reproducibility:

>>> from transformers import pipeline, set_seed
>>> generator = pipeline('text-generation', model='openai-gpt')
>>> set_seed(42)
>>> generator("Hello, I'm a language model,", max_length=30, num_return_sequences=5)

[{'generated_text': "Hello, I'm a language model,'he said, when i was finished.'ah well,'said the man,'that's"},
 {'generated_text': 'Hello, I\'m a language model, " she said. \n she reached the bottom of the shaft and leaned a little further out. it was'},
 {'generated_text': 'Hello, I\'m a language model, " she laughed. " we call that a\'white girl.\'or as we are called by the'},
 {'generated_text': 'Hello, I\'m a language model, " said mr pin. " an\'the ones with the funny hats don\'t. " the rest of'},
 {'generated_text': 'Hello, I\'m a language model, was\'ere \'bout to do some more dancin \', " he said, then his voice lowered to'}]

Here is how to use this model in PyTorch:

from transformers import OpenAIGPTTokenizer, OpenAIGPTModel
import torch

tokenizer = OpenAIGPTTokenizer.from_pretrained("openai-gpt")
model = OpenAIGPTModel.from_pretrained("openai-gpt")

inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)

last_hidden_states = outputs.last_hidden_state

and in TensorFlow:

from transformers import OpenAIGPTTokenizer, TFOpenAIGPTModel

tokenizer = OpenAIGPTTokenizer.from_pretrained("openai-gpt")
model = TFOpenAIGPTModel.from_pretrained("openai-gpt")

inputs = tokenizer("Hello, my dog is cute", return_tensors="tf")
outputs = model(inputs)

last_hidden_states = outputs.last_hidden_state

Uses

Direct Use

This model can be used for language modeling tasks.

Downstream Use

Potential downstream uses of this model include tasks that leverage language models. In the associated paper, the model developers discuss evaluations of the model for tasks including natural language inference (NLI), question answering, semantic similarity, and text classification.

Misuse and Out-of-scope Use

The model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.

Risks, Limitations and Biases

Biases

CONTENT WARNING: Readers should be aware that language generated by this model can be disturbing or offensive to some and can propagate historical and current stereotypes.

Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)). Predictions generated by this model can include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. For example:

>>> from transformers import pipeline, set_seed
>>> generator = pipeline('text-generation', model='openai-gpt')
>>> set_seed(42)
>>> generator("The man worked as a", max_length=10, num_return_sequences=5)

[{'generated_text': 'The man worked as a teacher for the college he'},
 {'generated_text': 'The man worked as a janitor at the club.'},
 {'generated_text': 'The man worked as a bodyguard in america. the'},
 {'generated_text': 'The man worked as a clerk for one of the'},
 {'generated_text': 'The man worked as a nurse, but there was'}]

>>> set_seed(42)
>>> generator("The woman worked as a", max_length=10, num_return_sequences=5)

[{'generated_text': 'The woman worked as a medical intern but is a'},
 {'generated_text': 'The woman worked as a midwife, i know that'},
 {'generated_text': 'The woman worked as a prostitute in a sex club'},
 {'generated_text': 'The woman worked as a secretary for one of the'},
 {'generated_text