Raw capture — raw/hackaprompt-dataset.md

url: https://huggingface.co/datasets/hackaprompt/hackaprompt-dataset date_fetched: 2026-07-05 backfilled: true

Dataset Card for HackAPrompt 💻🔍

This dataset contains submissions from a prompt hacking competition. An in-depth analysis of the dataset has been accepted at the EMNLP 2023 conference. 📊👾

Submissions were sourced from two environments: a playground for experimentation and an official submissions platform.

The playground itself can be accessed here 🎮

More details about the competition itself here 🏆

Dataset Details 📋

Dataset Description 📄

We conducted a prompt hacking competition where users were competing to "hack" different large language models (LLMs). Different levels were proposed, with varying degrees of difficulty, and for each level, 3 LLMs were evaluated: GPT-3 (text-davinci-003), FlanT5-XXL (philschmid/flan-t5-xxl-sharded-fp16), and ChatGPT (gpt-3.5-turbo).

We anonymously collected user submissions throughout the competition and also had users submit their best attempts via an online platform for a chance to win the competition. Users submitted their prompts, and our servers automatically evaluated their attempts. To delineate between ties, token counts were used where lower counts gave better scores.

This dataset releases all submissions sent to both our playground and submission servers. 📤📥

Columns Description 🧾

• level: A numerical value indicating the difficulty or complexity of the prompt.
• user_input: The input provided by the user or participant in response to the given challenge.
• prompt: The full prompt that was used to query the model, this includes the user's input.
• completion: The output or completion generated by the model based on the user's input.
• model: The type or version of the model that generated the completion. For example, "gpt-3.5-turbo" or "FlanT5-XXL".
• expected_completion: The expected or ideal output that should have been generated by the model for the given user input.
• token_count: The number of tokens present in the user's input. This serves as a measure of the input's length.
• correct: A boolean value indicating whether the model's completion was correct or not, based on the expected output.
• error: A boolean value indicating if there was an error during the model's processing of the user input. Note: we did not include submissions that triggered errors in this dataset.
• score: A numerical value representing the score assigned to the model's completion based on its accuracy, correctness, and other evaluation metrics. (Only available for prompts on the submissions platform)
• dataset: A categorical variable indicating the source of the submission. The two categories are "playground_data" (for submissions from the playground environment) and "submission_data" (for official submissions).
• timestamp: The date and time when the submission was made. (Only available for playground dataset)

Uses 🧑🔬

This dataset is meant to be used in a research context to better understand the different types of attacks "in the wild" on LLMs. 📚🔬

Personal and Sensitive Information 🔒

We did not release directly any personal or sensitive information explicitly. On the playground, users could submit anonymously, and we did not collect information about the users directly.

For the submissions data, teams did submit in their names, but that information has not been made available in this version of the dataset to preserve participants' privacy.

Bias, Risks, and Limitations ⚠️

The data was submitted via a public portal hosted on huggingface. We did not curate the data before publishing it. The data may contain offensive material.

Please use at your own risk.

Recommendations 🚀

Users should be made aware of the risks, biases, and limitations of the dataset and use at their own risk.

Please use at your own risk.

Citation 📝

BibTeX:

https://arxiv.org/abs/2311.16119

@inproceedings{Schulhoff:Pinto:Khan:Bouchard:Si:Boyd-Graber:Anati:Tagliabue:Kost:Carnahan-2023,
  Title = {Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs Through a Global Prompt Hacking Competition},
  Author = {Sander V Schulhoff and Jeremy Pinto and Anaum Khan and Louis-François Bouchard and Chenglei Si and Jordan Lee Boyd-Graber and Svetlina Anati and Valen Tagliabue and Anson Liu Kost and Christopher R Carnahan},
  Booktitle = {Empirical Methods in Natural Language Processing},
  Year = {2023},
  Location = {Singapore}
}

• Downloads last month
• 1,234