An Open-Source Toolkit for LLM Watermarking

🎉 We welcome PRs! If you have implemented a LLM watermarking algorithm or are interested in contributing one, we'd love to include it in MarkLLM. Join our community and help make text watermarking more accessible to everyone!

🔥 If you are interested in watermarking for diffusion models (image/video watermark), please refer to the MarkDiffusion toolkit from our group.

💡 Some other watermark papers from our team that may interest you ✨

1. (ICLR 2024) A Semantic Invariant Robust Watermark for Large Language Models

   Aiwei Liu, Leyi Pan, Xuming Hu, Shiao Meng, Lijie Wen

2. (ICLR 2024) An Unforgeable Publicly Verifiable Watermark for Large Language Models

   Aiwei Liu, Leyi Pan, Xuming Hu, Shu'ang Li, Lijie Wen, Irwin King, Philip S. Yu

3. (ACM Computing Surveys) A Survey of Text Watermarking in the Era of Large Language Models

   Aiwei Liu*, Leyi Pan*, Yijian Lu, Jingjing Li, Xuming Hu, Xi Zhang, Lijie Wen, Irwin King, Hui Xiong, Philip S. Yu

4. (ICLR 2025 Spotlight) Can Watermarked LLMs be Identified by Users via Crafted Prompts?

   Aiwei Liu, Sheng Guan, Yiming Liu, Leyi Pan, Yifei Zhang, Liancheng Fang, Lijie Wen, Philip S. Yu, Xuming Hu

5. (ACL 2025 Main) Can LLM Watermarks Robustly Prevent Unauthorized Knowledge Distillation?

   Leyi Pan, Aiwei Liu, Shiyu Huang, Yijian Lu, Xuming Hu, Lijie Wen, Irwin King, Philip S. Yu

6. (NAACL 2025 Findings) WaterSeeker: Pioneering Efficient Detection of Watermarked Segments in Large Documents

   Leyi Pan, Aiwei Liu, Yijian Lu, Zitian Gao, Yichen Di, Lijie Wen, Irwin King, Philip S. Yu

7. (ACL 2024 Main) An Entropy-based Text Watermarking Detection Method

   Yijian Lu, AIwei Liu, Dianzhi Yu, Jingjing Li, Irwin King

8. (ACL 2024 Main) Can Watermarks Survive Translation? On the Cross-lingual Consistency of Text Watermark for Large Language Models

   Zhiwei He, Binglin Zhou, Hongkun Hao, Aiwei Liu, Xing Wang, Zhaopeng Tu, Zhuosheng Zhang, Rui Wang

Contents

• MarkLLM: An Open-Source Toolkit for LLM Watermarking
  • Contents
  • Notes
  • Updates
  • Introduction to MarkLLM
    • Overview
    • Key Features of MarkLLM
  • How to use the toolkit in your own code
    • Setting up the environment
    • Invoking watermarking algorithms
    • Visualizing mechanisms
    • Applying evaluation pipelines
  • More user examples
  • Demo jupyter notebooks
  • Citations

❗❗❗ Notes

As the MarkLLM repository content becomes increasingly rich and its size grows larger, we have created a model storage repository on Hugging Face called Generative-Watermark-Toolkits to facilitate usage. This repository contains various default models for watermarking algorithms that involve self-trained models. We have removed the model weights from the corresponding model/ folders of these watermarking algorithms in the main repository. When using the code, please first download the corresponding models from the Hugging Face repository according to the config paths and save them to the model/ directory before running the code.

Updates

• 🎉 (2025.09.22) Add SemStamp watermark method. Thanks Huan Wang for her PR!
• 🎉 (2025.09.17) Add IE watermark method. Thanks Tianle Gu for her PR!
• 🎉 (2025.09.14) Add Watermark Stealing attack method. Thanks Shuhao Zhang for his PR!
• 🎉 (2025.07.17) Add k-SemStamp watermarking method. Thanks Huan Wang for her PR!
• 🎉 (2025.07.17) Add Adaptive Watermark watermarking method. Thanks Yepeng Liu for his PR!
• 🎉 (2025.05.24) Add MorphMark watermarking method. Thanks Zongqi Wang for his PR!
• 🎉 (2025.03.12) Add Permute-and-Flip (PF) watermarking method. Thanks Zian Wang for his PR!
• 🎉 (2025.02.27) Add δ-reweight and LLR score detection for Unbiased watermarking method.
• 🎉 (2025.01.08) Add AutoConfiguration for watermarking methods.
• 🎉 (2024.12.21) Provide example code for integrating VLLM with MarkLLM in MarkvLLM_demo.py. Thanks to @zhangjf-nlp for his PR!
• 🎉 (2024.11.21) Support distortionary version of SynthID-Text method (Nature).
• 🎉 (2024.11.03) Add SynthID-Text method (Nature) and support detection methods including mean, weighted mean, and bayesian.
• 🎉 (2024.11.01) Add TS-Watermark method (ICML 2024). Thanks to Kyle Zheng and Minjia Huo for their PR!
• 🎉 (2024.10.07) Provide an alternative, equivalent implementation of the EXP watermarking algorithm (EXPGumbel) utilizing Gumbel noise. With this implementation, users should be able to modify the watermark strength by adjusting the sampling temperature in the configuration file.
• 🎉 (2024.10.07) Add Unbiased watermarking method.
• 🎉 (2024.10.06) We are excited to announce that our paper "MarkLLM: An Open-Source Toolkit for LLM Watermarking" has been accepted by EMNLP 2024 Demo!
• 🎉 (2024.08.08) Add DiPmark watermarking method. Thanks to Sheng Guan for his PR!
• 🎉 (2024.08.01) Released as a python package! Try pip install markllm. We provide a user example at the end of this file.
• 🎉 (2024.07.13) Add ITSEdit watermarking method. Thanks to Yiming Liu for his PR!
• 🎉 (2024.07.09) Add more hashing schemes for KGW (skip, min, additive, selfhash). Thanks to Yichen Di for his PR!
• 🎉 (2024.07.08) Add top-k filter for watermarking methods in Christ family. Thanks to Kai Shi for his PR!
• 🎉 (2024.07.03) Updated Back-Translation Attack. Thanks to Zihan Tang for his PR!
• 🎉 (2024.06.19) Updated Random Walk Attack from the impossibility results of strong watermarking paper at ICML, 2024. (Blog). Thanks to Hanlin Zhang for his PR!
• 🎉 (2024.05.23) We're thrilled to announce the release of our website demo!

Introduction to MarkLLM

Overview

MarkLLM is an open-source toolkit developed to facilitate the research and application of watermarking technologies within large language models (LLMs). As the use of large language models (LLMs) expands, ensuring the authenticity and origin of machine-generated text becomes critical. MarkLLM simplifies the access, understanding, and assessment of watermarking technologies, making it accessible to both researchers and the broader community.

Key Features of MarkLLM

• Implementation Framework: MarkLLM provides a unified and extensible platform for the implementation of various LLM watermarking algorithms. It currently supports nine specific algorithms from two prominent families, facilitating the integration and expansion of watermarking techniques.

  Framework Design:

   ![unified_implementation](images/unified_implementation.png) 
  

  Currently Supported Algorithms:

  Algorithm Name	Publication	Link
  KGW	ICML 2023	[2301.10226] A Watermark for Large Language Models (arxiv.org)
  Unigram	ICLR 2024	[2306.17439] Provable Robust Watermarking for AI-Generated Text (arxiv.org)
  SWEET	ACL 2024	[2305.15060] Who Wrote this Code? Watermarking for Code Generation (arxiv.org)
  UPV	ICLR 2024	[2307.16230] An Unforgeable Publicly Verifiable Watermark for Large Language Models (arxiv.org)
  EWD	ACL 2024	[2403.13485] An Entropy-based Text Watermarking Detection Method (arxiv.org)
  SIR	ICLR 2024	[2310.06356] A Semantic Invariant Robust Watermark for Large Language Models (arxiv.org)
  X-SIR	ACL 2024	[2402.14007] Can Watermarks Survive Translation? On the Cross-lingual Consistency of Text Watermark for Large Language Models (arxiv.org)
  DiPmark	ICML 2024	[2310.07710] A Resilient and Accessible Distribution-Preserving Watermark for Large Language Models (arxiv.org)
  Unbiased Watermark	ICLR 2024	[2310.10669] Unbiased Watermark for Large Language Models (arxiv.org)
  TS Watermark	ICML 2024	[2402.18059] Token-Specific Watermarking with Enhanced Detectability and Semantic Coherence for Large Language Models (arxiv.org)
  SynthID-Text	Nature 2024	Scalable Watermarking for Identifying Large Language Model Outputs (Nature)
  PF Watermark	ICLR 2025	[2402.05864] Permute-and-Flip: An Optimally Stable and Watermarkable Decoder for LLMs
  MorphMark	ACL 2025	[2505.11541] MorphMark: Flexible Adaptive Watermarking for Large Language Models
  Adaptive Watermark	ICML 2024	[2401.13927] Adaptive Text Watermark for Large Language Models
  SemStamp	NAACL 2024	[2310.03991]SemStamp: A Semantic Watermark with Paraphrastic Robustness for Text Generation
  k-SemStamp	ACL 2024 (Findings)	[2402.11399] k-SemStamp: A Clustering-Based Semantic Watermark for Detection of Machine-Generated Text
  EXP/EXPGumbel	Lecture Note	https://www.scottaaronson.com/talks/watermark.ppt
  EXP-Edit	TMLR 2024	[2307.15593] Robust Distortion-free Watermarks for Language Models (arxiv.org)
  ITS-Edit	TMLR 2024	[2307.15593] Robust Distortion-free Watermarks for Language Models (arxiv.org)
  IE	Arxiv Preprint	[2505.14112] Invisible Entropy: Towards Safe and Efficient Low-Entropy LLM Watermarking (arxiv.org)
  TextSeal	Arxiv Preprint 2026	[2605.12456] TextSeal: A Dual-Key Watermark for Large Language Models
  E2E-LLM-Watermark	ICML 2025	[2505.02344] An End-to-End Model For Logits Based Large Language Models Watermarking
  DEW	Arxiv Preprint	[2606.31602] Robust Text Watermarking for Large Language Models via Dual Semantic Embeddings (arxiv.org)

• Visualization Solutions: The toolkit includes custom visualization tools that enable clear and insightful views into how different watermarking algorithms operate under various scenarios. These visualizations help demystify the algorithms' mechanisms, making them more understandable for users.

• Evaluation Module: With 12 evaluation tools that cover detectability, robustness, and impact on text quality, MarkLLM stands out in its comprehensive approach to assessing watermarking technologies. It also features customizable automated evaluation pipelines that cater to diverse needs and scenarios, enhancing the toolkit's practical utility.

  Tools:

  • Success Rate Calculator of Watermark Detection: FundamentalSuccessRateCalculator, DynamicThresholdSuccessRateCalculator
  • Text Editor: WordDeletion, SynonymSubstitution, ContextAwareSynonymSubstitution, GPTParaphraser, DipperParaphraser, RandomWalkAttack
  • Text Quality Analyzer: PPLCalculator, LogDiversityAnalyzer, BLEUCalculator, PassOrNotJudger, GPTDiscriminator

  Pipelines:

  • Watermark Detection Pipeline: WatermarkedTextDetectionPipeline, UnwatermarkedTextDetectionPipeline
  • Text Quality Pipeline: DirectTextQualityAnalysisPipeline, ReferencedTextQualityAnalysisPipeline, ExternalDiscriminatorTextQualityAnalysisPipeline

How to use the toolkit in your own code

Setting up the environment

• python 3.10
• pytorch
• pip install -r requirements.txt

Tips: If you wish to utilize the EXPEdit or ITSEdit algorithm, you will need to import for .pyx file, take EXPEdit as an example:

• run python watermark/exp_edit/cython_files/setup.py build_ext --inplace
• move the generated .so file into watermark/exp_edit/cython_files/

Invoking watermarking algorithms

import torch
from watermark.auto_watermark import AutoWatermark
from utils.transformers_config import TransformersConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

# Device
device = "cuda" if torch.cuda.is_available() else "cpu"

# Transformers config
transformers_config = TransformersConfig(model=AutoModelForCausalLM.from_pretrained('facebook/opt-1.3b').to(device),
                                         tokenizer=AutoTokenizer.from_pretrained('facebook/opt-1.3b'),
                                         vocab_size=50272,
                                         device=device,
                                         max_new_tokens=200,
                                         min_length=230,
                                         do_sample=True,
                                         no_repeat_ngram_size=4)
  
# Load watermark algorithm
myWatermark = AutoWatermark.load('KGW', 
                                 algorithm_config='config/KGW.json',
                                 transformers_config=transformers_config)

# Prompt
prompt = 'Good Morning.'

# Generate and detect
watermarked_text = myWatermark.generate_watermarked_text(prompt)
detect_result = myWatermark.detect_watermark(watermarked_text)
unwatermarked_text = myWatermark.generate_unwatermarked_text(prompt)
detect_result = myWatermark.detect_watermark(unwatermarked_text)

Visualizing mechanisms

Assuming you already have a pair of watermarked_text and unwatermarked_text, and you wish to visualize the differences and specifically highlight the watermark within the watermarked text using a watermarking algorithm, you can utilize the visualization tools available in the visualize/ directory.

KGW Family

import torch
from visualize.font_settings import FontSettings
from watermark.auto_watermark import AutoWatermark
from utils.transformers_config import TransformersConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from visualize.visualizer import DiscreteVisualizer
from visualize.legend_settings import DiscreteLegendSettings
from visualize.page_layout_settings import PageLayoutSettings
from visualize.color_scheme import ColorSchemeForDiscreteVisualization

# Load watermark algorithm
device = "cuda" if torch.cuda.is_available() else "cpu"
transformers_config = TransformersConfig(
    						model=AutoModelForCausalLM.from_pretrained('facebook/opt-1.3b').to(device),
                            tokenizer=AutoTokenizer.from_pretrained('facebook/opt-1.3b'),
                            vocab_size=50272,
                            device=device,
                            max_new_tokens=200,
                            min_length=230,
                            do_sample=True,
                            no_repeat_ngram_size=4)
myWatermark = AutoWatermark.load('KGW', 
                                 algorithm_config='config/KGW.json',
                                 transformers_config=transformers_config)
# Get data for visualization
watermarked_data = myWatermark.get_data_for_visualization(watermarked_text)
unwatermarked_data = myWatermark.get_data_for_visualization(unwatermarked_text)

# Init visualizer
visualizer = DiscreteVisualizer(color_scheme=ColorSchemeForDiscreteVisualization(),
                                font_settings=FontSettings(), 
                                page_layout_settings=PageLayoutSettings(),
                                legend_settings=DiscreteLegendSettings())
# Visualize
watermarked_img = visualizer.visualize(data=watermarked_data, 
                                       show_text=True, 
                                       visualize_weight=True, 
                                       display_legend=True)

unwatermarked_img = visualizer.visualize(data=unwatermarked_data,
                                         show_text=True, 
                                         visualize_weight=True, 
                                         display_legend=True)
# Save
watermarked_img.save("KGW_watermarked.png")
unwatermarked_img.save("KGW_unwatermarked.png")

Christ Family

import torch
from visualize.font_settings import FontSettings
from watermark.auto_watermark import AutoWatermark
from utils.transformers_config import TransformersConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from visualize.visualizer import ContinuousVisualizer
from visualize.legend_settings import ContinuousLegendSettings
from visualize.page_layout_settings import PageLayoutSettings
from visualize.color_scheme import ColorSchemeForContinuousVisualization

# Load watermark algorithm
device = "cuda" if torch.cuda.is_available() else "cpu"
transformers_config = TransformersConfig(
    						model=AutoModelForCausalLM.from_pretrained('facebook/opt-1.3b').to(device),
                            tokenizer=AutoTokenizer.from_pretrained('facebook/opt-1.3b'),
                            vocab_size=50272,
                            device=device,
                            max_new_tokens=200,
                            min_length=230,
                            do_sample=True,
                            no_repeat_ngram_size=4)
myWatermark = AutoWatermark.load('EXP', 
                                 algorithm_config='config/EXP.json',
                                 transformers_config=transformers_config)
# Get data for visualization
watermarked_data = myWatermark.get_data_for_visualization(watermarked_text)
unwatermarked_data = myWatermark.get_data_for_visualization(unwatermarked_text)

# Init visualizer
visualizer = ContinuousVisualizer(color_scheme=ColorSchemeForContinuousVisualization(),
                                  font_settings=FontSettings(), 
                                  page_layout_settings=PageLayoutSettings(),
                                  legend_settings=ContinuousLegendSettings())
# Visualize
watermarked_img = visualizer.visualize(data=watermarked_data, 
                                       show_text=True, 
                                       visualize_weight=True, 
                                       display_legend=True)

unwatermarked_img = visualizer.visualize(data=unwatermarked_data,
                                         show_text=True, 
                                         visualize_weight=True, 
                                         display_legend=True)
# Save
watermarked_img.save("EXP_watermarked.png")
unwatermarked_img.save("EXP_unwatermarked.png")

For more examples on how to use the visualization tools, please refer to the test/test_visualize.py script in the project directory.

Applying evaluation pipelines

Using Watermark Detection Pipelines

import torch
from evaluation.dataset import C4Dataset
from watermark.auto_watermark import AutoWatermark
from utils.transformers_config import TransformersConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from evaluation.tools.text_editor import TruncatePromptTextEditor, WordDeletion
from evaluation.tools.success_rate_calculator import DynamicThresholdSuccessRateCalculator
from evaluation.pipelines.detection import WatermarkedTextDetectionPipeline, UnWatermarkedTextDetectionPipeline, DetectionPipelineReturnType

# Load dataset
my_dataset = C4Dataset('dataset/c4/processed_c4.json')

# Device
device = 'cuda' if torch.cuda.is_available() else 'cpu'

# Transformers config
transformers_config = TransformersConfig(
    model=AutoModelForCausalLM.from_pretrained('facebook/opt-1.3b').to(device),
    tokenizer=AutoTokenizer.from_pretrained('facebook/opt-1.3b'),
    vocab_size=50272,
    device=device,
    max_new_tokens=200,
    do_sample=True,
    min_length=230,
    no_repeat_ngram_size=4)

# Load watermark algorithm
my_watermark = AutoWatermark.load('KGW', 
                                  algorithm_config='config/KGW.json',
                                  transformers_config=transformers_config)

# Init pipelines
pipeline1 = WatermarkedTextDetectionPipeline(
    dataset=my_dataset, 
    text_editor_list=[TruncatePromptTextEditor(), WordDeletion(ratio=0.3)],
    show_progress=True, 
    return_type=DetectionPipelineReturnType.SCORES) 

pipeline2 = UnWatermarkedTextDetectionPipeline(dataset=my_dataset, 
                                               text_editor_list=[],
                                               show_progress=True,
                                               return_type=DetectionPipelineReturnType.SCORES)

# Evaluate
calculator = DynamicThresholdSuccessRateCalculator(labels=['TPR', 'F1'], rule='best')
print(calculator.calculate(pipeline1.evaluate(my_watermark), pipeline2.evaluate(my_watermark)))

Using Text Quality Analysis Pipeline

import torch
from evaluation.dataset import C4Dataset
from watermark.auto_watermark import AutoWatermark
from utils.transformers_config import TransformersConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from evaluation.tools.text_editor import TruncatePromptTextEditor
from evaluation.tools.text_quality_analyzer import PPLCalculator
from evaluation.pipelines.quality_analysis import DirectTextQualityAnalysisPipeline, QualityPipelineReturnType

# Load dataset
my_dataset = C4Dataset('dataset/c4/processed_c4.json')

# Device
device = 'cuda' if torch.cuda.is_available() else 'cpu'

# Transformer config
transformers_config = TransformersConfig(
    model=AutoModelForCausalLM.from_pretrained('facebook/opt-1.3b').to(device),                             	tokenizer=AutoTokenizer.from_pretrained('facebook/opt-1.3b'),
    vocab_size=50272,
    device=device,
    max_new_tokens=200,
    min_length=230,
    do_sample=True,
    no_repeat_ngram_size=4)

# Load watermark algorithm
my_watermark = AutoWatermark.load('KGW', 
                                  algorithm_config='config/KGW.json',
                                  transformers_config=transformers_config)

# Init pipeline
quality_pipeline = DirectTextQualityAnalysisPipeline(
    dataset=my_dataset, 
    watermarked_text_editor_list=[TruncatePromptTextEditor()],
    unwatermarked_text_editor_list=[],                                             
    analyzer=PPLCalculator(
        model=AutoModelForCausalLM.from_pretrained('..model/llama-7b/', device_map='auto'),                 		tokenizer=LlamaTokenizer.from_pretrained('..model/llama-7b/'),
        device=device),
    unwatermarked_text_source='natural', 
    show_progress=True, 
    return_type=QualityPipelineReturnType.MEAN_SCORES)

# Evaluate
print(quality_pipeline.evaluate(my_watermark))

For more examples on how to use the pipelines, please refer to the test/test_pipeline.py script in the project directory.

Leveraging example scripts for evaluation

In the evaluation/examples/ directory of our repository, you will find a collection of Python scripts specifically designed for systematic and automated evaluation of various algorithms. By using these examples, you can quickly and effectively gauge the d etectability, robustness and impact on text quality of each algorithm implemented within our toolkit.

Note: To execute the scripts in evaluation/examples/, first run the following command to set the environment variables.

export PYTHONPATH="path_to_the_MarkLLM_project:$PYTHONPATH"

More user examples

Additional user examples are available in test/. To execute the scripts contained within, first run the following command to set the environment variables.

export PYTHONPATH="path_to_the_MarkLLM_project:$PYTHONPATH"

Demo jupyter notebooks

In addition to the Colab Jupyter notebook we provide (some models cannot be downloaded due to storage limits), you can also easily deploy using MarkLLM_demo.ipynb on your local machine.

Citations

@inproceedings{pan-etal-2024-markllm,
    title = "{M}ark{LLM}: An Open-Source Toolkit for {LLM} Watermarking",
    author = "Pan, Leyi  and
      Liu, Aiwei  and
      He, Zhiwei  and
      Gao, Zitian  and
      Zhao, Xuandong  and
      Lu, Yijian  and
      Zhou, Binglin  and
      Liu, Shuliang  and
      Hu, Xuming  and
      Wen, Lijie  and
      King, Irwin  and
      Yu, Philip S.",
    editor = "Hernandez Farias, Delia Irazu  and
      Hope, Tom  and
      Li, Manling",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.emnlp-demo.7",
    pages = "61--71",
    abstract = "Watermarking for Large Language Models (LLMs), which embeds imperceptible yet algorithmically detectable signals in model outputs to identify LLM-generated text, has become crucial in mitigating the potential misuse of LLMs. However, the abundance of LLM watermarking algorithms, their intricate mechanisms, and the complex evaluation procedures and perspectives pose challenges for researchers and the community to easily understand, implement and evaluate the latest advancements. To address these issues, we introduce MarkLLM, an open-source toolkit for LLM watermarking. MarkLLM offers a unified and extensible framework for implementing LLM watermarking algorithms, while providing user-friendly interfaces to ensure ease of access. Furthermore, it enhances understanding by supporting automatic visualization of the underlying mechanisms of these algorithms. For evaluation, MarkLLM offers a comprehensive suite of 12 tools spanning three perspectives, along with two types of automated evaluation pipelines. Through MarkLLM, we aim to support researchers while improving the comprehension and involvement of the general public in LLM watermarking technology, fostering consensus and driving further advancements in research and application. Our code is available at https://github.com/THU-BPM/MarkLLM.",
}