Tranformer-based Denoising AutoEncoder for Sentence Transformers Unsupervised pre-training

The acquisition of sentence embeddings often necessitates a substantial volume of labeled data. However, in many cases and fields, labeled data is rarely accessible, and the procurement of such data is costly. In this project, we employ an unsupervised process grounded in pre-trained Transformers-based Sequential Denoising Auto-Encoder (TSDAE), introduced by the Ubiquitous Knowledge Processing Lab of Darmstadt, which can realize a performance level reaching 93.1% of in-domain supervised methodologies.

The TSDAE schema comprises two components: an encoder and a decoder. Throughout the training process, TSDAE translates tainted sentences into uniform-sized vectors, necessitating the decoder to reconstruct the original sentences utilizing this sentence embedding. For good reconstruction quality, the semantics must be captured well in the sentence embeddings from the encoder. Subsequently, during inference, the encoder is solely utilized to form sentence embeddings.

Moreover, TSDAE serves as an effective pre-training technique, surpassing the classical Mask Language Model (MLM) pre-training task in performance.

Dependencies

Below is a list of the main dependencies for TSDAE:

• nltk: The Natural Language Toolkit (NLTK) is a suite of libraries and programs for symbolic and statistical natural language processing. In TSDAE, it's primarily used for text preprocessing tasks, such as tokenization.
• re: A library for regular expression operations in Python, utilized for text cleaning and splitting operations within TSDAE to prepare text data for further processing.
• random: Provides functionalities for generating random numbers, used in TSDAE for shuffling datasets and sampling data subsets in various preprocessing steps.
• logging: Facilitates logging events for debugging and tracking the execution process. TSDAE uses the logging module to record critical information, errors, and progress updates during the model's operation.
• datasets: Part of the Hugging Face ecosystem, this library simplifies the loading and processing of large-scale datasets. TSDAE uses it to fetch and prepare datasets for training the sentence embeddings.
• sentence_transformers: A framework for state-of-the-art sentence, text, and image embeddings. TSDAE leverages this library to create and train the underlying Transformer models for generating sentence embeddings.
• torch: The PyTorch library provides a wide array of deep learning and tensor computation tools with GPU acceleration support. It is central to TSDAE for modeling and training the denoising autoencoder.
• ssl: Used for handling Secure Sockets Layer (SSL) and Transport Layer Security (TLS) encryption in Python. In TSDAE, it is optionally employed to create unverified HTTPS contexts, facilitating dataset retrieval in environments with strict SSL certificate requirements.

The dependencies above are instrumental in dataset handling, model training, and embedding processes, constituting the backbone of TSDAE's operational infrastructure.

Installation

Before proceeding with TSDAE, ensure that all dependencies are installed through the Python package manager pip:

pip install tsdae nltk datasets sentence-transformers torch

Note: Additional steps, such as configuring unverified HTTPS contexts, may be necessary depending on your execution environment.

Usage

Here's how you can use tsdae:

1. Installation: Install the required libraries, including Torch, Transformers...
2. Initialization: Create an instance of the TSDAE class.

Model Architecture

The TSDAE model is bifurcated into two primary components:

• Encoder: The encoder processes input sentences that have been deliberately corrupted, converting them into fixed-sized sentence embeddings. Essential to the model's success is the encoder's ability to distill and encode the semantic essence of the sentences into these embeddings.

• Decoder: Tasked with the challenge of reconstruction, the decoder utilizes the sentence embeddings generated by the encoder to recreate the original sentences. The quality of reconstruction is directly proportional to the semantic information retained within the sentence embeddings.

Usage Example

Below is a concise illustration of employing TSDAE to train a model on a dataset named "louisbrulenaudet/cgi", showcasing the seamless integration of components from dataset preparation to model training.

from tsdae import TSDAE

# Initialize an instance of TSDAE
instance = TSDAE()

# Load a dataset
train_dataset = instance.load_dataset_from_hf(
    dataset="louisbrulenaudet/cgi"
)

# Train the model with the dataset
model = instance.train(
    train_dataset=train_dataset,
    model_name="bert-base-multilingual-uncased",
    column="output",
    output_path="output/tsdae-lemon-mbert-base"
)

This example encapsulates the simplicity and power of TSDAE, guiding users from dataset acquisition to model optimization with minimal overhead.

References

Wang, K., Reimers, N., & Gurevych, I. (2021). TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning. arXiv. Retrieved from http://arxiv.org/abs/2104.06979.

Citing this project

If you use this code in your research, please use the following BibTeX entry.

@misc{louisbrulenaudet2023,
	author = {Louis Brulé Naudet},
	title = {Tranformer-based Denoising AutoEncoder for Sentence Transformers Unsupervised pre-training},
	howpublished = {\url{https://github.com/louisbrulenaudet/tsdae}},
	year = {2024}
}

Feedback

If you have any feedback, please reach out at [email protected].