fix: Regex bug in Precompiled Normalizer

Author: CodeWithKyrian

examples/pipelines/summarization.php

@@ -13,22 +13,49 @@
 
 
 $summarizer = pipeline('summarization', 'Xenova/distilbart-cnn-6-6');
+//$summarizer = pipeline('summarization', 'Xenova/bart-large-cnn');
+//$summarizer = pipeline('summarization', 'Falconsai/text_summarization', quantized: false);
 
 $streamer = TextStreamer::make();
 
-$article = 'The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, ' .
-    'and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. ' .
-    'During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest ' .
-    'man-made structure in the world, a title it held for 41 years until the Chrysler Building in New ' .
-    'York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to ' .
-    'the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the ' .
-    'Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second-tallest ' .
-    'free-standing structure in France after the Millau Viaduct.';
-
-
-//$article = "I called my friend to see if he wanted to go to the movies. However, he was busy and said he would " .
-//    "call me back. I didn't hear back from him, so I called him again. He didn't answer, so I decided to go to the movies by myself.";
-
-$summary = $summarizer($article, streamer: $streamer, maxNewTokens: 512, temperature: 0.7);
+//$article = 'The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, ' .
+//    'and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. ' .
+//    'During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest ' .
+//    'man-made structure in the world, a title it held for 41 years until the Chrysler Building in New ' .
+//    'York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to ' .
+//    'the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the ' .
+//    'Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second-tallest ' .
+//    'free-standing structure in France after the Millau Viaduct.';
+
+$article = <<<EOF
+Hugging Face: Revolutionizing Natural Language Processing
+Introduction
+In the rapidly evolving field of Natural Language Processing (NLP), Hugging Face has emerged as a prominent and innovative force. This article will explore the story and significance of Hugging Face, a company that has made remarkable contributions to NLP and AI as a whole. From its inception to its role in democratizing AI, Hugging Face has left an indelible mark on the industry.
+The Birth of Hugging Face
+Hugging Face was founded in 2016 by Clément Delangue, Julien Chaumond, and Thomas Wolf. The name "Hugging Face" was chosen to reflect the company's mission of making AI models more accessible and friendly to humans, much like a comforting hug. Initially, they began as a chatbot company but later shifted their focus to NLP, driven by their belief in the transformative potential of this technology.
+Transformative Innovations
+Hugging Face is best known for its open-source contributions, particularly the "Transformers" library. This library has become the de facto standard for NLP and enables researchers, developers, and organizations to easily access and utilize state-of-the-art pre-trained language models, such as BERT, GPT-3, and more. These models have countless applications, from chatbots and virtual assistants to language translation and sentiment analysis.
+Key Contributions:
+1. **Transformers Library:** The Transformers library provides a unified interface for more than 50 pre-trained models, simplifying the development of NLP applications. It allows users to fine-tune these models for specific tasks, making it accessible to a wider audience.
+2. **Model Hub:** Hugging Face's Model Hub is a treasure trove of pre-trained models, making it simple for anyone to access, experiment with, and fine-tune models. Researchers and developers around the world can collaborate and share their models through this platform.
+3. **Hugging Face Transformers Community:** Hugging Face has fostered a vibrant online community where developers, researchers, and AI enthusiasts can share their knowledge, code, and insights. This collaborative spirit has accelerated the growth of NLP.
+Democratizing AI
+Hugging Face's most significant impact has been the democratization of AI and NLP. Their commitment to open-source development has made powerful AI models accessible to individuals, startups, and established organizations. This approach contrasts with the traditional proprietary AI model market, which often limits access to those with substantial resources.
+By providing open-source models and tools, Hugging Face has empowered a diverse array of users to innovate and create their own NLP applications. This shift has fostered inclusivity, allowing a broader range of voices to contribute to AI research and development.
+Industry Adoption
+The success and impact of Hugging Face are evident in its widespread adoption. Numerous companies and institutions, from startups to tech giants, leverage Hugging Face's technology for their AI applications. This includes industries as varied as healthcare, finance, and entertainment, showcasing the versatility of NLP and Hugging Face's contributions.
+Future Directions
+Hugging Face's journey is far from over. As of my last knowledge update in September 2021, the company was actively pursuing research into ethical AI, bias reduction in models, and more. Given their track record of innovation and commitment to the AI community, it is likely that they will continue to lead in ethical AI development and promote responsible use of NLP technologies.
+Conclusion
+Hugging Face's story is one of transformation, collaboration, and empowerment. Their open-source contributions have reshaped the NLP landscape and democratized access to AI. As they continue to push the boundaries of AI research, we can expect Hugging Face to remain at the forefront of innovation, contributing to a more inclusive and ethical AI future. Their journey reminds us that the power of open-source collaboration can lead to groundbreaking advancements in technology and bring AI within the reach of many.
+EOF;
+
+
+//
+//$article = "Existing law authorizes state agencies to enter into contracts for the acquisition of goods or services upon approval by the Department of General Services. Existing law sets forth various requirements and prohibitions for those contracts, including, but not limited to, a prohibition on entering into contracts for the acquisition of goods or services of $100,000 or more with a contractor that discriminates between spouses and domestic partners or same-sex and different-sex couples in the provision of benefits. Existing law provides that a contract entered into in violation of those requirements and prohibitions is void and authorizes the state or any person acting on behalf of the state to bring a civil action seeking a determination that a contract is in violation and therefore void. Under existing law, a willful violation of those requirements and prohibitions is a misdemeanor.\nThis bill would also prohibit a state agency from entering into contracts for the acquisition of goods or services of $100,000 or more with a contractor that discriminates between employees on the basis of gender identity in the provision of benefits, as specified. By expanding the scope of a crime, this bill would impose a state-mandated local program.\nThe California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.\nThis bill would provide that no reimbursement is required by this act for a specified reason.',
+// 'text': 'The people of the State of California do enact as follows:\n\n\nSECTION 1.\nSection 10295.35 is added to the Public Contract Code, to read:\n10295.35.\n(a) (1) Notwithstanding any other law, a state agency shall not enter into any contract for the acquisition of goods or services in the amount of one hundred thousand dollars ($100,000) or more with a contractor that, in the provision of benefits, discriminates between employees on the basis of an employee’s or dependent’s actual or perceived gender identity, including, but not limited to, the employee’s or dependent’s identification as transgender.\n(2) For purposes of this section, “contract” includes contracts with a cumulative amount of one hundred thousand dollars ($100,000) or more per contractor in each fiscal year.\n(3) For purposes of this section, an employee health plan is discriminatory if the plan is not consistent with Section 1365.5 of the Health and Safety Code and Section 10140 of the Insurance Code.\n(4) The requirements of this section shall apply only to those portions of a contractor’s operations that occur under any of the following conditions:\n(A) Within the state.\n(B) On real property outside the state if the property is owned by the state or if the state has a right to occupy the property, and if the contractor’s presence at that location is connected to a contract with the state.\n(C) Elsewhere in the United States where work related to a state contract is being performed.\n(b) Contractors shall treat as confidential, to the maximum extent allowed by law or by the requirement of the contractor’s insurance provider, any request by an employee or applicant for employment benefits or any documentation of eligibility for benefits submitted by an employee or applicant for employment.\n(c) After taking all reasonable measures to find a contractor that complies with this section, as determined by the state agency, the requirements of this section may be waived under any of the following circumstances:\n(1) There is only one prospective contractor willing to enter into a specific contract with the state agency.\n(2) The contract is necessary to respond to an emergency, as determined by the state agency, that endangers the public health, welfare, or safety, or the contract is necessary for the provision of essential services, and no entity that complies with the requirements of this section capable of responding to the emergency is immediately available.\n(3) The requirements of this section violate, or are inconsistent with, the terms or conditions of a grant, subvention, or agreement, if the agency has made a good faith attempt to change the terms or conditions of any grant, subvention, or agreement to authorize application of this section.\n(4) The contractor is providing wholesale or bulk water, power, or natural gas, the conveyance or transmission of the same, or ancillary services, as required for ensuring reliable services in accordance with good utility practice, if the purchase of the same cannot practically be accomplished through the standard competitive bidding procedures and the contractor is not providing direct retail services to end users.\n(d) (1) A contractor shall not be deemed to discriminate in the provision of benefits if the contractor, in providing the benefits, pays the actual costs incurred in obtaining the benefit.\n(2) If a contractor is unable to provide a certain benefit, despite taking reasonable measures to do so, the contractor shall not be deemed to discriminate in the provision of benefits.\n(e) (1) Every contract subject to this chapter shall contain a statement by which the contractor certifies that the contractor is in compliance with this section.\n(2) The department or other contracting agency shall enforce this section pursuant to its existing enforcement powers.\n(3) (A) If a contractor falsely certifies that it is in compliance with this section, the contract with that contractor shall be subject to Article 9 (commencing with Section 10420), unless, within a time period specified by the department or other contracting agency, the contractor provides to the department or agency proof that it has complied, or is in the process of complying, with this section.\n(B) The application of the remedies or penalties contained in Article 9 (commencing with Section 10420) to a contract subject to this chapter shall not preclude the application of any existing remedies otherwise available to the department or other contracting agency under its existing enforcement powers.\n(f) Nothing in this section is intended to regulate the contracting practices of any local jurisdiction.\n(g) This section shall be construed so as not to conflict with applicable federal laws, rules, or regulations. In the event that a court or agency of competent jurisdiction holds that federal law, rule, or regulation invalidates any clause, sentence, paragraph, or section of this code or the application thereof to any person or circumstances, it is the intent of the state that the court or agency sever that clause, sentence, paragraph, or section so that the remainder of this section shall remain in effect.\nSEC. 2.\nSection 10295.35 of the Public Contract Code shall not be construed to create any new enforcement authority or responsibility in the Department of General Services or any other contracting agency.\nSEC. 3.\nNo reimbursement is required by this act pursuant to Section 6 of Article XIII\u2009B of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIII\u2009B of the California Constitution.',
+// 'title': 'An act to add Section 10295.35 to the Public Contract Code, relating to public contracts.";
+
+$summary = $summarizer($article, streamer: $streamer, maxNewTokens: 512);
 
 dd("Done", timeUsage(), memoryUsage());

src/Generation/Streamers/Streamer.php

@@ -23,17 +23,7 @@ abstract class Streamer
 
     public static function make(): static
     {
-        $streamer =  new static();
-
-        $streamer->onStreamCallback ??= function ($value) {
-            echo $value;
-        };
-
-        $streamer->onStreamEndCallback ??= function () {
-            echo PHP_EOL;
-        };
-
-        return $streamer;
+        return new static();
     }
 
     public function setTokenizer(PretrainedTokenizer $tokenizer): static

src/Generation/Streamers/TextStreamer.php

@@ -18,6 +18,21 @@ class TextStreamer extends Streamer
     protected int $lastDecodedCheckpointForToken = 0;
     protected int $lastDecodedCheckpointForText = 0;
 
+    public static function make(): static
+    {
+        $streamer = parent::make();
+
+        $streamer->onStreamCallback ??= function ($value) {
+            echo $value;
+        };
+
+        $streamer->onStreamEndCallback ??= function () {
+            echo PHP_EOL;
+        };
+
+        return $streamer;
+    }
+
     public function put(mixed $value): void
     {
         if (count($value) > 1) {
@@ -43,8 +58,8 @@ public function put(mixed $value): void
         $punctuationMarks = ['.', ',', '!', '?', ';', ':'];
 
         $this->printedText = mb_substr($this->printedText, 0, $this->lastDecodedCheckpointForText)
-            . ($this->lastDecodedCheckpointForToken == 0 ? '' : ' ')
-            . $decodedText;
+            .($this->lastDecodedCheckpointForToken == 0 ? '' : ' ')
+            .$decodedText;
 
         $newText = mb_substr($this->printedText, $this->printedLength);
 

src/Normalizers/Precompiled.php

@@ -22,7 +22,9 @@ public function __construct(array $config)
 
     /**
      * Normalizes the given text by applying the precompiled charsmap.
+     *
      * @param string $text The text to normalize.
+     *
      * @return string The normalized text.
      */
     public function normalize(string $text): string
@@ -37,14 +39,13 @@ public function normalize(string $text): string
         //
         // For now, we only implement the default (nmt_nfkc).
         // See https://raw.githubusercontent.com/google/sentencepiece/master/data/nmt_nfkc.tsv for the full list of rules.
-        // TODO: detect when a different `this.charsmap` is used.
-
+        // TODO: detect when a different `$this->charsMap` is used.
 
         // Remove control characters
         $text = preg_replace('/[\x01-\x08\x0B\x0E-\x1F\x7F\x8F\x9F]/u', '', $text);
 
         // Replace certain characters with a space
-        $text = preg_replace('/[\x09\x0A\x0C\x0D\x1680\x200B\x200C\x200E\x200F\x2028\x2029\x2581\xFEFF\xFFFD]/u', ' ', $text);
+        $text = preg_replace('/[\x09\x0A\x0C\x0D\x{1680}\x{200B}\x{200C}\x{200E}\x{200F}\x{2028}\x{2029}\x{2581}\x{FEFF}\x{FFFD}]/u', ' ', $text);
 
         if (mb_strpos($text, '～') !== false) {
             // To match the sentencepiece implementation 100%, we must handle a very strange edge-case.