Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: CodeWithKyrian/transformers-php
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 0.3.1
Choose a base ref
...
head repository: CodeWithKyrian/transformers-php
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref
Loading
Showing with 41,850 additions and 3,853 deletions.
  1. +41 −0 .github/workflows/release.yml
  2. +15 −4 .gitignore
  3. +225 −0 CHANGELOG.md
  4. +1 −1 LICENSE
  5. +4 −4 README.md
  6. +1 −0 VERSION
  7. +1 −1 bin/transformers
  8. +20 −9 composer.json
  9. +8 −0 docs/.vitepress/config.mts
  10. +110 −0 docs/audio-classification.md
  11. +146 −0 docs/automatic-speech-recognition.md
  12. +25 −28 docs/configuration.md
  13. +31 −9 docs/getting-started.md
  14. +1 −1 docs/image-classification.md
  15. +11 −2 docs/pipelines.md
  16. +122 −1 docs/text-generation.md
  17. +3 −3 docs/text-to-text-generation.md
  18. +3 −3 docs/tokenizers.md
  19. +223 −1 docs/utils/generation.md
  20. +64 −29 docs/utils/image.md
  21. +300 −41 docs/utils/tensor.md
  22. +5 −3 examples/bootstrap.php
  23. +33 −27 examples/composer.json
  24. +0 −50 examples/image_detection.php
  25. BIN examples/images/astronaut.png
  26. BIN examples/images/beach.png
  27. BIN examples/images/butterfly.jpg
  28. BIN examples/images/cats.jpg
  29. BIN examples/images/code-screenshot.png
  30. BIN examples/images/corgi.jpg
  31. BIN examples/images/handwriting.jpg
  32. BIN examples/images/handwriting2.jpeg
  33. BIN examples/images/handwriting3.png
  34. BIN examples/images/handwriting4.jpeg
  35. BIN examples/images/kyrian-cartoon.jpeg
  36. BIN examples/images/tiger.jpg
  37. BIN examples/images/typed-text2.png
  38. BIN examples/images/typed-text3.png
  39. BIN examples/images/typed.png
  40. +31 −0 examples/misc/background-removal.php
  41. +72 −0 examples/misc/custom-object-detection.php
  42. +17 −0 examples/misc/general-test.php
  43. +15 −10 examples/{ → misc}/image-test.php
  44. +38 −0 examples/pipelines/asr.php
  45. +18 −0 examples/pipelines/audio-classification.php
  46. +1 −1 examples/pipelines/image-to-image.php
  47. +3 −3 examples/pipelines/image-to-text.php
  48. +14 −13 examples/pipelines/object-detection.php
  49. +45 −18 examples/pipelines/summarization.php
  50. +3 −7 examples/pipelines/text-classification.php
  51. +12 −8 examples/pipelines/text-generation.php
  52. +9 −9 examples/pipelines/text2text-generation.php
  53. +3 −3 examples/pipelines/token-classification.php
  54. +6 −4 examples/pipelines/translation.php
  55. +2 −2 examples/tokenizers/apply-chat-template.php
  56. +2 −0 libs/.gitignore
  57. +28 −55 scripts/convert_upload_hf.ipynb
  58. +63 −11 src/Commands/DownloadModelCommand.php
  59. +69 −0 src/Commands/InstallCommand.php
  60. +8 −4 src/DataStructures/CharTrie.php
  61. +2 −2 src/DataStructures/TokenLattice.php
  62. +26 −17 src/Decoders/ByteFallback.php
  63. +263 −16 src/Decoders/ByteLevelDecoder.php
  64. +3 −3 src/Decoders/CTCDecoder.php
  65. +1 −1 src/Decoders/FuseDecoder.php
  66. +18 −14 src/Decoders/ReplaceDecoder.php
  67. +7 −6 src/Decoders/StripDecoder.php
  68. +20 −0 src/Decoders/VitsDecoder.php
  69. +3 −3 src/Decoders/WordPieceDecoder.php
  70. +68 −0 src/FFI/Libc.php
  71. +759 −0 src/FFI/OnnxRuntime.php
  72. +141 −0 src/FFI/Samplerate.php
  73. +181 −0 src/FFI/Sndfile.php
  74. +116 −0 src/FFI/TransformersUtils.php
  75. +75 −0 src/FeatureExtractors/ASTFeatureExtractor.php
  76. +1 −2 src/FeatureExtractors/DetrFeatureExtractor.php
  77. +3 −1 src/FeatureExtractors/FeatureExtractor.php
  78. +139 −174 src/FeatureExtractors/ImageFeatureExtractor.php
  79. +11 −5 src/FeatureExtractors/Swin2SRImageProcessor.php
  80. +44 −0 src/FeatureExtractors/Wav2Vec2FeatureExtractor.php
  81. +74 −0 src/FeatureExtractors/WhisperFeatureExtractor.php
  82. +1 −1 src/{Utils → Generation}/AggregationStrategy.php
  83. +3 −4 src/Generation/LogitsProcessors/BadWordsLogitsProcessor.php
  84. +7 −8 src/Generation/LogitsProcessors/ForceTokensLogitsProcessor.php
  85. +3 −6 src/Generation/LogitsProcessors/ForcedBOSTokenLogitsProcessor.php
  86. +3 −6 src/Generation/LogitsProcessors/ForcedEOSTokenLogitsProcessor.php
  87. +2 −3 src/Generation/LogitsProcessors/LogitsProcessor.php
  88. +1 −7 src/Generation/LogitsProcessors/LogitsProcessorList.php
  89. +2 −3 src/Generation/LogitsProcessors/MinLengthLogitsProcessor.php
  90. +2 −3 src/Generation/LogitsProcessors/MinNewTokensLengthLogitsProcessor.php
  91. +3 −3 src/Generation/LogitsProcessors/NoRepeatNGramLogitsProcessor.php
  92. +2 −3 src/Generation/LogitsProcessors/RepetitionPenaltyLogitsProcessor.php
  93. +2 −3 src/Generation/LogitsProcessors/SuppressTokensAtBeginLogitsProcessor.php
  94. +114 −0 src/Generation/LogitsProcessors/WhisperTimeStampLogitsProcessor.php
  95. +9 −11 src/Generation/Samplers/BeamSearchSampler.php
  96. +2 −3 src/Generation/Samplers/GreedySampler.php
  97. +10 −11 src/Generation/Samplers/MultinomialSampler.php
  98. +6 −15 src/Generation/Samplers/Sampler.php
  99. +0 −20 src/Generation/Streamers/StdOutStreamer.php
  100. +52 −3 src/Generation/Streamers/Streamer.php
  101. +32 −122 src/Generation/Streamers/TextStreamer.php
  102. +138 −0 src/Generation/Streamers/WhisperTextStreamer.php
  103. +4 −0 src/Models/Auto/AutoModel.php
  104. +18 −0 src/Models/Auto/AutoModelForAudioClassification.php
  105. +17 −0 src/Models/Auto/AutoModelForCTC.php
  106. +1 −0 src/Models/Auto/AutoModelForCausalLM.php
  107. +17 −0 src/Models/Auto/AutoModelForSpeechSeq2Seq.php
  108. +18 −14 src/Models/Auto/PretrainedMixin.php
  109. +13 −19 src/Models/ModelArchitecture.php
  110. +1 −1 src/Models/Output/BaseModelOutput.php
  111. +20 −0 src/Models/Output/CasualLMOutput.php
  112. +1 −1 src/Models/Output/DetrSegmentationOutput.php
  113. +1 −1 src/Models/Output/MaskedLMOutput.php
  114. +1 −1 src/Models/Output/ObjectDetectionOutput.php
  115. +1 −1 src/Models/Output/QuestionAnsweringModelOutput.php
  116. +1 −1 src/Models/Output/SequenceClassifierOutput.php
  117. +1 −1 src/Models/Output/TokenClassifierOutput.php
  118. +11 −0 src/Models/Pretrained/ASTForAudioClassification.php
  119. +11 −0 src/Models/Pretrained/ASTModel.php
  120. +14 −0 src/Models/Pretrained/ASTPretrainedModel.php
  121. +1 −1 src/Models/Pretrained/BartForConditionalGeneration.php
  122. +12 −3 src/Models/Pretrained/CLIPVisionModelWithProjection.php
  123. +1 −1 src/Models/Pretrained/CodeGenPretrainedModel.php
  124. +1 −1 src/Models/Pretrained/GPT2PretrainedModel.php
  125. +1 −1 src/Models/Pretrained/GPTBigCodePretrainedModel.php
  126. +1 −1 src/Models/Pretrained/GPTJPretrainedModel.php
  127. +11 −0 src/Models/Pretrained/LlamaForCausalLM.php
  128. +14 −0 src/Models/Pretrained/LlamaModel.php
  129. +40 −0 src/Models/Pretrained/LlamaPretrainedModel.php
  130. +1 −1 src/Models/Pretrained/M2M100ForConditionalGeneration.php
  131. +120 −80 src/Models/Pretrained/PretrainedModel.php
  132. +4 −4 src/Models/Pretrained/Qwen2PreTrainedModel.php
  133. +12 −3 src/Models/Pretrained/SiglipTextModel.php
  134. +12 −3 src/Models/Pretrained/SiglipVisionModel.php
  135. +1 −1 src/Models/Pretrained/T5ForConditionalGeneration.php
  136. +4 −4 src/Models/Pretrained/TrOCRPretrainedModel.php
  137. +4 −2 src/Models/Pretrained/VisionEncoderDecoderModel.php
  138. +20 −0 src/Models/Pretrained/Wav2Vec2ForAudioFrameClassification.php
  139. +17 −0 src/Models/Pretrained/Wav2Vec2ForCTC.php
  140. +17 −0 src/Models/Pretrained/Wav2Vec2ForSequenceClassification.php
  141. +37 −0 src/Models/Pretrained/Wav2Vec2Model.php
  142. +11 −0 src/Models/Pretrained/Wav2Vec2PretrainedModel.php
  143. +328 −0 src/Models/Pretrained/WhisperForConditionalGeneration.php
  144. +11 −0 src/Models/Pretrained/WhisperModel.php
  145. +11 −0 src/Models/Pretrained/WhisperPretrainedModel.php
  146. +8 −51 src/Normalizers/BertNormalizer.php
  147. +166 −39 src/Normalizers/Precompiled.php
  148. +14 −3 src/Normalizers/Replace.php
  149. +3 −3 src/Normalizers/StripAccents.php
  150. +82 −0 src/Pipelines/AudioClassificationPipeline.php
  151. +267 −0 src/Pipelines/AutomaticSpeechRecognitionPipeline.php
  152. +6 −5 src/Pipelines/FeatureExtractionPipeline.php
  153. +14 −11 src/Pipelines/FillMaskPipeline.php
  154. +10 −9 src/Pipelines/ImageClassificationPipeline.php
  155. +1 −1 src/Pipelines/ImageToImagePipeline.php
  156. +1 −2 src/Pipelines/ImageToTextPipeline.php
  157. +7 −8 src/Pipelines/Pipeline.php
  158. +8 −12 src/Pipelines/QuestionAnsweringPipeline.php
  159. +73 −38 src/Pipelines/Task.php
  160. +7 −14 src/Pipelines/Text2TextGenerationPipeline.php
  161. +14 −15 src/Pipelines/TextClassificationPipeline.php
  162. +9 −19 src/Pipelines/TextGenerationPipeline.php
  163. +5 −5 src/Pipelines/TokenClassificationPipeline.php
  164. +10 −17 src/Pipelines/ZeroShotClassificationPipeline.php
  165. +2 −2 src/Pipelines/ZeroShotImageClassificationPipeline.php
  166. +6 −7 src/Pipelines/ZeroShotObjectDetectionPipeline.php
  167. +1 −0 src/PostProcessors/PostProcessor.php
  168. +68 −0 src/PostProcessors/PostProcessorSequence.php
  169. +3 −3 src/PreTokenizers/BertPreTokenizer.php
  170. +264 −15 src/PreTokenizers/ByteLevelPreTokenizer.php
  171. +8 −4 src/PreTokenizers/DigitsPreTokenizer.php
  172. +0 −1 src/PreTokenizers/MetaspacePreTokenizer.php
  173. +2 −1 src/PreTokenizers/PreTokenizer.php
  174. +5 −4 src/PreTokenizers/PunctuationPreTokenizer.php
  175. +31 −11 src/PreTokenizers/SplitPreTokenizer.php
  176. +19 −0 src/PreTokenizers/WhitespacePreTokenizer.php
  177. +2 −1 src/PreTokenizers/WhitespaceSplit.php
  178. +3 −3 src/{PretrainedTokenizers → PreTrainedTokenizers}/AlbertTokenizer.php
  179. +116 −0 src/PreTrainedTokenizers/AutoTokenizer.php
  180. +10 −0 src/PreTrainedTokenizers/BartTokenizer.php
  181. +13 −0 src/PreTrainedTokenizers/BertTokenizer.php
  182. +10 −0 src/PreTrainedTokenizers/BlenderbotSmallTokenizer.php
  183. +10 −0 src/PreTrainedTokenizers/BlenderbotTokenizer.php
  184. +2 −2 src/{PretrainedTokenizers → PreTrainedTokenizers}/BloomTokenizer.php
  185. +11 −0 src/PreTrainedTokenizers/CLIPTokenizer.php
  186. +10 −0 src/PreTrainedTokenizers/CamembertTokenizer.php
  187. +11 −0 src/PreTrainedTokenizers/CodeGenTokenizer.php
  188. +2 −2 src/{PretrainedTokenizers → PreTrainedTokenizers}/CodeLlamaTokenizer.php
  189. +10 −0 src/PreTrainedTokenizers/CohereTokenizer.php
  190. +11 −0 src/PreTrainedTokenizers/ConvBertTokenizer.php
  191. +11 −0 src/PreTrainedTokenizers/DebertaTokenizer.php
  192. +11 −0 src/PreTrainedTokenizers/DebertaV2Tokenizer.php
  193. +10 −0 src/PreTrainedTokenizers/DistilBertTokenizer.php
  194. +11 −0 src/PreTrainedTokenizers/ElectraTokenizer.php
  195. +10 −0 src/PreTrainedTokenizers/EsmTokenizer.php
  196. +10 −0 src/PreTrainedTokenizers/FalconTokenizer.php
  197. +3 −3 src/{PretrainedTokenizers → PreTrainedTokenizers}/GPT2Tokenizer.php
  198. +10 −0 src/PreTrainedTokenizers/GPTNeoXTokenizer.php
  199. +10 −0 src/PreTrainedTokenizers/GemmaTokenizer.php
  200. +9 −0 src/PreTrainedTokenizers/Grok1Tokenizer.php
  201. +11 −0 src/PreTrainedTokenizers/HerbertTokenizer.php
  202. +12 −10 src/{PretrainedTokenizers → PreTrainedTokenizers}/LlamaTokenizer.php
  203. +5 −5 src/{PretrainedTokenizers → PreTrainedTokenizers}/M2M100Tokenizer.php
  204. +2 −2 src/{PretrainedTokenizers → PreTrainedTokenizers}/MBart50Tokenizer.php
  205. +5 −5 src/{PretrainedTokenizers → PreTrainedTokenizers}/MBartTokenizer.php
  206. +10 −0 src/PreTrainedTokenizers/MPNetTokenizer.php
  207. +11 −0 src/PreTrainedTokenizers/MobileBertTokenizer.php
  208. +6 −6 src/{PretrainedTokenizers → PreTrainedTokenizers}/NllbTokenizer.php
  209. +10 −0 src/PreTrainedTokenizers/NougatTokenizer.php
  210. +125 −122 src/{PretrainedTokenizers/PretrainedTokenizer.php → PreTrainedTokenizers/PreTrainedTokenizer.php}
  211. +11 −0 src/PreTrainedTokenizers/Qwen2Tokenizer.php
  212. +11 −0 src/PreTrainedTokenizers/RoFormerTokenizer.php
  213. +10 −0 src/PreTrainedTokenizers/RobertaTokenizer.php
  214. +10 −0 src/PreTrainedTokenizers/SiglipTokenizer.php
  215. +10 −0 src/PreTrainedTokenizers/SpeechT5Tokenizer.php
  216. +11 −0 src/PreTrainedTokenizers/SqueezeBertTokenizer.php
  217. +10 −0 src/PreTrainedTokenizers/T5Tokenizer.php
  218. +18 −0 src/PreTrainedTokenizers/VitsTokenizer.php
  219. +11 −0 src/PreTrainedTokenizers/Wav2Vec2CTCTokenizer.php
  220. +812 −0 src/PreTrainedTokenizers/WhisperTokenizer.php
  221. +10 −0 src/PreTrainedTokenizers/XLMRobertaTokenizer.php
  222. +3 −3 src/{PretrainedTokenizers → PreTrainedTokenizers}/XLMTokenizer.php
  223. +0 −110 src/PretrainedTokenizers/AutoTokenizer.php
  224. +0 −10 src/PretrainedTokenizers/BartTokenizer.php
  225. +0 −14 src/PretrainedTokenizers/BertTokenizer.php
  226. +0 −11 src/PretrainedTokenizers/CLIPTokenizer.php
  227. +0 −10 src/PretrainedTokenizers/CamembertTokenizer.php
  228. +0 −11 src/PretrainedTokenizers/CodeGenTokenizer.php
  229. +0 −11 src/PretrainedTokenizers/ConvBertTokenizer.php
  230. +0 −11 src/PretrainedTokenizers/DebertaTokenizer.php
  231. +0 −11 src/PretrainedTokenizers/DebertaV2Tokenizer.php
  232. +0 −10 src/PretrainedTokenizers/DistilBertTokenizer.php
  233. +0 −11 src/PretrainedTokenizers/ElectraTokenizer.php
  234. +0 −10 src/PretrainedTokenizers/EsmTokenizer.php
  235. +0 −10 src/PretrainedTokenizers/FalconTokenizer.php
  236. +0 −10 src/PretrainedTokenizers/GPTNeoXTokenizer.php
  237. +0 −11 src/PretrainedTokenizers/HerbertTokenizer.php
  238. +0 −10 src/PretrainedTokenizers/MPNetTokenizer.php
  239. +0 −11 src/PretrainedTokenizers/MobileBertTokenizer.php
  240. +0 −11 src/PretrainedTokenizers/Qwen2Tokenizer.php
  241. +0 −11 src/PretrainedTokenizers/RoFormerTokenizer.php
  242. +0 −10 src/PretrainedTokenizers/RobertaTokenizer.php
  243. +0 −11 src/PretrainedTokenizers/SqueezeBertTokenizer.php
  244. +0 −10 src/PretrainedTokenizers/T5Tokenizer.php
  245. +0 −10 src/PretrainedTokenizers/XLMRobertaTokenizer.php
  246. +6 −4 src/Processors/AutoProcessor.php
  247. +7 −5 src/Processors/Processor.php
  248. +11 −0 src/Processors/Wav2Vec2ProcessorWithLM.php
  249. +11 −0 src/Processors/WhisperProcessor.php
  250. +21 −0 src/Tensor/MatrixOperator.php
  251. +68 −0 src/Tensor/OpenBLASFactory.php
  252. +1,681 −0 src/Tensor/Tensor.php
  253. +218 −0 src/Tensor/TensorBuffer.php
  254. +18 −0 src/Tensor/TensorBufferFactory.php
  255. +27 −0 src/Tensor/TensorService.php
  256. +8 −15 src/Tokenizers/{BPETokenizer.php → BPEModel.php}
  257. +17 −10 src/Tokenizers/{LegacyTokenizer.php → LegacyModel.php}
  258. +148 −89 src/Tokenizers/{Tokenizer.php → TokenizerModel.php}
  259. +13 −16 src/Tokenizers/{UnigramTokenizer.php → UnigramModel.php}
  260. +3 −3 src/Tokenizers/{WordPieceTokenizer.php → WordPieceModel.php}
  261. +86 −27 src/Transformers.php
  262. +590 −0 src/Utils/Audio.php
  263. +3 −4 src/Utils/AutoConfig.php
  264. +140 −0 src/Utils/Downloader.php
  265. +7 −5 src/Utils/GenerationConfig.php
  266. +72 −8 src/Utils/Helpers.php
  267. +61 −58 src/Utils/Hub.php
  268. +455 −196 src/Utils/Image.php
  269. +632 −0 src/Utils/InferenceSession.php
  270. +161 −0 src/Utils/LibsChecker.php
  271. +276 −0 src/Utils/StreamLogger.php
  272. +0 −1,237 src/Utils/Tensor.php
  273. +33 −0 tests/Expectations.php
  274. +0 −7 tests/Pest.php
  275. +0 −59 tests/TokenizerTest.php
  276. +14 −14 tests/Utils/HubTest.php
  277. +0 −162 tests/Utils/MathTest.php
  278. +83 −0 tests/Utils/StreamLoggerTest.php
  279. +0 −135 tests/Utils/TensorTest.php
  280. +39 −0 tests/tensors/TensorBufferTest.php
  281. +300 −0 tests/tensors/TensorTest.php
  282. +30 −0 tests/tokenizers/Datasets.php
  283. +213 −0 tests/tokenizers/TokenizersTest.php
  284. +27,849 −0 tests/tokenizers/dataset-regular.json
  285. +951 −0 tests/tokenizers/dataset-templates.json
41 changes: 41 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
name: Build and Release Libraries

permissions:
contents: write
packages: read

on:
release:
types:
- published

workflow_dispatch:
inputs:
tag:
description: 'Release Tag'
required: true


jobs:
add-libs:
runs-on: ubuntu-latest

steps:
- name: Log in to GHCR
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Build Libraries
run: |
TAG=${{ startsWith(github.ref, 'refs/tags/') && github.ref_name || github.event.inputs.tag }}
docker run --rm -v ./libs:/libs -e TAG=$TAG ghcr.io/codewithkyrian/transformers-php:latest
ls libs
- name: Add Libraries to Release
uses: softprops/action-gh-release@v2
with:
files: |
libs/*
19 changes: 15 additions & 4 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,11 +1,22 @@
/.phpunit.cache
/.php-cs-fixer.cache
/.php-cs-fixer.php
/composer.lock
.phpunit.cache
.phpunit.result.cache
.php-cs-fixer.cache
.php-cs-fixer.php

composer.lock
/vendor/

.DS_Store
Thumbs.db

*.swp
*.swo
playground/*

.idea
.fleet
.vscode

.transformers-cache/*
tests/models/*
dist
225 changes: 225 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -2,6 +2,201 @@

All notable changes to `transformers-php` will be documented in this file.

## TransformersPHP v0.5.3 - 2024-09-27

This release brings new features, critical bug fixes, and improvements to enhance the functionality and performance of the package. Below is a summary of the changes.

### What's New

- **feat**: Add support for PostProcessor Sequence by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/commit/2cf18ccd83ad84ad8389a7f8c0198252d523f3f6
- **feat**: New tensor method `random` by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/commit/0564dd1609db203d4665a4c040cf1b55e4503ac9
- **feat**: Correctly parse and use the Precompiled Normalizer by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/commit/7e880dd9917b074d995feb98aed9bd41754429ff
- **feat**: Update conversion notebook to include task by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/commit/ca6fc3b72a688ba739a6d62c062b406e7d821d25

### Bug Fixes

- **fix**: Regex bug in Precompiled Normalizer by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/commit/69089b124b299a7b7bef49b676ee5951ac7ef7fb
- **fix**: Improve Unigram Tokenizer handling of multibyte strings by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/commit/e8a8a9ab48e81692d77260197b42e15f768df086
- **fix**: Correct WhitespaceSplit Pretokenizer handling of invisible space characters by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/commit/6ec3e3e9016952913a951d838b237bfb425e07a9
- **fix**: Correctly handle multibyte strings in Precompiled Normalizer by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/commit/bee47e01858b4c69e34eb061c571498c9e0a4e14
- **fix**: Fuse function not combining unknown token IDs correctly by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/commit/300801334cce6f67651665957e776764584db8bc
- **fix**: Tensor topK error when -1 is passed by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/commit/0564dd1609db203d4665a4c040cf1b55e4503ac9

### Improvements

- **fix**: Precompiled Normalizer improvements by @CodeWithKyrian https://github.com/CodeWithKyrian/transformers-php/commit/b01240076ddd50637b1135d3c5e9e55dab08a592

I encourage everyone to update to this latest version and explore the new features. As always, feel free to report any issues or contribute to the project.

**Full Changelog**: https://github.com/CodeWithKyrian/transformers-php/compare/0.5.2...0.5.3

## TransformersPHP v0.5.2 - 2024-08-29

### What's Changed

* Use a static list for byte-unicode and unicode-byte conversion by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/commit/75f5d9cd671791f0542963f4541dfc6d8dad5dcb
* Fix Vips RGBA -> RGBA conversion error by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/commit/54bdee0a2b0542e57922ede1b633e462de80b0e1
* Show output progress when downloading the shared libraries by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/commit/d613a0d1c2774ca83bb6743862c12c259cfddf71
* Obviate the need for autoload.php in establishing library base path by @timwhitlock in https://github.com/CodeWithKyrian/transformers-php/pull/65

### New Contributors

* @timwhitlock made their first contribution in https://github.com/CodeWithKyrian/transformers-php/pull/65

**Full Changelog**: https://github.com/CodeWithKyrian/transformers-php/compare/0.5.1...0.5.2

## TransformersPHP v0.5.1 - 2024-08-24

### What's new

- **Tensor Operations**: `magnitude`, `sqrt` and `cosSimilarity` added.
- **Vips Binaries**: - Vips binaries are now bundled by default, eliminating need to modify anything on the system to use libvips.

### Bug Fixes

- **Error Handling**: - Adjusted error level to a warning for unknown model types, providing clearer feedback without interrupting the workflow.

### Reversions

- **Dependencies**: Reverted `rokka/vips` from dev back to normal dependencies. Since vips binaries are bundled by default, use of vips is now encouraged.

**Full Changelog**: https://github.com/CodeWithKyrian/transformers-php/compare/0.5.0...0.5.1

## TransformersPHP v0.5.0 - 2024-08-21

I'm excited to announce the latest version of TransformersPHP, packed with new features, improvements, and bug fixes. This release brings powerful enhancements to your machine-learning-driven PHP applications, enabling more efficient and versatile operations.

### New Features

- **New Pipeline: Audio Classification** - Easily classify audio clips with a pre-trained model.

```php
$classifier = pipeline('audio-classification', 'Xenova/ast-finetuned-audioset-10-10-0.4593');
$audioUrl = __DIR__ . '/../sounds/cat_meow.wav';
$output = $classifier($audioUrl);
// [
// [
// "label" => "Meow"
// "score" => 0.6109990477562
// ]
// ]




```
- **New Pipeline: Automatic Speech Recognition (ASR)** - Supports models like `wav2vec` and `whisper` for transcribing speech to text. If a specific model is not officially supported, please open an issue with a feature request.

- Example:
```php
$transcriber = pipeline('asr', 'Xenova/whisper-tiny.en');
$audioUrl = __DIR__ . '/../sounds/preamble.wav';
$output = $transcriber($audioUrl, maxNewTokens: 256);
// [
// "text" => "We, the people of the United States, ..."
// ]




```


### Enhancements

- **Shared Libraries Dependencies:** - A revamped workflow for downloading shared libraries dependencies ensures they are versioned correctly, reducing download sizes. These binaries are now thoroughly tested on Apple Silicon, Intel Macs, Linux x86_64, Linux aarch64, and Windows platforms.

- **`Transformers::setup` Simplified** - `Transformers::setup()` is now optional. Default settings are automatically applied if not called. The` apply()` method is no longer necessary, but still available for backward compatibility.

- **Immutable Image Utility** - The Image utility class is now immutable. Each operation returns a new instance, allowing for method chaining and a more predictable workflow.

```php
$image = Image::read($url);
$resizedImage = $image->resize(100, 100);
// $image remains unchanged




```
- **New Tensor Operations** - New operations were added: `copyTo`, `log`, `exp`, `pow`, `sum`, `reciprocal`, `stdMean`. Additionally, overall performance improvements have been made to Tensor operations.

- **TextStreamer Improvements** - TextStreamer now prints to stdout by default. You can override this behavior using the `onStream(callable $callback)` method. Consequently, the `StdoutStreamer` class is now obsolete.

- **VIPS PHP Driver Update** - The VIPS PHP driver is no longer bundled by default in `composer.json`. Detailed documentation is provided for installing the Vips PHP driver and setting up Vips on your machine.

- **ONNX Runtime Upgrade** - Upgraded to version 1.19.0, bringing more performance and compatibility with newer models.

- Bug Fixes & Performance Improvements - Various bug fixes have been implemented to enhance stability and performance across the package.


I hope you enjoy these updates and improvements. If you encounter any issues or have any suggestions, please don’t hesitate to reach out through our [Issue Tracker](https://github.com/CodeWithKyrian/transformers-php/issues)

**Full Changelog**: https://github.com/CodeWithKyrian/transformers-php/compare/0.4.4...0.5.0

## v0.4.4 - 2024-08-14

### What's Changed

* feat: add optional host argument for model download by @k99k5 in https://github.com/CodeWithKyrian/transformers-php/pull/56

### New Contributors

* @k99k5 made their first contribution in https://github.com/CodeWithKyrian/transformers-php/pull/56

**Full Changelog**: https://github.com/CodeWithKyrian/transformers-php/compare/0.4.3...0.4.4

## v0.4.3 - 2024-07-31

### What's Changed

* Fix typo in docs by @BlackyDrum in https://github.com/CodeWithKyrian/transformers-php/pull/42
* fix: statically calling FFI::new deprecated in PHP 8.3 by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/pull/48
* fix: improve regex for detecting language codes in NllbTokenizer by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/pull/49
* fix: digits pre-tokenizer returning empty array for text with no digits by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/pull/51
* [feat: allow passing model filename when downloading a model from CLI](https://github.com/CodeWithKyrian/transformers-php/commit/91db063eab90da732f301a028e23a0a00ee25979)
* fix: preTokenizer null error when there's no text pair](https://github.com/CodeWithKyrian/transformers-php/commit/901a049b8bd837c83d3edcd517dd76cf8e3ba6b9)
* feat: implement enforce size divisibility for image feature extractor by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/pull/53

### New Contributors

* @BlackyDrum made their first contribution in https://github.com/CodeWithKyrian/transformers-php/pull/42

**Full Changelog**: https://github.com/CodeWithKyrian/transformers-php/compare/0.4.2...0.4.3

## v0.4.2 - 2024-06-05

### What's Changed

* bugfix: Repository url resolution not working properly in Windows by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/pull/41

**Full Changelog**: https://github.com/CodeWithKyrian/transformers-php/compare/0.4.1...0.4.2

## v0.4.1 - 2024-05-24

### What's Changed

* configuration.md: fix indentation of Transformers::setup() by @k00ni in https://github.com/CodeWithKyrian/transformers-php/pull/35
* PretrainedTokenizer::truncateHelper: prevent array_slice() error for flawed text input (summarization) by @k00ni in https://github.com/CodeWithKyrian/transformers-php/pull/36
* Fix bug with Download CLI - use named parameters for model construct by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/pull/39

### New Contributors

* @k00ni made their first contribution in https://github.com/CodeWithKyrian/transformers-php/pull/35

**Full Changelog**: https://github.com/CodeWithKyrian/transformers-php/compare/0.4.0...0.4.1

## v0.3.1 - 2024-04-22

### What's Changed

* Add Qwen2 model support by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/pull/20
* Add chat input detection for text generation, and refactor streamer API. by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/pull/21
* bugfix: Fix error that occurs when streamer is not used by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/pull/22
* bugfix: Decoder sequence not calling the right method by @CodeWithKyrian in https://github.com/CodeWithKyrian/transformers-php/pull/23

**Full Changelog**: https://github.com/CodeWithKyrian/transformers-php/compare/0.3.0...0.3.1

## v0.3.0 - 2024-04-13

### What's Changed
@@ -83,6 +278,16 @@ composer require codewithkyrian/transformers













```
And you must initialize the library to download neccesary libraries for ONNX

@@ -93,6 +298,16 @@ And you must initialize the library to download neccesary libraries for ONNX













```
#### Checkout the Documentation

@@ -109,6 +324,16 @@ To ensure a smooth user experience, especially with larger models, we recommend













```
#### What's Next?

2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -187,7 +187,7 @@
same "printed page" as the copyright notice for easier
identification within third-party archives.

Copyright [yyyy] [name of copyright owner]
Copyright 2024 Kyrian Obikwelu

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -152,7 +152,7 @@ Transformers::setup()
->setAuthToken('...') // Set the auth token for downloading models. Defaults to `null`
->setUserAgent('...') // Set the user agent for downloading models. Defaults to `transformers-php/{version}`
->setImageDriver('...') // Set the image driver for processing images. Defaults to `IMAGICK'
->apply(); // Apply the configuration
->setLogger('...'); // Set the logger for TransformersPHP. Defaults to `null`
```

You can call the `set` methods in any order, or leave any out entirely, in which case, it uses the default values. For
@@ -251,8 +251,8 @@ This package is a WIP, but here's a list of tasks and architectures currently te
| [Document Question Answering](https://huggingface.co/tasks/document-question-answering) | `document-question-answering` | Answering questions on document images. ||
| [Feature Extraction](https://codewithkyrian.github.io/transformers-php/feature-extraction) | `feature-extraction` | Transforming raw data into numerical features that can be processed while preserving the information in the original dataset. ||
| [Image Feature Extraction](https://codewithkyrian.github.io/transformers-php/image-feature-extraction) | `image-feature-extraction` | Extracting features from images. ||
| [Image-to-Text](https://codewithkyrian.github.io/transformers-php/image-to-text) | `image-to-text` | Output text from a given image. | |
| [Text-to-Image](https://huggingface.co/tasks/text-to-image) | `text-to-image` | Generates images from input text. | |
| [Image-to-Text](https://codewithkyrian.github.io/transformers-php/image-to-text) | `image-to-text` | Output text from a given image. | |
| [Text-to-Image](https://huggingface.co/tasks/text-to-image) | `text-to-image` | Generates images from input text. | |
| [Visual Question Answering](https://huggingface.co/tasks/visual-question-answering) | `visual-question-answering` | Answering open-ended questions based on an image. ||
| [Zero-Shot Audio Classification](https://huggingface.co/learn/audio-course/chapter4/classification_models#zero-shot-audio-classification) | `zero-shot-audio-classification` | Classifying audios into classes that are unseen during training. ||
| [Zero-Shot Image Classification](https://codewithkyrian.github.io/transformers-php/zero-shot-image-classification) | `zero-shot-image-classification` | Classifying images into classes that are unseen during training. ||
@@ -399,4 +399,4 @@ This package is a WIP, but here's a list of tasks and architectures currently te
1. **[YOLOS](https://huggingface.co/docs/transformers/model_doc/yolos)** (from Huazhong University of Science &
Technology) released with the
paper [You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection](https://arxiv.org/abs/2106.00666)
by Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, Wenyu Liu.
by Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, Wenyu Liu.
1 change: 1 addition & 0 deletions VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0.5.3
2 changes: 1 addition & 1 deletion bin/transformers
Original file line number Diff line number Diff line change
@@ -12,7 +12,7 @@ $application = new Application();
try {
$application->setName('Transformers PHP CLI');

// $application->add(new Codewithkyrian\Transformers\Commands\InitCommand());
$application->add(new Codewithkyrian\Transformers\Commands\InstallCommand());
$application->add(new Codewithkyrian\Transformers\Commands\DownloadModelCommand());

$application->run();
Loading