Add Transformer.js as a backend for image classification

[pearmini]

For single image:

function preload() {
  classifier = ml5.imageClassifier("VisionTransformer");
  img = loadImage("images/bird.jpg");
}

function setup() {
  createCanvas(400, 400);
  classifier.classify(img, gotResult);
  image(img, 0, 0, width, height);
}

For webcam:

function preload() {
  classifier = ml5.imageClassifier("VisionTransformer");
}

function setup() {
  createCanvas(640, 480);
  video = createCapture(VIDEO);
  video.size(640, 480);
  video.hide();
  classifier.classifyStart(video, gotResult);
}

Specify options:

function preload() {
  classifier = ml5.imageClassifier("VisionTransformer", {dtype: "32",  device: "wasm", topK: 5});
}

TODO

• Classify single image
• Handle Video Input
• Add more examples
• Support top_k input
• Docs

[Copilot]

Pull request overview

This pull request adds Transformer.js as a new backend for image classification in ml5.js, introducing a Vision Transformer model alongside the existing TensorFlow.js-based models. The implementation creates a new ImageClassifierTransformer class that wraps Hugging Face's Transformers.js library and integrates it into the existing imageClassifier API through a factory pattern.

Key changes:

• Added @huggingface/transformers dependency (v3.7.6) with associated dependencies for ONNX runtime and image processing
• Implemented ImageClassifierTransformer class with support for WebGPU/WASM inference
• Created three example sketches demonstrating single image classification, top-k results, and webcam classification

Reviewed changes

Copilot reviewed 11 out of 14 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
package.json	Added @huggingface/transformers v3.7.6 dependency
yarn.lock	Added all transitive dependencies for Transformers.js including ONNX runtime, Sharp, and Protobuf packages
webpack.config.js	Added warning suppression for ESM import.meta usage in Transformers.js
src/utils/imageUtilities.js	Exported existing drawToCanvas helper function for video-to-canvas conversion
src/ImageClassifier/transformer.js	New implementation of Vision Transformer-based image classifier
src/ImageClassifier/index.js	Modified factory function to route "VisionTransformer" model name to new implementation
examples/imageClassifier-transformer-single-image/*	Example demonstrating single image classification with default top-k
examples/imageClassifier-transformer-single-image-topk/*	Example demonstrating custom top-k parameter usage
examples/imageClassifier-transformer-webcam/*	Example demonstrating continuous webcam classification with start/stop

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

@pearmini I've opened a new pull request, #293, to work on those changes. Once the pull request is ready, I'll request review from you.

[gohai]

This might be just for testing - thought I point this out just in case: I don't think it's worth having a separate example just to demonstrate the topk option (documenting this seems sufficient imho)

[gohai]

Elsewhere in the codebase, we use topk as an option with lowercase k. Probably good to retain that for consistency?

[gohai]

add using a transformer model (here and in <title>, sketch.js)?

[gohai]

nitpick: extra newline

[gohai]

nitpick: extra newline

[gohai]

Git prefers to have a newline character at the end of each file (since diff operates on whole lines)

[gohai]

Does this limit WASM to 1 fps (regardless of compute power)? Is there any way to schedule this dynamically instead?

[gohai]

Personally a bit on the fence if "VisionTransformer" is beneficial or not vs "vit-base-patch16-224" ... searching for the former brings up some articles of the general architecture (that by now different models implement) - only the latter lets me know that this was e.g. trained on 14 million images, with 21 thousand classes, and uses a resolution of 224x224.

If we'll be using "VisionTransformer": how about printing the actual name of the model that is being used to the console?

[gohai]

Thank you @pearmini - added a few comments to specific lines.

I had two general observations:

In China, using NYU Shanghai's otherwise excellent VPN, loading the 345 MB model took me 3.3 minutes - which is significantly worse than with Google's tensorflow.js models. (Most users will already have given up at this point.) Curious if this is a known issue (throttling?) in our corner of the woods, or if there are other CDNs in front of HuggingFace we might want to use?

In my Chrome, I received a warning message and two error messages in the console while it ran. Ideally, we don't print those, since people using the library might think those are their fault, or something they might need to address..? Is there any way to reduce transformers.js verbosity, or otherwise filtering its output?

[nasif-co]

In my Chrome, I received a warning message and two error messages in the console while it ran. Ideally, we don't print those, since people using the library might think those are their fault, or something they might need to address..? Is there any way to reduce transformers.js verbosity, or otherwise filtering its output?

These console warnings seem to be an ongoing problem with the onnxruntime itself, here's a github issue on the matter: huggingface/transformers.js#270