diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index d07a15a71b..e9a89aaa44 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -208,6 +208,12 @@ Voice and Tone: - Formatting standards: **Bold text** for UI elements (buttons, menu items, field names), *Italic text* for emphasis and new terms, `Code formatting` for file names, commands, code elements. - Use shortcodes for common pitfalls, warnings, important notes. +### Code fence integrity + +- Every fenced code block opened with triple backticks (```) MUST be explicitly closed with matching triple backticks before any non-code content resumes +- Never generate unterminated or partial code fences +- Do not rely on implicit closure, indentation, or surrounding formatting to end a code block + ## Arm naming and architecture terms - Use Arm for the brand in prose (for example, "Arm processors", "Arm servers"). diff --git a/.github/workflows/test-lp.yml b/.github/workflows/test-lp.yml index 4ed6b1d352..5abdc17914 100644 --- a/.github/workflows/test-lp.yml +++ b/.github/workflows/test-lp.yml @@ -31,7 +31,7 @@ jobs: tmpfile=$(mktemp) - git diff --name-only origin/${{ github.base_ref }}...HEAD | + git diff --name-only --diff-filter=d origin/${{ github.base_ref }}...HEAD | grep '^content/' | while read -r path; do name=$(basename "$path") diff --git a/.wordlist.txt b/.wordlist.txt index 3a2774a56f..f8afdacc88 100644 --- a/.wordlist.txt +++ b/.wordlist.txt @@ -5493,4 +5493,102 @@ preselected reconfiguring torchscript xxxxxx -Modescope \ No newline at end of file +Modescope + +APerf's +AdamW +AndroidX +AssetManager +BGA +BatchNormalization +BoardRenderer +CVVXMAB +DigitNet +EPs +IC +ImageFolder +MAB +MIMX +NXP's +Netron +ONNX's +OTP +Opset +Opsets +Otsu +PGDATA +PSK +PrepareData +PrepareModelForAndroid +QDQ +QuantizeModel +Relu +RemainAfterExit +RunSudokuProcessor +Runlevel +Runtime's +SBBM +SSID +SSIDs +ScrollView +SessionOptions +SmallNet +SudokuEngine +SudokuProcessor +SudokuSolver +SudokuSolverOnnx +TCMalloc +VVX +WantedBy +YOLOv +accuracies +acyclic +androidx +aug +bitnami +claude +dataloaders +deployability +deterministically +digitnet +dirs +drawable +drawables +flto +frdm +hpa +httproute +ifconfig +initDebug +interprocedural +mgmt +misrecognized +mlan +moal +modprobe +netdev +netron +oneshot +onnxscript +opset +opsets +picocom +psk +pvc +reimplemented +scarthgap +serviceaccount +silabs +smallnet +sourcecode +ssid +stepwise +sudoku +sudokusolveronnx +tensor's +thumbdrive +ttyLP +udhcpc +usb +vcp +Aperf's \ No newline at end of file diff --git a/content/install-guides/_images/about-wpa.png b/content/install-guides/_images/about-wpa.png deleted file mode 100644 index 7cd5131059..0000000000 Binary files a/content/install-guides/_images/about-wpa.png and /dev/null differ diff --git a/content/install-guides/_images/aperf.webp b/content/install-guides/_images/aperf.webp deleted file mode 100644 index 85f67ed286..0000000000 Binary files a/content/install-guides/_images/aperf.webp and /dev/null differ diff --git a/content/install-guides/_images/aperf0.webp b/content/install-guides/_images/aperf0.webp deleted file mode 100644 index 4db30ede31..0000000000 Binary files a/content/install-guides/_images/aperf0.webp and /dev/null differ diff --git a/content/install-guides/_images/aperf_report_aligned_graphs.png b/content/install-guides/_images/aperf_report_aligned_graphs.png new file mode 100644 index 0000000000..98306da434 Binary files /dev/null and b/content/install-guides/_images/aperf_report_aligned_graphs.png differ diff --git a/content/install-guides/_images/aperf_report_help_panel.png b/content/install-guides/_images/aperf_report_help_panel.png new file mode 100644 index 0000000000..79a3c3a435 Binary files /dev/null and b/content/install-guides/_images/aperf_report_help_panel.png differ diff --git a/content/install-guides/_images/aperf_report_home.png b/content/install-guides/_images/aperf_report_home.png new file mode 100644 index 0000000000..160cd651ad Binary files /dev/null and b/content/install-guides/_images/aperf_report_home.png differ diff --git a/content/install-guides/_images/aperf_report_statistical_findings.png b/content/install-guides/_images/aperf_report_statistical_findings.png new file mode 100644 index 0000000000..09d9780fcb Binary files /dev/null and b/content/install-guides/_images/aperf_report_statistical_findings.png differ diff --git a/content/install-guides/_images/ChromeOSpf.png b/content/install-guides/_images/chromeospf.png similarity index 100% rename from content/install-guides/_images/ChromeOSpf.png rename to content/install-guides/_images/chromeospf.png diff --git a/content/install-guides/_images/download-win-armpl_23.10.png b/content/install-guides/_images/download-win-armpl_23.10.png deleted file mode 100644 index 2b7df887af..0000000000 Binary files a/content/install-guides/_images/download-win-armpl_23.10.png and /dev/null differ diff --git a/content/install-guides/_images/download_ecosys_fvp.png b/content/install-guides/_images/download_ecosys_fvp.png deleted file mode 100644 index 4489dbb54b..0000000000 Binary files a/content/install-guides/_images/download_ecosys_fvp.png and /dev/null differ diff --git a/content/install-guides/_images/MCUXpresso_Installer.png b/content/install-guides/_images/mcuxpresso_installer.png similarity index 100% rename from content/install-guides/_images/MCUXpresso_Installer.png rename to content/install-guides/_images/mcuxpresso_installer.png diff --git a/content/install-guides/_images/win-sys-path.png b/content/install-guides/_images/win-sys-path.png deleted file mode 100644 index 40810ef574..0000000000 Binary files a/content/install-guides/_images/win-sys-path.png and /dev/null differ diff --git a/content/install-guides/_images/windows-sys-env_23.10.png b/content/install-guides/_images/windows-sys-env_23.10.png deleted file mode 100644 index 4153d46fb9..0000000000 Binary files a/content/install-guides/_images/windows-sys-env_23.10.png and /dev/null differ diff --git a/content/install-guides/_images/windows-sys-prop.png b/content/install-guides/_images/windows-sys-prop.png deleted file mode 100644 index e504711317..0000000000 Binary files a/content/install-guides/_images/windows-sys-prop.png and /dev/null differ diff --git a/content/install-guides/aperf.md b/content/install-guides/aperf.md index 9ad8337985..7e766fcb3c 100644 --- a/content/install-guides/aperf.md +++ b/content/install-guides/aperf.md @@ -8,24 +8,20 @@ official_docs: https://github.com/aws/aperf test_images: - ubuntu:latest test_maintenance: true -title: AWS Perf (APerf) +title: APerf tool_install: true weight: 1 --- -APerf (AWS Perf) is an open source command line performance analysis tool which saves time by collecting information which is normally collected by multiple tools such as `perf`, `sysstat`, and `sysctl`. +APerf is an open source command line tool maintained by AWS. It helps you monitor and debug performance on Linux systems by collecting a wide range of performance-related system metrics and data that traditionally require multiple tools, such as `perf`, `sysstat`, and `sysctl`. -APerf was created by AWS to help with Linux performance analysis. +APerf collects system data and saves it in an archive. It then generates a static HTML report from one or more archives to visualize the data. When you generate the report, APerf analyzes the data to automatically detect potential performance issues. You can open the report in a browser to view all collected data and analytical findings. -In addition to the CLI, APerf includes an HTML view to visualize the collected data. +## Install APerf -## What should I do before I begin installing APerf? +This guide provides a quick solution to install APerf on Arm Linux and get started. -APerf works on Linux, and is available as a single binary. - -APerf works best if `perf` is installed. Refer to the [Perf for Linux on Arm](/install-guides/perf) install guide for instructions. - -This article provides a quick solution to install APerf on Arm Linux and get started. +## Before you begin Confirm you are using an Arm machine by running: @@ -39,37 +35,52 @@ The output should be: aarch64 ``` -If you see a different result, you are not using an Arm computer running 64-bit Linux. +{{% notice Note %}} If you see a different result, you are not using an Arm computer running 64-bit Linux. APerf can only run on Linux.{{% /notice %}} + +To allow APerf to collect PMU (Processor Monitoring Unit) metrics without sudo or root permissions, set `/proc/sys/kernel/perf_event_paranoid` to -1: + +```bash +sudo sysctl -w kernel.perf_event_paranoid=-1 +``` + +To use APerf's CPU profiling option (`--profile`), install the `perf` binary. See the [Perf for Linux on Arm](/install-guides/perf) install guide for instructions. -## How do I download and install APerf? +For kernel address visibility, set `/proc/sys/kernel/kptr_restrict` to 0: -The easiest way to install APerf is to download a release from GitHub, extract it, and setup your `PATH` environment variable or copy the executable to a directory already in your search path. +```bash +sudo sysctl -w kernel.kptr_restrict=0 +``` + +To use APerf's Java profiling option (`--profile-java`), install the [async-profiler](https://github.com/async-profiler/async-profiler) tool. + +## Download and install APerf +The easiest way to install APerf is to download a release from GitHub and extract it. -Visit the [releases page](https://github.com/aws/aperf/releases/) to see a list of available releases. +Visit the [releases page](https://github.com/aws/aperf/releases/) to see available releases. -You can also download a release from the command line: +You can download a release from the command line: ```bash { target="ubuntu:latest" } -wget https://github.com/aws/aperf/releases/download/v0.1.15-alpha/aperf-v0.1.15-alpha-aarch64.tar.gz +wget https://github.com/aws/aperf/releases/download/v1.0.0/aperf-v1.0.0-aarch64.tar.gz ``` Extract the release: ```bash { target="ubuntu:latest" } -tar xvfz aperf-v0.1.15-alpha-aarch64.tar.gz +tar xvfz aperf-v1.0.0-aarch64.tar.gz ``` Add the path to `aperf` in your `.bashrc` file. ```console -echo 'export PATH="$PATH:$HOME/aperf-v0.1.15-alpha-aarch64"' >> ~/.bashrc +echo 'export PATH="$PATH:$HOME/aperf-v1.0.0-aarch64"' >> ~/.bashrc source ~/.bashrc ``` Alternatively, you can copy the `aperf` executable to a directory already in your search path. ```bash { target="ubuntu:latest" } -sudo cp aperf-v0.1.15-alpha-aarch64/aperf /usr/local/bin +sudo cp aperf-v1.0.0-aarch64/aperf /usr/local/bin ``` Confirm `aperf` is installed by printing the version: @@ -81,81 +92,100 @@ aperf --version The output should print the version: ```output -aperf 0.1.0 (4b910d2) +aperf 1.0.0 (4cf8d28) ``` -## How do I verify APerf is working? +## Verify APerf is working -### How do I create and view a report? +To confirm APerf is working, start a collection run with the default settings. The default interval is 1 second, and the default period is 10 seconds. -To confirm APerf is working, start it for 10 seconds and take a sample every 1 second. +Run the following command to start data collection: ```console -sudo aperf record -i 1 -p 10 -r run1 --profile +aperf record -r test_1 ``` -After 10 seconds `aperf` completes and you see a directory named `run1` and a tar file named `run1.tar.gz`. +After 10 seconds, the collection completes. APerf creates a directory named `test_1` and a tar file named `test_1.tar.gz`. + +If you need CPU profiling, add the `--profile` flag. For Java profiling, add the `--profile-java` flag. + +### How do I create and view a report? -Next, generate a report from the recorded data: +Generate a report from the recorded data: ```console -sudo aperf report -r run1 -n report1 +aperf report -r test_1 -n test_report ``` -The name of the report is `report1` and you will see a `report1` directory and a tar file named `report1.tar.gz`. - -The tar files are useful if you want to copy them to another machine. +APerf creates a directory named `test_report` and a tar file named `test_report.tar.gz`. The tar file is useful when you want to copy the report to another machine. -Using a web browser, open the file `index.html` in the `report1/` directory. To open the file use `Ctrl+O` for Linux and Windows and use `⌘+O` for macOS. +To view the report, open the `index.html` file in the `test_report/` directory using a web browser. Press `Ctrl+O` on Linux and Windows, or `⌘+O` on macOS. -The report is now visible in the browser. +The report's home page displays system information from the APerf run, followed by analytical findings that highlight potential performance issues: -There are a number of tabs on the left side showing the collected data. +![APerf report home page showing system information and analytical findings alt-txt#center](/install-guides/_images/aperf_report_home.png "APerf report home page") -You can browse the data and see what has been collected. +You can browse through all collected data using the navigation panel on the left. -![APerf #center](/install-guides/_images/aperf0.webp) +To learn more about a specific metric, select the info button next to it to open the help panel: -{{% notice Note %}} -The Kernel Config and Sysctl Data tabs are blank unless you click No. -{{% /notice %}} +![APerf report help panel showing detailed metric information alt-txt#center](/install-guides/_images/aperf_report_help_panel.png "APerf report help panel") -### How do I create and view a report containing 2 runs? +### How do I compare multiple runs? -To demonstrate comparing 2 runs, create a second run with `aperf record`: +To demonstrate comparing multiple runs, create a second run with `aperf record`: ```console -sudo aperf record -i 1 -p 10 -r run2 --profile +aperf record -r test_2 ``` -After 10 seconds `aperf` completes and you see a directory named `run2` and a tar file named `run2.tar.gz`. +Similarly, after 10 seconds the collection completes, and APerf produces a directory named `test_2` and a tar file named `test_2.tar.gz`. -Generate a report with both the first and second runs included: +Generate a report that includes both runs. The first run in the `-r` arguments becomes the base run for automatic comparisons: ```console -sudo aperf report -r run1 -r run2 -n compare +aperf report -r test_1 test_2 -n compare_report ``` -The name of the report is `compare` and you will see a `compare` directory and a tar file named `compare.tar.gz`. +APerf creates a directory named `compare_report` and a tar file named `compare_report.tar.gz`. + +Open the `index.html` file in the `compare_report/` directory using a web browser. + +Because the report includes multiple runs, APerf compares all runs against the base run and displays statistical findings on the home page: -Open the `index.html` file in the `compare/` directory to see the 2 runs side by side. +![APerf report home page showing statistical comparisons between multiple runs alt-text#center](/install-guides/_images/aperf_report_statistical_findings.png "APerf report statistical findings") -A screenshot is shown below: +When you view metric graphs, APerf aligns graphs of the same metric from different runs side by side for easy comparison: -![APerf #center](/install-guides/_images/aperf.webp) +![APerf report showing aligned metric graphs from multiple runs for comparison alt-text#center](/install-guides/_images/aperf_report_aligned_graphs.png "APerf report aligned graphs") -### How do I use an HTTP server to view reports? +### How do I view reports from a remote system? -If you are doing performance analysis on a remote system or cloud instance without a remote desktop, you can view the APerf reports from your local browser by running a simple web server on the remote machine. +If you're working on a remote system or cloud instance without a desktop environment, you can view APerf reports in your local browser by running a web server on the remote machine. + +Navigate to the directory containing the report and the `index.html` file: + +```console +cd test_report +``` -In the directory with the report data and the `index.html` file run a simple web server: +Start a simple HTTP server: ```console python -m http.server 3000 ``` -Make sure port 3000 is open on the remote system and enter the IP address of the remote system followed by `:3000` in your browser address bar. +The server starts on port 3000. Make sure this port is open in your firewall or security group settings. + +Open a web browser on your local machine and navigate to: + +```output +http://:3000 +``` + +Replace `` with the IP address of your remote system. + +The APerf report opens in your browser without needing to copy files to your local machine. -You will see the same APerf report, and avoid the need to copy files to your local machine from the remote system for viewing. +You're now ready to use APerf for performance analysis on your Arm Linux system. -You are ready to use APerf for performance analysis on your Arm Linux system. diff --git a/content/install-guides/claude-code.md b/content/install-guides/claude-code.md new file mode 100644 index 0000000000..5f8c92b3c3 --- /dev/null +++ b/content/install-guides/claude-code.md @@ -0,0 +1,300 @@ +--- +title: Claude Code + +author: Pareena Verma +minutes_to_complete: 10 +official_docs: https://code.claude.com/docs + +layout: installtoolsall +multi_install: false +multitool_install_part: false +tool_install: true +weight: 1 +--- + +Claude Code is an AI-powered command-line tool that helps you build features, debug code, and navigate codebases directly from your terminal. It provides autonomous coding assistance and integrates with your existing development workflow. + +Claude Code works seamlessly on Arm-based systems, including Linux distributions running on Arm servers, macOS on Apple Silicon, and Windows on Arm devices. + +## Review prerequisites + +You need a Claude account to use Claude Code. A Claude.ai account is recommended, though you can also use a Claude Console account. + +If you don't have a Claude account, visit [Claude.ai](https://claude.ai/) and sign up. + +Claude Code is only available for paid Pro and Max accounts, if not using API credits. Visit [Claude Pricing](https://www.anthropic.com/pricing) to review the options. + +## Install Claude Code + +Claude Code is a terminal application that works on macOS, Linux, and Windows systems, including Arm-based platforms. + +### Install on Linux (Arm) + +The recommended installation method for Linux uses the installation script: + +```bash { target="ubuntu:latest" } +curl -fsSL https://claude.ai/install.sh | bash +``` + +This script automatically detects your system architecture and installs the appropriate version for Arm64 systems. + +Add Claude Code to your PATH: + +```bash +echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc +``` + +## Install on macOS (Apple Silicon) + +On macOS, you can use the installation script: + +```bash +curl -fsSL https://claude.ai/install.sh | bash +``` + +Or install using Homebrew: + +```bash +brew install --cask claude-code +``` + +## Install on Windows on Arm + +On Windows systems, including Windows on Arm, run the following PowerShell command: + +```console +irm https://claude.ai/install.ps1 | iex +``` + +For other options, please [see the Claude Code setup page](https://code.claude.com/docs/en/setup). + +## Verify installation + +Confirm Claude Code is installed by checking the version: + +```console +claude --version +``` + +The output shows the installed version: + +```output +2.1.7 (Claude Code) +``` + +## Authenticate Claude Code + +After installing Claude Code, you need to authenticate: + +Navigate to a project directory: + +```console +cd your-project +``` + +Start Claude Code: + +```console +claude +``` + +Configure your preferences when prompted (dark mode, editor settings, etc.). + +On first use, Claude Code prompts you to authenticate: + +- If using Claude.ai, authenticate through your browser +- If on a remote machine, paste the provided link into a local browser, then enter the authentication code in Claude Code + +Accept the acknowledgements to complete setup. + +Claude Code automatically saves your authentication credentials for future sessions. + +## Confirm Claude Code is working + +Test Claude Code by asking it to perform a simple task. + +Start Claude Code in a project directory: + +```console +claude +``` + +Type a request, for example: + +```console +> Create a Python function to calculate fibonacci numbers for my Arm machine +``` + +Claude Code analyzes your request, creates a plan, and generates the code. + +Review the proposed changes before accepting them. Claude Code shows you a preview of changes before applying them, giving you control over what gets modified in your codebase. + +If Claude Code doesn't respond: +- Verify you're authenticated (run `claude` and check for authentication prompts) +- Check your internet connection +- Ensure your Claude account is active +- Try restarting Claude Code + + +You’re now ready to use Claude Code. +{{% notice Note %}} The sections below are optional and cover advanced integrations, including using MCP servers with Claude Code on Arm.{{% /notice %}} + + +## Use MCP Servers with Claude Code + +Model Context Protocol (MCP) Servers extend Claude Code's capabilities by providing specialized tools and knowledge bases. Claude Code can connect to MCP servers to access domain-specific expertise and functionality. + +The Arm MCP Server provides AI assistants with tools and knowledge for Arm architecture development, migration, and optimization. This is particularly useful when working on Arm-based systems. + +## Arm MCP Server tools + +The Arm MCP Server includes several tools designed for Arm development: + +- migrate-ease scan: Analyzes codebases for x86-specific code that needs updating for Arm compatibility +- skopeo: Inspects container images to check for ARM64 architecture support +- knowledge_base_search: Searches Arm documentation and learning resources +- mca (Machine Code Analyzer): Analyzes assembly code for performance on Arm architectures +- check_image: Verifies Docker image architecture compatibility + +## Configure the Arm MCP Server with Claude Code + +You need Docker running on your system to use the Arm MCP Server. See the [Docker install guide](/install-guides/docker/) for instructions. + +First, pull the Arm MCP Server image: + +```console +docker pull armlimited/arm-mcp:latest +``` + +Configure the Arm MCP Server using the `claude mcp add` command. You can configure MCP servers at three different scopes: + +- Local scope (default): Available only to you in the current project +- Project scope: Shared with everyone in the project via `.mcp.json` file +- User scope: Available to you across all projects + +{{% notice Note %}} +Choose the appropriate scope based on your needs. Project scope is recommended for team collaboration, while user scope is useful for personal tools you use across multiple projects. +{{% /notice %}} + +**Configure for a specific project (local scope)** + +Navigate to your project directory and add the Arm MCP Server: + +```console +cd your-project +claude mcp add --transport stdio arm-mcp -- docker run --rm -i --pull=always -v "$(pwd):/workspace" armlimited/arm-mcp:latest +``` + +This configuration is stored in `~/.claude.json` under your project's path and is only accessible when working in this directory. + +**Configure for all projects (user scope)** + +To make the Arm MCP Server available across all your projects: + +```console +claude mcp add --scope user --transport stdio arm-mcp -- docker run --rm -i --pull=always -v "$(pwd):/workspace" armlimited/arm-mcp:latest +``` + +This configuration is stored in `~/.claude.json` and is accessible from any project directory. + +**Configure for team sharing (project scope)** + +To share the MCP server configuration with your team via version control: + +```console +cd your-project +claude mcp add --scope project --transport stdio arm-mcp -- docker run --rm -i --pull=always -v "$(pwd):/workspace" armlimited/arm-mcp:latest +``` + +This creates a `.mcp.json` file in your project root that can be committed to version control. + +## Analyze a local codebase with the Arm MCP Server + +The Arm MCP Server automatically mounts your current working directory to the `/workspace` folder inside the Docker container when you use the configuration commands shown above. + +To analyze a different directory, modify the volume mount in the `docker run` command. For example, to analyze `/Users/username/myproject`: + +```console +claude mcp add --transport stdio arm-mcp -- docker run --rm -i -v "/Users/username/myproject:/workspace" armlimited/arm-mcp:latest +``` + +## Verify the Arm MCP Server is working + +List configured MCP servers: + +```console +claude mcp list +``` + +You should see `arm-mcp` in the list of configured servers. + +Get details about the Arm MCP Server configuration: + +```console +claude mcp get arm-mcp +``` + +To test the server's functionality, start Claude Code and ask it to use the Arm MCP tools: + +```console +claude +``` + +Then try one of these prompts: + +```console +> Use the Arm MCP Server to scan my codebase for x86-specific code +``` + +or + +```console +> Check if the nginx:latest Docker image supports Arm64 +``` + +You can also use the `/mcp` command within Claude Code to see the status of all connected MCP servers and their available tools. + +## Example prompts using the Arm MCP Server + +Here are some example prompts that use the Arm MCP Server tools: + +- `Scan my workspace for code that needs updating for Arm compatibility` +- `Check if the postgres:latest container image supports Arm64 architecture` +- `Search the Arm knowledge base for NEON intrinsics examples` +- `Find learning resources about migrating from x86 to Arm` +- `Analyze this assembly code for performance on Arm processors` + +## Manage MCP servers + +Remove an MCP server: + +```console +claude mcp remove arm-mcp +``` + +Update an MCP server configuration by removing and re-adding it with new settings. + +Check MCP server status within Claude Code: + +```console +> /mcp +``` + +## Troubleshoot MCP Server connections + +If the Arm MCP Server doesn't connect: + +- Verify Docker is running: `docker ps` +- Check that the image was pulled successfully: `docker images | grep arm-mcp` +- Ensure the volume mount path exists and is accessible +- Check that the Docker daemon is running and accessible +- Try restarting Claude Code after configuration changes +- Review the output of `claude mcp get arm-mcp` for configuration errors + +If you encounter issues or have questions, reach out to mcpserver@arm.com. + + +## Custom prompts and workflows + +Create custom prompts for common tasks in your workflow. Refer to the [Claude Code documentation](https://code.claude.com/docs) for advanced configuration options. + diff --git a/content/install-guides/fvps-on-macos.md b/content/install-guides/fvps-on-macos.md index da67fa3389..363d72a3b3 100644 --- a/content/install-guides/fvps-on-macos.md +++ b/content/install-guides/fvps-on-macos.md @@ -26,7 +26,8 @@ multi_install: false # Set to true if first page of multi-page articl multitool_install_part: false # Set to true if a sub-page of a multi-page article, else false layout: installtoolsall # DO NOT MODIFY. Always true for tool install articles --- -This guide is intended to get you up and running with the [Arm Virtual Hardware (AVH) Fixed Virtual Platforms (FVPs)](https://www.arm.com/products/development-tools/simulation/virtual-hardware) on macOS. For a thorough review of all options, refer to the official documentation. +This guide shows you how to use [Arm Virtual Hardware (AVH) Fixed Virtual Platforms (FVPs)](https://www.arm.com/products/development-tools/simulation/virtual-hardware) on macOS by running them in Docker containers. The [official repository](https://github.com/Arm-Examples/FVPs-on-Mac/blob/main/README.md) provides additional technical details. + ## What are the prerequisites for running AVH FVPs on macOS? diff --git a/content/install-guides/mcuxpresso_vs.md b/content/install-guides/mcuxpresso_vs.md index 30cb891d56..3b9b16731c 100644 --- a/content/install-guides/mcuxpresso_vs.md +++ b/content/install-guides/mcuxpresso_vs.md @@ -65,7 +65,7 @@ Download and run the installer. Select one or more packages and click `Install`. -![MCUXpresso Installer #center](/install-guides/_images/MCUXpresso_Installer.png) +![MCUXpresso Installer #center](/install-guides/_images/mcuxpresso_installer.png) ### Are there other embedded development extensions for VS Code? diff --git a/content/install-guides/openvscode-server.md b/content/install-guides/openvscode-server.md index 6405293b2a..65e8513fc4 100644 --- a/content/install-guides/openvscode-server.md +++ b/content/install-guides/openvscode-server.md @@ -105,7 +105,7 @@ With the port open, substitute the public IP address of the instance instead of On ChromeOS you can use the Linux configuration settings to automatically do port forwarding. No SSH connection is needed. -![port forwarding #center](/install-guides/_images/ChromeOSpf.png) +![port forwarding #center](/install-guides/_images/chromeospf.png) ## What other configuration options are available? diff --git a/content/learning-paths/automotive/zenacssdebug/configdb.png b/content/learning-paths/automotive/zenacssdebug/configdb.png deleted file mode 100644 index 819071a7ff..0000000000 Binary files a/content/learning-paths/automotive/zenacssdebug/configdb.png and /dev/null differ diff --git a/content/learning-paths/cross-platform/gitlab-managed-runners/_index.md b/content/learning-paths/cross-platform/gitlab-managed-runners/_index.md index 2f39932a2f..e4615f81fa 100644 --- a/content/learning-paths/cross-platform/gitlab-managed-runners/_index.md +++ b/content/learning-paths/cross-platform/gitlab-managed-runners/_index.md @@ -5,14 +5,14 @@ draft: true cascade: draft: true -minutes_to_complete: 30 +minutes_to_complete: 40 who_is_this_for: This is an Introductory topic for DevOps professionals who are looking to build a CI/CD pipeline with GitLab on Google Axion using GitLab-Hosted runners. learning_objectives: - Create a GitLab Project - Understand basic pipeline script structure and how to use it - - Build and test a simple CI/CD pipeline Using Gitlab-hosted runners + - Build and test a simple CI/CD pipeline using Gitlab-hosted runners which will build and produce a tiny docker image from a simple "Hello world" "C" language program. The image will be built to run on Arm64 machines and will be saved in Gitlab Registery to be used later. prerequisites: @@ -26,10 +26,12 @@ subjects: CI-CD cloud_service_providers: Google Cloud armips: - - Neoverse + - Neoverse-N1 tools_software_languages: - GitLab + - Docker + - C operatingsystems: - Linux diff --git a/content/learning-paths/cross-platform/gitlab-managed-runners/info.md b/content/learning-paths/cross-platform/gitlab-managed-runners/info.md index cc40e6d597..57578b3f4e 100644 --- a/content/learning-paths/cross-platform/gitlab-managed-runners/info.md +++ b/content/learning-paths/cross-platform/gitlab-managed-runners/info.md @@ -15,11 +15,9 @@ A GitLab Runner works with GitLab CI/CD to run jobs in a pipeline. It acts as an 3. Multi-architecture support: GitLab runners support multiple architectures including - **`x86/amd64`** and **`arm64`**. -## What is Google Axion? -Axion is Google's first Arm-based server processor, built using the Armv9 Neoverse V2 CPU. The VM instances are part of the **`C4A`** family of compute instances. To learn more about Google Axion refer to this [page](http://cloud.google.com/products/axion/) . {{% notice Note %}} -All The information provided in the next section are from GitLab official Pages and it's provided here for convenience and can be changed by Gitlab at anytime. Please refer to the [Gitlab Documentation](https://docs.gitlab.com/ci/runners/hosted_runners/) for more details and for the latest updates. +All The information provided in the next section are from GitLab official Pages and it's provided here for convenience and can be changed by Gitlab at anytime. Please refer to the [Gitlab Documentation](https://docs.gitlab.com/ci/runners/hosted_runners/) for more details and for the latest updates. This section is optional. {{% /notice %}} ## GitLab-Hosted Runners Facts diff --git a/content/learning-paths/cross-platform/gitlab-managed-runners/pipeline.md b/content/learning-paths/cross-platform/gitlab-managed-runners/pipeline.md index 2a51972858..713ccd3fdf 100644 --- a/content/learning-paths/cross-platform/gitlab-managed-runners/pipeline.md +++ b/content/learning-paths/cross-platform/gitlab-managed-runners/pipeline.md @@ -8,83 +8,122 @@ layout: learningpathall ## How to Create a CI/CD Pipeline with Gitlab-hosted Runners? -To create the pipeline we only need to create a new **`.gitlab-ci.yml`** file in our Project and define it's stages. Nothing else is needed since Gtilab-hosted runners are readily available to any Project and doesn't need to be created or instantiated by the Gitlab users. +To create the pipeline you only need to create a new **`.gitlab-ci.yml`** file in your Project and define it's stages. Nothing else is needed since Gtilab-hosted runners are readily avilable to any Project and doesn't need to be created or instantiated by the Gitlab users. -Once we run our pipeline with the correct **`tags`** Gitlab will create everything that we need and yep it is as simple as that. +Once you run your pipeline with the correct **`tags`** Gitlab will create everything that you need and yep it is as simple as that. + +## How are you going to test your pipeline functionality? + +You will test the pipeline by building a Docker image from a simple C language "Hello World" program which can run on Arm64 instances/machines and to do that you will need to create the following files: + +1) **`main.c`** File: which is the main program that will get executed when we will run your Docker image later. I only provided a simple example but please feel free to use any program that you like. Although, I advise to start with this simple program to test that everything is working then use anything later after by changing the **`main.c`** file. + +```c +//main.c +#include + +int main(void) { + printf("Hello from an Arm64 Docker image built on GitLab hosted Arm runners!\n"); + return 0; +} +``` + +2) **`DockerFile`** File: This file has a set of instruction for Docker on how to create a Docker image. It simply instructs the Docker on your runner on how to build and package your **``hello``** app into a Docker image. This produces a tiny image and will run on Arm64 hosts as long as the binary is Arm64 (which it will be, since we’re building on an Arm runner). + +```DockerFile +# DockerFile +# syntax=docker/dockerfile:1 + +FROM alpine:3.20 AS build +RUN apk add --no-cache build-base +WORKDIR /src +COPY main.c . +RUN gcc -O2 -static -s -o hello main.c + +FROM scratch +COPY --from=build /src/hello /hello +ENTRYPOINT ["/hello"] + +``` + +3) Optionally **`.dockerignore`** file: This file instructs Docker to ignor certain files that has nothing to do with the image that it will create. + +```.dockerignore +.git +.gitlab-ci.yml +``` + +4) You will also need to create a YML file as I mentioned before which I will explain in more details in the next section. + +To Create any of those files simply follow the same steps in the next section but instead of choosing **`.gitlab-ci.yml`** file just change the name to each of the corresponding file names above. It is very important to create the 3 files from this section first because once you create and commit/save the **`.gitlab-ci.yml`** file, it will simply run the pipeline. If the other 3 files don't exist at that time then the pipeline will fail. ## How to Create .gitlab-ci.yml file in a Gitlab Project? -1. Start by going to the main project page where we will need to Create the CI/CD pipeline. +1. Start by going to the main project page where you will need to Create the CI/CD pipeline. -2. We can choose to create **`.gitlab-ci.yml`** file by using one of the 2 options circled in red in the image below. -![CI-CD-New #center](_images/ci-cd-new.webp) +2. You can choose to create **`.gitlab-ci.yml`** file by using one of the 2 options circled in red in the image below. +![CI-CD-New #center](_images/ci-cd-new.png) -Option1: We can Click on **`Set up CI/CD`** button/link and follow the wizard to create an empty **`.gitlab-ci.yml`** file. +Option1: You can Click on **`Set up CI/CD`** button/link and follow the wizad to create an empty **`.gitlab-ci.yml`** file. Option2: Click on the "+" button. From the popup menu click on **`New File`** option and name the file **`.gitlab-ci.yml`** and then click on **`Commit Changes`** button on the top right hand side like in the image below (Add any message as your commit message). ![New-YML #center](_images/new-yml.png) -3. A page like the one in the image below will be visible with our **`.gitlab-ci.yml`** file. From here, we will need to Click on the **`Edit`** button. A menu will pop up, We will click on **`Edit in pipeline Editor`** which will allow us to add our CD/CD script. -![Editor-YML #center](_images/editor-yml.webp) +3. A page like the one in the image below will be visible with your **`.gitlab-ci.yml`** file. From here, you will need to Click on the **`Edit`** button. A menu will pop up, you will click on **`Edit in pipeline Editor`** which will allow you to add your CD/CD script. +![Editor-YML #center](_images/editor-yml.png) 4. In the pipeline editor, just copy and paste the following YML script and click on commit changes (Add any relevent message as your commit update message). ```YML -# This file is a template, and might need editing before it works on your project. -# This is a sample GitLab CI/CD configuration file that should run without any modifications. -# It demonstrates a basic 3 stage CI/CD pipeline. Instead of real tests or scripts, -# it uses echo commands to simulate the pipeline execution. -# -# A pipeline is composed of independent jobs that run scripts, grouped into stages. -# Stages run in sequential order, but jobs within stages run in parallel. -# -# For more information, see: https://docs.gitlab.com/ee/ci/yaml/#stages -# -# You can copy and paste this template into a new `.gitlab-ci.yml` file. -# You should not add this template to an existing `.gitlab-ci.yml` file by using the `include:` keyword. -# -# To contribute improvements to CI/CD templates, please follow the Development guide at: -# https://docs.gitlab.com/development/cicd/templates/ -# This specific template is located at: -# https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Getting-Started.gitlab-ci.yml - -stages: # List of stages for jobs, and their order of execution - - build - - test - - deploy - -build-job: # This job runs in the build stage, which runs first. - stage: build - tags: - - saas-linux-small-arm64 #Instruct Gitlab to use it's own Hosted runners that use Linux on Arm64 instance of size small. - script: - - echo "Compiling the code..." - - echo "Compile complete." +#First Section +stages: [build, test, push] -unit-test-job: # This job runs in the test stage. - stage: test # It only starts when the job in the build stage completes successfully. - tags: - - saas-linux-small-arm64 #Instruct Gitlab to use it's own Hosted runners that use Linux on Arm64 instance of size small. - script: - - echo "Running unit tests... This will take about 60 seconds." - - sleep 60 - - echo "Code coverage is 90%" +variables: + IMAGE_TAG: "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA" + IMAGE_LATEST: "$CI_REGISTRY_IMAGE:latest" -lint-test-job: # This job also runs in the test stage. - stage: test # It can run at the same time as unit-test-job (in parallel). - tags: - - saas-linux-small-arm64 #Instruct Gitlab to use it's own Hosted runners that use Linux on Arm64 instance of size small. + # Talk to docker:dind over TLS (default behavior for docker:dind) + DOCKER_HOST: "tcp://docker:2376" + DOCKER_TLS_CERTDIR: "/certs" + DOCKER_CERT_PATH: "/certs/client" + DOCKER_TLS_VERIFY: "1" + +#Second Section +build_test_push: + stage: build + tags: # This tag is used to specify the size of the runner for each stage but it can be defined on the top of the file if you want to use the same exact runner size for all the stages in your pipeline + - saas-linux-small-arm64 + image: docker:27 + services: + - name: docker:27-dind + before_script: + - uname -m + # install lscpu (provided by util-linux) which is used to identify the CPU used for this stage runner + - apk add --no-cache util-linux + - lscpu + - docker version + - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY" script: - - echo "Linting code... This will take about 10 seconds." - - sleep 10 - - echo "No lint issues found." + - docker build --pull -t "$IMAGE_TAG" . + - docker run --rm "$IMAGE_TAG" + - docker push "$IMAGE_TAG" -deploy-job: # This job runs in the deploy stage. - stage: deploy # It only runs when *both* jobs in the test stage complete successfully. +#Third Section +push_latest: + stage: push tags: - - saas-linux-small-arm64 #Instruct Gitlab to use it's own Hosted runners that use Linux on Arm64 instance of size small. - environment: production + - saas-linux-small-arm64 + image: docker:27 + services: + - name: docker:27-dind + before_script: + - apk add --no-cache util-linux #since each stage is using a different runner then we need to check this CPU as well + - lscpu + - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY" script: - - echo "Deploying application..." - - echo "Application successfully deployed." + - docker pull "$IMAGE_TAG" + - docker tag "$IMAGE_TAG" "$IMAGE_LATEST" + - docker push "$IMAGE_LATEST" + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH ``` 5. Once you commit the file updates, Gitlab will check the scripts for errors and will try to execute the pipeline directly. diff --git a/content/learning-paths/cross-platform/gitlab-managed-runners/project.md b/content/learning-paths/cross-platform/gitlab-managed-runners/project.md index 1ad8bea1a2..aa1c7c196b 100644 --- a/content/learning-paths/cross-platform/gitlab-managed-runners/project.md +++ b/content/learning-paths/cross-platform/gitlab-managed-runners/project.md @@ -6,13 +6,13 @@ weight: 10 layout: learningpathall --- -## Where Should We Start? +## Where Should you Start? Start by logging into your GitLab account or create a new one in the [Gitlab](https://gitlab.com/) main page. -We will need to a new project/repo that will contain all our project files including our **`CI/CD`** pipeline configuration file. +You will need to a new project/repo that will contain all your project files including your **`CI/CD`** pipeline configuration file. -We can also choose to use any previously created projects in our Gitlab account. Simply open your previously created project. If that is the case then skip the rest of the steps in the current page and move to the next steps in this tutorial. +You can also choose to use any previously created projects in your Gitlab account. Simply open your previously created project, If that is the case then skip the rest of the steps in the current page and move to the next steps in this tutorial. ## Create a New Project in Gitlab @@ -21,17 +21,17 @@ We can also choose to use any previously created projects in our Gitlab account. 2. Click on the **`New Project`** button on the top right hand side as the image below. ![Gitlab-Projects #center](_images/gitlab-projects.png) -3. We will get a new screen like the image below with multiple options. You can choose any of the 2 options highlighted in red from the image below. +3. You will get a new screen like the image below with multiple options. You can choose any of the 2 options highlighted in red from the image below. {{% notice Note %}} -If we chose option 2 then we will need to choose **`GitLab CI/CD components`** option from the list of templates. +If you chose option 2 then you will need to choose the **`GitLab CI/CD components`** option from the list of templates. {{%/notice%}} ![New-Project #center](_images/new-project.png) -4. Regardles of which option we choose, We will get a screen like the image below where we need to fill-in fields highlighted in red. The first field is the **`Project Name`** which we will name it **`CI-CD Runner`**. In the second field we need to choose any option from the **`Project Url`** list then click on the **`Create Project`** button at the end of the page. +4. Regardles of which option you choose, you will get a screen like the image below where you will need to fill-in the fields highlighted in red. The first field is the **`Project Name`** which you can name it **`CI-CD Runner`**. In the second field you need to choose any option from the **`Project Url`** list then click on the **`Create Project`** button at the end of the page. ![Project-Info #center](_images/project-info.png) -##### **If we did everything correctly then we should get a screen like the one in the image below.** +##### **If you did everything correctly then you should get a screen like the one in the image below.** ![Project-Done #center](_images/project-done.png) diff --git a/content/learning-paths/cross-platform/gitlab-managed-runners/results.md b/content/learning-paths/cross-platform/gitlab-managed-runners/results.md index afde4d2267..67ab88e0e0 100644 --- a/content/learning-paths/cross-platform/gitlab-managed-runners/results.md +++ b/content/learning-paths/cross-platform/gitlab-managed-runners/results.md @@ -10,16 +10,18 @@ layout: learningpathall The Pipeline script has multiple sections where each section instructs the pipeline operator on what todo or use and how each Stage looks like. -### First Section: Stages +### First Section: The stages -In this section we are describing how many sequential stages will our pipeline have and what are their names (ex. **`Build, Test and Deploy`**). If we would like all the stages or jobs to run simultaneously then we simply don't define this section. +In this section you are describing how many squentional stages that your pipeline will have and what are their names (ex. **`Build, Test and Push`**). If you would like all the stages or jobs to run simultinously then you simply don't define this section. -### Second Section: Build-Job part of the Build stage +I am also defining some **`Variables`** that I will use in the other sections for simplisty. -In this section we are defining the Build-Job as part of the Build stage. This stage will run on Gitlab-Hosted runner that uses Linux OS on Arm64 instance of size small. +### Second Section: build_test_push part for the Build stage + +In this section you are defining the build_test_push as the Build stage. This stage will run on Gitlab-Hosted runner that uses Linux OS on Arm64 instance of size small. I used the **`lscpu`** command to print out the CPU information for reference. {{% notice Important Note %}} -Gitlab offers 3 Arm64 based Instances that use Linux as their OS. +Gitlab offers 3 Arm64 based Instances that use Linux as their OS. In the Free tier you can only use the small version. - saas-linux-small-arm64 - saas-linux-medium-arm64 @@ -29,7 +31,11 @@ For more information about all Arm and other available Gitlab-hosted runners che {{%/notice%}} -### Other Sections: +I am also saving my Docker image in GitLab registery because it's the easiest way to do that but you can modify your pipeline to save your image in any other registery that you prefer. To get the Gitlab registery creditiationals I am using **`$CI`** variables that are defined in the Gitlab enviornment and saving them in Docker for simplisty. Please note it's always recommended to encrypt your information or save it in a sercrets/passwords manages. + +### Third Section: + +You will notice that in this stage I am simply testing that I am able to get the image that I saved in my registery before and pushing it back as the latest version. The rest of the other sections follow the same pattern. You will notice that the **`Test`** stage for example has 2 Jobs in it (unit-test-job and lint-test-job). The **`Deploy`** stage here has only 1 Job called **`deploy-job`**. As you get to learn more YML scripting you will be able to add a lot more complex functionality to your pipelines. @@ -46,6 +52,29 @@ From the left hand side panel, Navigate to **`Build`** then to **`Pipeline`** th To check the status of your pipeline and to check the output of any of it's Jobs simply click on any of the **`Jobs`** as the image below (with red rectangle around them). ![pipeline-execution #center](_images/pipeline-execution.webp) +You can also download the docker image that you saved in your gitlab registery and run it on an Arm64 instance/box for testing using the following bash script. + +```bash +docker login registry.gitlab.com +docker pull registry.gitlab.com//:latest +docker run --rm registry.gitlab.com//:latest +``` + +If everything works correctly you should see an output like in the box below. + +```output +Hello from an Arm64 Docker image built on GitLab hosted Arm runners! +``` + +You can also check your Gitlab Registery by going to your project then: + +- Go to Deploy → Container Registry +- You should see new-docker +- With tags like latest and **``** + +If it’s there, the registry did its job. + + ## Gitlab Helpful tools If you navigate to your pipeline editor from before you will notice that there are more tabs in that page other than the **`Edit`** tab. ![visual-pipeline #center](_images/visual-pipeline.webp) diff --git a/content/learning-paths/cross-platform/kleidiai-explainer/Arm_KleidiAI_square_color.png b/content/learning-paths/cross-platform/kleidiai-explainer/Arm_KleidiAI_square_color.png deleted file mode 100644 index b55c12e40e..0000000000 Binary files a/content/learning-paths/cross-platform/kleidiai-explainer/Arm_KleidiAI_square_color.png and /dev/null differ diff --git a/content/learning-paths/cross-platform/kleidiai-explainer/KleidiAI-src-matmul.JPG b/content/learning-paths/cross-platform/kleidiai-explainer/KleidiAI-src-matmul.JPG deleted file mode 100644 index 96a15e9636..0000000000 Binary files a/content/learning-paths/cross-platform/kleidiai-explainer/KleidiAI-src-matmul.JPG and /dev/null differ diff --git a/content/learning-paths/cross-platform/kleidiai-explainer/KleidiAI-src.JPG b/content/learning-paths/cross-platform/kleidiai-explainer/kleidiai-src.JPG similarity index 100% rename from content/learning-paths/cross-platform/kleidiai-explainer/KleidiAI-src.JPG rename to content/learning-paths/cross-platform/kleidiai-explainer/kleidiai-src.JPG diff --git a/content/learning-paths/cross-platform/kleidiai-explainer/page2.md b/content/learning-paths/cross-platform/kleidiai-explainer/page2.md index fe1c0be900..1e25fa8da1 100644 --- a/content/learning-paths/cross-platform/kleidiai-explainer/page2.md +++ b/content/learning-paths/cross-platform/kleidiai-explainer/page2.md @@ -16,7 +16,7 @@ There are essentially two types of KleidiAI micro-kernels today: 2. Matrix Multiplication routines - the three directories with the prefix `matmul_clamp`. Each directory contains routines specialized for a specific input data type. -![KleidiAI stuff](KleidiAI-src.JPG "KleidiAI src directory") +![KleidiAI stuff](kleidiai-src.JPG "KleidiAI src directory") ### What are the quantization levels that KleidiAI supports? KleidiAI has multiple matrix multiplication micro-kernels, and dynamic quantization routines, to optimally support all model quantization levels. To learn more about model quantization and how selecting the right quantization level affects your AI-based application, refer to [this Learning Path](/learning-paths/servers-and-cloud-computing/llama-cpu/llama-chatbot#quantization-format). diff --git a/content/learning-paths/cross-platform/mca-godbolt/running_mca.md b/content/learning-paths/cross-platform/mca-godbolt/running_mca.md index 4936df31d2..b16ee0cc4e 100644 --- a/content/learning-paths/cross-platform/mca-godbolt/running_mca.md +++ b/content/learning-paths/cross-platform/mca-godbolt/running_mca.md @@ -393,6 +393,12 @@ You can see by looking at the timeline view that instructions no longer depend o Instructions also spend less time waiting in the scheduler's queue. This explains why the performance of `sum_test2.s` is so much better than `sum_test1.s`. -Note the use of the flag `-mcpu=neoverse-v2` throughout all of those examples. This flag tells MCA to simulate the performance of the code in `sum_test1.s` and `sum_test2.s` on a Neoverse V2 core. This flag can be changed to any core supported in MCA. You can find what cores are supported in MCA by running `llvm-mca -mcpu=help <<<''`. You can also look at the LLVM sources in [llvm-project](https://github.com/llvm/llvm-project/tree/main/llvm/test/tools/llvm-mca/AArch64), which will give you more detailed examples. For instance, when looking at the Neoverse cores, there is currently support for the N1, N2, N3 and the V1, V2, V3 cores. +Note the use of the flag `-mcpu=neoverse-v2` throughout all of those examples. This flag tells MCA to simulate the performance of the code in `sum_test1.s` and `sum_test2.s` on a Neoverse V2 core. This flag can be changed to any core supported in MCA. + +You can find what cores are supported in MCA by running `llvm-mca -mcpu=help`. + +If you are using an older version of `llvm-mca`, you may have to add an empty string to print the help, use `llvm-mca -mcpu=help <<<''`. + +You can also look at the LLVM sources in [llvm-project](https://github.com/llvm/llvm-project/tree/main/llvm/test/tools/llvm-mca/AArch64), which will give you more detailed examples. For instance, when looking at the Neoverse cores, there is currently support for the N1, N2, N3 and the V1, V2, V3 cores. In the next section, you can try running `llvm-mca` with Compiler Explorer. diff --git a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/1-get-started.md b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/1-get-started.md index 8c5b2e3bfe..d826539d12 100644 --- a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/1-get-started.md +++ b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/1-get-started.md @@ -299,7 +299,7 @@ Then select the **Reopen in Container** menu entry as shown below. It automatically finds and uses `.devcontainer/devcontainer.json`: -![VSCode Docker alt-text#center](VSCode.png "Figure 1: Setting up the Docker container.") +![VSCode Docker alt-text#center](vscode.png "Figure 1: Setting up the Docker container.") All your commands now run within the container, so there is no need to prepend them with a Docker invocation, as VS Code handles all this seamlessly for you. diff --git a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/VSCode.png b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/vscode.png similarity index 100% rename from content/learning-paths/cross-platform/multiplying-matrices-with-sme2/VSCode.png rename to content/learning-paths/cross-platform/multiplying-matrices-with-sme2/vscode.png diff --git a/content/learning-paths/cross-platform/remoteit/connections.md b/content/learning-paths/cross-platform/remoteit/connections.md index eb5287ef43..dbfa4e7b93 100644 --- a/content/learning-paths/cross-platform/remoteit/connections.md +++ b/content/learning-paths/cross-platform/remoteit/connections.md @@ -18,7 +18,7 @@ The advantages are: Peer to Peer connections are created by installing a software package with initiator support on the initiator device. Install the Remote.It Desktop application or Remote.It CLI on your initiator device to support Peer to Peer connection. -![example image alt-text#center](Remote.It-Connections.png "Proxy and Peer to Peer Connection Illustration") +![example image alt-text#center](remote.it-connections.png "Proxy and Peer to Peer Connection Illustration") ## Peer to Peer Connections diff --git a/content/learning-paths/cross-platform/remoteit/Remote.It-Connections.png b/content/learning-paths/cross-platform/remoteit/remote.it-connections.png similarity index 100% rename from content/learning-paths/cross-platform/remoteit/Remote.It-Connections.png rename to content/learning-paths/cross-platform/remoteit/remote.it-connections.png diff --git a/content/learning-paths/cross-platform/remoteit/targets.png b/content/learning-paths/cross-platform/remoteit/targets.png deleted file mode 100644 index 4d576b10e3..0000000000 Binary files a/content/learning-paths/cross-platform/remoteit/targets.png and /dev/null differ diff --git a/content/learning-paths/cross-platform/woa_azure/images/azure2.png b/content/learning-paths/cross-platform/woa_azure/images/azure2.png deleted file mode 100644 index 7e1572b843..0000000000 Binary files a/content/learning-paths/cross-platform/woa_azure/images/azure2.png and /dev/null differ diff --git a/content/learning-paths/cross-platform/woa_azure/images/click_create.png b/content/learning-paths/cross-platform/woa_azure/images/click_create.png deleted file mode 100644 index b7e1acc668..0000000000 Binary files a/content/learning-paths/cross-platform/woa_azure/images/click_create.png and /dev/null differ diff --git a/content/learning-paths/cross-platform/woa_azure/images/select_win_arm_64.png b/content/learning-paths/cross-platform/woa_azure/images/select_win_arm_64.png deleted file mode 100644 index 7dfe7d5355..0000000000 Binary files a/content/learning-paths/cross-platform/woa_azure/images/select_win_arm_64.png and /dev/null differ diff --git a/content/learning-paths/cross-platform/woa_azure/images/vm_search.png b/content/learning-paths/cross-platform/woa_azure/images/vm_search.png deleted file mode 100644 index dcc22795d3..0000000000 Binary files a/content/learning-paths/cross-platform/woa_azure/images/vm_search.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/_index.md b/content/learning-paths/embedded-and-microcontrollers/_index.md index 49c994ac52..c7b2a30c7d 100644 --- a/content/learning-paths/embedded-and-microcontrollers/_index.md +++ b/content/learning-paths/embedded-and-microcontrollers/_index.md @@ -11,8 +11,8 @@ maintopic: true operatingsystems_filter: - Android: 2 - Baremetal: 30 -- Linux: 34 -- macOS: 7 +- Linux: 35 +- macOS: 8 - RTOS: 10 - Windows: 5 subjects_filter: @@ -20,7 +20,7 @@ subjects_filter: - Containers and Virtualization: 6 - Embedded Linux: 4 - Libraries: 3 -- ML: 18 +- ML: 19 - Performance and Architecture: 22 - RTOS Fundamentals: 5 - Security: 2 @@ -39,7 +39,9 @@ tools_software_languages_filter: - Arm Streamline: 1 - Arm Virtual Hardware: 12 - Assembly: 1 -- C: 5 +- Baremetal: 1 +- Bash: 1 +- C: 6 - ChatGPT: 1 - Clang: 1 - CMSIS: 4 @@ -50,7 +52,7 @@ tools_software_languages_filter: - Containerd: 1 - CPP: 1 - DetectNet: 1 -- Docker: 10 +- Docker: 11 - DSTREAM: 2 - Edge AI: 3 - Edge Impulse: 2 @@ -86,13 +88,14 @@ tools_software_languages_filter: - Paddle: 1 - Performance analysis: 1 - Porcupine: 1 -- Python: 9 +- Python: 10 - PyTorch: 4 - QEMU: 1 - Raspberry Pi: 7 - Remote.It: 1 - Runbook: 4 - STM32: 2 +- systemd: 1 - TensorFlow: 3 - TensorRT: 1 - tinyML: 2 diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/connecting_peripheral.md b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/connecting_peripheral.md index 86fdf0c0a1..08644f427f 100644 --- a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/connecting_peripheral.md +++ b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/connecting_peripheral.md @@ -12,23 +12,23 @@ Follow the steps outline below to connect the AXI4 peripheral you created in the 1. Right-click on the empty space again in the diagram and choose “Create Port...” (Ctrl+K). Create 2 ports with the following settings and names: - ![Creating output port](images/Picture15.png) + ![Creating output port](images/picture15.png) *Figure 3.1. Creating output port* - ![Creating input port](images/Picture16.png) + ![Creating input port](images/picture16.png) *Figure 3.2. Creating input port* 2. Wire the “sw[3:0]” input to the “sw[3:0]” port of the “axi_gpio_asoc_0” block and the same for the “led[3:0]” output to the equivalent port of the block as shown in the diagram below. This connects them to external ports of the ZYNQ chip: - ![Vivado block diagram](images/Picture17.png) + ![Vivado block diagram](images/picture17.png) *Figure 3.3. Vivado block diagram* 3. Two IP blocks will be generated automatically. The “Processor System Reset” IP is used to generate reset signals for different peripherals. The “AXI Interconnect” IP here is used to interconnect AXI4-Lite Subordinate and AXI Manager. Select the “Address Editor” tab next to “Diagram” and change the “64K” to “4K”. Save all your progress. - ![Changing peripheral address settings](images/Picture18.png) + ![Changing peripheral address settings](images/picture18.png) *Figure 3.4. Changing peripheral address settings* @@ -40,26 +40,26 @@ Follow the steps outline below to connect the AXI4 peripheral you created in the 5. Right-click the “Generate Bitstream” option in the “Flow Navigator” on the left and select “Bitstream settings”. Click the three dots next to “tcl.pre”: - ![Changing bitstream settings](images/Picture19.png) + ![Changing bitstream settings](images/picture19.png) *Figure 3.5. Changing bitstream settings* 6. Select the “New Script” option, click the three dots next to the empty box, choose the “pins.tcl” file you created earlier and click “Ok” on all windows. Right-click the “Constraints” under the sources tab and select “Add sources”: - ![Adding sources](images/Picture20.png) + ![Adding sources](images/picture20.png) *Figure 3.6. Adding sources* 7. Select “Add or create constraints” and click “Next”. Select “Create File”, give any name to the file for example pin_constraints, and click “Finish”: - ![Creating a constraints file](images/Picture21.png) + ![Creating a constraints file](images/picture21.png) *Figure 3.7. Creating a constraints file* 8. Expand the “Constraints” folder within the “Sources” tab and double-click the file you just created to open it. Add the following constraints from [Digilent/Zybo-Z7-10-Pmod-VGA/blob/master/src/constraints/Zybo-Z7-Master.xdc](https://github.com/Digilent/Zybo-Z7-10-Pmod-VGA/blob/master/src/constraints/Zybo-Z7-Master.xdc), and save the file: - ![Editing constraints file](images/Picture22.png) + ![Editing constraints file](images/picture22.png) *Figure 3.8. Editing constraints file* diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/creating_peripheral.md b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/creating_peripheral.md index 1498a78246..7b569a65dc 100644 --- a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/creating_peripheral.md +++ b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/creating_peripheral.md @@ -12,19 +12,19 @@ The Xilinx Vivado tools provide a simplified way to create an AXI4 peripheral. Y 1. Start by clicking “Tools” -> “Create and Package New IP”. Click “Next” and choose the following option: - ![Creating AXI4 peripheral](images/Picture7.png) + ![Creating AXI4 peripheral](images/picture7.png) *Figure 2.1. Creating AXI4 peripheral* 2. Give the following name to the IP (you may keep the IP location path as provided by default): - ![Adding peripheral details](images/Picture8.jpg) + ![Adding peripheral details](images/picture8.jpg) *Figure 2.2. Adding peripheral details* 3. You need a Subordinate interface and four 32-bit registers for the switches and LEDs. - ![Peripheral settings for Subordinate interface](images/Picture9.png) + ![Peripheral settings for Subordinate interface](images/picture9.png) *Figure 2.3. Peripheral settings for Subordinate interface* @@ -36,7 +36,7 @@ These are the basic template files generated for an AXI-Lite peripheral. You can 5. Double-click to open the top-level Verilog file called “AUP_advanced_SoC_v1_0.v”: - ![Editing top-level Verilog file](images/Picture10.png) + ![Editing top-level Verilog file](images/picture10.png) *Figure 2.4. Editing top-level Verilog file* @@ -64,7 +64,7 @@ These are the basic template files generated for an AXI-Lite peripheral. You can ``` 8. Save the changes in the file (Ctrl+S). Next, expand and open the other Verilog file (AUP_advanced_SoC_v1_0_S00_AXI.v) shown below: - ![Edit the second Verilog file](images/Picture11.png) + ![Edit the second Verilog file](images/picture11.png) *Figure 2.5. Edit the second Verilog file* @@ -93,7 +93,7 @@ These are the basic template files generated for an AXI-Lite peripheral. You can 12. Save the changes in the file (Ctrl+S). Next, go to the “Package IP – AUP_advanced_SoC” tab, choose the “Customization Parameters” option on the left and click “Merge Changes from Customization Parameters Wizard” to update the IP package with the changes made in HDL files: - ![. Saving all the changes](images/Picture12.png) + ![. Saving all the changes](images/picture12.png) *Figure 2.6. Saving all the changes* @@ -127,7 +127,7 @@ These are the basic template files generated for an AXI-Lite peripheral. You can ``` 15. Then, click on “Settings” (under “Project Manager”) in the “Flow Navigator” menu on the left. Expand the “IP” section in the new window that appears and choose the “Repository” option. - ![Adding IP Repository](images/Picture13.png) + ![Adding IP Repository](images/picture13.png) *Figure 2.7. Adding IP Repository* @@ -135,7 +135,7 @@ These are the basic template files generated for an AXI-Lite peripheral. You can 17. Click “Run Connection Automation” and then click “OK” to connect the AXI-Lite Subordinate interface on GPIO peripheral to the AXI Manager interface on Arm processor. - ![Connect AXI-Lite Subordinate interface (Custom IP) to AXI Manager interface](images/Picture14.png) + ![Connect AXI-Lite Subordinate interface (Custom IP) to AXI Manager interface](images/picture14.png) *Figure 2.8. Connect AXI-Lite Subordinate interface (Custom IP) to AXI Manager interface* diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/generating_bitstream.md b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/generating_bitstream.md index 89982fca43..69fd4994d2 100644 --- a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/generating_bitstream.md +++ b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/generating_bitstream.md @@ -12,56 +12,56 @@ In this section you will generate the bitstream for this project and write a C p 1. Under the “Sources” tab on the left, expand the “Design Sources” folder, right-click the design1.bd file, choose the “Create HDL Wrapper” and select all default options. - ![Creating HDL Wrapper](images/Picture23.png) + ![Creating HDL Wrapper](images/picture23.png) *Figure 3.1. Creating HDL Wrapper* 2. Save any other changes and click “Generate Bitstream” on the left and click “Ok” for all pop-ups. This process takes some time. Once the process is done, select “Open Hardware Manager”: - ![Generating bitstream and uploading to board](images/Picture24.png) + ![Generating bitstream and uploading to board](images/picture24.png) *Figure 3.2. Generating bitstream and uploading to board* 3. Connect the board and make sure the jumper (JP3) above the red LED on the Zybo board is in JTAG mode. Then, in Vivado, click “Auto Connect” in the Hardware Manager as shown below: - ![Connecting the board](images/Picture25.png) + ![Connecting the board](images/picture25.png) *Figure 3.3. Connecting the board* 4. Right-click on the board, select “Program Device” as shown below and click “Program” on the pop-up window. - ![Programming the board](images/Picture26.png) + ![Programming the board](images/picture26.png) *Figure 3.4. Programming the board* 5. Once the board is programmed, the green LED labeled “LD12” should light up on the board. Click “File” on the main menu bar and select “Export” -> “Export Hardware” and click “Next” on the pop-up window. Choose the following option on the next page: - ![Exporting hardware and bitstream file](images/Picture27.png) + ![Exporting hardware and bitstream file](images/picture27.png) *Figure 3.5. Exporting hardware and bitstream file* 6. Choose the “export to” location as the project folder and save the file. Then click “Finish”. Next, click “Tools” on the main menu bar and select “Launch Vitis IDE”. Choose the same project folder as your workspace. Click “File” -> “New” -> “Application Project”. - ![Creating a new application project](images/Picture28.png) + ![Creating a new application project](images/picture28.png) *Figure 3.6. Creating a new application project* 7. Select the “Create a new platform from hardware (XSA)” tab and click browse to select the XSA file you saved earlier: - ![Adding the XSA file](images/Picture29.png) + ![Adding the XSA file](images/picture29.png) *Figure 3.7. Adding the XSA file* 8. Click next and give a name (e.g. led_system) to the application project. Click “Next” until you reach the following page and choose “Empty Application(C)” and click “Finish”: - ![Creating an empty C Application](images/Picture30.png) + ![Creating an empty C Application](images/picture30.png) *Figure 3.8. Creating an empty C Application* 9. Then right-click the “src” folder within the application project you created and add a new file called “main.c”. - ![Adding a main.c file](images/Picture31.png) + ![Adding a main.c file](images/picture31.png) *Figure 3.9. Adding a main.c file* @@ -81,7 +81,7 @@ In this section you will generate the bitstream for this project and write a C p 12. Right-click the application project in the explorer tab, select “Build Project” and ensure that the build is successful. Then right click again and select “Run As” and then “1 Launch Hardware” to upload everything to the board. - ![Running the program on the board.](images/Picture32.png) + ![Running the program on the board.](images/picture32.png) *Figure 3.10. Running the program on the board.* diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture10.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture10.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture10.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture10.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture11.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture11.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture11.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture11.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture12.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture12.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture12.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture12.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture13.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture13.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture13.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture13.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture14.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture14.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture14.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture14.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture15.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture15.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture15.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture15.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture16.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture16.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture16.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture16.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture17.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture17.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture17.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture17.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture18.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture18.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture18.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture18.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture19.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture19.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture19.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture19.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture2.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture2.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture2.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture2.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture20.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture20.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture20.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture20.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture21.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture21.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture21.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture21.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture22.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture22.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture22.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture22.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture23.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture23.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture23.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture23.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture24.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture24.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture24.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture24.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture25.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture25.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture25.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture25.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture26.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture26.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture26.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture26.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture27.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture27.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture27.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture27.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture28.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture28.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture28.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture28.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture29.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture29.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture29.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture29.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture3.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture3.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture3.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture3.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture30.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture30.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture30.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture30.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture31.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture31.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture31.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture31.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture32.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture32.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture32.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture32.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture4.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture4.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture4.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture4.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture5.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture5.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture5.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture5.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture6.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture6.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture6.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture6.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture7.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture7.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture7.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture7.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture8.jpg b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture8.jpg similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture8.jpg rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture8.jpg diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture9.png b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture9.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/Picture9.png rename to content/learning-paths/embedded-and-microcontrollers/advanced_soc/images/picture9.png diff --git a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/setup.md b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/setup.md index 8ba3dd68ef..ce962fc650 100644 --- a/content/learning-paths/embedded-and-microcontrollers/advanced_soc/setup.md +++ b/content/learning-paths/embedded-and-microcontrollers/advanced_soc/setup.md @@ -16,31 +16,31 @@ After installation is complete, follow the steps outlined below to setup a works 2. Open Xilinx Vivado and create a new project in your workspace as shown below: - ![Creating a project](images/Picture2.png) + ![Creating a project](images/picture2.png) *Figure 1.1. Creating a project* 3. Give it a meaningful name and click "Next". Select RTL Project and click “Next”. You will not be adding any sources or constraints so click “Next” for the next two pages. In the next page, click the “Boards” tab and search for “Zybo” in the search bar and click the download button next to “Zybo Z7-10” if the option is available. Then select it and click “Next”. Do not click on the hyperlink but click on the empty area next to “Zybo Z7-10”. - ![Board Selection](images/Picture3.png) + ![Board Selection](images/picture3.png) *Figure 1.2. Board Selection* 4. Click “Finish”. In the “Flow Navigator” menu on the left, click “Create Block Design” under “IP Integrator”: - ![Creating a new block design](images/Picture4.png) + ![Creating a new block design](images/picture4.png) *Figure 1.3. Creating a new block design* 5. Choose a design name or use the default one and click “Ok”. Within the empty “Diagram” box on the right-hand side, right-click and select “Add IP”. Enter “Zynq” in the search box and choose “ZYNQ7 Processing System”. Click the “Run Block Automation” option that has now appeared. - ![Running Block Automation](images/Picture5.png) + ![Running Block Automation](images/picture5.png) *Figure 1.4. Running Block Automation* 6. Click “Ok” on the pop-up window. - ![Run Block Automation default settings](images/Picture6.png) + ![Run Block Automation default settings](images/picture6.png) Figure 1.5. Run Block Automation default settings* diff --git a/content/learning-paths/embedded-and-microcontrollers/arduino-pico/_images/Breadboard.jpeg b/content/learning-paths/embedded-and-microcontrollers/arduino-pico/_images/breadboard.jpeg similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/arduino-pico/_images/Breadboard.jpeg rename to content/learning-paths/embedded-and-microcontrollers/arduino-pico/_images/breadboard.jpeg diff --git a/content/learning-paths/embedded-and-microcontrollers/arduino-pico/_images/complete.webp b/content/learning-paths/embedded-and-microcontrollers/arduino-pico/_images/complete.webp deleted file mode 100644 index b654878a73..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/arduino-pico/_images/complete.webp and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/arduino-pico/_images/PIR-Sensor-Pinout.png b/content/learning-paths/embedded-and-microcontrollers/arduino-pico/_images/pir-sensor-pinout.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/arduino-pico/_images/PIR-Sensor-Pinout.png rename to content/learning-paths/embedded-and-microcontrollers/arduino-pico/_images/pir-sensor-pinout.png diff --git a/content/learning-paths/embedded-and-microcontrollers/arduino-pico/_images/traditional_arm.png b/content/learning-paths/embedded-and-microcontrollers/arduino-pico/_images/traditional_arm.png deleted file mode 100644 index 9c1270ee43..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/arduino-pico/_images/traditional_arm.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/arduino-pico/raspberrypi_pico.md b/content/learning-paths/embedded-and-microcontrollers/arduino-pico/raspberrypi_pico.md index 9ce571da90..73540ff9f9 100644 --- a/content/learning-paths/embedded-and-microcontrollers/arduino-pico/raspberrypi_pico.md +++ b/content/learning-paths/embedded-and-microcontrollers/arduino-pico/raspberrypi_pico.md @@ -20,7 +20,7 @@ You were probably expecting an Arduino board, rather than a Raspberry Pi board b For interacting with the physical world you can use two cheap commodity components. -![PIR Sensor](_images/PIR-Sensor-Pinout.png) +![PIR Sensor](_images/pir-sensor-pinout.png) First a PIR motion sensor. This sensor reacts to infrared photons emitted by a warm moving object, like a person or animal. The interface is simple, it has one pin for input voltage, one pin for ground to complete the circuit, and a third pin that will have the same voltage as the input pin when motion is detected, and the same voltage as the ground pin when it isn't. @@ -29,7 +29,7 @@ First a PIR motion sensor. This sensor reacts to infrared photons emitted by a w Second, a very simple electric buzzer. You could get fancy with one of these and make it play different sounds with something called Pulse Width Modulation (PWM) but, for simplicity, you can give it a constant voltage which will result in a high-pitched beeping noise. -![Breadboard](_images/Breadboard.jpeg) +![Breadboard](_images/breadboard.jpeg) Finally, you can use a breadboard to connect the components together without having to do any soldering. diff --git a/content/learning-paths/embedded-and-microcontrollers/asm/images/AddSource.png b/content/learning-paths/embedded-and-microcontrollers/asm/images/AddSource.png deleted file mode 100644 index e6700e4f54..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/asm/images/AddSource.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/asm/images/AddSource2.png b/content/learning-paths/embedded-and-microcontrollers/asm/images/AddSource2.png deleted file mode 100644 index bb5bc3d418..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/asm/images/AddSource2.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/asm/images/CallStack.png b/content/learning-paths/embedded-and-microcontrollers/asm/images/CallStack.png deleted file mode 100644 index 983b8d835f..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/asm/images/CallStack.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/asm/images/CallStack2.png b/content/learning-paths/embedded-and-microcontrollers/asm/images/CallStack2.png deleted file mode 100644 index bab1cf685c..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/asm/images/CallStack2.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/asm/images/CallStack3.png b/content/learning-paths/embedded-and-microcontrollers/asm/images/CallStack3.png deleted file mode 100644 index eb4e1f2e6a..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/asm/images/CallStack3.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/asm/images/Flash.png b/content/learning-paths/embedded-and-microcontrollers/asm/images/Flash.png deleted file mode 100644 index 34664fe284..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/asm/images/Flash.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/asm/images/NewKeilProject.png b/content/learning-paths/embedded-and-microcontrollers/asm/images/NewKeilProject.png deleted file mode 100644 index cea8ac39dc..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/asm/images/NewKeilProject.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/asm/images/ProjectExplorer.png b/content/learning-paths/embedded-and-microcontrollers/asm/images/ProjectExplorer.png deleted file mode 100644 index 97dfc9d378..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/asm/images/ProjectExplorer.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/asm/images/SelectDevice1.png b/content/learning-paths/embedded-and-microcontrollers/asm/images/SelectDevice1.png deleted file mode 100644 index a8430717d1..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/asm/images/SelectDevice1.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/asm/images/SelectDevice2.png b/content/learning-paths/embedded-and-microcontrollers/asm/images/SelectDevice2.png deleted file mode 100644 index e03bea4ed0..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/asm/images/SelectDevice2.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/asm/images/SoftwareComponents.png b/content/learning-paths/embedded-and-microcontrollers/asm/images/SoftwareComponents.png deleted file mode 100644 index 138298cdb7..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/asm/images/SoftwareComponents.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/asm/images/TargetOptions.png b/content/learning-paths/embedded-and-microcontrollers/asm/images/TargetOptions.png deleted file mode 100644 index 1780023801..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/asm/images/TargetOptions.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/asm/images/TargetOptions2.png b/content/learning-paths/embedded-and-microcontrollers/asm/images/TargetOptions2.png deleted file mode 100644 index 58198dc09e..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/asm/images/TargetOptions2.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/asm/images/TargetOptions3.png b/content/learning-paths/embedded-and-microcontrollers/asm/images/TargetOptions3.png deleted file mode 100644 index fa0a8d0474..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/asm/images/TargetOptions3.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/asm/images/Build.png b/content/learning-paths/embedded-and-microcontrollers/asm/images/build.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/asm/images/Build.png rename to content/learning-paths/embedded-and-microcontrollers/asm/images/build.png diff --git a/content/learning-paths/embedded-and-microcontrollers/asm/images/Debug.png b/content/learning-paths/embedded-and-microcontrollers/asm/images/debug.png similarity index 100% rename from content/learning-paths/embedded-and-microcontrollers/asm/images/Debug.png rename to content/learning-paths/embedded-and-microcontrollers/asm/images/debug.png diff --git a/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/Figure1.png b/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/Figure1.png deleted file mode 100644 index 45715e64b9..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/Figure1.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/Figure2.png b/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/Figure2.png deleted file mode 100644 index 01931b8594..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/Figure2.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/Figure4.png b/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/Figure4.png deleted file mode 100644 index 8c22788772..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/Figure4.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/Figure6.png b/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/Figure6.png deleted file mode 100644 index 1f1c6be737..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/Figure6.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/Figure7.png b/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/Figure7.png deleted file mode 100644 index d48dd2bf4e..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/Figure7.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/end-to-end_workflow.md b/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/end-to-end_workflow.md index c4ad31b306..3104f754a8 100644 --- a/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/end-to-end_workflow.md +++ b/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/end-to-end_workflow.md @@ -80,7 +80,7 @@ Training the model usually takes a lot of time. In step 2, an already trained En By default, the script uses the image shown below (QBHOUSE) as an example to verify the inference results on the Corstone-300 FVP with Arm Cortex-M55. -![QBHOUSE#center](./Figure4.png) +![QBHOUSE#center](./figure4.png) Make the script executable with `chmod`. diff --git a/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/figure5.webp b/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/figure5.webp deleted file mode 100644 index 92eb070e1e..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/figure5.webp and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/figure6.png b/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/figure6.png deleted file mode 100644 index 1f1c6be737..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/figure6.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/figure7.png b/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/figure7.png deleted file mode 100644 index d48dd2bf4e..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/figure7.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/overview.md b/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/overview.md index 3efb086329..6abe12100b 100644 --- a/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/overview.md +++ b/content/learning-paths/embedded-and-microcontrollers/avh_ppocr/overview.md @@ -16,7 +16,7 @@ OCR has been widely used in many industry scenarios such as ticket information e Text recognition is a sub-task of OCR. It's the step after text detection in OCR's two-stage algorithm which converts image information into text information. -![Example of English text recognition #center](./Figure1.png "Figure 1. Example of English text recognition (Image source: https://iapr.org/archives/icdar2015/index.html)") +![Example of English text recognition #center](./figure1.png "Figure 1. Example of English text recognition (Image source: https://iapr.org/archives/icdar2015/index.html)") In this Learning Path, you will learn how to apply deep learning (DL) to the OCR text recognition task and setup a development flow from model training to application deployment. @@ -39,7 +39,7 @@ As seen in Figure 2, the overall pipeline of PP-OCRv3 is similar to PP-OCRv2 wit For example, the text recognition model introduces [SVTR](https://arxiv.org/abs/2205.00159) (Scene Text Recognition with a Single Visual Model) based on PP-OCRv2. The model also uses [GTC](https://arxiv.org/pdf/2002.01276.pdf) (Guided Training of CTC) to guide training and model distillation. For more details, please refer to this PP-OCRv3 [technical report](https://arxiv.org/abs/2206.03001v2). -![PP-OCRv3 pipeline diagram #center](./Figure2.png "Figure 2. PP-OCRv3 pipeline diagram (Image source: https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/doc/doc_en/PP-OCRv3_introduction_en.md)") +![PP-OCRv3 pipeline diagram #center](./figure2.png "Figure 2. PP-OCRv3 pipeline diagram (Image source: https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/doc/doc_en/PP-OCRv3_introduction_en.md)") In the next section, you will deploy a trained PP-OCR text recognition model on the Arm Corstone-300 FVP. diff --git a/content/learning-paths/embedded-and-microcontrollers/edge/images/3b.webp b/content/learning-paths/embedded-and-microcontrollers/edge/images/3b.webp deleted file mode 100644 index 5f1554f555..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/edge/images/3b.webp and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/introduction-to-tinyml-on-arm/Connect.png b/content/learning-paths/embedded-and-microcontrollers/introduction-to-tinyml-on-arm/Connect.png deleted file mode 100644 index 6af713b403..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/introduction-to-tinyml-on-arm/Connect.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/introduction-to-tinyml-on-arm/Overview.png b/content/learning-paths/embedded-and-microcontrollers/introduction-to-tinyml-on-arm/Overview.png deleted file mode 100644 index cbcd944107..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/introduction-to-tinyml-on-arm/Overview.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/1-overview.md b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/1-overview.md new file mode 100644 index 0000000000..d03d9757ad --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/1-overview.md @@ -0,0 +1,41 @@ +--- +title: Overview +weight: 2 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Hardware Overview - NXP's FRDM i.MX 93 Board + +Selecting the best hardware for machine learning (ML) models depends on effective tools. You can visualize ML performance early in the development cycle by using NXP's [FRDM i.MX 93](https://www.nxp.com/design/design-center/development-boards-and-designs/frdm-i-mx-93-development-board:FRDM-IMX93) board. + +
+ + +*Unboxing NXP's FRDM i.MX 93 board* +
+ +![NXP FRDM i.MX 93 Board SoC Highlighted alt-text#center](./nxp-frdm-imx93-board-soc-highlighted.png "Arm Ethos-U65 NPU location") + +### NXP's FRDM i.MX 93 Processor Decoded + +![i.MX 93 Processor SoC alt-text#center](./imx93-application-processor-soc.png "NXP's FRDM i.MX 93 processor") + +**NXP's Processor Labeling Convention:** +|Line|Meaning| +|----|-------| +|MIMX9352|• MI – Microcontroller IC
• MX93 – i.MX 93 family
• 52 – Variant:
• Dual-core Arm Cortex-A55
• Single Cortex-M33
• Includes **Ethos-U65 NPU**| +|CVVXMAB|• C - Commercial temperature grade (0°C to 95°C)
• VVX - Indicates package type and pinout (BGA, pitch, etc.)
• MAB - Specific configuration (e.g., NPU present, security level, memory interfaces) +| +|1P87F|• Silicon mask set identifier| +|SBBM2410E|• NXP traceability code| + +## Benefits and applications + +NPUs, like Arm's [Ethos-U65](https://www.arm.com/products/silicon-ip-cpu/ethos/ethos-u65) NPU are available on physical devices specifically made for developers. Development boards like NXP's [FRDM i.MX 93](https://www.nxp.com/design/design-center/development-boards-and-designs/frdm-i-mx-93-development-board:FRDM-IMX93) also connect to displays via a HDMI cable. Additionally the board accepts video inputs. This is useful for for ML performance visualization due to: +- visual confirmation that your ML model is running on the physical device, +- image and video inputs for computer vision models running on the device, +- clearly indicated instruction counts, +- confirmation of total execution time and +- visually appealing output for prototypes and demos. \ No newline at end of file diff --git a/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/2-boot-nxp.md b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/2-boot-nxp.md new file mode 100644 index 0000000000..8765903b69 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/2-boot-nxp.md @@ -0,0 +1,103 @@ +--- +# User change +title: "Boot the NXP FRDM i.MX 93 Board" + +weight: 3 + +# Do not modify these elements +layout: "learningpathall" +--- + +In this section, you will prepare the NXP [FRDM i.MX 93](https://www.nxp.com/design/design-center/development-boards-and-designs/frdm-i-mx-93-development-board:FRDM-IMX93) board for ML development. + +## Unbox the NXP Board + +Follow NXP's getting started instructions: [Getting Started with FRDM-IMX93](https://www.nxp.com/document/guide/getting-started-with-frdm-imx93:GS-FRDM-IMX93): +* Stop when you complete section "1.6 Connect Power Supply" + +## Connect to the NXP Board + +Prior to logging in to the NXP board, you need to configure `picocom`. This allows you to connect to the board using a USB cable. + +{{% notice macOS %}} + +1. Install the Silicon Labs driver: + + https://www.silabs.com/developer-tools/usb-to-uart-bridge-vcp-drivers?tab=downloads + +2. Install [picocom](https://github.com/npat-efault/picocom): + ```bash + brew install picocom + ``` + +3. Establish a USB-to-UART (serial) connection: + - Connect the board's "DEBUG" USB-C connector to your Mac + - Find the NXP board's USB connections in your computer's terminal: + ```bash { output_lines = "2-7" } + ls /dev/tty.* + # output lines + ... + /dev/tty.debug-console + /dev/tty.usbmodem56D70442811 + /dev/tty.usbmodem56D70442813 + ... + ``` + + - Connect to the NXP board: + ```bash { output_lines = "2-5" } + sudo picocom -b 115200 /dev/tty.usbmodem56D70442811 + # output lines + picocom v3.1 + ... + Terminal ready + ``` + +4. Go straight to step 2, in the below instructions + +{{% /notice %}} + +1. Establish a USB-to-UART (serial) connection: + - Connect the board's "DEBUG" USB-C connector to your Linux machine + - Find the NXP board's USB connections in your computer's terminal: + ```bash { output_lines = "2-3" } + ls /dev/ttyUSB* /dev/ttyACM* 2>/dev/null + # output lines + /dev/ttyACM0 /dev/ttyACM1 + ``` + + - Connect to the NXP board: + ```bash { output_lines = "2-5" } + sudo picocom -b 115200 /dev/ttyACM0 + # output lines + picocom v3.1 + ... + Terminal ready + ``` +2. Log in to Linux on the NXP board: + - Connect the board's "POWER" USB-C connector to your laptop + - At this point you should see one red and one white light on the board + - Next you should see scrolling text in your `picocom` window, as the NXP board boots + - The last line should say `login:` + ```bash { output_lines = "1-9" } + # output lines + ... + [ OK ] Reached target Graphical Interface. + Starting Record Runlevel Change in UTMP... + [ OK ] Finished Record Runlevel Change in UTMP. + + NXP i.MX Release Distro 6.6-scarthgap imx93frdm ttyLP0 + + imx93frdm login: + ``` +3. Type `root` to log in as root. There is no password + +### Troubleshooting +* Restart the NXP board, to get to the `login:` prompt: + * Hold the NXP board's power button for 2-seconds, until the lights turn off + * Hold the NXP board's power button again for 2-seconds, until the lights turn on + +## [Optional] Run the Built-In NXP Demos +* Connect the NXP board to a monitor via HDMI +* Connect a mouse to the NXP board's USB-A port + +![NXP board built-in ML demos alt-text#center](./nxp-board-built-in-ml-demos.png "NXP board built-in ML demos") diff --git a/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/3-create-super-user.md b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/3-create-super-user.md new file mode 100644 index 0000000000..927a029927 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/3-create-super-user.md @@ -0,0 +1,36 @@ +--- +# User change +title: "Create a Linux Super User" + +weight: 4 + +# Do not modify these elements +layout: "learningpathall" +--- + +On the NXP board, create a non-root super user (if you do not already have one): + +1. While [logged in as root]( {{< relref "2-boot-nxp.md" >}} ): + + * Enable super user privileges: + ```bash + sudo visudo + ``` + * In the vi editor that opens up, uncomment the below line: + ```bash { output_lines = "1" } + %wheel ALL=(ALL:ALL) ALL # uncomment this line + ``` + +2. Add a super user: + ```bash + sudo adduser testuser + sudo usermod -aG wheel testuser + ``` + +3. While still logged in as root, confirm successful super user creation: + ```bash + su - testuser + sudo whoami # should return "root" + ``` + +4. Log out of the NXP board and log back in to Linux as the super user \ No newline at end of file diff --git a/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/4-enable-wifi.md b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/4-enable-wifi.md new file mode 100644 index 0000000000..12ce8e216d --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/4-enable-wifi.md @@ -0,0 +1,88 @@ +--- +# User change +title: "Enable WiFi" + +weight: 5 # 1 is first, 2 is second, etc. + +# Do not modify these elements +layout: "learningpathall" +--- + +{{% notice Note %}} + +* WiFi network connectivity **does not persist** on NXP board reboot +* It **does persist** on logging out and then logging back in as the same Linux user + +{{% /notice %}} + +1. [Log in to Linux]( {{< relref "2-boot-nxp.md" >}} ) on the board, as a [super user]( {{< relref "3-create-super-user" >}} ) + +2. Run the below terminal commands: + ```bash + sudo /usr/sbin/modprobe moal mod_para=nxp/wifi_mod_para.conf + sudo connmanctl + ``` + +3. The prompt will change to `connmanctl>`, where you will enter the following commands: + + ```bash + enable wifi + scan wifi + services + ``` + +4. Your available WiFi networks will be listed in the following form: + + ```bash { output_lines = "1-3" } + wifi_0123456789ab_cdef0123456789_managed_psk + wifi_abcdef012345_6789abcdef0123_managed_psk + wifi_fedcba987654_3210fedcba9876_managed_psk + ``` + + {{% notice Note %}} + + Duplicate SSIDs may appear, so you will have to experiment with the different `wifi_..._managed_psk` names, when you try to connect in the next step + + {{% /notice %}} + +5. Still within the `connmanctl>` prompt, enter the following commands: + + ```bash + agent on + connect wifi_0123456789ab_cdef0123456789_managed_psk # Your wifi_..._managed_ps name will be different + Agent RequestInput wifi_0123456789ab_cdef0123456789_managed_psk + Passphrase = [ Type=psk, Requirement=mandatory ] + Passphrase? # Enter your WiFi password + connmanctl> quit + ``` + +6. Assuming your WiFi network is connected to the Internet, test connectivity: + + ```bash + curl -I http://www.example.com + ``` + + If WiFi is configured correctly, you will see the example.com web page load: + + ```bash { output_lines = "1-2" } + HTTP/1.1 200 OK + ... + ``` + +7. [optional] If your WiFi network is not connected to the internet, test connectivity this way: + + ```bash + ifconfig | grep RUNNING -A 1 + ``` + + If WiFi is configured correctly, you will see a list of `RUNNING` network adapters: + * one for `127.0.0.1` (`localhost`) and + * a second for the NXP board's assigned IP address on the WiFi network + * Example output, where `192.168.1.89` is the NXP board's successfully assigned IP address: + ```bash { output_lines = "1-5" } + lo: flags=73 mtu 65536 + inet 127.0.0.1 netmask 255.0.0.0 + -- + mlan0: flags=-28605 mtu 1500 + inet 192.168.1.89 netmask 255.255.255.0 broadcast 192.168.1.255 + ``` \ No newline at end of file diff --git a/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/5-transfer-files-wifi.md b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/5-transfer-files-wifi.md new file mode 100644 index 0000000000..8eaa0c09ed --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/5-transfer-files-wifi.md @@ -0,0 +1,30 @@ +--- +# User change +title: "Transfer Files Over WiFi" + +weight: 6 # 1 is first, 2 is second, etc. + +# Do not modify these elements +layout: "learningpathall" +--- + +1. [Log in to Linux]( {{< relref "2-boot-nxp.md" >}} ) on the board, as a [super user]( {{< relref "3-create-super-user" >}} ) + +2. [Enable Wifi]( {{< relref "4-enable-wifi.md" >}} ) on the NXP board + +3. Note down the NXP board's IP address on your WiFi network: + ```bash + ifconfig | grep RUNNING -A 1 + ``` + +4. Open a terminal window on the machine with the source file + +5. Navigate to the source file directory and copy the file to the NXP board's destination directory: + ```bash + # On your machine, in the source file directory + scp @:/home/nxp_user/path/to/destination/directory/ + ``` + Example: + ```bash { output_lines = "1" } + scp install.sh testuser@192.168.1.1:/home/testuser/apps/test_app/ + ``` \ No newline at end of file diff --git a/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/6-tranfer-files-usb.md b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/6-tranfer-files-usb.md new file mode 100644 index 0000000000..89ee721ff8 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/6-tranfer-files-usb.md @@ -0,0 +1,32 @@ +--- +# User change +title: "Transfer Files Over USB" + +weight: 7 # 1 is first, 2 is second, etc. + +# Do not modify these elements +layout: "learningpathall" +--- + +1. [Log in to Linux]( {{< relref "2-boot-nxp.md" >}} ) on the board, as a [super user]( {{< relref "3-create-super-user" >}} ) + +2. On your machine with the source file, copy the source file to a USB-A thumb drive: + +3. Insert the thumb drive into the NXP board's USB-A port + +4. Mount the thumb drive and then copy the files to the board: + ```bash { output_lines = "1" } + # Execute these commands on the board + mount /dev/sda1 /mnt + cp /mnt/ /path/to/destination/directory/ + ``` + + Example: + ```bash { output_lines = "1" } + cp /mnt/install.sh ./apps/test_app/ + ``` + +5. [optional] Unmount the thumbdrive and then remove it from the NXP board + ```bash + umount /mnt + ``` \ No newline at end of file diff --git a/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/7-enable-persistent-wifi.md b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/7-enable-persistent-wifi.md new file mode 100644 index 0000000000..d8b2f7a6cf --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/7-enable-persistent-wifi.md @@ -0,0 +1,148 @@ +--- +# User change +title: "(Optional) Enable Persistent WiFi" + +weight: 8 # 1 is first, 2 is second, etc. + +# Do not modify these elements +layout: "learningpathall" +--- + +On this page you will configure the NXP board to connect to a specific WiFi network on boot. + +1. [Log in to Linux]( {{< relref "2-boot-nxp.md" >}} ) on the board, as `root` + +2. Create a `wpa_supplicant.conf`: + ```bash + touch /etc/wpa_supplicant.conf + nano /etc/wpa_supplicant.conf + ``` + Enter your WiFi credentials into the `wpa_supplicant.conf` file: + ```bash + ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev + update_config=1 + + network={ + ssid="YOUR_SSID" + psk="YOUR_PASSWORD" + key_mgmt=WPA-PSK + } + ``` + +3. Test the `wpa_supplicant.conf` file: + ```bash + modprobe moal mod_para=nxp/wifi_mod_para.conf + ifconfig mlan0 up + wpa_supplicant -B -i mlan0 -c /etc/wpa_supplicant.conf + udhcpc -i mlan0 + ``` + * mlan0 is the WiFi interface on i.MX93 + * If this connects to WiFi, we’re ready for automation + +4. Configure DNS server IP addresses, so that the NXP board can resolve Internet addresses: + ```bash + touch /usr/share/udhcpc/default.script + nano /usr/share/udhcpc/default.script + ``` + and add in the following `udhcpc` script: + ```bash + #!/bin/sh + # udhcpc script + case "$1" in + deconfig) + ip addr flush dev $interface + ;; + bound|renew) + ip addr add $ip/$subnet dev $interface + ip route add default via $router + echo "nameserver 8.8.8.8" > /etc/resolv.conf + echo "nameserver 1.1.1.1" >> /etc/resolv.conf + ;; + esac + ``` + Make the `default.script` executable: + ```bash + chmod +x /usr/share/udhcpc/default.script + ``` + +5. Create a `nxp-wifi-setup.sh` script: + ```bash + touch /usr/bin/nxp-wifi-setup.sh + nano /usr/bin/nxp-wifi-setup.sh + ``` + and add in the following lines: + ```bash + #!/bin/sh + # Load WiFi driver + /usr/sbin/modprobe moal mod_para=nxp/wifi_mod_para.conf + + # Bring interface up + /usr/bin/ifconfig mlan0 up + + # Connect to WiFi + /usr/sbin/wpa_supplicant -B -i mlan0 -c /etc/wpa_supplicant.conf + + # Obtain DHCP IP + DNS + /usr/sbin/udhcpc -i mlan0 -s /usr/share/udhcpc/default.script + ``` + Make the `nxp-wifi-setup.sh` executable: + ```bash + chmod +x /usr/bin/nxp-wifi-setup.sh + ``` + +6. Create a `nxp-wifi-setup.service`: + ```bash + touch /etc/systemd/system/nxp-wifi-setup.service + nano /etc/systemd/system/nxp-wifi-setup.service + ``` + Enter the following systemd commands into the `nxp-wifi-setup.service` file: + ```bash + [Unit] + Description=WiFi Setup for NXP FRDM i.MX93 + After=network.target + + [Service] + Type=oneshot + ExecStart=/usr/bin/nxp-wifi-setup.sh + RemainAfterExit=yes + + [Install] + WantedBy=multi-user.target + ``` + +7. Create a `wpa_supplicant.service`: + ```bash + touch /etc/systemd/system/wpa_supplicant.service + nano /etc/systemd/system/wpa_supplicant.service + ``` + Enter the following systemd commands into the `wpa_supplicant.service` file: + ```bash + [Unit] + Description=WPA Supplicant daemon + After=network.target + + [Service] + Type=simple + ExecStart=/usr/sbin/wpa_supplicant -i mlan0 -c /etc/wpa_supplicant.conf + Restart=always + + [Install] + WantedBy=multi-user.target + ``` + +8. Enable and Start the `nxp-wifi-setup.service`: + ```bash + systemctl daemon-reload + systemctl enable nxp-wifi-setup.service wpa_supplicant.service + systemctl start nxp-wifi-setup.service wpa_supplicant.service + ``` + +10. Check status: + ```bash + systemctl status nxp-wifi-setup.service + systemctl status wpa_supplicant.service + ``` + and confirm Internet connectivity: + ```bash + curl -I http://www.example.com + ``` \ No newline at end of file diff --git a/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/_index.md b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/_index.md new file mode 100644 index 0000000000..9040c9ca8e --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/_index.md @@ -0,0 +1,64 @@ +--- +title: Using Linux on the NXP FRDM IMX93 Board + +draft: true +cascade: + draft: true + +minutes_to_complete: 120 + +who_is_this_for: This is an advanced topic for embedded device developers and machine learning engineers, who want need a professional-grade Arm embedded device development platform. + +learning_objectives: + - Identify suitable physical Arm-based devices for TinyML applications. + - Configure physical embedded devices. + - Enable hardware modules on embedded devices. + - Deploy a apps to NXP's FRDM i.MX 93 board. + - Using Linux on an embedded device's Arm Cortex-A processor(s). + - Deploying machine learning models to Arm Cortex-M and Ethos-U processors will be covered in a later learning path. + +prerequisites: + - Purchase of a NXP [FRDM i.MX 93](https://www.nxp.com/design/design-center/development-boards-and-designs/frdm-i-mx-93-development-board:FRDM-IMX93) board. + - A computer running Linux or macOS. + +author: Waheed Brown + +### Tags +skilllevels: Introductory +subjects: ML +armips: + - Cortex-A + +operatingsystems: + - Linux + - macOS + +tools_software_languages: + - Baremetal + - Python + - Bash + - systemd + +further_reading: + - resource: + title: TinyML Brings AI to Smallest Arm Devices + link: https://newsroom.arm.com/blog/tinyml + type: blog + - resource: + title: Arm Machine Learning Resources + link: https://www.arm.com/developer-hub/embedded-and-microcontrollers/ml-solutions/getting-started + type: documentation + - resource: + title: Arm Developers Guide for Cortex-M Processors and Ethos-U NPU + link: https://developer.arm.com/documentation/109267/0101 + type: documentation + + + + +### FIXED, DO NOT MODIFY +# ================================================================================ +weight: 1 # _index.md always has weight of 1 to order correctly +layout: "learningpathall" # All files under learning paths have this same wrapper +learning_path_main_page: "yes" # This should be surfaced when looking for related content. Only set for _index.md of learning path content. +--- diff --git a/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/_next-steps.md b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/_next-steps.md new file mode 100644 index 0000000000..c3db0de5a2 --- /dev/null +++ b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/_next-steps.md @@ -0,0 +1,8 @@ +--- +# ================================================================================ +# FIXED, DO NOT MODIFY THIS FILE +# ================================================================================ +weight: 21 # Set to always be larger than the content in this path to be at the end of the navigation. +title: "Next Steps" # Always the same, html page title. +layout: "learningpathall" # All files under learning paths have this same wrapper for Hugo processing. +--- diff --git a/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/imx93-application-processor-soc.png b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/imx93-application-processor-soc.png new file mode 100644 index 0000000000..838d47f6d5 Binary files /dev/null and b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/imx93-application-processor-soc.png differ diff --git a/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/nxp-board-built-in-ml-demos.png b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/nxp-board-built-in-ml-demos.png new file mode 100644 index 0000000000..e50d656b13 Binary files /dev/null and b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/nxp-board-built-in-ml-demos.png differ diff --git a/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/nxp-frdm-imx93-board-soc-highlighted.png b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/nxp-frdm-imx93-board-soc-highlighted.png new file mode 100644 index 0000000000..b50ace3a21 Binary files /dev/null and b/content/learning-paths/embedded-and-microcontrollers/linux-nxp-board/nxp-frdm-imx93-board-soc-highlighted.png differ diff --git a/content/learning-paths/embedded-and-microcontrollers/linux-on-fvp/Cortex-a53_MPIDR_EL1.png b/content/learning-paths/embedded-and-microcontrollers/linux-on-fvp/Cortex-a53_MPIDR_EL1.png deleted file mode 100644 index 1f204e5b88..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/linux-on-fvp/Cortex-a53_MPIDR_EL1.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/linux-on-fvp/armds_ide.webp b/content/learning-paths/embedded-and-microcontrollers/linux-on-fvp/armds_ide.webp deleted file mode 100644 index 48e4f5652a..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/linux-on-fvp/armds_ide.webp and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/linux-on-fvp/cortex-a53_mpidr_el1.png b/content/learning-paths/embedded-and-microcontrollers/linux-on-fvp/cortex-a53_mpidr_el1.png deleted file mode 100644 index 1f204e5b88..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/linux-on-fvp/cortex-a53_mpidr_el1.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/linux-on-fvp/cortex-a55_MPIDR_EL1.png b/content/learning-paths/embedded-and-microcontrollers/linux-on-fvp/cortex-a55_MPIDR_EL1.png deleted file mode 100644 index eea3e19a05..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/linux-on-fvp/cortex-a55_MPIDR_EL1.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/linux-on-fvp/cortex-a55_mpidr_el1.png b/content/learning-paths/embedded-and-microcontrollers/linux-on-fvp/cortex-a55_mpidr_el1.png deleted file mode 100644 index eea3e19a05..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/linux-on-fvp/cortex-a55_mpidr_el1.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/linux-on-fvp/debug_config.webp b/content/learning-paths/embedded-and-microcontrollers/linux-on-fvp/debug_config.webp deleted file mode 100644 index 8e5c97a234..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/linux-on-fvp/debug_config.webp and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/mlek/term.png b/content/learning-paths/embedded-and-microcontrollers/mlek/term.png deleted file mode 100644 index 7ebc52be3e..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/mlek/term.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/mlek/vis.png b/content/learning-paths/embedded-and-microcontrollers/mlek/vis.png deleted file mode 100644 index 4658424ac0..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/mlek/vis.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/CMSIS-RTX_missing.png b/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/CMSIS-RTX_missing.png deleted file mode 100644 index 23a31f7c89..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/CMSIS-RTX_missing.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/CoreDebug_uvision.png b/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/CoreDebug_uvision.png deleted file mode 100644 index 1706587529..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/CoreDebug_uvision.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/Device_missing.png b/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/Device_missing.png deleted file mode 100644 index 9a9b1ee390..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/Device_missing.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/EventRecorder_migration.png b/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/EventRecorder_migration.png deleted file mode 100644 index fb4f06a639..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/EventRecorder_migration.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/IO_migration.png b/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/IO_migration.png deleted file mode 100644 index 2b5aa12196..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/IO_migration.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/OS_Tick_missing.png b/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/OS_Tick_missing.png deleted file mode 100644 index 5f262287ad..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/OS_Tick_missing.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/troubleshooting.md b/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/troubleshooting.md index 30f740436a..42a5277aa4 100644 --- a/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/troubleshooting.md +++ b/content/learning-paths/embedded-and-microcontrollers/project-migration-cmsis-v6/troubleshooting.md @@ -24,7 +24,7 @@ This section provides an overview of the most common errors you might encounter The following error occurs when you try to migrate a project to CMSIS v6 but you have not installed the Cortex_DFP pack: -![Requested device not found for target](./Device_missing.png) +![Requested device not found for target](./device_missing.png) {{% notice Resolution %}} Install [ARM.Cortex_DFP.1.0.0.pack](https://www.keil.arm.com/packs/cortex_dfp-arm) or above. @@ -34,7 +34,7 @@ Install [ARM.Cortex_DFP.1.0.0.pack](https://www.keil.arm.com/packs/cortex_dfp-ar The following error occurs when you try to migrate a Keil RTX-based project but you have not installed the CMSIS-RTX pack: -![Keil RTX5 component not available for target](./CMSIS-RTX_missing.png) +![Keil RTX5 component not available for target](./cmsis-rtx_missing.png) {{% notice Resolution %}} Install [ARM.CMSIS-RTX.5.8.0.pack](https://www.keil.arm.com/packs/cmsis-rtx-arm) or above. @@ -46,7 +46,7 @@ Install [ARM.CMSIS-RTX.5.8.0.pack](https://www.keil.arm.com/packs/cmsis-rtx-arm) The following warning is shown in the Validation Output window when you try to migrate a Keil RTX-based project: -![Additional software components required](./OS_Tick_missing.png) +![Additional software components required](./os_tick_missing.png) {{% notice Resolution %}} Use the **Resolve** button to select the missing component automatically. @@ -98,12 +98,12 @@ As the components from the Keil.ARM_Compiler pack do not have 1:1 replacements, Deselect the `Compiler:Event Recorder` component and select the `CMSIS-View:Event Recorder` component: - ![Event Recorder migration](./EventRecorder_migration.png) + ![Event Recorder migration](./eventrecorder_migration.png) 2. Compiler:I/O migration Deselect any component of `Compiler:I/O` and select the corresponding `CMSIS-Compiler` component: - ![I/O migration](./IO_migration.png) + ![I/O migration](./io_migration.png) This table helps you to identify the correct components: @@ -245,7 +245,7 @@ For more information on how to configure your *.cproject.yml file, please refer 1. Go to **Project - Options for Target** and switch to the **C/C++ (AC6)** tab. Add `-include \cmsis_5_to_6_patch.h` to the **Misc Controls**: - ![Add patch file](./CoreDebug_uvision.png) + ![Add patch file](./coredebug_uvision.png) 1. Rebuild the project. {{% /notice %}} diff --git a/content/learning-paths/embedded-and-microcontrollers/raspberry-pi-smart-home/4-smart-home-assistant.md b/content/learning-paths/embedded-and-microcontrollers/raspberry-pi-smart-home/4-smart-home-assistant.md index 3ec24fd82c..45c88a4d11 100644 --- a/content/learning-paths/embedded-and-microcontrollers/raspberry-pi-smart-home/4-smart-home-assistant.md +++ b/content/learning-paths/embedded-and-microcontrollers/raspberry-pi-smart-home/4-smart-home-assistant.md @@ -78,7 +78,7 @@ Try asking the assistant to `turn on living room light`. If you've connected add Open your browser and navigate to `http://0.0.0.0:8000`, or as printed in the terminal output. -![Web interface of the smart home assistant showing device control through LLM commands alt-text#center](UI3.png "Interacting with the LLM through the web interface") +![Web interface of the smart home assistant showing device control through LLM commands alt-text#center](ui3.png "Interacting with the LLM through the web interface") ## Command line interface diff --git a/content/learning-paths/embedded-and-microcontrollers/raspberry-pi-smart-home/UI3.png b/content/learning-paths/embedded-and-microcontrollers/raspberry-pi-smart-home/UI3.png deleted file mode 100644 index bc4dfecd52..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/raspberry-pi-smart-home/UI3.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/uv_debug/2_basics.md b/content/learning-paths/embedded-and-microcontrollers/uv_debug/2_basics.md index fa81e39f5f..6c69f237d8 100644 --- a/content/learning-paths/embedded-and-microcontrollers/uv_debug/2_basics.md +++ b/content/learning-paths/embedded-and-microcontrollers/uv_debug/2_basics.md @@ -104,7 +104,7 @@ There is a global variable `g_msTicks` located in `Blinky.c` near line 11 that y 1. Click on `` twice and enter: `SystemCoreClock` 2. Right click on the `Value` and deselect **Hexadecimal Display**. 32 MHz will be displayed: -![SystemCoreClock in Watch 1 Window](./SystemCoreClockWatch.png) +![SystemCoreClock in Watch 1 Window](./systemcoreclockwatch.png) {{% notice Note %}} You do not need to stop the program execution to enter variables, raw addresses or structures in a **Watch** or **Memory** window. @@ -140,7 +140,7 @@ The example application uses the Arm Cortex-M system tick timer. 1. ![System Viewer](./b_uv4_systemviewer.png) Go to **Peripherals - Core Peripherals** and then select **System Tick Timer S (SysTick)**. 2. The **SysTick Timer** window opens: -![SysTick Timer Window](./SysTickTimerWindow.png) +![SysTick Timer Window](./systicktimerwindow.png) 1. ![Run](./b_uv4_run.png) **Run (F5)** the application. 6. While the program is running, type `0x10000` in the `SysTick -> LOAD` register and click in another register or press Enter. 7. The program execution will speed up. This is the power of Arm CoreSight debugging. diff --git a/content/learning-paths/embedded-and-microcontrollers/uv_debug/SysTickTimerWindow.png b/content/learning-paths/embedded-and-microcontrollers/uv_debug/SysTickTimerWindow.png deleted file mode 100644 index 1a38093d8d..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/uv_debug/SysTickTimerWindow.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/uv_debug/SystemCoreClockWatch.png b/content/learning-paths/embedded-and-microcontrollers/uv_debug/SystemCoreClockWatch.png deleted file mode 100644 index ebf593c735..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/uv_debug/SystemCoreClockWatch.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/uv_debug/b_uv4_memory_window.png b/content/learning-paths/embedded-and-microcontrollers/uv_debug/b_uv4_memory_window.png deleted file mode 100644 index b00af3bac8..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/uv_debug/b_uv4_memory_window.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/uv_debug/b_uv4_rebuild_all.png b/content/learning-paths/embedded-and-microcontrollers/uv_debug/b_uv4_rebuild_all.png deleted file mode 100644 index 68aa04275a..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/uv_debug/b_uv4_rebuild_all.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/uv_debug/blinky_c_proj_win.png b/content/learning-paths/embedded-and-microcontrollers/uv_debug/blinky_c_proj_win.png deleted file mode 100644 index 4f12e62851..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/uv_debug/blinky_c_proj_win.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/uvprojx-conversion/convert_project.png b/content/learning-paths/embedded-and-microcontrollers/uvprojx-conversion/convert_project.png deleted file mode 100644 index a4e0910f32..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/uvprojx-conversion/convert_project.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/yolo-on-himax/pose_estimation.jpg b/content/learning-paths/embedded-and-microcontrollers/yolo-on-himax/pose_estimation.jpg deleted file mode 100644 index 09e24fcbcf..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/yolo-on-himax/pose_estimation.jpg and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/zephyr_vsworkbench/images/build_application.png b/content/learning-paths/embedded-and-microcontrollers/zephyr_vsworkbench/images/build_application.png deleted file mode 100644 index 4d83aa2f3f..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/zephyr_vsworkbench/images/build_application.png and /dev/null differ diff --git a/content/learning-paths/embedded-and-microcontrollers/zephyr_vsworkbench/images/install_debug_tools.png b/content/learning-paths/embedded-and-microcontrollers/zephyr_vsworkbench/images/install_debug_tools.png deleted file mode 100644 index 63b23b78e4..0000000000 Binary files a/content/learning-paths/embedded-and-microcontrollers/zephyr_vsworkbench/images/install_debug_tools.png and /dev/null differ diff --git a/content/learning-paths/iot/azure-iot/image.webp b/content/learning-paths/iot/azure-iot/image.webp deleted file mode 100644 index 6bcb5b038c..0000000000 Binary files a/content/learning-paths/iot/azure-iot/image.webp and /dev/null differ diff --git a/content/learning-paths/laptops-and-desktops/_index.md b/content/learning-paths/laptops-and-desktops/_index.md index 6e75fe728b..8c972241f1 100644 --- a/content/learning-paths/laptops-and-desktops/_index.md +++ b/content/learning-paths/laptops-and-desktops/_index.md @@ -29,7 +29,7 @@ tools_software_languages_filter: - Arm64EC: 1 - Assembly: 1 - Bash: 3 -- C: 10 +- C: 11 - C#: 6 - CCA: 1 - Clang: 13 @@ -37,7 +37,7 @@ tools_software_languages_filter: - CPP: 12 - CSS: 1 - Daytona: 1 -- Docker: 5 +- Docker: 6 - FFmpeg: 1 - GCC: 12 - Git: 1 diff --git a/content/learning-paths/laptops-and-desktops/win_aws_iot/image.webp b/content/learning-paths/laptops-and-desktops/win_aws_iot/image.webp deleted file mode 100644 index fe5742b2e3..0000000000 Binary files a/content/learning-paths/laptops-and-desktops/win_aws_iot/image.webp and /dev/null differ diff --git a/content/learning-paths/laptops-and-desktops/windows_armpl/figures/vs_console_code.png b/content/learning-paths/laptops-and-desktops/windows_armpl/figures/vs_console_code.png deleted file mode 100755 index c3075df38d..0000000000 Binary files a/content/learning-paths/laptops-and-desktops/windows_armpl/figures/vs_console_code.png and /dev/null differ diff --git a/content/learning-paths/laptops-and-desktops/windows_armpl/vs_new_project1.png b/content/learning-paths/laptops-and-desktops/windows_armpl/vs_new_project1.png deleted file mode 100755 index e6cb43321c..0000000000 Binary files a/content/learning-paths/laptops-and-desktops/windows_armpl/vs_new_project1.png and /dev/null differ diff --git a/content/learning-paths/laptops-and-desktops/windows_cicd_github/images/ghrunner_1.png b/content/learning-paths/laptops-and-desktops/windows_cicd_github/images/ghrunner_1.png deleted file mode 100644 index 8aa65170a0..0000000000 Binary files a/content/learning-paths/laptops-and-desktops/windows_cicd_github/images/ghrunner_1.png and /dev/null differ diff --git a/content/learning-paths/laptops-and-desktops/windowsperf-vs-extension/sampling-settings.png b/content/learning-paths/laptops-and-desktops/windowsperf-vs-extension/sampling-settings.png deleted file mode 100644 index 68bcc1cf5c..0000000000 Binary files a/content/learning-paths/laptops-and-desktops/windowsperf-vs-extension/sampling-settings.png and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/_index.md b/content/learning-paths/mobile-graphics-and-gaming/_index.md index 9fec7623ba..c7f4d24740 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/_index.md +++ b/content/learning-paths/mobile-graphics-and-gaming/_index.md @@ -10,23 +10,23 @@ key_ip: maintopic: true operatingsystems_filter: - Android: 35 -- Linux: 35 -- macOS: 15 -- Windows: 14 +- Linux: 36 +- macOS: 16 +- Windows: 15 subjects_filter: - Gaming: 6 - Graphics: 6 -- ML: 17 +- ML: 18 - Performance and Architecture: 36 subtitle: Optimize Android apps and build faster games using cutting-edge Arm tech title: Mobile, Graphics, and Gaming tools_software_languages_filter: - 7-Zip: 1 - adb: 2 -- Android: 4 +- Android: 5 - Android NDK: 2 - Android SDK: 1 -- Android Studio: 12 +- Android Studio: 13 - Arm Development Studio: 1 - Arm Mobile Studio: 1 - Arm Performance Studio: 3 @@ -50,10 +50,10 @@ tools_software_languages_filter: - Google Test: 1 - Halide: 1 - Hugging Face: 6 -- Java: 6 +- Java: 7 - Jupyter Notebook: 1 - KleidiAI: 2 -- Kotlin: 8 +- Kotlin: 9 - LiteRT: 1 - llama.cpp: 1 - LLM: 1 @@ -62,11 +62,11 @@ tools_software_languages_filter: - MediaPipe: 2 - MTE: 2 - NEON: 1 -- ONNX Runtime: 1 -- ONNX runtime: 1 +- ONNX: 1 +- ONNX Runtime: 2 - OpenGL ES: 1 -- Python: 8 -- PyTorch: 2 +- Python: 9 +- PyTorch: 3 - QEMU: 1 - RenderDoc: 1 - RME: 1 @@ -74,6 +74,7 @@ tools_software_languages_filter: - Rust: 2 - SDDiskTool: 1 - SVE2: 1 +- TensorFlow: 1 - Trusted Firmware: 1 - Unity: 6 - Unreal Engine: 4 diff --git a/content/learning-paths/mobile-graphics-and-gaming/ams/fa.md b/content/learning-paths/mobile-graphics-and-gaming/ams/fa.md index 77d77a8f6e..5d83889000 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/ams/fa.md +++ b/content/learning-paths/mobile-graphics-and-gaming/ams/fa.md @@ -59,7 +59,7 @@ Explore each frame to evaluate how efficiently they were rendered on the device. 1. Look at the Render Graph to see how the frame was constructed. - ![Render Graph](images/FA_render_graph_1.1.gif) + ![Render Graph](images/fa_render_graph_1.1.gif) Evaluate the render graph to look for render passes or input or output attachments that aren’t used in the final output, and could be removed, saving processing power and bandwidth. diff --git a/content/learning-paths/mobile-graphics-and-gaming/ams/images/FA_render_graph_1.1.gif b/content/learning-paths/mobile-graphics-and-gaming/ams/images/FA_render_graph_1.1.gif deleted file mode 100644 index 9e74966d16..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/ams/images/FA_render_graph_1.1.gif and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/ams/images/fa_render_graph.png b/content/learning-paths/mobile-graphics-and-gaming/ams/images/fa_render_graph.png deleted file mode 100644 index 32fce118d1..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/ams/images/fa_render_graph.png and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/ams/images/ga_device_manager.png b/content/learning-paths/mobile-graphics-and-gaming/ams/images/ga_device_manager.png deleted file mode 100644 index 3e0616181e..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/ams/images/ga_device_manager.png and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/FA_Sphinx.png b/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/FA_Sphinx.png deleted file mode 100644 index de50e00fac..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/FA_Sphinx.png and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/FA_render_graph_1.1.gif b/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/FA_render_graph_1.1.gif deleted file mode 100644 index 9e74966d16..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/FA_render_graph_1.1.gif and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/FA_step_drawcalls.gif b/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/FA_step_drawcalls.gif deleted file mode 100644 index 4c8d438306..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/FA_step_drawcalls.gif and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/Pillars.gif b/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/Pillars.gif deleted file mode 100644 index 3090674db0..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/Pillars.gif and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/Render_graph_egypt_redundant_attachments.png b/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/Render_graph_egypt_redundant_attachments.png deleted file mode 100644 index e267572f04..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/Render_graph_egypt_redundant_attachments.png and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/analyze.md b/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/analyze.md index 100ca052ff..14b9196090 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/analyze.md +++ b/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/analyze.md @@ -15,7 +15,7 @@ When the analysis completes, you will see Frame Advisor's `Analysis` screen. 1. You can see all the render passes that make up the frame. Expand a render pass to see the draw calls within it. Step through the draw calls to see how the scene is built. - ![Stepping through draw calls alt-text#center](FA_step_drawcalls.gif "Figure 1. Stepping through draw calls in Frame Advisor") + ![Stepping through draw calls alt-text#center](fa_step_drawcalls.gif "Figure 1. Stepping through draw calls in Frame Advisor") Draw calls are expensive for the CPU to process, so it is important to reduce the number of them where possible. Look for draw calls that don’t render visible changes to the framebuffer. If you don’t see any change, draws could be outside of the frustum or behind other objects. Use software culling techniques to eliminate them. @@ -23,4 +23,4 @@ When the analysis completes, you will see Frame Advisor's `Analysis` screen. 1. Look for instances where many identical objects are being drawn individually, like these pillars. There could be an opportunity to reduce the number of draw calls by batching multiple objects into a single combined mesh or by using an instanced draw call. - ![Framebuffers view alt-text#center](Pillars.gif "Figure 1. Framebuffers view in Frame Advisor") + ![Framebuffers view alt-text#center](pillars.gif "Figure 1. Framebuffers view in Frame Advisor") diff --git a/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/analyze_geometry.md b/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/analyze_geometry.md index 87408a96c3..95217e96bf 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/analyze_geometry.md +++ b/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/analyze_geometry.md @@ -13,7 +13,7 @@ Use the Content Metrics view in Frame Advisor to find geometry-related problems 1. Right-click the draw call at the top of the list and choose `Navigate to call`. This complex object is now selected in the `Frame Hierarchy` view and you can see it in the `Framebuffers` view. This object is the Sphinx model and it is built using almost 23,000 primitives. This is a high number for a game object on mobile, so the first step is to see whether this model can be simplified. Fewer primitives reduces GPU processing cost and memory bandwidth. - ![The Sphinx model shown in the Framebuffers view alt-text#center](FA_Sphinx.png "Figure 2. The Sphinx model shown in the Framebuffers view") + ![The Sphinx model shown in the Framebuffers view alt-text#center](fa_sphinx.png "Figure 2. The Sphinx model shown in the Framebuffers view") In cases where the model cannot be simplified any further, there are other options to consider. diff --git a/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/analyze_render_graph.md b/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/analyze_render_graph.md index afc773038e..dec17523f6 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/analyze_render_graph.md +++ b/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/analyze_render_graph.md @@ -9,11 +9,11 @@ The render graph in Frame Advisor shows a visualization of the rendering operati Render passes flow from left to right. The render pass that outputs to the swapchain is the final render pass that outputs to the screen. -![The Render Graph view in Frame Advisor alt-text#center](FA_render_graph_1.1.gif "Figure 1. The Render Graph view") +![The Render Graph view in Frame Advisor alt-text#center](fa_render_graph_1.1.gif "Figure 1. The Render Graph view") 1. Here, we can see some output attachments that are not used in a future render pass. - ![Redundant output attachments alt-text#center](Render_graph_egypt_redundant_attachments.png "Figure 3. Redundant output attachments") + ![Redundant output attachments alt-text#center](render_graph_egypt_redundant_attachments.png "Figure 3. Redundant output attachments") You should clear or invalidate input and output attachments that are not used to avoid unnecessary memory accesses. If clear or invalidate calls are present within a render pass, they are shown in the `Frame Hierarchy` view. diff --git a/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/fa_show_descriptions.png b/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/fa_show_descriptions.png deleted file mode 100644 index 695daffd15..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/fa_show_descriptions.png and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/render_graph_egypt.webp b/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/render_graph_egypt.webp deleted file mode 100644 index c92c9aa72e..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/analyze_a_frame_with_frame_advisor/render_graph_egypt.webp and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/android_webgpu_dawn/3-integrate-dawn.md b/content/learning-paths/mobile-graphics-and-gaming/android_webgpu_dawn/3-integrate-dawn.md index 544058f2ac..dfd0a822ef 100755 --- a/content/learning-paths/mobile-graphics-and-gaming/android_webgpu_dawn/3-integrate-dawn.md +++ b/content/learning-paths/mobile-graphics-and-gaming/android_webgpu_dawn/3-integrate-dawn.md @@ -26,7 +26,7 @@ GameActivity is a Jetpack library designed to assist Android games in processing GameActivity is a direct descendant of NativeActivity and shares a similar architecture: -![Game Activity Architecture #center](./images/GameActivityArchitecture.png "Figure 5: Game Activity Architecture") +![Game Activity Architecture #center](./images/gameactivityarchitecture.png "Figure 5: Game Activity Architecture") With GameActivity, you can focus on game development and avoid spending excessive amounts of time dealing with the Java Native Interface (JNI) code. diff --git a/content/learning-paths/mobile-graphics-and-gaming/android_webgpu_dawn/7-profiling-app-using-streamline.md b/content/learning-paths/mobile-graphics-and-gaming/android_webgpu_dawn/7-profiling-app-using-streamline.md index 1f5326e38d..87a2ae3b62 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/android_webgpu_dawn/7-profiling-app-using-streamline.md +++ b/content/learning-paths/mobile-graphics-and-gaming/android_webgpu_dawn/7-profiling-app-using-streamline.md @@ -36,13 +36,13 @@ Wait until Streamline completes processing the data. Switch to **Mali Timeline** view as shown below: -!["Mali Timeline Streamline" #center](images/Streamline-mali-timeline.png "Figure 13: Mali Timeline Streamline") +!["Mali Timeline Streamline" #center](images/streamline-mali-timeline.png "Figure 13: Mali Timeline Streamline") You might have to zoom into the data up to the maximum (**500 us**), as you are rendering a simple 3D object. You can analyze two consecutive frames as shown below: -!["Two Consecutive Frames" #center](./images/Streamline-mali-analysis.png "Figure 14: Two Consecutive Frames") +!["Two Consecutive Frames" #center](./images/streamline-mali-analysis.png "Figure 14: Two Consecutive Frames") Arm has worked with the Dawn team to optimize data uploading to GPU buffers for Mali GPUs. diff --git a/content/learning-paths/mobile-graphics-and-gaming/android_webgpu_dawn/images/GameActivityArchitecture.png b/content/learning-paths/mobile-graphics-and-gaming/android_webgpu_dawn/images/GameActivityArchitecture.png deleted file mode 100644 index 717875772e..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/android_webgpu_dawn/images/GameActivityArchitecture.png and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/android_webgpu_dawn/images/Streamline-mali-analysis.png b/content/learning-paths/mobile-graphics-and-gaming/android_webgpu_dawn/images/Streamline-mali-analysis.png deleted file mode 100644 index a7ec578686..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/android_webgpu_dawn/images/Streamline-mali-analysis.png and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/android_webgpu_dawn/images/Streamline-mali-timeline.png b/content/learning-paths/mobile-graphics-and-gaming/android_webgpu_dawn/images/Streamline-mali-timeline.png deleted file mode 100644 index 799734c347..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/android_webgpu_dawn/images/Streamline-mali-timeline.png and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/add_asr_feature.png b/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/add_asr_feature.png deleted file mode 100644 index 979e73abf5..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/add_asr_feature.png and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/arm_asr_debugger.png b/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/arm_asr_debugger.png deleted file mode 100644 index 7ed7c26e2f..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/arm_asr_debugger.png and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/arm_asr_settings.png b/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/arm_asr_settings.png deleted file mode 100644 index 7f79d38e62..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/arm_asr_settings.png and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/arm_asr_view.webp b/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/arm_asr_view.webp deleted file mode 100644 index 8eeb583ea6..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/arm_asr_view.webp and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/camera_settings.png b/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/camera_settings.png deleted file mode 100644 index 7da6fb34ef..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/camera_settings.png and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/create_renderer.png b/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/create_renderer.png deleted file mode 100644 index e09399b81c..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/create_renderer.png and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/disable_opaque_downsampling.png b/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/disable_opaque_downsampling.png deleted file mode 100644 index 0a3b1187c9..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/disable_opaque_downsampling.png and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/project_settings.png b/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/project_settings.png deleted file mode 100644 index 968df94bed..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/project_settings.png and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/select_arm_asr.png b/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/select_arm_asr.png deleted file mode 100644 index 3ee3202255..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/select_arm_asr.png and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/third_person_pack_opening_screen.webp b/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/third_person_pack_opening_screen.webp deleted file mode 100644 index 67e500192d..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-arm-asr/images/third_person_pack_opening_screen.webp and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-unity-on-android/1-setup.md b/content/learning-paths/mobile-graphics-and-gaming/get-started-with-unity-on-android/1-setup.md index b8a99fbadc..95c1ddac24 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/get-started-with-unity-on-android/1-setup.md +++ b/content/learning-paths/mobile-graphics-and-gaming/get-started-with-unity-on-android/1-setup.md @@ -77,7 +77,7 @@ A simple project is provided to accompany this Learning Path. To open it in Unit You will see your project listed in the _Projects_ tab in Unity Hub. -- You can now click on the project to open it. The sample was created with Unity 2022.3.18f1; if you use a different version, you will get a warning. The project is very simple and should be safe to convert. However, if in doubt, install 2022.3.18f1 via the Unity Hub as already shown. +- You can now click on the project to open it. The sample was created with Unity 2022.3.18f1; if you use a different version, you will get a warning. The project is simple and should be safe to open in your installed version. However, if in doubt, install 2022.3.18f1 via the Unity Hub as already shown. - The project will now open in Unity. Once loaded (the first time can take a while) find the scene folder, open the sample scene, and then click the _Play_ button to run the sample. This will run the project inside the editor. You will see a spinning cube. diff --git a/content/learning-paths/mobile-graphics-and-gaming/mte_on_pixel8/pictures/01_mte_option_in_developer_options.png b/content/learning-paths/mobile-graphics-and-gaming/mte_on_pixel8/pictures/01_mte_option_in_developer_options.png deleted file mode 100644 index 1ba96d1862..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/mte_on_pixel8/pictures/01_mte_option_in_developer_options.png and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/1-install-plugin.md b/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/1-install-plugin.md index d4ce0ad4ca..7a7e51a8c0 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/1-install-plugin.md +++ b/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/1-install-plugin.md @@ -1,6 +1,6 @@ --- title: Introduction to neural graphics and Neural Super Sampling (NSS) -weight: 2 +weight: 3 ### FIXED, DO NOT MODIFY layout: learningpathall @@ -27,20 +27,4 @@ Under the hood, Neural Super Sampling for Unreal Engine (NSS for UE) runs its ne With these resources, you can seamlessly integrate NSS into any Unreal Engine project. The setup is designed to work with Vulkan as your rendering backend, and you don’t need to overhaul your workflow - just plug it in and start leveraging ML-powered upscaling right away. The technology is available as a source-code implementation that you will build with Visual Studio. -## Download required artifacts - -Before you begin, download the required plugins and dependencies. These two repositories contain everything you need to set up NSS for Unreal Engine, including the VGF model file, and the ML Emulations Layers for Vulkan. - -### 1. Download the NSS plugin - -[**Neural Super Sampling Unreal Engine Plugin** → GitHub Repository](https://github.com/arm/neural-graphics-for-unreal) - -Download the latest release package and extract it on your Windows machine. Use the folder corresponding to your Unreal version. - - -### 2. Download the runtime for ML Extensions for Vulkan -[**Unreal NNE Runtime RDG for ML Extensions for Vulkan** → GitHub Repository](https://github.com/arm/ml-extensions-for-vulkan-unreal-plugin). - -Download and extract the release package on your Windows machine. - -Once you’ve extracted both repositories, proceed to the next section to set up your development environment and enable the NSS plugin. \ No newline at end of file +Proceed to the next section to set up your development environment and enable the NSS plugin. \ No newline at end of file diff --git a/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/2-emulation-layer.md b/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/2-emulation-layer.md index 4d9b825d88..655c2fab3c 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/2-emulation-layer.md +++ b/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/2-emulation-layer.md @@ -1,6 +1,6 @@ --- title: Setting up the emulation layers -weight: 3 +weight: 4 ### FIXED, DO NOT MODIFY layout: learningpathall @@ -8,11 +8,11 @@ layout: learningpathall ## Install dependencies -To run NSS in your Unreal Engine project, install and configure the following: +To run NSS in your Unreal Engine project, you will need to install and configure the following: - **Vulkan SDK**: Required for development of applications that use Vulkan, and to enable the Vulkan Configurator. The latter sets up the emulation layers used for running ML extensions for Vulkan workloads. -- **ML Emulation Layer for Vulkan**: These layers allows neural inference to run in emulation through Vulkan’s compute backend. They are activated by Vulkan Configurator to run with the Unreal Engine plugin. The layers are included in the `NNERuntimeRDGMLExtensionsForVulkan` zip you downloaded in a previous step. The Vulkan layer configuration activates the ML Emulation Layer for Vulkan, which implements the ML extensions for Vulkan. -- **NSS for Unreal Engine plugins**: These include `NSS` (the inference and model interface) and `NNERuntimeRDGMLExtensionsForVulkan` (which connects Unreal’s Render Dependency Graph to the ML extensions for Vulkan). +- **ML Emulation Layer for Vulkan**: These layers allows neural inference to run in emulation through Vulkan’s compute backend. They are activated by Vulkan Configurator to run with the Unreal Engine plugin. The Vulkan layer configuration activates the ML Emulation Layer for Vulkan, which implements the ML extensions for Vulkan. +- **NSS for Unreal Engine plugin**: You will download and integrate the plugin in the next section. These components allow you to run NSS in Unreal Engine, using ML emulation layers for Vulkan for development and testing. @@ -20,6 +20,15 @@ These components allow you to run NSS in Unreal Engine, using ML emulation layer Go to the [Vulkan SDK landing page](https://vulkan.lunarg.com/sdk/home) and download the SDK Installer for Windows. After you have run the installer, you can move on to the next step. +## Download the emulation layers + +For this Learning Path, a pre-built of package of the emulation layers is available. Download them by clicking the link. + +[**ML Emulation Layer for Vulkan** → Arm Developer Downloads](https://www.arm.com/-/media/Files/developer/MLEmulationLayerForVulkan20251107) + +Extract the downloaded file in a location of your choice. + + ## Configure Vulkan Layers Vulkan Configurator is a program that will run the emulation layers in the background when you want to utilize them with Unreal Engine. @@ -28,9 +37,9 @@ To emulate the ML extensions for Vulkan: 1. Launch the **Vulkan Configurator** (bundled with the Vulkan SDK) from the Windows **Start** menu. 2. In the **Apply a Vulkan Loader Configuration** list, right-click and choose **Create a new Configuration**. You can give the new configuration any name, for example `NSS`. 3. Navigate to the **Vulkan Layers Location** tab. -4. Append a user-defined path pointing to the emulation layers you downloaded in the previous section: +4. Append a user-defined path pointing to the emulation layers you downloaded in the previous step: ``` - /NNERuntimeRDGMLExtensionsForVulkan/MLEmulationLayerForVulkan + /MLEmulationLayerForVulkan20251107 ``` ![Add user-defined Vulkan layers path in Vulkan Configurator#center](./images/load_layers.png "Figure 1: Add Vulkan layer path.") @@ -38,56 +47,10 @@ To emulate the ML extensions for Vulkan: ![Layer configuration showing Graph above Tensor#center](./images/verify_layers.png "Figure 2: Verify layer ordering and scope.") - -{{% notice %}} -Keep the Vulkan Configurator running to enable the emulation layers during engine execution. +{{% notice Before you move on %}} +Make sure you keep Vulkan Configurator running in the background as you go through the next steps. {{% /notice %}} -## Enable NSS for Unreal Engine - -1. Open Unreal Engine and create a new **Third Person** template project using the **C++** option. - -![Unreal Engine project selection screen showing C++ Third Person template#center](./images/unreal_startup.webp "Figure 3: Create a new C++ project in Unreal Engine.") - -2. Open the project in **Visual Studio**. Build it from source through **Build** > **Build Solution** or with `Ctrl+Shift+B`. - -After the build is finished, open your project in Unreal Engine. - -## Change Unreal’s Rendering Interface to Vulkan - -By default, Unreal uses DirectX. Instead, you need to choose Vulkan as the default RHI: -1. Go to: - ``` - Project Settings > Platform > Windows > Targeted RHIs > Default RHI - ``` -2. Select **Vulkan**. -3. Restart Unreal Engine to apply the change. - -![Project Settings with Vulkan selected as Default RHI under Targeted RHIs#center](./images/targeted_rhis.png "Figure 4: Set Vulkan as the default RHI.") - -## Create the Plugins directory - -Open your project directory in Windows explorer, and create a new folder called `Plugins`. - -Enabling the plugin will look slightly different depending on what Unreal version you are using. Follow the steps corresponding to your setup. - -## For Unreal 5.4 and 4.27 - -1. Copy the downloaded and extracted `.zip` archive into the new `Plugins` directory: - - `UE5.4` or `UE4.27` -2. Re-open Unreal Engine. When prompted, confirm plugin integration. -3. Rebuild your project in Visual Studio from source. -4. Verify the installation by opening the Plugins view in Unreal Engine, and making sure the checkbox is selected for `NSS`. Restart Unreal Engine if prompted. - -## For Unreal 5.5 - -1. Copy the downloaded and extracted `.zip` archives into the new `Plugins` directory: - - `UE5.5` - - `NNERuntimeRDGMLExtensionsForVulkan` -2. Re-open Unreal Engine. When prompted, confirm plugin integration. -3. Rebuild your project in Visual Studio from source. -4. Verify the installation by opening the Plugins view in Unreal Engine, and making sure the checkbox is selected for both `NSS` and `NNERuntimeRDGMLExtensionsForVulkan` as shown. Restart Unreal Engine if prompted. - -![Unreal Engine plugins window showing NSS and NNERuntimeRDGMLExtensionsForVulkan enabled#center](./images/verify_plugin_enabled.png "Figure 5: Verify plugin installation in Unreal Engine.") +With the ML emulation layers configured, Vulkan is now able to run machine learning workloads through the ML extensions for Vulkan. This enables neural inference to execute alongside the graphics pipeline during development, without requiring access to hardware with dedicated neural accelerators. -With the emulation layers and plugins configured, you're ready to run Neural Super Sampling in Unreal Engine. Continue to the next section to test the integration. +The next step is to integrate Neural Super Sampling into an Unreal Engine project. You’ll do this by installing the NSS plugin and creating a simple example game that lets you verify the setup and visualize the upscaling in action. \ No newline at end of file diff --git a/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/3-create-example.md b/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/3-create-example.md new file mode 100644 index 0000000000..d4089fdb75 --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/3-create-example.md @@ -0,0 +1,60 @@ +--- +title: Create an example game +weight: 5 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- +## Download the NSS plugin + +This repository's release package contains everything you need to set up NSS for Unreal Engine, including the plugin and the VGF model file. + +[**Neural Super Sampling Unreal Engine Plugin** → GitHub Repository](https://github.com/arm/neural-graphics-for-unreal) + +Download the latest release `.zip` and extract it on your Windows machine. + +{{% notice Unreal Engine 5.5 plugin %}} +For this UE version, the steps will be slightly different. Refer to the repository documentation for more information. +{{% /notice %}} + + +## Enable NSS for Unreal Engine + +1. Open Unreal Engine and create a new **Third Person** template project using the **C++** option. + +![Unreal Engine project selection screen showing C++ Third Person template#center](./images/unreal_startup.webp "Figure 3: Create a new C++ project in Unreal Engine.") + +2. Open the project in **Visual Studio**. Build it from source through **Build** > **Build Solution** or with `Ctrl+Shift+B`. + +After the build is finished, open your project in Unreal Engine. + +## Change Unreal’s Rendering Interface to Vulkan + +By default, Unreal uses DirectX. Instead, you need to choose Vulkan as the default RHI: +1. Go to: + ``` + Project Settings > Platform > Windows > Targeted RHIs > Default RHI + ``` +2. Select **Vulkan**. +3. Restart Unreal Engine to apply the change. + +![Project Settings with Vulkan selected as Default RHI under Targeted RHIs#center](./images/targeted_rhis.png "Figure 4: Set Vulkan as the default RHI.") + +## Create the Plugins directory + +Open your project directory in Windows explorer, and create a new folder called `Plugins`. + +![Windows File Explorer showing project directory with newly created Plugins folder alongside other project directories#center](./images/plugins_dir.png "Figure 5: The new Plugins directory") + +## Enable the plugin + +The plugin is included in the release package you downloaded in the previous section. The package contains a separate folder for each supported Unreal Engine version. Make sure you use the folder that matches your engine version (for example, UE5.5 for Unreal Engine 5.5). + +1. Copy the appropriate engine-version folder from the extracted .zip archive into your project's Plugins directory. +2. Reopen Unreal Engine. When prompted, confirm that you want to enable the plugin. +3. Rebuild your project from source in Visual Studio. +4. Verify the installation by opening Edit → Plugins in Unreal Engine and confirming that the NSS plugin is enabled. Restart Unreal Engine if prompted. +![Unreal Engine plugins window showing NSS enabled#center](./images/verify_plugin_enabled.png "Figure 5: Verify plugin installation in Unreal Engine.") + +With the emulation layers and plugins configured, you're ready to run Neural Super Sampling in Unreal Engine. Continue to the next section to test the integration. + diff --git a/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/3-run-example.md b/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/3-run-example.md index b1c3304a64..6c8ea8333f 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/3-run-example.md +++ b/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/3-run-example.md @@ -1,6 +1,6 @@ --- title: Run the example -weight: 4 +weight: 6 ### FIXED, DO NOT MODIFY layout: learningpathall @@ -20,13 +20,43 @@ In **Project Settings > Plugins > Neural Super Sampling**, you can view and conf Run `ShowFlag.VisualizeTemporalUpscaler 0` to disable the overview. To visualize the NSS model output in real-time, run the following command: ``` - r.NSS.Debug 2 + r.NSS.Debug 1 ``` This will add real-time views showing the model’s processed outputs, such as predicted filter coefficients and feedback, as below. In the [Wrapping up section](/learning-paths/mobile-graphics-and-gaming/nss-unreal/6-wrapping-up), you will find links to learn more about what the debug outputs mean. ![Debug view of Neural Super Sampling model output in Unreal Engine#center](./images/nss_debug.png "Figure 6: Visualize NSS model debug output in real time.") +## Troubleshooting tips + +If the example does not behave as expected, check the following common issues before continuing. + +### Check for build issues in Visual Studio +- Build failures related to `AutomationTool`, `Gauntlet`, or other `*.Automation` projects can be ignored. +- Focus on whether the project itself, named as `Editor`, builds successfully. + +### Check you Unreal Engine configuration +- Verify that Vulkan is selected as the **Default RHI**. +- Confirm the NSS plugin is enabled and that Unreal Engine was restarted after enabling it. +- Check **Project Settings → Plugins → Neural Super Sampling** to confirm a model is selected and active. + +If the NSS plugin is enabled but appears to have no effect: +- Ensure Vulkan Configurator is running. +- Verify that the correct layer configuration is selected and active. +- Double-check that: + - The emulation layer path is correct + - The Graph layer is ordered above the Tensor layer + +Refer back to the [emulation layer section](/learning-paths/mobile-graphics-and-gaming/nss-unreal/2-emulation-layer/) for the full Vulkan Configurator setup and validation steps. + +### Check the software and hardware setup +- Confirm that the plugin version exactly matches your Unreal Engine version. +- Verify that your GPU driver supports Vulkan. +- Verify that your Visual Studio version aligns with the Unreal Engine version you are using. +- Return to the Visual Studio build output and inspect the logs carefully to identify the first reported error + +Build or startup failures are often caused by version mismatches or missing dependencies. + ## NSS model on Hugging Face The model that powers NSS is published on Hugging Face in the [VGF format](https://github.com/arm/ai-ml-sdk-vgf-library). This format is optimized for inference via ML extensions for Vulkan. diff --git a/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/5-renderdoc.md b/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/5-renderdoc.md index 0a099cc209..219fb6f59a 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/5-renderdoc.md +++ b/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/5-renderdoc.md @@ -1,6 +1,6 @@ --- title: Using RenderDoc for Debugging and Analysis -weight: 6 +weight: 7 ### FIXED, DO NOT MODIFY layout: learningpathall diff --git a/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/6-wrapping-up.md b/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/6-wrapping-up.md index e75384be86..d635bba4d5 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/6-wrapping-up.md +++ b/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/6-wrapping-up.md @@ -1,6 +1,6 @@ --- title: Wrapping up -weight: 7 +weight: 8 ### FIXED, DO NOT MODIFY layout: learningpathall diff --git a/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/_index.md b/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/_index.md index 852e91d22c..9ded8ab2a2 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/_index.md +++ b/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/_index.md @@ -15,8 +15,8 @@ learning_objectives: prerequisites: - Windows 11 - - Unreal Engine 4.27, 5.4 or 5.5 (with the Templates and Feature Pack enabled) - - Visual Studio 2022 (with Desktop Development with C++ and .NET desktop build tools) + - Unreal Engine 4.27 or 5.4-5.6 (with the Templates and Feature Pack enabled) + - Visual Studio (with Desktop Development with C++ and .NET desktop build tools) author: Annie Tallund @@ -26,6 +26,7 @@ skilllevels: Introductory subjects: ML armips: - Mali + - Immortalis tools_software_languages: - Unreal Engine - Vulkan SDK diff --git a/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/images/add_plugin_folder.png b/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/images/add_plugin_folder.png deleted file mode 100644 index 3bacc9afc5..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/images/add_plugin_folder.png and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/images/confirm_layers.png b/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/images/confirm_layers.png deleted file mode 100644 index 86ed7de624..0000000000 Binary files a/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/images/confirm_layers.png and /dev/null differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/images/plugins_dir.png b/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/images/plugins_dir.png new file mode 100644 index 0000000000..d8799bf429 Binary files /dev/null and b/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/images/plugins_dir.png differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/images/verify_plugin_enabled.png b/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/images/verify_plugin_enabled.png index 96d8e1e267..0c1aea78a6 100644 Binary files a/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/images/verify_plugin_enabled.png and b/content/learning-paths/mobile-graphics-and-gaming/nss-unreal/images/verify_plugin_enabled.png differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/onnx/01_fundamentals.md b/content/learning-paths/mobile-graphics-and-gaming/onnx/01_fundamentals.md new file mode 100644 index 0000000000..0dcca61c93 --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/onnx/01_fundamentals.md @@ -0,0 +1,103 @@ +--- +# User change +title: "ONNX Fundamentals" + +weight: 2 + +layout: "learningpathall" +--- +The goal of this tutorial is to provide developers with a practical, end-to-end pathway for working with Open Neural Network Exchange (ONNX) in real-world scenarios. Starting from the fundamentals, we will build a simple neural network model in Python, export it to the ONNX format, and demonstrate how it can be used for both inference and training on Arm64 platforms. Along the way, we will cover model optimization techniques such as layer fusion, and conclude by deploying the optimized model into a fully functional Android application. By following this series, you will gain not only a solid understanding of ONNX’s philosophy and ecosystem but also the hands-on skills required to integrate ONNX into your own projects from prototyping to deployment. + +In this first step, we will introduce the ONNX standard and explain why it has become a cornerstone of modern machine learning workflows. You will learn what ONNX is, how it represents models in a framework-agnostic format, and why this matters for developers targeting different platforms such as desktops, Arm64 devices, or mobile environments. We will also discuss the role of ONNX Runtime as the high-performance engine that brings these models to life, enabling efficient inference and even training across CPUs, GPUs, and specialized accelerators. Finally, we will outline the typical ONNX workflow, from training in frameworks like PyTorch or TensorFlow, through export and optimization, to deployment on edge and Android devices, which we will gradually demonstrate throughout the tutorial. + +## What is ONNX +The ONNX is an open standard for representing machine learning models in a framework-independent format. Instead of being tied to the internal model representation of a specific framework—such as PyTorch, TensorFlow, or scikit-learn—ONNX provides a universal way to describe models using a common set of operators, data types, and computational graphs. + +At its core, an ONNX model is a directed acyclic graph (DAG) where nodes represent mathematical operations (e.g., convolution, matrix multiplication, activation functions) and edges represent tensors flowing between these operations. This standardized representation allows models trained in one framework to be exported once and executed anywhere, without requiring the original framework at runtime. + +ONNX was originally developed by Microsoft and Facebook to address a growing need in the machine learning community: the ability to move models seamlessly between training environments and deployment targets. Today, it is supported by a wide ecosystem of contributors and hardware vendors, making it the de facto choice for interoperability and cross-platform deployment. + +For developers, this means flexibility. You can train your model in PyTorch, export it to ONNX, run it with ONNX Runtime on an Arm64 device such as a Raspberry Pi, and later deploy it inside an Android application without rewriting the model. This portability is the main reason ONNX has become a central building block in modern AI workflows. + +A useful way to think of ONNX is to compare it to a PDF for machine learning models. Just as a PDF file ensures that a document looks the same regardless of whether you open it in Adobe Reader, Preview on macOS, or a web browser, ONNX ensures that a machine learning model behaves consistently whether you run it on a server GPU, a Raspberry Pi, or an Android phone. It is this “write once, run anywhere” principle that makes ONNX especially powerful for developers working across diverse hardware platforms. + +At the same time, ONNX is not a closed box. Developers can extend the format with custom operators or layers when standard ones are not sufficient. This flexibility makes it possible to inject novel research ideas, proprietary operations, or hardware-accelerated kernels into an ONNX model while still benefiting from the portability of the core standard. In other words, ONNX gives you both consistency across platforms and extensibility for innovation. + +## Why ONNX Matters +Machine learning today is not limited to one framework or one device. A model might be trained in PyTorch on a GPU workstation, tested in TensorFlow on a cloud server, and then finally deployed on an Arm64-based edge device or Android phone. Without a common standard, moving models between these environments would be complex, error-prone, and often impossible. ONNX solves this problem by acting as a universal exchange format, ensuring that models can flow smoothly across the entire development and deployment pipeline. + +The main reasons ONNX matters are: +1. Interoperability – ONNX eliminates framework lock-in. You can train in PyTorch, validate in TensorFlow, and deploy with ONNX Runtime on almost any device, from servers to IoT boards. +2. Performance – ONNX Runtime includes highly optimized execution backends, supporting hardware acceleration through Arm NEON, CUDA, DirectML, and Android NNAPI. This means the same model can run efficiently across a wide spectrum of hardware. +3. Portability – Once exported to ONNX, the model can be deployed to Arm64 devices (like Raspberry Pi or AWS Graviton servers) or even embedded in an Android app, without rewriting the code. +4. Ecosystem – The ONNX Model Zoo provides ready-to-use, pre-trained models for vision, NLP, and speech tasks, making it easy to start from state-of-the-art baselines. +5. Extensibility – Developers can inject their own layers or custom operators when the built-in operator set is not sufficient, enabling innovation while preserving compatibility. + +In short, ONNX matters because it turns the fragmented ML ecosystem into a cohesive workflow, empowering developers to focus on building applications rather than wrestling with conversion scripts or hardware-specific code. + +## ONNX Model Structure +An ONNX model is more than just a collection of weights—it is a complete description of the computation graph that defines how data flows through the network. Understanding this structure is key to seeing why ONNX is both portable and extensible. + +At a high level, an ONNX model consists of three main parts: +1. Graph, which is the heart of the model, represented as a directed acyclic graph (DAG). In this graph nodes correspond to operations (e.g., Conv, Relu, MatMul), edges represent tensors flowing between nodes, carrying input and output data. +2. Opset (Operator Set), which is a versioned collection of supported operations. Opsets guarantee that models exported with one framework will behave consistently when loaded by another, as long as the same opset version is supported. +3. Metadata, which contains information about inputs, outputs, tensor shapes, and data types. Metadata can also include custom annotations such as the model author, domain, or framework version. + +This design allows ONNX to describe anything from a simple logistic regression to a deep convolutional neural network. For example, a single ONNX graph might define: +* An input tensor representing a camera image. +* A sequence of convolution and pooling layers. +* Fully connected layers leading to classification probabilities. +* An output tensor with predicted labels. + +Because the ONNX format is based on a standardized graph representation, it is both human-readable (with tools like Netron for visualization) and machine-executable (parsed directly by ONNX Runtime or other backends). + +Importantly, ONNX models are not static. Developers can insert, remove, or replace nodes in the graph, making it possible to add new layers, prune unnecessary ones, or fuse operations for optimization. This graph-level flexibility is what enables many of the performance improvements we’ll explore later in this tutorial, such as layer fusion and quantization. + +## ONNX Runtime +While ONNX provides a standard way to represent models, it still needs a high-performance engine to actually execute them. This is where ONNX Runtime (ORT) comes in. ONNX Runtime is the official, open-source inference engine for ONNX models, designed to run them quickly and efficiently across a wide variety of hardware. + +At its core, ONNX Runtime is optimized for speed, portability, and extensibility: +1. Cross-platform support. ORT runs on Windows, Linux, and macOS, as well as mobile platforms like Android and iOS. It supports both x86 and Arm64 architectures, making it suitable for deployment from cloud servers to edge devices such as Raspberry Pi boards and smartphones. + +2. Hardware acceleration. ORT integrates with a wide range of execution providers (EPs) that tap into hardware capabilities: +* Arm Kleidi kernels accelerated with Arm NEON, SVE2, and SME2 instructions for efficient CPU execution on Arm64. +* CUDA for NVIDIA GPUs. +* DirectML for Windows. +* NNAPI on Android, enabling direct access to mobile accelerators (DSPs, NPUs). + +3. Inference and training. ONNX Runtime also supports training and fine-tuning, making it possible to use the same runtime across the entire ML lifecycle. + +4. Optimization built in. ORT can automatically apply graph optimizations such as constant folding, operator fusion, or memory layout changes to squeeze more performance out of your model. + +For developers, this means you can take a model trained in PyTorch, export it to ONNX, and then run it with ONNX Runtime on virtually any device—without worrying about the underlying hardware differences. The runtime abstracts away the complexity, choosing the best available execution provider for your environment. + +This flexibility makes ONNX Runtime a powerful bridge between training frameworks and deployment targets, and it is the key technology that allows ONNX models to run effectively on Arm64 platforms and Android devices. + +## How ONNX Fits into the Workflow + +One of the biggest advantages of ONNX is how naturally it integrates into a developer’s machine learning workflow. Instead of locking you into a single framework from training to deployment, ONNX provides a bridge that connects different stages of the ML lifecycle. + +A typical ONNX workflow looks like this: +1. Train the model. You first use your preferred framework (e.g., PyTorch, TensorFlow, or scikit-learn) to design and train a model. At this stage, you benefit from the flexibility and ecosystem of the framework of your choice. +2. Export to ONNX. Once trained, the model is exported into the ONNX format using built-in converters (such as torch.onnx.export for PyTorch). This produces a portable .onnx file describing the network architecture, weights, and metadata. +3. Run inference with ONNX Runtime. The ONNX model can now be executed on different devices using ONNX Runtime. On Arm64 hardware, ONNX Runtime can take advantage of Arm Kleidi kernels accelerated with NEON, SVE2, and SME2 instructions, while on Android devices it can leverage NNAPI to access mobile accelerators (where available). +4. Optimize the model. Apply graph optimizations like layer fusion, constant folding, or quantization to improve performance and reduce memory usage, making the model more suitable for edge and mobile deployments. +5. Deploy. Finally, the optimized ONNX model is packaged into its target environment. This could be an Arm64-based embedded system (e.g., Raspberry Pi), a server powered by Arm CPUs (e.g., AWS Graviton), or an Android application distributed via the Play Store. + +This modularity means developers are free to mix and match the best tools for each stage: train in PyTorch, optimize with ONNX Runtime, and deploy to Android—all without rewriting the model. By decoupling training from inference, ONNX enables efficient workflows that span from research experiments to production-grade applications. + +## Example Use Cases +ONNX is already widely adopted in real-world applications where portability and performance are critical. A few common examples include: +1. Computer Vision at the Edge – Running an object detection model (e.g., YOLOv5 exported to ONNX) on a Raspberry Pi 4 or NVIDIA Jetson, enabling low-cost cameras to detect people, vehicles, or defects in real time. +2. Mobile Applications – Deploying face recognition or image classification models inside an Android app using ONNX Runtime Mobile, with NNAPI acceleration for efficient on-device inference. +3. Natural Language Processing (NLP) – Running BERT-based models on Arm64 cloud servers (like AWS Graviton) to provide fast, low-cost inference for chatbots and translation services. +4. Healthcare Devices – Using ONNX to integrate ML models into portable diagnostic tools or wearable sensors, where Arm64 processors dominate due to their low power consumption. +5. Cross-platform Research to Production – Training experimental architectures in PyTorch, exporting them to ONNX, and validating them across different backends to ensure consistent performance. +6. AI Accelerator Integration – ONNX is especially useful for hardware vendors building custom AI accelerators. Since accelerators often cannot support the full range of ML operators, ONNX’s extensible operator model allows manufacturers to plug in custom kernels where hardware acceleration is available, while gracefully falling back to the standard runtime for unsupported ops. This makes it easier to adopt new hardware without rewriting entire models. + +## Summary +In this section, we introduced ONNX as an open standard for representing machine learning models across frameworks and platforms. We explored its model structure—graphs, opsets, and metadata—and explained the role of ONNX Runtime as the high-performance execution engine. We also showed how ONNX fits naturally into the ML workflow: from training in PyTorch or TensorFlow, to exporting and optimizing the model, and finally deploying it on Arm64 or Android devices. + +A useful way to think of ONNX is as the PDF of machine learning models—a universal, consistent format that looks the same no matter where you open it, but with the added flexibility to inject your own layers and optimizations. + +Beyond portability for developers, ONNX is also valuable for hardware and AI-accelerator builders. Because accelerators often cannot support every possible ML operator, ONNX’s extensible operator model allows manufacturers to seamlessly integrate custom kernels where acceleration is available, while relying on the runtime for unsupported operations. This combination of consistency, flexibility, and extensibility makes ONNX a cornerstone technology for both AI application developers and hardware vendors. \ No newline at end of file diff --git a/content/learning-paths/mobile-graphics-and-gaming/onnx/02_setup.md b/content/learning-paths/mobile-graphics-and-gaming/onnx/02_setup.md new file mode 100644 index 0000000000..fe9179c87c --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/onnx/02_setup.md @@ -0,0 +1,130 @@ +--- +# User change +title: "Environment Setup" + +weight: 3 + +layout: "learningpathall" +--- + +## Objective +This step gets you ready to build, export, run, and optimize ONNX models on Arm64. You’ll set up Python, install ONNX & ONNX Runtime, confirm hardware-backed execution providers. + +## Choosing the hardware +You can choose a variety of hardware, including: +* Edge boards (Linux/Arm64) - Raspberry Pi 4/5 (64-bit OS), Jetson (Arm64 CPU; GPU via CUDA if using NVIDIA stack), Arm servers (e.g., AWS Graviton). +* Apple Silicon (macOS/Arm64) - Great for development, deploy to Arm64 Linux later. +* Windows on Arm - Dev/test on WoA, deploy to Linux Arm64 for production if desired. + +The nice thing about ONNX is that the **same model file** can run across all of these, so your setup is flexible. + +## Install Python +Depending on the hardware you use you follow different installation paths + +1. Linux (Arm64). In the console type: +```console +sudo apt update +sudo apt install -y python3 python3-venv python3-pip build-essential libopenblas-dev +``` + +2. macOS (Apple Sillicon): +```console +brew install python +``` + +3. Windows on Arm: +* Install Python 3.10+ from python.org (Arm64 build). +* Ensure pip is on PATH. + +After installing Python, open a terminal or console, create a clean virtual environment, and update pip and wheel: + +```console +python3 -m venv .venv +source .venv/bin/activate # on Windows use: .venv\Scripts\activate +python -m pip install --upgrade pip wheel +``` + +Using a virtual environment keeps dependencies isolated and avoids conflicts with system-wide Python packages. + +## Install Core Packages +Start by installing the minimal stack: +```console +pip install onnx onnxruntime onnxscript netron numpy +``` +The above will install the following: +* onnx – core library for loading/saving ONNX models. +* onnxruntime – high-performance runtime to execute models. +* onnxscript – required for the new Dynamo-based exporter. +* netron – tool for visualizing ONNX models. +* numpy – used for tensor manipulation. + +Now, install PyTorch (we’ll use it later to build and export a sample model): + +```console +pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu +``` + +## Verify the installation +Let’s verify everything works end-to-end by training a toy network and exporting it to ONNX. + +Create a new file 01_Init.py and add the following code + +```python +import torch, torch.nn as nn +import onnx, onnxruntime as ort +import numpy as np + +class SmallNet(nn.Module): + def __init__(self): + super().__init__() + self.seq = nn.Sequential( + nn.Conv2d(1, 8, 3, padding=1), + nn.ReLU(), + nn.AdaptiveAvgPool2d((1,1)), + nn.Flatten(), + nn.Linear(8, 10) + ) + def forward(self, x): return self.seq(x) + +m = SmallNet().eval() +dummy = torch.randn(1, 1, 28, 28) + +torch.onnx.export( + m, dummy, "smallnet.onnx", + input_names=["input"], output_names=["logits"], + opset_version=19, + do_constant_folding=True, + keep_initializers_as_inputs=False, + dynamo=True +) + +# Quick sanity run +sess = ort.InferenceSession("smallnet.onnx", providers=["CPUExecutionProvider"]) +out = sess.run(["logits"], {"input": dummy.numpy()})[0] +print("Output shape:", out.shape, "Providers:", sess.get_providers()) +``` + +Then, run it as follows + +```console +python3 01_Init.py +``` + +You should see the following output: +```output +python3 01_Init.py +[torch.onnx] Obtain model graph for `SmallNet([...]` with `torch.export.export(..., strict=False)`... +[torch.onnx] Obtain model graph for `SmallNet([...]` with `torch.export.export(..., strict=False)`... ✅ +[torch.onnx] Run decomposition... +[torch.onnx] Run decomposition... ✅ +[torch.onnx] Translate the graph into ONNX... +[torch.onnx] Translate the graph into ONNX... ✅ +Output shape: (1, 10) Providers: ['CPUExecutionProvider'] +``` + +The 01_Init.py script serves as a quick end-to-end validation of your ONNX environment. It defines a very small convolutional neural network (SmallNet) in PyTorch, which consists of a convolution layer, activation function, pooling, flattening, and a final linear layer that outputs 10 logits. Instead of training the model, we simply run it in evaluation mode on a random input tensor to make sure the graph structure works. This model is then exported to the ONNX format using PyTorch’s new Dynamo-based exporter, producing a portable smallnet.onnx file. + +After export, the script immediately loads the ONNX model with ONNX Runtime and executes a forward pass using the CPU execution provider. This verifies that the installation of ONNX, ONNX Runtime, and PyTorch is correct and that models can flow seamlessly from definition to inference. By printing the output tensor’s shape and the active execution provider, the script demonstrates that the toolchain is fully functional on your Arm64 device, giving you a solid baseline before moving on to more advanced models and optimizations. + +## Summary +You now have a fully functional ONNX development environment on Arm64. Python and all required packages are installed, and you successfully exported a small PyTorch model to ONNX using the new Dynamo exporter, ensuring forward compatibility. Running the model with ONNX Runtime confirmed that inference works end-to-end with the CPU execution provider, proving that your toolchain is correctly configured. With this foundation in place, the next step is to build and export a more complete model and run it on Arm64 hardware to establish baseline performance before applying optimizations. \ No newline at end of file diff --git a/content/learning-paths/mobile-graphics-and-gaming/onnx/03_preparingdata.md b/content/learning-paths/mobile-graphics-and-gaming/onnx/03_preparingdata.md new file mode 100644 index 0000000000..b5bea9f3b2 --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/onnx/03_preparingdata.md @@ -0,0 +1,261 @@ +--- +# User change +title: "Preparing a Synthetic Sudoku Digit Dataset" + +weight: 4 + +layout: "learningpathall" +--- + +## Big picture +Our end goal is a camera-to-solution Sudoku app that runs efficiently on Arm64 devices (e.g., Raspberry Pi or Android phones). ONNX is the glue: we’ll train the digit recognizer in PyTorch, export it to ONNX, and run it anywhere with ONNX Runtime (CPU EP on edge devices, NNAPI EP on Android). Everything around the model—grid detection, perspective rectification, and solving—stays deterministic and lightweight. + +## Objective +In this step, we will generate a custom dataset of Sudoku puzzles and their digit crops, which we’ll use to train a digit recognition model. Starting from a Hugging Face parquet dataset that provides paired puzzle/solution strings, we transform raw boards into realistic, book-style Sudoku pages, apply camera-like augmentations to mimic mobile captures, and automatically slice each page into 81 labeled cell images. This yields a large, diverse, perfectly labeled set of digits (0–9 with 0 = blank) without manual annotation. By the end, you’ll have a structured dataset ready to train a lightweight model in the next section. + +## Why Synthetic Generation? +When building a Sudoku digit recognizer, the hardest part is obtaining a well-labeled dataset that matches real capture conditions. MNIST contains handwritten digits, which differ from printed, grid-aligned Sudoku digits; relying on it alone hurts real-world performance. + +By generating synthetic Sudoku pages directly from the parquet dataset, we get: +1. Perfect labeling. Since the puzzle content is known, every cropped cell automatically comes with the correct label (digit or blank), eliminating manual annotation. +2. Control over style. We can render Sudoku pages to look like those in printed books, with realistic fonts, grid lines, and difficulty levels controlled by how many cells are left blank. +3. Robustness through augmentation: By applying perspective warps, blur, noise, and lighting variations, we simulate how a smartphone camera might capture a Sudoku page, improving the model’s ability to handle real-world photos. +4. Scalability. With millions of Sudoku solutions available, we can easily generate tens of thousands of training samples in minutes, ensuring a dataset that is both large and diverse. + +This synthetic data generation strategy allows us to create a custom-fit dataset for our Sudoku digit recognition problem, bridging the gap between clean digital puzzles and noisy real-world inputs. + +## What we’ll produce +By the end of this step, you will have two complementary datasets: +1. Digit crops for training the classifier. A folder tree structured for torchvision.datasets.ImageFolder, containing tens of thousands of labeled 28×28 images of Sudoku digits (0–9, with 0 meaning blank): + +```console +data/ + train/ + 0/....png (blank) + 1/....png + ... + 9/....png + val/ + 0/....png + ... + 9/....png +``` + +These will be used in next step to train a lightweight model for digit recognition. + +2. Rendered Sudoku grids for camera simulation. Full-page Sudoku images (both clean book-style and augmented camera-like versions) stored in: +```console +data/ + grids/ + train/ + 000001_clean.png + 000001_cam.png + ... + val/ + ... +``` + +These grid images allow us to later test the end-to-end pipeline: detect the board with OpenCV, rectify perspective, classify each cell using the ONNX digit recognizer, and then solve the Sudoku puzzle. + +Together, these datasets provide both the micro-level data needed to train the digit recognizer and the macro-level data to simulate the camera pipeline for testing and deployment. + +## Implementation +Start by creating a new file 02_PrepareData.py and modify it as follows: +```python +import os, random, pathlib +import numpy as np +import cv2 as cv +import pandas as pd +from tqdm import tqdm + +random.seed(0) + +# Parameters +PARQUET_PATH = "train_1.parquet" # path to your downloaded HF Parquet +OUT_DIR = pathlib.Path("data") +N_TRAIN = 1000 # how many puzzles to render for training +N_VAL = 100 # how many for validation +IMG_H, IMG_W = 1200, 800 # page size (portrait-ish) +GRID_MARGIN = 60 # outer margin, px +CELL_SIZE = 28 # output crop size for classifier (MNIST-like) +FONT = cv.FONT_HERSHEY_SIMPLEX +# + +def str_to_grid(s: str): + """81-char '012345678' string -> 9x9 list of ints.""" + s = s.strip() + assert len(s) == 81, f"bad length: {len(s)}" + return [[int(s[9*r+c]) for c in range(9)] for r in range(9)] + +def load_puzzles(parquet_path, n_train, n_val): + """Load puzzles/solutions; return two lists of 9x9 int grids for train/val.""" + df = pd.read_parquet(parquet_path, engine="pyarrow") + # Shuffle reproducibly + df = df.sample(frac=1.0, random_state=0).reset_index(drop=True) + # Keep only needed columns if present + need_cols = [c for c in ["puzzle", "solution"] if c in df.columns] + if not need_cols or "puzzle" not in need_cols: + raise ValueError(f"Expected 'puzzle' (and optionally 'solution') columns; got: {list(df.columns)}") + + # Slice train/val partitions + df_train = df.iloc[:n_train] + df_val = df.iloc[n_train:n_train+n_val] + + puzzles_train = [str_to_grid(p) for p in df_train["puzzle"].astype(str)] + puzzles_val = [str_to_grid(p) for p in df_val["puzzle"].astype(str)] + + # Solutions are optional (useful later for solver validation) + solutions_train = [str_to_grid(s) for s in df_train["solution"].astype(str)] if "solution" in df_train else None + solutions_val = [str_to_grid(s) for s in df_val["solution"].astype(str)] if "solution" in df_val else None + + return (puzzles_train, solutions_train), (puzzles_val, solutions_val) + +def draw_grid(img, size=9, margin=GRID_MARGIN): + H, W = img.shape[:2] + step = (min(H, W) - 2*margin) // size + x0 = (W - size*step) // 2 + y0 = (H - size*step) // 2 + for i in range(size+1): + thickness = 3 if i % 3 == 0 else 1 + # vertical + cv.line(img, (x0 + i*step, y0), (x0 + i*step, y0 + size*step), (0, 0, 0), thickness) + # horizontal + cv.line(img, (x0, y0 + i*step), (x0 + size*step, y0 + i*step), (0, 0, 0), thickness) + return (x0, y0, step) + +def put_digit(img, r, c, d, x0, y0, step): + if d == 0: + return # blank cell + text = str(d) + scale = step / 60.0 + thickness = 2 + (tw, th), base = cv.getTextSize(text, FONT, scale, thickness) + cx = x0 + c*step + (step - tw)//2 + cy = y0 + r*step + (step + th)//2 - th//4 + cv.putText(img, text, (cx, cy), FONT, scale, (0, 0, 0), thickness, cv.LINE_AA) + +def render_page(puzzle9x9): + page = np.full((IMG_H, IMG_W, 3), 255, np.uint8) + x0, y0, step = draw_grid(page, 9, GRID_MARGIN) + for r in range(9): + for c in range(9): + put_digit(page, r, c, puzzle9x9[r][c], x0, y0, step) + return page, (x0, y0, step) + +def aug_camera(img): + """Light camera-like augmentation: perspective jitter + optional Gaussian blur.""" + H, W = img.shape[:2] + def jitter(pt, s=20): + return (pt[0] + random.randint(-s, s), pt[1] + random.randint(-s, s)) + src = np.float32([(0, 0), (W, 0), (W, H), (0, H)]) + dst = np.float32([jitter((0,0)), jitter((W,0)), jitter((W,H)), jitter((0,H))]) + M = cv.getPerspectiveTransform(src, dst) + warped = cv.warpPerspective(img, M, (W, H), flags=cv.INTER_LINEAR, borderValue=(220, 220, 220)) + if random.random() < 0.5: + k = random.choice([1, 2]) + warped = cv.GaussianBlur(warped, (2*k+1, 2*k+1), 0) + return warped + +def ensure_dirs(split): + for cls in range(10): # 0..9 (0 == blank) + (OUT_DIR / split / str(cls)).mkdir(parents=True, exist_ok=True) + +def save_crops(page, geom, puzzle9x9, split, base_id): + x0, y0, step = geom + idx = 0 + for r in range(9): + for c in range(9): + x1, y1 = x0 + c*step, y0 + r*step + roi = page[y1:y1+step, x1:x1+step] + g = cv.cvtColor(roi, cv.COLOR_BGR2GRAY) + g = cv.resize(g, (CELL_SIZE, CELL_SIZE), interpolation=cv.INTER_AREA) + label = puzzle9x9[r][c] # 0 for blank, 1..9 digits + out_path = OUT_DIR / split / str(label) / f"{base_id}_{idx:02d}.png" + cv.imwrite(str(out_path), g) + idx += 1 + +def process_split(puzzles, split_name, n_limit): + ensure_dirs(split_name) + grid_dir = OUT_DIR / "grids" / split_name + grid_dir.mkdir(parents=True, exist_ok=True) + + N = min(n_limit, len(puzzles)) + for i in tqdm(range(N), desc=f"render {split_name}"): + puzzle = puzzles[i] + + # Clean page + page, geom = render_page(puzzle) + save_crops(page, geom, puzzle, split_name, base_id=f"{i:06d}_clean") + cv.imwrite(str(grid_dir / f"{i:06d}_clean.png"), page) + + # Camera-like + warped = aug_camera(page) + save_crops(warped, geom, puzzle, split_name, base_id=f"{i:06d}_cam") + cv.imwrite(str(grid_dir / f"{i:06d}_cam.png"), warped) + +def main(): + (p_train, _s_train), (p_val, _s_val) = load_puzzles(PARQUET_PATH, N_TRAIN, N_VAL) + process_split(p_train, "train", N_TRAIN) + process_split(p_val, "val", N_VAL) + print("Done. Output under:", OUT_DIR.resolve()) + +if __name__ == "__main__": + main() +``` + +At the top, you set basic knobs for the generator: where to read the Parquet file, where to write outputs, how many puzzles to render for train/val, page size, grid margin, crop size, and the OpenCV font. Tweaking these lets you control dataset scale, visual style, and classifier input size (e.g., CELL_SIZE=32 if you want a slightly larger digit crop). + +The method str_to_grid(s) converts an 81-character Sudoku string into a 9×9 list of integers. Each character represents a cell: 0 is blank, 1–9 are digits. This is the canonical internal representation used throughout the script. + +Then, we have load_puzzles(parquet_path, n_train, n_val), which loads the dataset from Parquet, shuffles it deterministically, and slices it into train/val partitions. It returns the puzzles (and, if present, solutions) as 9×9 integer grids. In this step we only need puzzle for rendering and labeling digit crops (blanks included); solution is useful later for solver validation. + +Subsequently, draw_grid(img, size=9, margin=GRID_MARGIN) draws a Sudoku grid on a blank page image. It computes the step size from the page dimensions and margin, then draws both thin inner lines and thick 3×3 box boundaries. It returns the top-left corner (x0, y0) and the cell size (step), which are reused to place digits and to locate each cell for cropping. + +Next, put_digit(img, r, c, d, x0, y0, step) renders a single digit d at row r, column c inside the grid. The text is centered in the cell using the font metrics; if d == 0, it leaves the cell blank. This mirrors printed-book Sudoku styling so our crops look realistic. + +Another method, render_page(puzzle9x9) builds a complete “book-style” Sudoku page: creates a white canvas, draws the grid, loops over all 81 cells, and writes digits using put_digit. It returns the page plus the grid geometry (x0, y0, step) for subsequent cropping. + +A method aug_camera(img) applies a light, camera-like augmentation to mimic smartphone captures: a small perspective warp (random corner jitter) and optional Gaussian blur. The warp uses a light gray border fill so any exposed areas look like paper rather than colored artifacts. This produces a second version of each page that’s closer to real-world inputs. + +Afterward, ensure_dirs(split) makes the class directories for a given split (train or val) so that crops can be saved in data/{split}/{class}/.... The classes are 0..9 with 0 = blank. + +A method save_crops(page, geom, puzzle9x9, split, base_id) slices the page into 81 cell crops using the grid geometry, converts each crop to grayscale, resizes it to CELL_SIZE × CELL_SIZE, and saves it into the appropriate class directory based on the puzzle’s value at that cell (0..9). Using the puzzle for labels ensures we learn to recognize blanks as well as digits. + +Then, process_split(puzzles, split_name, n_limit) is the workhorse for each partition. For each puzzle, it (1) renders a clean page, saves its 81 crops, and writes the full page under data/grids/{split}; then (2) generates an augmented “camera-like” version and saves its crops and full page too. This gives you both micro-level training data (crops) and macro-level test images (full grids) for the later camera pipeline. + +Finally, main() loads train/val puzzles from Parquet and calls process_split for each. When it finishes, you’ll have: +```console +data/ + train/ + 0/… 1/… … 9/… + val/ + 0/… … 9/… + grids/ + train/ (..._clean.png, ..._cam.png) + val/ (..._clean.png, ..._cam.png) +``` + +## Launching instructions +1. Install dependencies (inside your virtual env): +```console +pip install pandas pyarrow opencv-python tqdm numpy +``` + +2. Place the Parquet file (e.g., train_1.parquet) next to the script or update PARQUET_PATH accordingly. Here we used the file from [this location](https://huggingface.co/datasets/Ritvik19/Sudoku-Dataset/blob/main/train_1.parquet). + +3. Run the generator +```console +python3 02_PrepareData.py +``` + +4. Inspect outputs: +* Digit crops live under data/train/{0..9}/ and data/val/{0..9}/. +* Full-page grids (clean + camera-like) live under data/grids/train/ and data/grids/val/. + +Tips +* Start small (N_TRAIN=1000, N_VAL=100) to verify everything, then scale up. +* If you want larger inputs for the classifier, increase CELL_SIZE to 32 or 40. +* To make augmentation a bit stronger (more realistic), slightly increase the perspective jitter in aug_camera, add brightness/contrast jitter, or a faint gradient shadow overlay. + +## Summary +After running this step you’ll have a robust, labeled, Sudoku-specific dataset: thousands of digit crops (including blanks) for training and realistic full-page grids for pipeline testing. You’re ready for the next step—training the digit recognizer and exporting it to ONNX. \ No newline at end of file diff --git a/content/learning-paths/mobile-graphics-and-gaming/onnx/04_training.md b/content/learning-paths/mobile-graphics-and-gaming/onnx/04_training.md new file mode 100644 index 0000000000..332db92273 --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/onnx/04_training.md @@ -0,0 +1,238 @@ +--- +# User change +title: "Train the Digit Recognizer" + +weight: 5 + +layout: "learningpathall" +--- + +## Objective ## +We will now train a small CNN to classify Sudoku cell crops into 10 classes (0=blank, 1..9=digit), verify accuracy, then export the model to ONNX using the Dynamo exporter and sanity-check parity with ONNX Runtime. This gives us a portable model ready for Arm64 inference and later Android deployment. + +## Creating a model +We use a tiny convolutional neural network (CNN) called DigitNet, designed to be both fast (so it runs efficiently on Arm64 and mobile) and accurate enough for recognizing 28×28 grayscale crops of Sudoku digits. It expects 1 input channel (in_channels=1) because we forced grayscale in the preprocessing step. + +We start by creating a new file digitnet_model.py and defining the DigitNet class: +```python +import torch +import torch.nn as nn + +class DigitNet(nn.Module): + """ + Tiny CNN for Sudoku digit classification. + Classes: 0..9 where 0 = blank. + Input: (N,1,H,W) grayscale (default 28x28). + """ + def __init__(self, num_classes: int = 10): + super().__init__() + self.net = nn.Sequential( + nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(), + nn.MaxPool2d(2), + nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), + nn.AdaptiveAvgPool2d((1,1)), + nn.Flatten(), + nn.Linear(32, num_classes), + ) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + return self.net(x) +``` + +We use a very compact convolutional neural network (CNN), which we call DigitNet, to recognize Sudoku digits. The goal is to have a model that is simple enough to run efficiently on Arm64 and mobile devices, but still powerful enough to tell apart the ten classes we care about (0 for blank, and digits 1 through 9). + +The network expects each input to be a 28×28 grayscale crop, so it begins with a convolution layer that has one input channel and sixteen filters. This first convolution is responsible for learning very low-level patterns such as strokes or edges. Immediately after, a ReLU activation introduces non-linearity, which allows the network to combine those simple features into more expressive ones. A max-pooling layer then reduces the spatial resolution by half, making the representation more compact and less sensitive to small translations. + +At this point, the feature maps are passed through a second convolutional layer with thirty-two filters. This stage learns richer patterns, for example combinations of edges that form loops or intersections that distinguish an “8” from a “0” or a “6”. Another ReLU activation adds the necessary non-linearity to these higher-level features. + +Instead of flattening the entire feature map, we apply an adaptive average pooling operation that squeezes each of the thirty-two channels down to a single number. This effectively summarizes the information across the whole image and ensures the model produces a fixed-length representation regardless of the exact input size. After pooling, the features are flattened into a one-dimensional vector. + +The final step is a fully connected layer that maps the thirty-two features to ten output values, one for each class. These values are raw scores (logits) that indicate how strongly the model associates the input crop with each digit. During training, a cross-entropy loss will turn these logits into probabilities and guide the model to adjust its weights. + +In practice, this means that when you feed in a batch of grayscale Sudoku cells of shape [N, 1, 28, 28], DigitNet transforms them step by step into a batch of [N, 10] outputs, where each row contains the scores for the ten possible classes. Despite its simplicity, this small CNN strikes a balance between speed and accuracy that makes it ideal for Sudoku digit recognition on resource-constrained devices. + +## Training a model +We will now prepare the self-containing script that trains the above model on the data prepared earlier. Start by creating the new file 03_Training.py and modify it as follows: +```python +import os, random, numpy as np +import torch as tr +import torch.nn as nn +import torch.nn.functional as F +from torch.utils.data import DataLoader +from torchvision import datasets, transforms +from tqdm import tqdm +from torch.onnx import dynamo_export +from torch.export import Dim +import onnxruntime as ort + +from digitnet_model import DigitNet + +# Configuration +random.seed(0); np.random.seed(0); tr.manual_seed(0) +DEVICE = "cpu" # keep CPU for portability +DATA_DIR = "data" # data/train/0..9, data/val/0..9 +ARTI_DIR = "artifacts" +os.makedirs(ARTI_DIR, exist_ok=True) + +BATCH = 256 +EPOCHS = 10 +LR = 1e-3 +WEIGHT_DECAY = 1e-4 +LABEL_SMOOTH = 0.05 + +# Datasets (force grayscale to match model) +tfm_train = transforms.Compose([ + transforms.Grayscale(num_output_channels=1), # force 1-channel input + transforms.ToTensor(), + transforms.Normalize((0.5,), (0.5,)), + transforms.RandomApply([transforms.GaussianBlur(3)], p=0.15), + transforms.RandomAffine(degrees=5, translate=(0.02,0.02), scale=(0.95,1.05)), +]) +tfm_val = transforms.Compose([ + transforms.Grayscale(num_output_channels=1), # force 1-channel input + transforms.ToTensor(), + transforms.Normalize((0.5,), (0.5,)), +]) + +train_ds = datasets.ImageFolder(os.path.join(DATA_DIR, "train"), transform=tfm_train) +val_ds = datasets.ImageFolder(os.path.join(DATA_DIR, "val"), transform=tfm_val) + +train_loader = DataLoader(train_ds, batch_size=BATCH, shuffle=True, num_workers=0) +val_loader = DataLoader(val_ds, batch_size=BATCH, shuffle=False, num_workers=0) + +def evaluate(model: nn.Module, loader: DataLoader) -> float: + model.eval() + correct = total = 0 + with tr.no_grad(): + for x, y in loader: + x, y = x.to(DEVICE), y.to(DEVICE) + pred = model(x).argmax(1) + correct += (pred == y).sum().item() + total += y.numel() + return correct / total if total else 0.0 + +def main(): + # Sanity: verify loader channels + xb, _ = next(iter(train_loader)) + print("Train batch shape:", xb.shape) # expect [B, 1, 28, 28] + + model = DigitNet(num_classes=10).to(DEVICE) + opt = tr.optim.AdamW(model.parameters(), lr=LR, weight_decay=WEIGHT_DECAY) + + best_acc, best_state = 0.0, None + for ep in range(1, EPOCHS + 1): + model.train() + for x, y in tqdm(train_loader, desc=f"epoch {ep}/{EPOCHS}"): + x, y = x.to(DEVICE), y.to(DEVICE) + opt.zero_grad() + logits = model(x) + loss = F.cross_entropy(logits, y, label_smoothing=LABEL_SMOOTH) + loss.backward() + opt.step() + + acc = evaluate(model, val_loader) + print(f"val acc: {acc:.4f}") + if acc > best_acc: + best_acc = acc + best_state = {k: v.cpu().clone() for k, v in model.state_dict().items()} + + if best_state is not None: + model.load_state_dict(best_state) + print(f"Best val acc: {best_acc:.4f}") + + # Save PyTorch weights (optional) + tr.save(model.state_dict(), os.path.join(ARTI_DIR, "digitnet_best.pth")) + + # Export to ONNX with dynamic batch using the Dynamo API + model.eval() + dummy = tr.randn(1, 1, 28, 28) + onnx_path = os.path.join(ARTI_DIR, "sudoku_digitnet.onnx") + + tr.onnx.export( + model, # model + dummy, # input tensor corresponds to arg name 'x' + onnx_path, # output .onnx + input_names=["input"], # ONNX *display* name (independent of arg name) + output_names=["logits"], + opset_version=19, + do_constant_folding=True, + keep_initializers_as_inputs=False, + dynamo=True, + dynamic_shapes={"x": {0: Dim("N")}} + ) + + print("Exported:", onnx_path) + + # quick parity with a big batch (proves dynamic batch works) + sess = ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"]) + x = tr.randn(512, 1, 28, 28) + onnx_logits = sess.run(["logits"], {"input": x.numpy().astype(np.float32)})[0] + pt_logits = model(x).detach().numpy() + print("Parity MAE:", np.mean(np.abs(onnx_logits - pt_logits))) + +if __name__ == "__main__": + main() +``` + +This file is a self-contained trainer for the Sudoku digit classifier. It starts by fixing random seeds for reproducibility and sets DEVICE="cpu" so the workflow runs the same on desktops and Arm64 boards. It expects the dataset from the previous step under data/train/0..9 and data/val/0..9, and creates an artifacts/ folder for all outputs. + +The script builds two dataloaders (train/val) with a preprocessing stack that forces grayscale (Grayscale(num_output_channels=1)) so inputs match the model’s first convolution, converts to tensors, and normalizes to a centered range. Light augmentations on the training split—small affine jitter and occasional blur—mimic camera variability without distorting the digits. Batch size, epochs, and learning rate are set to conservative defaults so training is smooth on CPU; you can scale them up later. + +Then, the script it instantiates DigitNet(num_classes=10) model. The optimizer is AdamW with mild weight decay to control overfitting. The loss is cross-entropy with label smoothing (e.g., 0.05), which reduces over-confidence and helps on easily confused shapes (like 6/8/9). + +The training loop runs for a fixed number of epochs, iterating mini-batches from the training set. After each epoch, it evaluates on the validation split and logs the accuracy. The script keeps track of the best model state seen so far (based on val accuracy) and restores it at the end, ensuring the final model corresponds to your best epoch, not just the last one. + +The file will create two artifacts: +1. digitnet_best.pth — the best PyTorch weights (handy for quick experiments, fine-tuning, or debugging later). +2. sudoku_digitnet.onnx — the exported ONNX model, produced with PyTorch’s Dynamo exporter and a dynamic batch dimension. Dynamic batch means the model accepts input of shape [N, 1, 28, 28] for any N, which is ideal for efficient batched inference on Arm64 and for Android integration. + +Right after export, the script runs a parity test: it feeds the same randomly generated batch through both the PyTorch model and the ONNX model (executed by ONNX Runtime) and prints the mean absolute error between their logits. A tiny value confirms the exported graph faithfully matches your trained network. + +## Running the script +To run the training script, type: + +```console +python3 03_Training.py +``` + +The script will train, validate, export, and verify the digit recognizer in one go. After it finishes, you’ll have both a portable ONNX model and a PyTorch checkpoint ready for the next step—building the image processor that detects the Sudoku grid, rectifies it, segments cells, and performs batched ONNX inference to reconstruct the board for solving. + +Here is a sample run: + +```output +python3 03_Training.py +Train batch shape: torch.Size([256, 1, 28, 28]) +epoch 1/10: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 1597/1597 [03:24<00:00, 7.82it/s] +val acc: 0.8099 +epoch 2/10: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 1597/1597 [03:18<00:00, 8.05it/s] +val acc: 0.8378 +epoch 3/10: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 1597/1597 [03:17<00:00, 8.09it/s] +val acc: 0.8855 +epoch 4/10: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 1597/1597 [03:20<00:00, 7.97it/s] +val acc: 0.9180 +epoch 5/10: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 1597/1597 [03:20<00:00, 7.97it/s] +val acc: 0.9527 +epoch 6/10: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 1597/1597 [03:22<00:00, 7.88it/s] +val acc: 0.9635 +epoch 7/10: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 1597/1597 [03:22<00:00, 7.88it/s] +val acc: 0.9777 +epoch 8/10: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 1597/1597 [03:21<00:00, 7.91it/s] +val acc: 0.9854 +epoch 9/10: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 1597/1597 [03:21<00:00, 7.91it/s] +val acc: 0.9912 +epoch 10/10: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 1597/1597 [03:21<00:00, 7.91it/s] +val acc: 0.9928 +Best val acc: 0.9928 +[torch.onnx] Obtain model graph for `DigitNet([...]` with `torch.export.export(..., strict=False)`... +[torch.onnx] Obtain model graph for `DigitNet([...]` with `torch.export.export(..., strict=False)`... ✅ +[torch.onnx] Run decomposition... +[torch.onnx] Run decomposition... ✅ +[torch.onnx] Translate the graph into ONNX... +[torch.onnx] Translate the graph into ONNX... ✅ +Applied 1 of general pattern rewrite rules. +Exported: artifacts/sudoku_digitnet.onnx +Parity MAE: 1.0251999e-05 +``` + +## Summary +By running the training script you train the DigitNet CNN on the Sudoku digit dataset, steadily improving accuracy across epochs until the model surpasses 99% validation accuracy. The process builds on the earlier steps where we first defined the model architecture in digitnet_model.py and then prepared a dedicated training script to handle data loading, augmentation, optimization, and evaluation. During training the best-performing model state is saved, and at the end it is exported to the ONNX format with dynamic batch support. A parity check confirms that the ONNX and PyTorch versions produce virtually identical outputs (mean error ~1e-5). You now have a validated ONNX model (artifacts/sudoku_digitnet.onnx) and a PyTorch checkpoint (digitnet_best.pth), both ready for integration into the Sudoku image processing pipeline. Before moving on to grid detection and solving, however, we will first run standalone inference to confirm the model’s predictions on individual digit crops. diff --git a/content/learning-paths/mobile-graphics-and-gaming/onnx/05_inference.md b/content/learning-paths/mobile-graphics-and-gaming/onnx/05_inference.md new file mode 100644 index 0000000000..3a8fdc1426 --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/onnx/05_inference.md @@ -0,0 +1,249 @@ +--- +# User change +title: "Inference and Model Evaluation" + +weight: 6 + +layout: "learningpathall" +--- + +## Objective ## +In this section, we validate the digit recognizer by running inference on the validation dataset using both the PyTorch checkpoint and the exported ONNX model. We verify that PyTorch and ONNX Runtime produce consistent results, analyze class-level behavior using a confusion matrix, and generate visual diagnostics for debugging and documentation. This step acts as a final verification checkpoint before integrating the model into the full OpenCV-based Sudoku processing pipeline. + +Before introducing geometric processing, grid detection, and perspective correction, it is important to confirm that the digit recognizer works reliably in isolation. By validating inference and analyzing errors at the digit level, we ensure that any future issues in the end-to-end system can be attributed to image processing or geometry rather than the classifier itself. + +## Inference and Evaluation Script +Create a new file named 04_Test.py and paste the script below into it. This script evaluates the digit recognizer in a way that closely mirrors deployment conditions. It compares PyTorch and ONNX Runtime inference, measures accuracy on the validation dataset, and generates visual diagnostics that reveal both strengths and remaining failure modes of the model. + +```python +import os, numpy as np, torch +from torchvision import datasets, transforms +from torch.utils.data import DataLoader +from tqdm import tqdm +import matplotlib.pyplot as plt + +from digitnet_model import DigitNet + +DATA_DIR = "data" +ARTI_DIR = "artifacts" +os.makedirs(ARTI_DIR, exist_ok=True) + +ONNX_PATH = os.path.join(ARTI_DIR, "sudoku_digitnet.onnx") # fp32 + +# Same normalization as training (and force grayscale → 1 channel) +tfm_val = transforms.Compose([ + transforms.Grayscale(num_output_channels=1), + transforms.ToTensor(), + transforms.Normalize((0.5,), (0.5,)) +]) +val_ds = datasets.ImageFolder(os.path.join(DATA_DIR, "val"), transform=tfm_val) +val_loader = DataLoader(val_ds, batch_size=512, shuffle=False, num_workers=0) + +DIGIT_NAMES = [str(i) for i in range(10)] # 0 = blank, 1..9 = digits + + +def evaluate_pytorch(model, loader): + model.eval() + correct = total = 0 + with torch.no_grad(): + for x, y in loader: + pred = model(x).argmax(1) + correct += (pred == y).sum().item() + total += y.numel() + return correct / total if total else 0.0 + + +def confusion_matrix_onnx(onnx_model_path, loader): + import onnxruntime as ort + sess = ort.InferenceSession(onnx_model_path, providers=["CPUExecutionProvider"]) + mat = np.zeros((10, 10), dtype=np.int64) + total = 0 + correct = 0 + for x, y in tqdm(loader, desc="ONNX eval"): + # x: torch tensor [N,1,28,28] normalized to [-1,1] + inp = x.numpy().astype(np.float32) + logits = sess.run(["logits"], {"input": inp})[0] # [N,10] + pred = logits.argmax(axis=1) + y_np = y.numpy() + for t, p in zip(y_np, pred): + mat[t, p] += 1 + correct += (pred == y_np).sum() + total += y_np.size + acc = float(correct) / float(total) if total else 0.0 + return acc, mat + + +def plot_confusion_matrix(cm, classes=DIGIT_NAMES, normalize=False, title="Confusion matrix", fname=None): + """Plot confusion matrix. If normalize=True, rows sum to 1.""" + cm_plot = cm.astype("float") + if normalize: + row_sums = cm_plot.sum(axis=1, keepdims=True) + 1e-12 + cm_plot = cm_plot / row_sums + + plt.figure(figsize=(6, 5)) + plt.imshow(cm_plot, interpolation="nearest") + plt.title(title) + plt.colorbar() + tick_marks = np.arange(len(classes)) + plt.xticks(tick_marks, classes) + plt.yticks(tick_marks, classes) + + # Label each cell + thresh = cm_plot.max() / 2.0 + for i in range(cm_plot.shape[0]): + for j in range(cm_plot.shape[1]): + txt = f"{cm_plot[i, j]:.2f}" if normalize else f"{int(cm_plot[i, j])}" + plt.text(j, i, txt, + horizontalalignment="center", + verticalalignment="center", + fontsize=7, + color="white" if cm_plot[i, j] > thresh else "black") + + plt.ylabel("True label") + plt.xlabel("Predicted label") + plt.tight_layout() + if fname: + plt.savefig(fname, dpi=150) + print(f"Saved: {fname}") + plt.show() + + +def sample_predictions_onnx(onnx_path, dataset, k=24, seed=0): + """Show a grid of sample predictions (mix of correct and misclassified).""" + import onnxruntime as ort + rng = np.random.default_rng(seed) + sess = ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"]) + + # Over-sample candidates then choose some wrong + some right + idxs = rng.choice(len(dataset), size=min(k * 2, len(dataset)), replace=False) + imgs, ys, preds = [], [], [] + + for i in idxs: + x, y = dataset[i] # x: [1,28,28] after transforms; y: int + x_np = x.unsqueeze(0).numpy().astype(np.float32) # [1,1,28,28] + logits = sess.run(["logits"], {"input": x_np})[0] # [1,10] + p = int(np.argmax(logits, axis=1)[0]) + imgs.append(x.squeeze(0).numpy()) # [28,28] + ys.append(int(y)) + preds.append(p) + + mis_idx = [i for i, (t, p) in enumerate(zip(ys, preds)) if t != p] + cor_idx = [i for i, (t, p) in enumerate(zip(ys, preds)) if t == p] + picked = (mis_idx[:k // 2] + cor_idx[:k - len(mis_idx[:k // 2])])[:k] + if not picked: # fallback + picked = list(range(min(k, len(imgs)))) + + # Plot grid + import math + cols = 8 + rows = math.ceil(len(picked) / cols) + plt.figure(figsize=(cols * 1.6, rows * 1.8)) + for j, idx in enumerate(picked): + plt.subplot(rows, cols, j + 1) + plt.imshow(imgs[idx], cmap="gray") + t, p = ys[idx], preds[idx] + title = f"T:{t} P:{p}" + color = "green" if t == p else "red" + plt.title(title, color=color, fontsize=9) + plt.axis("off") + plt.tight_layout() + out = os.path.join(ARTI_DIR, "samples_grid.png") + plt.savefig(out, dpi=150) + print(f"Saved: {out}") + plt.show() + +def main(): + # Optional: evaluate the best PyTorch checkpoint for reference + pt_ckpt = os.path.join(ARTI_DIR, "digitnet_best.pth") + if os.path.exists(pt_ckpt): + model = DigitNet() + model.load_state_dict(torch.load(pt_ckpt, map_location="cpu")) + pt_acc = evaluate_pytorch(model, val_loader) + print(f"PyTorch val acc: {pt_acc:.4f}") + else: + print("No PyTorch checkpoint found; skipping PT eval.") + + # Evaluate ONNX fp32 + if os.path.exists(ONNX_PATH): + acc, cm = confusion_matrix_onnx(ONNX_PATH, val_loader) + print(f"ONNX fp32 val acc: {acc:.4f}") + print("Confusion matrix (rows=true, cols=pred):\n", cm) + + # Plots: counts + normalized + plot_confusion_matrix(cm, normalize=False, + title="ONNX fp32 – Confusion (counts)", + fname=os.path.join(ARTI_DIR, "cm_fp32_counts.png")) + plot_confusion_matrix(cm, normalize=True, + title="ONNX fp32 – Confusion (row-normalized)", + fname=os.path.join(ARTI_DIR, "cm_fp32_norm.png")) + + # Sample predictions grid + try: + sample_predictions_onnx(ONNX_PATH, val_ds, k=24) + except Exception as e: + print("Sample grid skipped:", e) + else: + print("Missing ONNX model:", ONNX_PATH) + +if __name__ == "__main__": + main() +``` + +The script first loads the validation dataset using the same preprocessing pipeline as training, including forced grayscale conversion to ensure a single input channel. It then optionally evaluates the best PyTorch checkpoint (digitnet_best.pth) to establish a reference accuracy. + +Next, the exported ONNX model (sudoku_digitnet.onnx) is loaded using ONNX Runtime and evaluated in batches. Because the model was exported with a dynamic batch dimension, inference can be performed efficiently on larger batches, which is representative of how the model will be used later in the pipeline. + +The script expects two things from the earlier steps: +1. A validation dataset stored under data/val/0..9/… +2. A trained model exported in previous step and stored under artifacts/ + * artifacts/digitnet_best.pth (optional, PyTorch weights) + * artifacts/sudoku_digitnet.onnx (required, ONNX model) + +When you run the script, it first loads the validation dataset using the same preprocessing as training, including forcing grayscale so the input has a single channel. It then optionally evaluates the PyTorch checkpoint to provide a reference accuracy. After that, it runs batched inference with ONNX Runtime, computes an overall accuracy, and builds a confusion matrix (true class vs predicted class) that reveals which digits are being confused. + +In addition to printing accuracy metrics, the script generates two types of diagnostic outputs: +1. Confusion matrix visualizations, saved as: + * artifacts/cm_fp32_counts.png (raw counts) + * artifacts/cm_fp32_norm.png (row-normalized) +2. A grid of example predictions, saved as: + *artifacts/samples_grid.png + +These artifacts provide both quantitative and qualitative insight into model performance. + +In the sample grid, each tile shows one crop together with its True label (T:) and Predicted label (P:), with correct predictions highlighted in green and mistakes highlighted in red. This makes it easy to quickly verify that the classifier behaves sensibly and to spot remaining failure modes. + +## Running the script +Run the evaluation script from the project root: + +```console +python3 04_Test.py +``` + +In the example below, the PyTorch and ONNX accuracies match exactly, confirming that the export process preserved model behavior. + +```console +python3 04_Test.py +PyTorch val acc: 0.9928 +ONNX eval: 100%|███████████████████████████████████████████████████████████| 32/32 [00:01<00:00, 21.06it/s] +ONNX fp32 val acc: 0.9928 +Confusion matrix (rows=true, cols=pred): + [[12623 7 0 0 0 0 0 0 0 0] + [ 0 420 0 0 0 0 0 0 0 0] + [ 0 0 331 0 4 0 1 0 0 0] + [ 0 1 0 332 0 1 0 0 0 0] + [ 0 0 0 0 460 0 0 0 0 0] + [ 0 1 0 1 0 486 2 0 0 0] + [ 1 0 0 0 0 19 387 0 1 2] + [ 0 1 0 0 0 0 0 375 0 0] + [ 0 0 0 0 0 6 27 0 297 10] + [ 0 1 0 0 0 14 10 0 7 372]] +Saved: artifacts/cm_fp32_counts.png +``` + +![img1](figures/01.png) +The confusion matrix provides more insight than a single accuracy number. Each row corresponds to the true class, and each column corresponds to the predicted class. A strong diagonal indicates correct classification. In this output, blank cells (class 0) are almost always recognized correctly, while the remaining errors occur primarily between visually similar printed digits such as 6, 8, and 9. + +This behavior is expected and indicates that the model has learned meaningful digit features. The remaining confusions are rare and can be addressed later through targeted augmentation or higher-resolution crops if needed. + +## Summary +With inference validated and error modes understood, the digit recognizer is now ready to be embedded into the full Sudoku image-processing pipeline, where OpenCV will be used to detect the grid, rectify perspective, segment cells, and run batched ONNX inference to reconstruct and solve complete puzzles. diff --git a/content/learning-paths/mobile-graphics-and-gaming/onnx/06_sudokuprocessor.md b/content/learning-paths/mobile-graphics-and-gaming/onnx/06_sudokuprocessor.md new file mode 100644 index 0000000000..1f5e686a3b --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/onnx/06_sudokuprocessor.md @@ -0,0 +1,448 @@ +--- +# User change +title: "Sudoku Processor. From Image to Solution" +weight: 7 +layout: "learningpathall" + +--- + +## Objective ## + +In this section, we integrate all previous components into a complete Sudoku processing pipeline. Starting from a full Sudoku image, we detect and rectify the grid, split it into individual cells, recognize digits using the ONNX model, and finally solve the puzzle using a deterministic solver. By the end of this step, you will have an end-to-end system that takes a photograph of a Sudoku puzzle and produces a solved board, along with visual outputs for debugging and validation. + +## Context +So far, we have: +1. Generated a synthetic, well-labeled Sudoku digit dataset, +2. Trained a lightweight CNN (DigitNet) to recognize digits and blanks, +3. Exported the model to ONNX with dynamic batch support, +4. Validated inference correctness and analyzed errors using confusion matrices. + +At this point, the digit recognizer is reliable in isolation. The remaining challenge is connecting vision with reasoning: extracting the Sudoku grid from an image, mapping each cell to a digit, and applying a solver. This section bridges that gap. + +## Overview of the pipeline +To implement the Sudoku processor, create the file (sudoku_processor.py) and paste the implementation below: + +```python +import cv2 as cv +import numpy as np +import onnxruntime as ort + +class SudokuProcessor: + def __init__( + self, + onnx_path: str, + input_size: int = 28, + warp_size: int = 450, + blank_class: int = 0, + blank_conf_threshold: float = 0.65, + providers=("CPUExecutionProvider",), + ): + """ + onnx_path: path to sudoku_digitnet.onnx + input_size: model input spatial size (28) + warp_size: size of rectified square board (e.g., 450 => each cell ~50px) + blank_class: class index used for blanks (0) + blank_conf_threshold: if model confidence < threshold, treat as blank (helps noisy cells) + """ + self.onnx_path = onnx_path + self.input_size = input_size + self.warp_size = warp_size + self.blank_class = blank_class + self.blank_conf_threshold = blank_conf_threshold + + self.sess = ort.InferenceSession(onnx_path, providers=list(providers)) + self.input_name = self.sess.get_inputs()[0].name # typically "input" + self.output_name = self.sess.get_outputs()[0].name # typically "logits" + + def process_image(self, bgr: np.ndarray, overlay: bool = True): + """ + Returns: + board (9x9 ints with 0 for blank), + solved_board (9x9 ints, or None if unsolved), + debug dict (warped, contours, etc.), + overlay_bgr (optional solution overlay) + """ + warped, H, quad = self.detect_and_warp_board(bgr) + cells = self.split_cells(warped) + board, conf = self.recognize_board(cells) + + solved = [row[:] for row in board] + ok = solve_sudoku(solved) + + overlay_img = None + if overlay and ok: + overlay_img = self.overlay_solution(bgr, H, board, solved) + + debug = { + "warped": warped, + "homography": H, + "quad": quad, + "confidence": conf, + } + return board, (solved if ok else None), debug, overlay_img + + # ----------------------------- + # Board detection / rectification + # ----------------------------- + def detect_and_warp_board(self, bgr: np.ndarray): + """ + Finds the largest Sudoku-like quadrilateral and warps it to a square. + Returns warped_board, homography, quad_points. + """ + gray = cv.cvtColor(bgr, cv.COLOR_BGR2GRAY) + blur = cv.GaussianBlur(gray, (5, 5), 0) + + # Strong binary image helps contour finding (works well for printed grids) + thr = cv.adaptiveThreshold( + blur, 255, cv.ADAPTIVE_THRESH_GAUSSIAN_C, cv.THRESH_BINARY_INV, 31, 7 + ) + + # Remove small noise, connect lines a bit + kernel = cv.getStructuringElement(cv.MORPH_RECT, (3, 3)) + thr = cv.morphologyEx(thr, cv.MORPH_CLOSE, kernel, iterations=2) + + contours, _ = cv.findContours(thr, cv.RETR_EXTERNAL, cv.CHAIN_APPROX_SIMPLE) + if not contours: + raise RuntimeError("No contours found. Try a clearer image or different thresholding.") + + # Pick the largest contour that approximates to 4 points + contours = sorted(contours, key=cv.contourArea, reverse=True) + quad = None + for c in contours[:20]: + peri = cv.arcLength(c, True) + approx = cv.approxPolyDP(c, 0.02 * peri, True) + if len(approx) == 4: + quad = approx.reshape(4, 2).astype(np.float32) + break + + if quad is None: + raise RuntimeError("Could not find a 4-corner Sudoku grid. Try a more fronto-parallel image.") + + quad = order_quad_points(quad) + + dst = np.array( + [[0, 0], [self.warp_size - 1, 0], [self.warp_size - 1, self.warp_size - 1], [0, self.warp_size - 1]], + dtype=np.float32, + ) + H = cv.getPerspectiveTransform(quad, dst) + warped = cv.warpPerspective(bgr, H, (self.warp_size, self.warp_size)) + + return warped, H, quad + + # ----------------------------- + # Cell splitting / preprocessing + # ----------------------------- + def split_cells(self, warped_bgr: np.ndarray): + """ + Splits a rectified square board into 81 cell images. + Returns list of (r, c, cell_bgr). + """ + cells = [] + step = self.warp_size // 9 + for r in range(9): + for c in range(9): + y0, y1 = r * step, (r + 1) * step + x0, x1 = c * step, (c + 1) * step + cell = warped_bgr[y0:y1, x0:x1].copy() + cells.append((r, c, cell)) + return cells + + def preprocess_cell(self, cell_bgr: np.ndarray): + """ + Produces a 28x28 float32 tensor in the same normalization as training: + grayscale -> [0,1] -> normalize to [-1,1] via (x-0.5)/0.5 + Also tries to suppress grid lines / borders by cropping margins. + """ + g = cv.cvtColor(cell_bgr, cv.COLOR_BGR2GRAY) + + # Crop a margin to remove grid lines/borders + h, w = g.shape + m = int(0.12 * min(h, w)) # ~12% margin + g = g[m:h - m, m:w - m] + + # Binarize & clean (helps isolate printed digits) + g_blur = cv.GaussianBlur(g, (3, 3), 0) + bw = cv.adaptiveThreshold(g_blur, 255, cv.ADAPTIVE_THRESH_GAUSSIAN_C, cv.THRESH_BINARY_INV, 21, 5) + + # Remove small specks + bw = cv.morphologyEx(bw, cv.MORPH_OPEN, np.ones((2, 2), np.uint8), iterations=1) + + # If almost empty => likely blank + if (bw > 0).sum() < 15: + # Return a near-empty input; classifier should produce blank + resized = cv.resize(g, (self.input_size, self.input_size), interpolation=cv.INTER_AREA) + else: + # Use bw mask to focus on digit; keep as grayscale for the model + resized = cv.resize(g, (self.input_size, self.input_size), interpolation=cv.INTER_AREA) + + x = resized.astype(np.float32) / 255.0 + x = (x - 0.5) / 0.5 # [-1,1] + x = x[None, None, :, :] # [1,1,H,W] + return x + + # ----------------------------- + # Inference + # ----------------------------- + def recognize_board(self, cells): + """ + Runs batched ONNX inference on 81 cells and returns: + board[9][9] with 0 for blank + conf[9][9] with max softmax probability + """ + xs = [] + coords = [] + for r, c, cell in cells: + coords.append((r, c)) + xs.append(self.preprocess_cell(cell)) + + X = np.concatenate(xs, axis=0).astype(np.float32) # [81,1,28,28] + logits = self.sess.run([self.output_name], {self.input_name: X})[0] # [81,10] + probs = softmax(logits, axis=1) + pred = probs.argmax(axis=1) + conf = probs.max(axis=1) + + board = [[0 for _ in range(9)] for _ in range(9)] + conf_grid = [[0.0 for _ in range(9)] for _ in range(9)] + for i, (r, c) in enumerate(coords): + p = int(pred[i]) + cf = float(conf[i]) + + # Optional safety: low-confidence => blank + if cf < self.blank_conf_threshold: + p = self.blank_class + + board[r][c] = p + conf_grid[r][c] = cf + + return board, conf_grid + + # ----------------------------- + # Overlay + # ----------------------------- + def overlay_solution(self, original_bgr, H, board, solved): + """ + Overlays ONLY the filled-in digits (where original board has 0). + """ + invH = np.linalg.inv(H) + overlay = original_bgr.copy() + + step = self.warp_size // 9 + # Create a transparent layer in warped space then map back + layer = np.zeros((self.warp_size, self.warp_size, 3), dtype=np.uint8) + + for r in range(9): + for c in range(9): + if board[r][c] != 0: + continue + d = solved[r][c] + # text placement in warped coordinates + x = int(c * step + step * 0.32) + y = int(r * step + step * 0.72) + cv.putText(layer, str(d), (x, y), cv.FONT_HERSHEY_SIMPLEX, 1.2, (0, 200, 0), 2, cv.LINE_AA) + + # Warp overlay layer back to original image + h0, w0 = original_bgr.shape[:2] + back = cv.warpPerspective(layer, invH, (w0, h0)) + + # Blend + mask = (back.sum(axis=2) > 0).astype(np.uint8) * 255 + mask3 = cv.merge([mask, mask, mask]) + overlay = np.where(mask3 > 0, cv.addWeighted(overlay, 0.6, back, 0.4, 0), overlay) + return overlay + + +# ----------------------------- +# Solver (backtracking) +# ----------------------------- +def solve_sudoku(board): + pos = find_empty(board) + if pos is None: + return True + r, c = pos + for v in range(1, 10): + if valid(board, r, c, v): + board[r][c] = v + if solve_sudoku(board): + return True + board[r][c] = 0 + return False + + +def find_empty(board): + for r in range(9): + for c in range(9): + if board[r][c] == 0: + return (r, c) + return None + + +def valid(board, r, c, v): + # row + for j in range(9): + if board[r][j] == v: + return False + # col + for i in range(9): + if board[i][c] == v: + return False + # box + br, bc = 3 * (r // 3), 3 * (c // 3) + for i in range(br, br + 3): + for j in range(bc, bc + 3): + if board[i][j] == v: + return False + return True + + +# ----------------------------- +# Utilities +# ----------------------------- +def order_quad_points(pts): + """ + Orders 4 points into: top-left, top-right, bottom-right, bottom-left. + """ + pts = np.array(pts, dtype=np.float32) + s = pts.sum(axis=1) + diff = np.diff(pts, axis=1).reshape(-1) + + tl = pts[np.argmin(s)] + br = pts[np.argmax(s)] + tr = pts[np.argmin(diff)] + bl = pts[np.argmax(diff)] + + return np.array([tl, tr, br, bl], dtype=np.float32) + + +def softmax(x, axis=1): + x = x - np.max(x, axis=axis, keepdims=True) + e = np.exp(x) + return e / (np.sum(e, axis=axis, keepdims=True) + 1e-12) +``` + +The Sudoku processor follows a sequence of steps: +1. Grid detection – find the outer Sudoku grid in the input image. +2. Perspective rectification – warp the grid to a square, top-down view. +3. Cell extraction – split the rectified grid into 81 cell images. +4. Digit recognition – run batched ONNX inference to classify each cell. +5. Board reconstruction – assemble a 9×9 numeric board. +6. Solving – apply a backtracking Sudoku solver. +7. Visualization – overlay the solution and render clean board images. + +We encapsulate the entire pipeline in a reusable class called SudokuProcessor. This class loads the ONNX model once and exposes a single high-level method that processes an input image and returns both intermediate results and final outputs. + +Conceptually, the processor: +* Accepts a BGR image, +* Returns the recognized board, the solved board (if solvable), and optional visual overlays. + +This design keeps inference fast and makes the processor easy to integrate later into an Android application or embedded system. + +## Grid detection and rectification +The first task is to locate the Sudoku grid in the image. We convert the image to grayscale, apply adaptive thresholding, and use contour detection to find large rectangular shapes. The largest contour that approximates a quadrilateral is assumed to be the Sudoku grid. + +Once the four corners are identified, we compute a perspective transform and warp the grid into a square image. This rectified representation removes camera tilt and perspective distortion, allowing all subsequent steps to assume a fixed geometry. + +We order the four corners consistently (top-left → top-right → bottom-right → bottom-left) before computing the perspective transform. + +## Splitting the grid into cells +After rectification, the grid is divided evenly into a 9×9 array. Each cell is cropped based on its row and column index. At this stage, every cell corresponds to one Sudoku position and is ready for preprocessing and classification. + +Each cell undergoes light preprocessing before inference: +* Conversion to grayscale, +* Cropping of a small margin to suppress grid lines, +* Adaptive thresholding and morphological cleanup to isolate printed digits, +* Resizing to the model’s input size (28×28), +* Normalization to match the training distribution. + +We crop a margin to suppress grid lines, because grid strokes can dominate the digit pixels and cause systematic misclassification. Cells with very little foreground content are treated as blank candidates, reducing false digit detections in empty cells. + +## Batched ONNX inference +All 81 cell tensors are stacked into a single batch and passed to ONNX Runtime in one call. Because the model was exported with a dynamic batch dimension, this batched inference is efficient and mirrors how the model will be used in production. + +The output logits are converted to probabilities, and the most likely class is selected for each cell. Optionally, a confidence threshold can be applied so that low-confidence predictions are treated as blanks. + +The result is a 9×9 board where: +* 0 represents a blank cell, +* 1–9 represent recognized digits. + +## Solving the Sudoku +With the recognized board constructed, we apply a classic backtracking Sudoku solver. This solver deterministically fills empty cells while respecting Sudoku constraints (row, column, and 3×3 block rules). + +If the solver succeeds, we obtain a complete solution. If it fails, the failure usually indicates one or more recognition errors, which can be diagnosed using the intermediate visual outputs. + +## Visualization and outputs +The processor saves several artifacts to help debugging and demonstration: +- `artifacts/warped.png` – rectified top-down view of the Sudoku grid. +- `artifacts/overlay_solution.png` – solution digits overlaid onto the original image (if solved). +- (Optional) `artifacts/recognized_board.png`, `artifacts/solved_board.png`, `artifacts/boards_side_by_side.png` – clean board renderings if you enabled those helpers. + +The driver script below saves warped.png and overlay_solution.png by default. + +## Running the processor +A small driver script (05_RunSudokuProcessor.py) demonstrates how to use the SudokuProcessor: + +```python +import os +import cv2 as cv + +from sudoku_processor import SudokuProcessor + +def print_board(board, title="Board"): + print("\n" + title) + for r in range(9): + row = "" + for c in range(9): + v = board[r][c] + row += (". " if v == 0 else f"{v} ") + if c % 3 == 2 and c != 8: + row += "| " + print(row.strip()) + if r % 3 == 2 and r != 8: + print("-" * 21) + + +def main(): + # Use any image path you like: + # - a real photo + # - a synthetic grid, e.g. data/grids/val/000001_cam.png + img_path = "data/grids/val/000001_cam.png" + onnx_path = os.path.join("artifacts", "sudoku_digitnet.onnx") + + bgr = cv.imread(img_path) + if bgr is None: + raise RuntimeError(f"Could not read image: {img_path}") + + proc = SudokuProcessor(onnx_path=onnx_path, warp_size=450, blank_conf_threshold=0.65) + + board, solved, dbg, overlay = proc.process_image(bgr, overlay=True) + + print_board(board, "Recognized board") + if solved is None: + print("\nSolver failed (board might contain recognition errors).") + else: + print_board(solved, "Solved board") + + # Save debug outputs + cv.imwrite("artifacts/warped.png", dbg["warped"]) + if overlay is not None: + cv.imwrite("artifacts/overlay_solution.png", overlay) + print("\nSaved: artifacts/overlay_solution.png") + print("Saved: artifacts/warped.png") + +if __name__ == "__main__": + main() +``` + +You simply provide the path to a Sudoku image and the ONNX model, and the script saves all intermediate and final results to the artifacts/ directory. + +Representational result is shown below: + +![img2](figures/02.png) + +## Summary +By completing this section, you have built a full vision-to-solution Sudoku system: +1. A trained and validated ONNX digit recognizer, +2. A robust OpenCV-based image processing pipeline, +3. A deterministic solver, +4. Clear visual diagnostics at every stage. + +In the next step of the learning path, we will focus on optimization and deployment. diff --git a/content/learning-paths/mobile-graphics-and-gaming/onnx/07_optimisation.md b/content/learning-paths/mobile-graphics-and-gaming/onnx/07_optimisation.md new file mode 100644 index 0000000000..a8e38c6ca7 --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/onnx/07_optimisation.md @@ -0,0 +1,424 @@ +--- +title: "Model Enhancements and Optimizations" +weight: 8 +layout: "learningpathall" +--- + +## Objective +In this section, we improve the Sudoku system from a working prototype into something that is faster, smaller, and more robust on Arm64-class hardware. We start by measuring a baseline, then apply ONNX Runtime optimizations and quantization, and finally address the most common real bottleneck: image preprocessing. At each step we re-check accuracy and solve rate so performance gains don’t come at the cost of correctness. + +## Establish a baseline +Before applying any optimizations, it is essential to understand where time is actually being spent in the Sudoku pipeline. Without this baseline, it is impossible to tell whether an optimization is effective or whether it simply shifts the bottleneck elsewhere. + +In the current system, the total latency of processing a single Sudoku image is composed of four main stages: +* Grid detection and warping – locating the outer Sudoku grid and rectifying it using a perspective transform. This step relies entirely on OpenCV and depends on image resolution, lighting, and grid clarity. +* Cell preprocessing – converting each of the 81 cells into a normalized 28×28 grayscale input for the neural network. This includes cropping margins, thresholding, and morphological operations. In practice, this stage is often the dominant cost. +* ONNX inference – running the digit recognizer on all 81 cells as a single batch. Thanks to dynamic batch support, this step is typically fast compared to preprocessing. +* Solving – applying a backtracking Sudoku solver to the recognized board. This step is usually negligible in runtime, unless recognition errors lead to difficult or contradictory boards. + +To quantify these contributions, we will add simple timing measurements around each stage of the pipeline using a high-resolution clock (time.perf_counter()). For each processed image, we will print a breakdown: +* warp_ms – time spent on grid detection and perspective rectification +* preprocess_ms – total time spent preprocessing all 81 cells +* onnx_ms – time spent running batched ONNX inference +* solve_ms – time spent solving the Sudoku +* split_ms – time spent splitting the warped grid into 81 cells +* total_ms – end-to-end processing time + +## Performance measurements +Open the sudoku_processor.py and add the following import + +```python +import time +``` + +Then, modify the process_image as follows +```python +def process_image(self, bgr: np.ndarray, overlay: bool = True): + """ + Returns: + board (9x9 ints with 0 for blank), + solved_board (9x9 ints, or None if unsolved), + debug dict (warped, homography, confidence, timing), + overlay_bgr (optional solution overlay) + """ + timing = {} + + t_total0 = time.perf_counter() + + # --- Grid detection + warp --- + t0 = time.perf_counter() + warped, H, quad = self.detect_and_warp_board(bgr) + timing["warp_ms"] = (time.perf_counter() - t0) * 1000.0 + + # --- Cell splitting --- + t0 = time.perf_counter() + cells = self.split_cells(warped) + timing["split_ms"] = (time.perf_counter() - t0) * 1000.0 + + # --- Preprocessing (81 cells) --- + t0 = time.perf_counter() + xs = [] + coords = [] + for r, c, cell in cells: + coords.append((r, c)) + xs.append(self.preprocess_cell(cell)) + X = np.concatenate(xs, axis=0).astype(np.float32) # [81,1,28,28] + timing["preprocess_ms"] = (time.perf_counter() - t0) * 1000.0 + + # --- ONNX inference --- + t0 = time.perf_counter() + logits = self.sess.run([self.output_name], {self.input_name: X})[0] + timing["onnx_ms"] = (time.perf_counter() - t0) * 1000.0 + + # --- Postprocess predictions --- + probs = softmax(logits, axis=1) + pred = probs.argmax(axis=1) + conf = probs.max(axis=1) + + board = [[0 for _ in range(9)] for _ in range(9)] + conf_grid = [[0.0 for _ in range(9)] for _ in range(9)] + for i, (r, c) in enumerate(coords): + p = int(pred[i]) + cf = float(conf[i]) + if cf < self.blank_conf_threshold: + p = self.blank_class + board[r][c] = p + conf_grid[r][c] = cf + + # --- Solve --- + t0 = time.perf_counter() + solved = [row[:] for row in board] + ok = solve_sudoku(solved) + timing["solve_ms"] = (time.perf_counter() - t0) * 1000.0 + + # --- Overlay (optional) --- + overlay_img = None + if overlay and ok: + t0 = time.perf_counter() + overlay_img = self.overlay_solution(bgr, H, board, solved) + timing["overlay_ms"] = (time.perf_counter() - t0) * 1000.0 + else: + timing["overlay_ms"] = 0.0 + + timing["total_ms"] = (time.perf_counter() - t_total0) * 1000.0 + + debug = { + "warped": warped, + "homography": H, + "quad": quad, + "confidence": conf_grid, + "timing": timing, + } + + return board, (solved if ok else None), debug, overlay_img +``` + +Finally, print the timings in the 05_RunSudokuProcessor.py: +```python +def main(): + # Use any image path you like: + # - a real photo + # - a synthetic grid, e.g. data/grids/val/000001_cam.png + img_path = "data/grids/val/000002_cam.png" + onnx_path = os.path.join("artifacts", "sudoku_digitnet.onnx") + + bgr = cv.imread(img_path) + if bgr is None: + raise RuntimeError(f"Could not read image: {img_path}") + + proc = SudokuProcessor(onnx_path=onnx_path, warp_size=450, blank_conf_threshold=0.65) + + board, solved, dbg, overlay = proc.process_image(bgr, overlay=True) + + print_board(board, "Recognized board") + if solved is None: + print("\nSolver failed (board might contain recognition errors).") + else: + print_board(solved, "Solved board") + + # Save debug outputs + cv.imwrite("artifacts/warped.png", dbg["warped"]) + if overlay is not None: + cv.imwrite("artifacts/overlay_solution.png", overlay) + print("\nSaved: artifacts/overlay_solution.png") + print("Saved: artifacts/warped.png") + + tim = dbg["timing"] + print( + f"warp={tim['warp_ms']:.1f} ms | " + f"preprocess={tim['preprocess_ms']:.1f} ms | " + f"onnx={tim['onnx_ms']:.1f} ms | " + f"solve={tim['solve_ms']:.1f} ms | " + f"total={tim['total_ms']:.1f} ms" + ) + +if __name__ == "__main__": + main() +``` + +The sample output will look as follows: +```output +python3 05_RunSudokuProcessor.py + +Recognized board +. . . | 7 . . | 6 . . +. . 4 | . . . | 1 . 9 +. . . | 1 5 . | . . . +--------------------- +. . . | . 1 . | . . . +. . . | . . . | . . . +3 . . | . . . | . 6 . +--------------------- +7 . . | . . . | . . . +. . 9 | . . . | . . . +. . . | . . . | . . . + +Solved board +1 2 3 | 7 4 9 | 6 5 8 +5 6 4 | 2 3 8 | 1 7 9 +8 9 7 | 1 5 6 | 2 3 4 +--------------------- +2 4 5 | 6 1 3 | 8 9 7 +9 1 6 | 4 8 7 | 3 2 5 +3 7 8 | 5 9 2 | 4 6 1 +--------------------- +7 3 1 | 8 2 5 | 9 4 6 +4 5 9 | 3 6 1 | 7 8 2 +6 8 2 | 9 7 4 | 5 1 3 + +Saved: artifacts/overlay_solution.png +Saved: artifacts/warped.png +warp=11.9 ms | preprocess=3.3 ms | onnx=1.9 ms | solve=3.1 ms | total=48.2 ms +``` + +## Folder benchmark +The single-image measurements introduced earlier are useful for understanding the rough structure of the pipeline and for verifying that ONNX inference is not the main computational bottleneck. In our case, batched ONNX inference typically takes less than 2 ms, while grid detection, warping, and preprocessing dominate the runtime. However, individual measurements can be noisy due to caching effects, operating system scheduling, and Python overhead. + +To obtain more reliable performance numbers, we extend the evaluation to multiple images and compute aggregated statistics. This allows us to track not only average performance, but also variability and tail latency, which are particularly important for interactive applications. + +To do this, we add two helper functions to 05_RunSudokuProcessor.py, and make sure you have import glob and import numpy as np at the top of the runner script. + +The first function, summarize, computes basic statistics from a list of timing measurements: +* mean – average runtime +* median – robust central tendency +* p90 / p95 – tail latency (90th and 95th percentiles), which indicate how bad the slow cases are + +```python +def summarize(values): + values = np.asarray(values, dtype=np.float64) + return { + "mean": float(values.mean()), + "median": float(np.median(values)), + "p90": float(np.percentile(values, 90)), + "p95": float(np.percentile(values, 95)), + } +``` + +The second function, benchmark_folder, runs the full Sudoku pipeline on a collection of images and aggregates timing results across multiple runs: + +```python +def benchmark_folder(proc, folder_glob, limit=100, warmup=10, overlay=False): + paths = sorted(glob.glob(folder_glob)) + if not paths: + raise RuntimeError(f"No images matched: {folder_glob}") + paths = paths[:limit] + + # Warmup + for p in paths[:min(warmup, len(paths))]: + bgr = cv.imread(p) + if bgr is None: + continue + proc.process_image(bgr, overlay=overlay) + + # Benchmark + agg = {k: [] for k in ["warp_ms", "preprocess_ms", "onnx_ms", "solve_ms", "total_ms"]} + solved_cnt = 0 + total_cnt = 0 + + for p in paths: + bgr = cv.imread(p) + if bgr is None: + continue + + board, solved, dbg, _ = proc.process_image(bgr, overlay=overlay) + tim = dbg["timing"] + + for k in agg: + agg[k].append(tim[k]) + + total_cnt += 1 + if solved is not None: + solved_cnt += 1 + + print(f"\nSolved {solved_cnt}/{total_cnt} ({(solved_cnt/total_cnt*100.0 if total_cnt else 0):.1f}%)") + + print("\nTiming summary (ms):") + for k in ["warp_ms", "preprocess_ms", "onnx_ms", "solve_ms", "total_ms"]: + s = summarize(agg[k]) + print(f"{k:14s} mean={s['mean']:.2f} median={s['median']:.2f} p90={s['p90']:.2f} p95={s['p95']:.2f}") +``` + +Finally, we invoke the benchmark in the main() function: + +```python +def main(): + onnx_path = os.path.join("artifacts", "sudoku_digitnet.onnx") + + proc = SudokuProcessor(onnx_path=onnx_path, warp_size=450, blank_conf_threshold=0.65) + + benchmark_folder(proc, "data/grids/val/*_cam.png", limit=30, warmup=10, overlay=False) + +if __name__ == "__main__": + main() +``` + +This evaluates the processor on a representative subset of camera-like validation grids, prints aggregated timing statistics, and reports the overall solve rate. + +Aggregated benchmarks provide a much more accurate picture than single measurements, especially when individual stages take only a few milliseconds. By reporting median and tail latencies, you can see whether occasional slow cases exist and whether an optimization truly improves user-perceived performance. Percentiles are particularly useful when a few slow cases exist (e.g., harder solves), because they reveal tail latency. These results form a solid quantitative baseline that you can reuse to evaluate every optimization that follows. + +Here is the sample output of the updated script: +```output +python3 05_RunSudokuProcessor.py + +Solved 30/30 (100.0%) + +Timing summary (ms): +warp_ms mean=10.25 median=10.27 p90=10.57 p95=10.59 +preprocess_ms mean=3.01 median=2.98 p90=3.16 p95=3.21 +onnx_ms mean=1.27 median=1.24 p90=1.30 p95=1.45 +solve_ms mean=74.76 median=2.02 p90=48.51 p95=74.82 +total_ms mean=89.41 median=16.97 p90=62.95 p95=89.43 +``` + +Notice that solve_ms (and therefore total_ms) has a much larger mean than median. This indicates a small number of outliers where the solver takes significantly longer. In practice, this occurs when one or more digits are misrecognized, forcing the backtracking solver to explore many branches before finding a solution (or failing). For interactive applications, median and p95 latency are more informative than the mean, as they better reflect typical user experience. + +## ONNX Runtime session optimizations +Now that you can measure onnx_ms and total_ms, the first low-effort improvement is to enable ONNX Runtime’s built-in graph optimizations and tune CPU threading. These changes do not modify the model, but can reduce inference overhead and improve throughput. + +In sudoku_processor.py, update the ONNX Runtime session initialization in __init__ to use SessionOptions: +```python +so = ort.SessionOptions() +so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL + +self.sess = ort.InferenceSession(onnx_path, sess_options=so, providers=list(providers)) +``` + +Re-run 05_RunSudokuProcessor.py and compare onnx_ms and total_ms to the baseline. + +```output +python3 05_RunSudokuProcessor.py + +Solved 30/30 (100.0%) + +Timing summary (ms): +warp_ms mean=10.43 median=10.36 p90=10.89 p95=10.96 +preprocess_ms mean=3.13 median=3.11 p90=3.34 p95=3.42 +onnx_ms mean=1.28 median=1.26 p90=1.37 p95=1.47 +solve_ms mean=78.61 median=2.01 p90=50.15 p95=77.87 +total_ms mean=93.58 median=17.06 p90=65.10 p95=92.55 +``` + +This result is expected for such a small model: ONNX inference is already efficient, and the dominant costs lie in image preprocessing and occasional solver backtracking. This highlights why system-level profiling is essential before focusing on model-level optimizations. + +## Quantize the model (FP32 -> INT8) +Quantization is one of the most impactful optimizations for Arm64 and mobile deployments because it reduces both model size and compute cost. For CNNs, the most compatible approach is static INT8 quantization in QDQ format. This uses a small calibration set to estimate activation ranges and typically works well across runtimes. + +Create a small script 06_QuantizeModel.py: + +```python +import os, glob +import numpy as np +import cv2 as cv + +from onnxruntime.quantization import ( + quantize_static, CalibrationDataReader, QuantFormat, QuantType +) + +ARTI_DIR = "artifacts" +FP32_PATH = os.path.join(ARTI_DIR, "sudoku_digitnet.onnx") +INT8_PATH = os.path.join(ARTI_DIR, "sudoku_digitnet.int8.onnx") + +# ---- Calibration data reader ---- +class SudokuCalibReader(CalibrationDataReader): + def __init__(self, folder_glob="data/train/0/*.png", limit=500, input_name="input", input_size=28): + self.input_name = input_name + self.input_size = input_size + + paths = sorted(glob.glob(folder_glob))[:limit] + self._iter = iter(paths) + + def get_next(self): + try: + p = next(self._iter) + except StopIteration: + return None + + g = cv.imread(p, cv.IMREAD_GRAYSCALE) + if g is None: + return self.get_next() + + g = cv.resize(g, (self.input_size, self.input_size), interpolation=cv.INTER_AREA) + x = g.astype(np.float32) / 255.0 + x = (x - 0.5) / 0.5 + x = x[None, None, :, :] # [1,1,28,28] + return {self.input_name: x} + +# ---- Run quantization ---- +reader = SudokuCalibReader(folder_glob="data/train/*/*.png", limit=1000) + +print("Quantizing (QDQ static INT8)...") +quantize_static( + model_input=FP32_PATH, + model_output=INT8_PATH, + calibration_data_reader=reader, + quant_format=QuantFormat.QDQ, # key: keep Conv as Conv with Q/DQ wrappers + activation_type=QuantType.QInt8, + weight_type=QuantType.QInt8, + per_channel=True # usually helps conv accuracy +) + +print("Saved:", INT8_PATH) +``` + +Run python 06_QuantizeModel.py + +Then update the runner script to point to the quantized model: + +```python +onnx_path = os.path.join("artifacts", "sudoku_digitnet.int8.onnx") +``` + +Re-run the processor and compare: +* onnx_ms (should improve or remain similar) +* total_ms +* solve success (should remain stable) + +Also compare file sizes: +```console +ls -lh artifacts/sudoku_digitnet.onnx artifacts/sudoku_digitnet.int8.onnx +``` +Even when inference time changes only modestly, size reduction is typically significant and matters for Android packaging. + +In this pipeline, quantization primarily reduces model size and improves deployability, while runtime speedups may be modest because inference is already a small fraction of the total latency. + +## Preprocessing-focused optimizations (highest impact) +The measurements above show that ONNX inference accounts for only a small fraction of the total runtime. In practice, the largest performance gains come from optimizing image preprocessing. + +The most effective improvements include: +- Converting the rectified board to grayscale **once**, instead of converting each cell independently. +- Adding an early “blank cell” check to skip expensive thresholding and morphology for empty cells. +- Using simpler thresholding (e.g., Otsu) on clean images, and reserving adaptive thresholding for difficult lighting conditions. +- Reducing or conditionally disabling morphological operations when cells already appear clean. + +These changes typically reduce `preprocess_ms` more than any model-level optimization, and therefore have the greatest impact on end-to-end latency. + +## Summary +In this section, we transformed the Sudoku solver from a functional prototype into a system with measurable, well-understood performance characteristics. By instrumenting the pipeline with fine-grained timing, we identified where computation is actually spent and established a quantitative baseline. + +We showed that: +- Batched ONNX inference is already efficient (≈1–2 ms per board). +- Image preprocessing dominates runtime and offers the largest optimization potential. +- Solver backtracking introduces rare but significant tail-latency outliers. +- ONNX Runtime optimizations and INT8 quantization improve deployability, even when raw inference speed gains are modest. + +Most importantly, we demonstrated a systematic optimization workflow: **measure first, optimize second, and always re-validate correctness**. With performance, robustness, and accuracy validated, the Sudoku pipeline is now ready for its final step—deployment as a fully on-device Android application. \ No newline at end of file diff --git a/content/learning-paths/mobile-graphics-and-gaming/onnx/08_android.md b/content/learning-paths/mobile-graphics-and-gaming/onnx/08_android.md new file mode 100644 index 0000000000..b87e820384 --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/onnx/08_android.md @@ -0,0 +1,1059 @@ +--- +# User change +title: "Android Deployment. From Model to App" + +weight: 9 + +layout: "learningpathall" +--- + +## Objective ## +In this section, we transition from a desktop prototype to a fully on-device Android application. The goal is to demonstrate how the optimized Sudoku pipeline—image preprocessing, ONNX inference, and deterministic solving—can be packaged and executed entirely on a mobile device, without relying on any cloud services. + +Rather than starting with a live camera feed, we begin with a fixed input bitmap that was generated earlier in the learning path. This approach allows us to focus on correctness, performance, and integration details before introducing additional complexity such as camera permissions, real-time capture, and varying lighting conditions. By keeping the input controlled, we can verify that the Android implementation faithfully reproduces the behavior observed in Python. + +Over the course of this section, we will: +1. Create a new Android project and add the required dependencies. +2. Bundle the trained ONNX model and a sample Sudoku image with the application. +3. Implement a minimal user interface that loads the image and triggers the solver. +4. Re-implement the Sudoku processing pipeline on Android, including preprocessing, batched ONNX inference, and solving. +5. Display the solved result as an image, confirming that the entire pipeline runs locally on the device. + +By the end of this section, you will have a working Android app that takes a Sudoku image, runs neural network inference and solving on-device, and displays the solution. This completes the learning path by showing how a trained and optimized ONNX model can be deployed in a real mobile application, closing the loop from data generation and training to practical, end-user deployment. + +## Project creation +We start by creating a new Android project using Android Studio. This project will host the Sudoku solver application and serve as the foundation for integrating ONNX Runtime and OpenCV. + +1. Create a new project: +* Open Android Studio and click New Project. +* In the Templates screen, select Phone and Tablet, then choose Empty Views Activity. +![img3](figures/03.png) + +This template creates a minimal Android application without additional UI components, which is ideal for a focused, step-by-step integration. + +* Click Next to proceed to the project configuration screen. + +2. Configure the project. In the configuration screen, fill in the fields as follows: +* Name: SudokuSolverOnnx. This is the application name that will appear in Android Studio and on the device. +* Package name: com.arm.sudokusolveronnx. This package name clearly reflects the purpose of the app and its use of ONNX on Arm platforms. +* Save location. Choose a convenient directory on your system (for example, your repositories folder). +* Language: Kotlin. Kotlin is the recommended language for modern Android development and integrates cleanly with ONNX Runtime APIs. +* Minimum SDK: API 24 (Android 7.0 – Nougat). This provides wide device coverage while remaining compatible with ONNX Runtime and OpenCV. +* Build configuration language: Kotlin DSL (build.gradle.kts). We use the Kotlin DSL for Gradle, which is now the recommended option. + +![img4](figures/04.png) + +* After confirming these settings, click Finish. Android Studio will create the project and generate a basic MainActivity along with the necessary Gradle files. + +## View +We now define the user interface of the Android application. The goal of this view is to remain intentionally simple while clearly exposing the end-to-end Sudoku workflow. The interface will consist of: +* A button row at the top that allows the user to load a Sudoku image and trigger the solver. +* A status text area used to display short messages (for example, whether an image has been loaded or the puzzle has been solved). +* An input image view that displays the selected Sudoku bitmap. +* An output image view that displays the solved result. + +This layout is sufficient to validate that the ONNX model, preprocessing pipeline, and solver are all working correctly on Android before adding more advanced features such as camera input or animations. + +To define the view, open the file res/layout/activity_main.xml and replace its contents with the following layout definition: +```xml + + + + + + + + + +