Wokwi Featured Functions (RDT-568) #233

hfudev · 2023-10-24T08:56:27Z

Wokwi support more functionalities other than serial I/O. like pushing buttons, set/get the state of sensors (described here)

Once the API is settled, implement this in pytest-embedded-wokwi

hayschan · 2024-01-08T09:56:24Z

Hello Hanxi,

Thank you for your response in the previous discussion regarding the CI/CD docker.

Over the past few days, I've been delving into the Unity unit testing framework and pytest-embedded as part of our ESP-IDF software development. I'm currently working on a project that involves testing a matrix keyboard, and I've successfully completed the development of the test_app. To streamline the testing process and eliminate the need for manual intervention (i.e., physically pressing the keyboard), I am considering the implementation of Wokwi.

However, I've encountered a couple of areas where additional resources or documentation would be immensely helpful:

API Documentation: The API section for pytest-embedded on Wokwi appears to be incomplete or unpopulated at this time.
Practical Examples: The current repository lacks examples that demonstrate the integration of Wokwi with pytest-embedded.

Furthermore, the existing examples, such as confirming a "Hello World" output, don't fully represent complex, real-world scenarios. It would be highly advantageous to have examples that cover more intricate use cases, like I2C interaction or WiFi connectivity testing, timers.

I believe addressing these areas would significantly enhance the utility and applicability of pytest-embedded for more complex development environments.

Looking forward to further updates or guidance on these matters.

Best regards.

hfudev · 2024-01-08T13:43:05Z

Hi @hayschan! Thank you for your input.

API Documentation

pytest-embedded-wokwi is a wrapper of the wokwi-cli. For now both wokwi-cli and pytest-embedded-wokwi are in alpha. We're working together with @urish to expose more APIs for testing.

Practical Examples

Yes as you said, we're lack of real world examples here in this repo. We will improve it in a forseeable future. For now if you're working together with ESP-IDF projects, you may take a look at the example projects in ESP-IDF repo: wifi getting started example

please note that the pytest.mark.esp32 marker is not applicable with vanilla pytest-embedded. We wrote some customized conftest.py in the esp-idf repo to support the syntactic sugar. You may use

import pytest

pytest.mark.parametrize('target', [
    'esp32',
], indirect=True)

instead.

urish · 2024-01-15T12:06:53Z

Hello! I just created a demo repo, based on espressif's gh-esp-test-template, which shows a complete setup using pytest embedded together with Wokwi in GitHub CI:

https://github.com/wokwi/wokwi-esp-test-template

It supports (almost) all espressif's ESP32 chips (including the not-yet-released ESP32-P4), and could serve for building more elaborate examples in the future.

Example workflow run (with 7 targets): https://github.com/wokwi/wokwi-esp-test-template/actions/runs/7528389475

hayschan · 2024-01-16T13:27:19Z

based on espressif's gh-esp-test-template, which shows a complete setup using pytest embedded together with Wokwi in GitHub CI:

To make sure everything is working at least locally, I have tried to run the Wokwi CLI on my computer.

I have two files that is important:

test.cpp for writing the TEST_CASE
pytest_test.py for using the pytest-embedded API

I encountered some problem.

Original problem

In my test.cpp, I have serveral test cases and app_main.

The app_main() uses unity_run_menu().

extern "C" void app_main(void)
{
    unity_run_menu();
}

And the pytest-embedded python file uses run_all_single_board_cases().

import pytest
from pytest_embedded import Dut

def test_aip1629(dut: Dut)-> None:
    dut.run_all_single_board_cases()

Then, this will happen:

I built the app by idf.py build, and then run pytest and wokwi using pytest -s --embedded-services idf,wokwi --tb short.
The app is uploaded to Wokwi CLI with success

Received the following error very quickly

2024-01-16 21:21:05 Wokwi CLI v0.8.0
2024-01-16 21:21:06 Connected to Wokwi Simulation API 1.0.0-20240114
2024-01-16 21:12:18 Press ENTER to see the list of tests.
FAILED

=================================================== FAILURES =================================================== 
_________________________________________________ test_aip1629 _________________________________________________ 
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:75: in wrapper
    index = func(self, pattern, *args, **kwargs)  # noqa
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:120: in expect
    return self.pexpect_proc.expect(pattern, **kwargs)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\spawnbase.py:354: in expect
    return self.expect_list(compiled_pattern_list,
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\spawnbase.py:383: in expect_list
    return exp.expect_loop(timeout)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\expect.py:181: in expect_loop
    return self.timeout(e)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\expect.py:144: in timeout
    raise exc
E   pexpect.exceptions.TIMEOUT: <pytest_embedded.log.PexpectProcess object at 0x00000181F61CF350>
E   searcher: searcher_re:
E       0: re.compile(b"Here's the test menu, pick your combo:(.+)Enter test for running.")
E   <pytest_embedded.log.PexpectProcess object at 0x00000181F61CF350>
E   searcher: searcher_re:
E       0: re.compile(b"Here's the test menu, pick your combo:(.+)Enter test for running.")

The above exception was the direct cause of the following exception:
pytest_aip1629.py:38: in test_aip1629
    dut.run_all_single_board_cases()
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:428: in run_all_single_board_cases
    for case in self.test_menu:
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:243: in test_menu
    self._test_menu = self._parse_test_menu()
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:125: in _parse_test_menu
    res = self.confirm_write(trigger, expect_pattern=pattern)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:102: in confirm_write
    raise err
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:94: in confirm_write
    res = self.expect(expect_pattern, timeout=timeout)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:82: in wrapper
    raise e.__class__(debug_str) from e
E   pexpect.exceptions.TIMEOUT: Not found "Here's the test menu, pick your combo:(.+)Enter test for running."    
E   Bytes in current buffer (color code eliminated): .
=========================================== short test summary info ============================================ 
FAILED pytest_aip1629.py::test_aip1629 - pexpect.exceptions.TIMEOUT: Not found "Here's the test menu, pick your combo:(.+)Enter test for running."

My way to solve this

By simply changing the unity functions used in app_main from unity_run_menu() to unity_run_all_tests(), all test cases could be run successfully.

extern "C" void app_main(void)
{
-    unity_run_menu();
+    unity_run_all_tests();
}

New error occured

Strangely, even after running all tests, it will still fail with TIMEOUT with the following error code:

2024-01-16 21:21:32 ./main/aip1629_test.cpp:331:Test Display During Heating:PASS
2024-01-16 21:21:32 Running Test Invalid Inputs...
2024-01-16 21:21:32 ./main/aip1629_test.cpp:340:Test Invalid Inputs:PASS
2024-01-16 21:21:32 I (10950) main_task: Returned from app_main()
FAILED

=================================================== FAILURES =================================================== 
_________________________________________________ test_aip1629 _________________________________________________ 
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:75: in wrapper
    index = func(self, pattern, *args, **kwargs)  # noqa
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:141: in expect_exact
    return self.pexpect_proc.expect_exact(pattern, **kwargs)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\spawnbase.py:432: in expect_exact
    return exp.expect_loop(timeout)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\expect.py:181: in expect_loop
    return self.timeout(e)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\expect.py:144: in timeout
    raise exc
E   pexpect.exceptions.TIMEOUT: <pytest_embedded.log.PexpectProcess object at 0x0000019A33A84B50>
E   searcher: searcher_string:
E       0: b'Press ENTER to see the list of tests'
E   <pytest_embedded.log.PexpectProcess object at 0x0000019A33A84B50>
E   searcher: searcher_string:
E       0: b'Press ENTER to see the list of tests'

The above exception was the direct cause of the following exception:
pytest_aip1629.py:38: in test_aip1629
    dut.run_all_single_board_cases()
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:428: in run_all_single_board_cases
    for case in self.test_menu:
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:243: in test_menu
    self._test_menu = self._parse_test_menu()
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:124: in _parse_test_menu
    self.expect_exact(ready_line)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:82: in wrapper
    raise e.__class__(debug_str) from e
E   pexpect.exceptions.TIMEOUT: Not found "Press ENTER to see the list of tests"
E   Bytes in current buffer (color code eliminated): _task: Returned from app_main()
=========================================== short test summary info ============================================ 
FAILED pytest_aip1629.py::test_aip1629 - pexpect.exceptions.TIMEOUT: Not found "Press ENTER to see the list of tests"
============================================== 1 failed in 30.24s ==============================================

hayschan · 2024-01-16T13:31:51Z

Another combination but error persists

I have also tried changing the python test file. So my combination of Python and C++ looks is as followed.

from pytest_embedded_idf.dut import IdfDut

def test_app(dut: IdfDut):
    dut.expect_unity_test_output()

extern "C" void app_main(void)
{
    unity_run_all_tests();
}

An error about regex failing to find test cases results will show.

2024-01-16 21:28:15 ./main/aip1629_test.cpp:340:Test Invalid Inputs:PASS
2024-01-16 21:28:15 I (10950) main_task: Returned from app_main()
2024-01-16 21:28:36 Timeout: simulation did not finish in 30000ms
FAILED

=================================================== FAILURES =================================================== 
___________________________________________________ test_app ___________________________________________________ 
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:75: in wrapper
    index = func(self, pattern, *args, **kwargs)  # noqa
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:120: in expect
    return self.pexpect_proc.expect(pattern, **kwargs)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\spawnbase.py:354: in expect
    return self.expect_list(compiled_pattern_list,
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\spawnbase.py:383: in expect_list
    return exp.expect_loop(timeout)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\expect.py:181: in expect_loop
    return self.timeout(e)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\expect.py:144: in timeout
    raise exc
E   pexpect.exceptions.TIMEOUT: <pytest_embedded.log.PexpectProcess object at 0x000002AC105CF350>
E   searcher: searcher_re:
E       0: re.compile(b'^[-]+\\s*(\\d+) Tests (\\d+) Failures (\\d+) Ignored\\s*(?P<result>OK|FAIL)')
E   <pytest_embedded.log.PexpectProcess object at 0x000002AC105CF350>
E   searcher: searcher_re:
E       0: re.compile(b'^[-]+\\s*(\\d+) Tests (\\d+) Failures (\\d+) Ignored\\s*(?P<result>OK|FAIL)')

The above exception was the direct cause of the following exception:
pytest_aip1629.py:44: in test_app
    dut.expect_unity_test_output()
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:168: in expect_unity_test_output
    self.expect(UNITY_SUMMARY_LINE_REGEX, timeout=timeout)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:82: in wrapper
    raise e.__class__(debug_str) from e
E   pexpect.exceptions.TIMEOUT: Not found "re.compile(b'^[-]+\\s*(\\d+) Tests (\\d+) Failures (\\d+) Ignored\\s*(?P<result>OK|FAIL)', re.MULTILINE)"
E   Bytes in current buffer (color code eliminated): Wokwi CLI v0.8.0 (2a4bae54982c) Connected to Wokwi Simulation API 1.0.0-20240114-g15d4d362 Starting simulation... ESP-ROM:esp32s3-20210327 Build:Mar 27 2021 rst:0x1... (total 8457 bytes)
=========================================== short test summary info ============================================ 
FAILED pytest_aip1629.py::test_app - pexpect.exceptions.TIMEOUT: Not found "re.compile(b'^[-]+\\s*(\\d+) Tests (\\d+) Failures (\\d+) Ignored\\s*...
========================================= 1 failed in 60.26s (0:01:00) =========================================

hfudev · 2024-01-16T14:15:43Z

@hayschan For your c code, it looks good to me. Usually a simple unity_run_menu works

ref to our test app

seems like wokwi-cli does not support "write" yet. I tried calling the wokwi-cli locally without pytest-embedded, and the results are about the same

Running console component tests


Press ENTER to see the list of tests.


1

2

(The 1, 2, and empty lines are what I typed.)

Please note that wokwi-cli is still under active development, and we appreciate your patience as we work to improve it.

hayschan · 2024-01-16T14:19:49Z

Thanks for the confirmation, @hfudev

I have subscribed to Wokwi's club to support @urish 's work. Really excited about wokwi-cli. Once it is completed, it will make my life way easier. 😆

For now, I will be using the GitHub action self hosted runner until the completion of wokwi-cli.

urish · 2024-01-16T15:06:25Z

Thanks for confirming! What's the quickest way for me to reproduce this locally?

hayschan · 2024-01-16T16:02:55Z

Thanks for confirming! What's the quickest way for me to reproduce this locally?

I have created an example repo for you to clone. @urish

Please go to the test_apps path for the component test project.

I have already tried on my computer. The project can be built and has the symptoms I mentioned.

Redirects stdin into the simulated serial port Related: espressif/pytest-embedded#233

see espressif#233 (comment) for details

urish · 2024-01-16T18:47:13Z

Thank you so much @hayschan for creating a reproducible repo. There's now a fix - if you want to test it without waiting for a new pytest release, just clone this repo, switch to the issue-233 branch, go into the pytest-embedded-wokwi directory and run pip install .

Note that you should also update your wokwi-cli to the latest version, 0.9.0. To update, just follow the installation instructions which will install the newest version on top of the existing one.

hayschan · 2024-01-18T04:39:53Z

Thank you so much @hayschan for creating a reproducible repo. There's now a fix

I've just test it. It works flawlessly. 👍🏼

pytest -s --embedded-services idf,wokwi --tb short
============================================== test session starts ==============================================
platform win32 -- Python 3.11.2, pytest-7.4.4, pluggy-1.3.0 -- d:\.espressif\python_env\idf5.1_py3.11_env\Scripts\python.exe
cachedir: .pytest_cache
configfile: pytest.ini
plugins: embedded-1.6.2, rerunfailures-13.0, timeout-2.2.0
collected 1 item

pytest_max6675.py::test_max6675 <- test_apps\pytest_max6675.py
------------------------------------------------ live log setup ------------------------------------------------- 
2024-01-18 12:32:36 INFO Executing wokwi-cli --interactive
2024-01-18 12:32:37 Wokwi CLI v0.9.0
2024-01-18 12:32:38 Connected to Wokwi Simulation API 1.0.0
2024-01-18 12:32:40 Starting simulation...
2024-01-18 12:32:42 ESP-ROM:esp32s3
------------------------------------------------
2024-01-18 12:32:43 ./main/test_max6675.c:36:MAX6675 Sensor Test:PASS
2024-01-18 12:32:43 Test ran in 24ms
2024-01-18 12:32:43
2024-01-18 12:32:43 -----------------------
2024-01-18 12:32:43 1 Tests 0 Failures 0 Ignored
2024-01-18 12:32:43 OK
2024-01-18 12:32:43 Enter next test, or 'enter' to see menu
PASSED
----------------------------------------------- live log teardown ----------------------------------------------- 
2024-01-18 12:32:43 INFO Created unity output junit report

if you want to test it without waiting for a new pytest release

@hfudev will you integrate the wokwi fix into a new pytest-embedded version soon? The change of this fix is minimal, with an addition of the --interactive flag.

super().__init__(
-            cmd=[wokwi_cli, app.app_path],
+            cmd=[wokwi_cli, '--interactive', app.app_path],

hfudev · 2024-01-18T12:12:54Z

@hayschan Hi, after merging PR #261, version 1.6.3 has been released. Please have a try.

hfudev added the enhancement New feature or request label Oct 24, 2023

hfudev mentioned this issue Oct 24, 2023

feature: Wokwi simulation support (RDT-551) #230

Merged

github-actions bot changed the title ~~Wokwi Featured Functions~~ Wokwi Featured Functions (RDT-568) Oct 24, 2023

urish added a commit to wokwi/wokwi-cli that referenced this issue Jan 16, 2024

feat: --interactive flag

e848055

Redirects stdin into the simulated serial port Related: espressif/pytest-embedded#233

urish added a commit to urish/pytest-embedded that referenced this issue Jan 16, 2024

fix(wokwi): run in interactive mode

4de3586

see espressif#233 (comment) for details

urish mentioned this issue Jan 16, 2024

fix(wokwi): run in interactive mode (RDT-651) #261

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wokwi Featured Functions (RDT-568) #233

Wokwi Featured Functions (RDT-568) #233

hfudev commented Oct 24, 2023

hayschan commented Jan 8, 2024

hfudev commented Jan 8, 2024

urish commented Jan 15, 2024 •

edited

Loading

hayschan commented Jan 16, 2024

hayschan commented Jan 16, 2024

hfudev commented Jan 16, 2024

hayschan commented Jan 16, 2024

urish commented Jan 16, 2024

hayschan commented Jan 16, 2024

urish commented Jan 16, 2024 •

edited

Loading

hayschan commented Jan 18, 2024

hfudev commented Jan 18, 2024

Wokwi Featured Functions (RDT-568) #233

Wokwi Featured Functions (RDT-568) #233

Comments

hfudev commented Oct 24, 2023

hayschan commented Jan 8, 2024

hfudev commented Jan 8, 2024

urish commented Jan 15, 2024 • edited Loading

hayschan commented Jan 16, 2024

Original problem

My way to solve this

New error occured

hayschan commented Jan 16, 2024

Another combination but error persists

hfudev commented Jan 16, 2024

hayschan commented Jan 16, 2024

urish commented Jan 16, 2024

hayschan commented Jan 16, 2024

urish commented Jan 16, 2024 • edited Loading

hayschan commented Jan 18, 2024

hfudev commented Jan 18, 2024

urish commented Jan 15, 2024 •

edited

Loading

urish commented Jan 16, 2024 •

edited

Loading