The correct way to implement the simplest web server

[omri-madar]

Hi guys,

I'm new to ESP32 and micropython, and I'm still learning the basics.
Also, I have a very basic knowledge in the networking / web development world.

I've read several tutorials online about how to set up a web server on the board, and almost all of them use an infinite loop ("while True") to run the server.
I also saw some topics in the old forums where people used infinite loops.

So my question is - Is that the correct way to do it?
I guess that such an infinite loop is not the most efficient/smart thing to do.
However... if there's no other choice...

So what's the best practice to set up a web server?

[shariltumin]

There is no way to do this without an infinite loop in MicroPython.

Your server must wait for socket connections from clients.

Even if you use uasyncio.start_server() there is an infinite loop in the implementation (extmod/asyncio/stream.py), you just don't see it.

...
    async def _serve(self, s, cb):
        # Accept incoming connections
        while True:
            try:
                yield core._io_queue.queue_read(s)
            except core.CancelledError:
                # Shutdown server
                s.close()
                return
            try:
                s2, addr = s.accept()
            except:
                # Ignore a failed accept
                continue
            s2.setblocking(False)
            s2s = Stream(s2, {"peername": addr})
            core.create_task(cb(s2s, s2s))
...

Without an infinite loop, your server can only accept and serve a client once and then stop.

Smart or not, it is what it is.

[bixb922]

I'd use a package like microdot or perhaps tinyweb. Microdot will most probably do all that's needed and more.
At a low level I believe the asyncio approach is the most convenient (with few exceptions). Microdot and other Web server use this approach.

import uasyncio 
import network
async def coro_server( reader, writer ):
    data = await reader.read(1024) 
    print(f"received {data=}")
     # Give a response so this can be tested with
     # curl http://192.168.xxx.xxx/
     writer.write(b'HTTP/1.0 200 OK\n\nHola mundo\n')
     await writer.wait_closed()
async def do_connect():
    # see standard example of do_connect

async def main():
     await do_connect()
     server = uasyncio.start_server( coro_server, "0.0.0.0", 80 )
     await uasyncio.gather( server, other_tasks() ) 
main() 

[robert-hh]

With the ESP32 port and some others, there is a way to register a handler, which is called when some instance connects to a socket. This method is e.g. used by WEBREPL, which is a server. Or ftpd. It is done by a call with option 20 to setsockopt(). See e.g. the code snippet below from the ftp server. _SO_REGISTER_HANDLER has teh value of 20.

    def __init__(self, ftpsocket, local_addr):
        self.command_client, self.remote_addr = ftpsocket.accept()
        self.remote_addr = self.remote_addr[0]
        self.command_client.settimeout(_COMMAND_TIMEOUT)
        log_msg(1, "FTP Command connection from:", self.remote_addr)
        self.command_client.setsockopt(socket.SOL_SOCKET,
                                       _SO_REGISTER_HANDLER,
                                       self.exec_ftp_command)

The handler runs in scheduled context and can run any Python script. To unregister a handler, call setsockopt() with the value 0f None for the handler. Other tasks on you machine will stop as long as the handler is active, unless you add other means for concurrency.

[Fisherman48]

Hi Robert
Really interesting topic.
Try to understard these vocabularies and concepts ' in the python world' and:

The handler runs in scheduled context and can run any Python script

Is this mechanism working for mqtt, using (client.check_msg() or client.wait_msg()), to avoid the while True: loop.

Br /Stefan

[robert-hh]

Scheduled context in contrast to interrupt context, like you get e.g. at the Pin.irq() with hard=True. In interrupt context the code is limited. It cannot for instance allocate memory, which happens quite often in Python code. And in any case the code must not block.

[shariltumin]

Hi Robert, thanks! Yes, we can drop the while True: server accept loop and use srv.setsockopt(socket.SOL_SOCKET, 20, handler) instead, and let the socket_events_handler() do the job instead.

For those who are building custom firmware and want to disable WEBREPL, you need to enable ```MICROPY_PY_SOCKET_EVENTS`` in "mpconfigport.h".

#define MICROPY_PY_WEBREPL                  (0)
#define MICROPY_PY_SOCKET_EVENTS            (1)

My earlier statement about the "infinite loop" was wrong. I apologise. With "SOCKET_EVENTS" enabled in the firmware for the ESP32, I can now do the following:

def handler(srv):
    print('Handler starting')
    cs,ca = srv.accept()
    print('Serving:', ca)
    cs.setblocking(False)
    rep = b'Hello\r\n'
    cs.write(rep)
    cs.close()

port = 8080
addr = socket.getaddrinfo('0.0.0.0', port)[0][-1]
srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind(addr)
srv.listen(5) # at most 5 clients
srv.setblocking(False)
srv.setsockopt(socket.SOL_SOCKET, 20, handler)

[shariltumin]

@omri-madar, I made wrong claim earlier, sorry. It is possible to use socket event callbacks to write request/response server without looping. The code below seems to work fine.

# To test on Linux
# curl --http0.9 http://10.0.1.73:8080
# telnet 10.0.1.73 8080

import socket, select, time, network
network.WLAN(network.AP_IF).active(False)
wlan = network.WLAN(network.STA_IF)
wlan.active(True)
wlan.connect("YOUR-WIFI", "YOUR-PWD")
time.sleep(5)
wlan.ifconfig()

def req_handler(cs):
    try:
       req = cs.read()
       if req:
          print('req:', req)
          rep = b'Hello\r\n'
          cs.write(rep)
       else:
          print('Client close connection')
    except Exception as e:
        print('Err:', e)
    cs.close()

def cln_handler(srv):
    cs,ca = srv.accept()
    print('Serving:', ca)
    cs.setblocking(False)
    cs.setsockopt(socket.SOL_SOCKET, 20, req_handler)

port = 8080
addr = socket.getaddrinfo('0.0.0.0', port)[0][-1]
srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind(addr)
srv.listen(5) # at most 5 clients
srv.setblocking(False)
srv.setsockopt(socket.SOL_SOCKET, 20, cln_handler)

Thanks to @robert-hh, I have learned something new today and I am happy.

[robert-hh]

Looks good.

[rkompass]

Thanks @shariltumin @robert-hh what a nice example.
I still don't understand why the cln_handler() is necessary/has advantages. Would someone like to explain it?
Given the above code it seems the next step would be to add the option to have different handlers for different routes?

[jchome]

Thanks for the solution. It only works on GET requests, without content.

With a content, where is more data to get. Let's suppose that the following data is on the socket:

POST /Acer HTTP/1.1\r\nHost: 192.168.4.1\r\nUser-Agent: python-requests/2.31.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 33\r\nContent-Type: application/json\r\n\r\n"content": "some dummy content"}

With the provided solution, the req variable is filled with:
POST /Acer HTTP/1.1\r\nHost: 192.168.4.1\r\nUser-Agent: python-requests/2.31.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 33\r\nContent-Type: application/json\r\n\r\n

The content is missing...

I've tried to add cs.read() to get the tail of the message, but I cannot make it work. How can I do this ?

[bulletmark]

Here's the way to implement it simply using microdot.

[shariltumin]

Hi Raul,
The cln_handler() is required to handle a new client connection (up to 5 clients connected at the same time).

When cln_handler() is called by the srv socket event, a new client socket cs is created. We then attach the req_handler callback to this socket.

Now we have two sockets, srv and cs, in our system. Each of these sockets can execute its callback. Say a new client connects, triggering cln_handler(). This handler will create a new socket for the new client.

We will then have three active sockets, srv, cs(client-1), cs(client-1). Given that srv.listen(5) can serve five clients at peak traffic, we will have six active sockets, all with callbacks.

Note that the example is a simple request/response setup. When a req_handler() is started by a cs socket event, it reads a single request, processes it (here just a print()), processes a response (here just assign 'Hello' to rep and write it to the socket). Close the socket immediately afterwards.

Here is a transcript with three active clients:

MicroPython v1.20.0-332-gb714f4181-dirty on 2023-08-13; ESP32 STD/BLE KAKI5 with ESP32
>>> os.listdir()
['boot.py', 'esp32_cb_srv.py']
>>> import esp32_cb_srv
('192.168.4.73', '255.255.252.0', '192.168.4.1', '192.168.4.1')
>>> Serving: ('192.168.4.27', 34536)
Serving: ('192.168.4.52', 54396)
Serving: ('192.168.4.27', 41712)
req: b'GET / HTTP/1.1\r\nHost: 192.168.4.73:8080\r\nUser-Agent: curl/7.68.0\r\nAccept: */*\r\n\r\n'
req: b'Hey\r\n'
req: b'Bye\r\n'

The first two are from telnet clients and the last one is from a curl client. The curl send request immediately, triggers req_handler() as you can see. We need to write something to the telnet session to make it send something to the server, hence the delay.

Below is the actual order in which the requests were made:

$ telnet 192.168.4.73 8080
Trying 192.168.4.73...
Connected to 192.168.4.73.
Escape character is '^]'.
-- wait 50 seconds --
Bye
Hello
Connection closed by foreign host.

Trying 192.168.4.73...
Connected to 192.168.4.73.
Escape character is '^]'.
-- wait 30 seconds --
Hey
Hello
Connection closed by foreign host.

$ curl --http0.9 http://192.168.4.73:8080

In the example we have a server listening only on port 8080. We can have another server listening on port 2323 for example. The server on port 8080 will serve http request/response traffic, while the server on port 2323 will serve plain request/response traffic.

The client handlers in this case will be http_cln_handler() and tcp_cln_handler().

A bit of a long explanation, but I hope it answers your question.

I have a question for @robert-hh, is there a limit to how many open sockets with callback we can have in ESP32 MicroPython?

[rkompass]

Thank you Sharil, very clear explanation.

[rkompass]

I still had not understood yet why we cannot replace cln_handler() and req_handler() with, say the combined version:

def clnreq_handler(srv):
    cs,ca = srv.accept()
    print('Serving:', ca)
    cs.setblocking(False)
    try:
       req = cs.read()
       if req:
          print('req:', req)
          rep = b'Hello\r\n'
          cs.write(rep)
       else:
          print('Client close connection')
    except Exception as e:
        print('Err:', e)
    cs.close()

so I tried it out. And this works for your curl example, but not for telnet.
Now I got the point:
For telnet we have the connection established after the execution of telnet 10.0.1.73 8080 but then there is nothing sent until I type something and send it with <enter>. Perhaps the telnet command needs even something from the server to send its string or is too slow, because while echo 'test' | telnet 10.0.1.73 8080 does not, an echo 'test' | nc 10.0.1.73 8080 returns the 'Hello'.

So the

req = cs.read()
       if req:

would have to be repeated in a loop, blocking the system (we are not using async yet..), which is avoided by the
setsockopt(socket.SOL_SOCKET, _SO_REGISTER_HANDLER, next_handler_function) which delegates the waiting for something to be read to the socket event functionality.

[omri-madar]

@shariltumin thank you so much for coming back with a better answer, and for wrapping it all up in one script.
I've learned a lot from this thread.

I used your code as a base, and implemented the most basic web server to handle HTTP requests.
After a small refactor that's what I got and it works fine:

import socket
import select
import time
import network


def req_handler(cs):
    try:
       req = cs.read()
       if req:
          print('Request:\n', req)
          cs.send('HTTP/1.1 200 OK\n')
          cs.send('Content-Type: text/html\n')
          cs.send('Connection: close\n\n')
          cs.sendall("<h1>Hello World!</h1>")
       else:
          print('Client close connection')
    except Exception as e:
        print('Err:', e)
    cs.close()


def cln_handler(srv):
    cs, ca = srv.accept()
    print('Serving:', ca)
    cs.setblocking(False)
    cs.setsockopt(socket.SOL_SOCKET, 20, req_handler)


if __name__ == "__main__":
    
    network.WLAN(network.AP_IF).active(False)
    wlan = network.WLAN(network.STA_IF)
    print('Connecting to network...')
    wlan.active(True)
    wlan.connect(<SSID>, <PW>)
    time.sleep(5)

    port = 8080
    address = socket.getaddrinfo(<Host>, port)[0][-1]
    srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    srv.bind(address)
    srv.listen(5) # at most 5 clients
    srv.setblocking(False)
    srv.setsockopt(socket.SOL_SOCKET, 20, cln_handler)
    

[rkompass]

is there a limit to how many open sockets with callback we can have in ESP32 MicroPython?

@shariltumin: I found #define CONFIG_LWIP_MAX_SOCKETS 10 in src/sdkconfig.h, which declares itself as as an automatically generated file.
Generation of it seems to use esp-idf/components/lwip/Kconfig
and therein you may change

    config LWIP_MAX_SOCKETS
        int "Max number of open sockets"
        range 1 16
        default 10
        help
            Sockets take up a certain amount of memory, and allowing fewer
            sockets to be open at the same time conserves memory. Specify
            the maximum amount of sockets here. The valid value is from 1
            to 16.

the default and/or the range if you like. From reading other posts it seems that lwip itself has no limit on that.
You will have less RAM in the end, of course.
One might try if the value of 10 is correct, perhaps with your approach of using telnet: try to open up to 11 sessions.

I would be interested in a method to check that value automatically from within a micropython program/session, as it is a parameter that has a certain descriptive value to characterise a port / configuration.

[shariltumin]

Hi Raul, it seems that we can have a maximum of 10 active (open) sockets. 1 for the server listening socket and 9 for connected sockets from clients.

>>> os.listdir()
['boot.py', 'esp32_cb_srv.py']
>>> import esp32_cb_srv
('192.168.4.73', '255.255.252.0', '192.168.4.1', '192.168.4.1')
>>> Serving: ('192.168.4.27', 34852)
Serving: ('192.168.4.27', 53788)
Serving: ('192.168.4.27', 53800)
Serving: ('192.168.4.27', 53812)
Serving: ('192.168.4.27', 35886)
Serving: ('192.168.4.27', 53196)
Serving: ('192.168.4.27', 53208)
Serving: ('192.168.4.27', 41142)
Serving: ('192.168.4.27', 41154)
Traceback (most recent call last):
  File "esp32_cb_srv.py", line 27, in cln_handler
OSError: 23

Nice of you to find LWIP_MAX_SOCKETS. We can probably do CONFIG_LWIP_MAX_SOCKETS=16 in "sdkconfig.board" to serve 15 clients (for a very busy server).

[rkompass]

I assume you changed srv.listen(5) # at most 5 clients**** to a higher value before that test.

[rkompass]

I repeated your test with the recent Winner w600 port and it shows here that I can have a maximum of 8 active (open) sockets. 1 for the server listening socket and 7 for connected sockets from clients. Which is in accordance with the respective lwip setting for the w600 which I found some time ago.
The srv.listen(5) did not have to be set to a higher value for that.

[shariltumin]

Sorry, I was not precise enough. srv.listen(5) means the server socket srv has 5 slots in its listen queue. Lets say within a network with many hosts. It can happen that 6 of these machines request connections to your server 'at the same time'. With only 5 slots your server has to reject 1.

A client connection in the listen queue get push to srv.accept() and the server now has 1 slot free for a new connection. The srv.accept() will create a client socket. The number 5 here is not the maximum number of client/server socket end-points we can have. In my case (esp32) I can have up to 9 (default).

[rkompass]

For someone learning (like me) how things work here a more elaborate example that includes a @route('/') magic,
the added logic being 'stolen' from wybirals async server.

# To test on Linux
# curl http://192.168.178.67:8080
# or browser at http://192.168.178.67:8080

import socket, select, time, network
# _SO_REGISTER_HANDLER = const(20)

our_handlers = []

def route(path, methods=['GET']):
    def wrapper(handler):
        our_handlers.append((path, methods, handler))
        return handler
    return wrapper

def req_handler(cs):
    try:
        line = cs.readline()
        print('line:', line)
        parts = line.decode().split()
        if len(parts) < 3:
            raise ValueError
        r={}
        r['method'] = parts[0]
        r['path'] = parts[1]
        parts = r['path'].split('?', 1)
        if len(parts) < 2:
            r['query'] = None
        else:
            r['path'] = parts[0]
            r['query'] = parts[1]
        r['headers'] = {}
        while True:
            line = cs.readline()
            if not line:
                break
            line = line.decode()
            if line == '\r\n':
                break
            key, value = line.split(':', 1)
            r['headers'][key.lower()] = value.strip()
        handled = False
        for path, methods, handler in our_handlers:
            if r['path'] != path:
                continue
            if r['method'] not in methods:
                continue
            handled = True
            handler(cs,r)
        if not handled:
            cs.write(b'HTTP/1.0 404 Not Found\r\n\r\nNot Found')
            print('No handler found')
    except Exception as e:
        print('Err:', e)
    cs.close()

def cln_handler(srv):
    cs,ca = srv.accept()
    print('Serving:', ca)
    cs.setblocking(False)
    cs.setsockopt(socket.SOL_SOCKET, 20, req_handler) # 20 = _SO_REGISTER_HANDLER
    # the above allows for waiting for something that is sent later.

port = 8080
addr = socket.getaddrinfo('0.0.0.0', port)[0][-1]
srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind(addr)
srv.listen(5) # at most 5 clients
srv.setblocking(False)
srv.setsockopt(socket.SOL_SOCKET, 20, cln_handler)  # 20 = _SO_REGISTER_HANDLER

@route('/')
def hello_handler(cs, r):  # cs: the socket to read/write from/to, r: dict with additional info
    print('r:', r)
    cs.write(b'HTTP/1.0 200 OK\r\n')
    cs.write(b'Content-Type: text/html; charset=utf-8\r\n')
    cs.write(b'\r\n')
    cs.write(b'<h1>Hello World!</h1>')

print('our_handlers:', our_handlers)