"cgi" standard library module used by bottle is deprecated in Python 3.11, to be removed in 3.13

[liyanage]

I'm running bottle with Python 3.11 for the first time and I see this deprecation warning:

/xxx/lib/bottle/bottle.py:72: DeprecationWarning: 'cgi' is deprecated and slated for removal in Python 3.13
  import base64, calendar, cgi, email.utils, functools, hmac, itertools,\

I guess bottle needs to stop using this module?

[defnull]

That's unfortunate, bottle uses the form-data/multipart parser from that library. The PEP says:

FieldStorage/MiniFieldStorage has no direct replacement, but can typically be replaced by using multipart (for POST and PUT requests) or urllib.parse.parse_qsl (for GET and HEAD requests)

Great. I'm the author of multipart, so I'm okay with that, but copy&pasting all that into bottle (to maintain the zero-dependency design goal) feels wrong.

[zb3]

Removing the multipart/form-data parser from Python standard library is unthinkable. I understand the cgi module was deprecated because CGI is itself deprecated, but the multipart encoding is not deprecated, it's still used today, hence I totally don't understand this..

[simsong]

PEP 594: Remove the cgi and cgitb modules, deprecated in Python 3.11.

cgi.FieldStorage can typically be replaced with urllib.parse.parse_qsl() for GET and HEAD requests, and the email.message module or multipart PyPI project for POST and PUT.

(per https://docs.python.org/3.13/whatsnew/3.13.html)

[fedepell]

Any news maybe on this? 3.13 is now getting closer with 3.12 being out in the meantime, albeit the release plan is around half of next year.

[defnull]

The only viable solution is to copy&paste code from multipart into bottle. I really really do not want to do that, but I have no other idea how to maintain the single-file no-deps aspect any other way.

[fedepell]

The only viable solution is to copy&paste code from multipart into bottle. I really really do not want to do that, but I have no other idea how to maintain the single-file no-deps aspect any other way.

Thanks for the fast reply! I see your point and absolutely agree is not "nice", but if we don't want to have dependencies, then I guess it's really the only way to go ("soon" or as Python 3.13 start getting traction).

Would you mind when you have time in case create a branch (possibly also for 0.12.x ?) with this so we can start testing? (I'm looking at this in the scope of future Fedora 41 planning to use it)

[aisk]

@defnull Hi can we using the email module in the standard library to avoid the copy / pasting?

[ilrico]

@defnull Since @aisk proposal does not work out (for large data as you underlined), what is your position on this issue ?

[simsong]

If the only requirement is to handle multipart and nothing else, it should not be hard to write a parser that does just that. multipart is not complex.

[ilrico]

I wonder if the no-deps line should not be open to compromise, as it is theorical: we see with this issue we are always dependant anyway, as least from std lib.

[simsong]

I'm also fine with copying the lines we from the cgi library into bottle.

[valq7711]

Here is the POC
https://github.com/valq7711/bottle/tree/multipart

Features:

• It parses body while writing request.body and produces a markup, which is stored in request.body.multipart_markup
• files are not tmp-files, but BytesIOProxy file-like objects, which are proxied to corresponding parts of request.body (no extra copying from request.body to tmp-files that cgi does), so it is ~30% faster
• nested multipart not supported
• only CRLF delimiters allowed (RFC 7578), single LFs not supported
• if there is an error while parsing, it is stored in request.body.multipart_markup.error and raised if only request.POST/forms/files touched, so if body has some exotic format, one can parse it using custom tools, without croaking of undesired errors (just do not touch request.POST/forms/files)

[simsong]

Looks good; are you going to commit a test as well?

[valq7711]

@simsong what do you mean? It passed the test https://github.com/valq7711/bottle/actions/runs/7160872053
Multipart test is in test_environ.py
The problem is that it is breaking changes as cgi supports nested multiparts and allows LF-delimiters (instead of CRLF)

[simsong]

Ah. I didn’t see that.

…

On Mon, Dec 11, 2023 at 1:25 AM valq7711 ***@***.***> wrote: @simsong <https://github.com/simsong> what do you mean? It passed the test https://github.com/valq7711/bottle/actions/runs/7160872053 Multipart test is in test_environ.py The problem is that it is breaking changes as cgi supports nested multiparts and allows LF-delimiters (instead of CRLF) — Reply to this email directly, view it on GitHub <#1403 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMFHLGUEIPLDPCVBTOOFXLYI2RNRAVCNFSM6AAAAAARZPDW7WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBZGQYDAMBSHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ilrico]

hi folks, a gentle reminder that 3.13 in alpha 4 and cgi confirmed removed!

PEP 594 (Removing dead batteries from the standard library) scheduled removals of many deprecated modules: aifc, audioop, chunk, cgi, cgitb, crypt, imghdr, mailcap, msilib, nis, nntplib, ossaudiodev, pipes, sndhdr, spwd, sunau, telnetlib, uu, xdrlib, lib2to3.

[fedepell]

Just a ping as Python 3.13 is closer and closer...

[opoplawski]

Python 3.13b2 has landed in Fedora Rawhide breaking bottle

[ilrico]

Hello, seems evolution to handle change in python 3.13 is stalled.
@defnull should be assumed that bottle maintenance is over?
I guess the community deserves a clear answer, some are already migrating to flask/fastAPI as we can see here.

That does not remove the fact that we are all grateful for the great work done for the last years on bottle.

[oz123]

For those interested, I intend on maintaining bottle under the fork here:

https://github.com/veilchen-web/veilchen

[simsong]

I guess the community deserves a clear answer, some are already migrating to flask/fastAPI as we can see here.

I have decided to migrate to fastAPI. It is not difficult, although some features of bottle are missing.

[defnull]

No the project is not dead, but not healthy either. My dilemma is as follows:

• Bottle cannot depend on multipart (my own library) because it has a strict no-external-dependency rule.
• We cannot just copy&paste multipart code into Bottle because that would break Python2 compatibility. Avoiding syntax errors on old python versions is not trivial, and testing those ancient versions gets harder and harder.
• We could just drop Python 2 support, and that is probably the sane choice. But that would be a major breaking change and require a significant version bump. The only clean solution would be to release Bottle 1.0 and make a clean cut. But that is actually a ton of work. A lot more than 'just' adding multipart code. Documentation needs an overhaul, removing all python 2 workarounds everywhere breaks a lot of tests, and test coverage is okay, but not perfect. There will be bugs, and I simply do not have the confidence to push out a half baked 1.0 release, and not the time to make it good enough. I started working on that in fix: cgi.FieldStorage not available in Python 3.13 #1438 but got stuck. I have a HUGE change-set that is not pushed yet, because I tried to fix everything at once for a 1.0 release. That was a mistake, obviously. Moving forward without breaking stuff is almost impossible after so many years and maybe I should just accept that a 1.0 version numbers means nothing and stuff can be fixed in Bottle 1.1 or 2.0 if needed.
• A fork does not have the restrictions that I imposed on myself and this project (e.g. backwards compatibility between releases). It's way easier to make a clean cut in a fork, because no one depends on it yet and you break no applications by changing things in a backwards incompatible way. So, maybe that's also an option. The License allows it. This does not help my own motivation to sink more work into this, though. Especially if it's a hard fork with no plans in contributing back into the original project.

tl;dr; Project is not dead, but struggling. I got stuck and need help. But before anyone can help me, I need to help myself and publish what I already have done.

[simsong]

@defnull - thank you for the heartfelt reply above. I appreciate all of the work that you have done on Bottle. and it is a great piece of work. The idea of having code that runs on both Python 3.13 and Python 2.7 is admirable, but I think that it is misguided. Running a production website on Python 2.7 would be a mistake because of the ever-increasing number of unfixable security bugs. I would encourage you to drop Python 2 support, as most other projects have done over the past decade.

[oz123]

@defnull you should let more people be involved in the process and let people co-author commits.
I've dropped all python 2 code from Veilchen and added your multipart implementation.
I've tried a few times to get more involved with bottle but your lack of response made me create my own fork. I'd love to contribute to bottle directly though.

[ilrico]

@defnull I totally agree with @simsong : we are in 2024 and python 2 is hardly a subject. And system stuck on python2 always the possibility to stay on current 0.12.x
Regarding versionning, my first thought is "this guy could never work at Microsoft" ;-) my second thought is, quoting a sticker seen on a laptop in a train trip "Done is better than perfect". You can always commit it under 1.0rc1 so people are aware that it's not guaranteed bug free for now. And hopefully 1.0 will follow soon.

Fork is not a good option IMHO. Bottle has a nice goodwill. It is known (not as flask, but still) and is respected for the choices your made (no dependency for instance).
So I push for option 3. Sanity matters ;-)
But there's no much time, people are moving away...

[simsong]

@ilrico - thanks for your comments. I agree that it would be easier for me to stay with Bottle than to move to FastAPI. But unless new projects have a positive reason to use Bottle, it's going to be a dwindling community.

The ease of getting an app running with Bottle is its big benefit. It's not just the lack of dependencies, it is the ease of integration.

[defnull]

Thanks a lot for the feedback and support. I just spend the last 2 hours reviewing what I had done already, cleaning everything up and pushing to #1438. The PR now contains more than it should (e.g. a complete documentation overhaul) but it's a bit late late for being picky. That's fine.

New plan:

• Merge fix: cgi.FieldStorage not available in Python 3.13 #1438 within the next 2 weeks (during my holiday) and release Bottle 0.13 shortly after. This will be an 'emergency release' that does break existing applications running on ancient Python versions, but still supports Python >=2.7.3 and a lot of 3.x releases (tested with 3.8-3.12). That should be fine for 99% of users, and those that are affected can and should stay on 0.12.x forever. Throwing out Python 2 support completely is the next step, but too much of a change for right now.
• Prepare Bottle 1.0: Drop Python 2.0 support and clean up all the workarounds. Make this a clean cut and move forward.

[defnull]

The PR now passes all tests. I'll be off the computer for a couple of days and ask everyone to have a look at #1438 and provide feedback. I'm actually quite happy with it now. If there are no major issues with it, I'd merge that into master next week and prepare a 0.13 release.

[oz123]

Thanks for your work so far and bring responsive now. Enjoy your time off, and please think about solving the human resource problem.
Currently you are the only person who is actively working on bottle with access to GitHub and pypi.
You should really involve at least one more person of trust. Maybe even apply for tidelift.

[defnull]

New implementation was just merged into master and I'm preparing a 0.13 release at this very moment.

[defnull]

Done. I'll finally close this issue now :) Thank you for all the feedback, your kind words and above all your patience. The next releases should be way easier, now that the 'big one' is out of the door.

I'll now start working on issues and PRs and see which ones can be closed or easily fixed. You can all help by commenting on issues that should be fixed in 0.13 and can be closed now. There sure is a lot of outdated stuff in the issue tracker.