What is the plan for Python 3 support? #657
Comments
|
Python 3 compatibility will likely happen during 2017. However, it is difficult to know exactly when. @gpotter2 does an amazing job at pushing Python 3 support by submitting dedicated PRs. The efforts currently happen on two labels:
Code coverage is now close to 80% (it was ~50% back in December 2016). I believe that it is a good indicator that Python 3 compatibility can be achieved in 2017 without introducing regressions. Feel free to contribute if you are interested ! |
|
Oh, that is awesome! I'm on packaging side here so it is sufficient for me to know that something is happening and that we do not need to drag scapy3k into Fedora (assuming the port will happen soon). Thank you very much! |
|
@pspacek Scapy3k was created a quite long time ago. It has not received all the bug fixes of scapy and its development is way less active than this one. There are no plan of merging both because of how old it is, and it wouldn't not make sense to fix the conflicts between them... Merging the different parts of python 3 support is slow, as we need to keep a stable environment. I have a roadmap of the next changes needed, and will pretty much work as quick as the review goes in, but there are quite a lot of things to fix :) Have you got any idea of when will python 2 be abandoned ? |
|
Does the scapy dev team plans for a Python 2 and 3 code base, or an alternative branch/project for Python 3 support maintained separately? |
|
The goal is to make the code Python 2 and 3 compatible. Some PRs were already merged while some are waiting for reviews, comments and modifications.
Feel free to have a look and contribute.
|
|
@guedou @gpotter2 This is a fork and is python 3 only https://github.com/phaethon/scapy and IMO scapy should by python 3 only as on this day https://pythonclock.org/ python 2 is EoL. With the changes in the fork, I do not see why you could not get python 3 support much quicker if you want to work with @phaethon to do so? |
|
Unfortunately, the creator of this fork never tried to help maintaining Scapy, and never pushed anything to us despite our calls to work on a shim layer. His behavior is quite aggressive and we don't want to work with him.
If you feel like helping us, we always welcome contributions !
|
|
My short version of phaethon/scapy branch and our relationship with @guedou and co. In 2015 an issue was raised (not by me) on bitbucket repository about python3 support and the answer of the core team was "not at this time, if you need it fork the repository". This is exactly what I did. After that I hoped that there would be some cooperation between the repos, but no - I asked to include info about python3 compatibility, but the suggestion was refused. I wanted people to find out that it is possible to use scapy with python3 (as many are tied to python3 for one reason or another) and was pushing information about the fork across different channels. The core team considered it being aggressive and in a passive aggressive form themselves kept ignoring if not hiding info about the python3 fork. I believe @guedou would more likely either reimplement from ground 0 python3 compatibility (as it was done for pcapng support after it was implemented in scapy3k) or wait for someone to bring the change from scapy3k to scapy (like it was done with Windows reimplementation, which eliminated libdnet, which was copied from scapy3k by a contributor). I would love to work in a cooperative way, but don't believe @guedou - he never invited me. Posting all scapy3k changes as one big PR would not make sense. I would believe it is possible to solve the situation by somebody from the community moderating the situation, but I wouldn't believe @guedou would let this happen. @guedou - fix me if I am wrong. |
|
and, btw, I wouldn't say it would be so hard to go through the differences and resync the repositories. With a trained eye it is quite quick to go through the commits. But for some people it would be too embarrassing to credit any of my effort, which made hundreds (measured by stars on github) or may be thousands of people to keep using scapy for last couple of years even if they are choosing to be with python3. |
|
@phaethon Hello, It does not really make sense to have two concurrent repositories. Yours has many many missing features/fixes compared to this one, but I'm sure that it also has great improvements. Moreover it's not useful to remind us what you did add first on yours, as yours remains a fork of this one... In a near future, this repository will support python 3, whatever you chose to act upon. We would be happy to get contributions until then. |
|
Well, that's a convenient way to relate what really happened. It is quite distracting and fun to read. Thanks that was refreshing ! |
|
@gpotter2 - a large part of the help you need is residing in a form of code in phaethon/scapy. You are welcome to ask me any questions that may arise. Fire up an issue on phaethon/scapy and I will prioritize responding to you over other issues. Actually, what I think has to be discussed first is the "interface" - how to treat strings and bytes, and what gets returned by different functions, and what expected at an input of the various function calls. scapy3k uses bytes for most things, but in some cases it may make sense to return/expect a string (e.g. domain name). It would be sad if secdev/scapy implemented incompatible interface to what scapy3k has, as I would believe many people working with python3 already have some code developed. And, yes, I would appreciate including credit to me in a Readme of this repository as it has de facto used (I guess it still uses) parts of code copied from phaethon/scapy. This would be a nice and friendly invitation for further cooperation. Is that unreasonable? |
|
I totally support @guedou point of view here. I don't think Scapy's code uses your code from phaethon/scapy, but I might be wrong (and actually I don't really care). |
|
If everyone can put things aside I would say that pulling in the code from the fork and having credit for it where credit is due would be I think for the best of everyone? and in turn the community. I do not mind acting as a mediator between the 2 parties.
At the end of the day I want everyone to be happy and the community to benefit from the work that people have done in the fork and original.
One other thing I would like to ask is that can we get more documentation for scapy.
Sent from phone
…________________________________
From: Pierre Lalet <[email protected]>
Sent: Monday, May 29, 2017 10:41:26 PM
To: secdev/scapy
Cc: J.Townsend; Comment
Subject: Re: [secdev/scapy] What is the plan for Python 3 support? (#657)
I totally support @guedou<https://github.com/guedou> point of view here. I don't think Scapy's code uses your code from phaethon/scapy, but I might be wrong.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<#657 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ADddQoK_-UIzjqiM6ahqToOsmZtSJM4nks5r-zuGgaJpZM4Nk_Qf>.
|
|
@L1ghtn1ng I don't think we need a mediator here, but thanks for your offer. We want to have Scapy work with both Python 2 & 3 and the same code base, and we are working on it. |
|
I had previously been told that Python 3 support was not a priority for Scapy (#87 (comment)) but it appears that there's been some backtracking here. As I have @phaethon's fork packaged as python3-scapy in Debian currently, that is as far as Debian is concerned the "official" Python 3 variant of Scapy. @phaethon Do you have a test suite that could be applied to any Python 3 porting work in this repository to ensure that if we change the upstream source for the python3-scapy package in Debian back to @secdev's repository that we wouldn't break existing uses of Scapy on Python 3 in Debian? |
|
@irl Scapy comes with an important test suite, and when it can use Python 3 we will be able to use the very same tests with both Python 2 and 3. |
|
@irl you're missing the point. The fork you're using as a base is not Scapy. It's a fork, based on Scapy, that should probably have another name to avoid confusions. If you're naming something python3-scapy, then you probably want to make sure that what works with Scapy also works with it, by the way. Its API and behavior already differ from Scapy's API. And Scapy's API and behavior are what exist in this repository. |
|
Based on the previous comment I had received, linked above, I did not believe that there would be Python 3 support in Scapy and so this was the only way to include it in Debian. The python3-scapy package provides a certain API and those that have installed the package in Debian will be upset when everything breaks because the API is different. I only care about what is in Debian and maintaining compatibility across versions within Debian. My priorities are the users of Debian and high quality free software. This is only an issue for the secdev developers if having their Scapy in Debian is the desirable outcome for them. Really, this is an issue for me, but it can be helped by a test suite from @phaethon so that I can come up with a series of patches to try to stop breakage from occurring due to a change in which project is considered the upstream for the package. At the very least, I'd like to have an idea of how much breakage there would be. If having Scapy in Debian is not something that really matters to you and you don't care, then that is also fine and I can follow up with @phaethon somewhere else. |
|
If you really care what is in Debian it would be best to transition the python3-scapy package to a different one with a different name. Because using the same name for two different (even if diverged) things is not meaningful. The debian package should have been named python3-scapy3k from the beginning because that's how it's called upstream... |
|
The code you are currently packaging as "python3-scapy" is not Scapy. This code is Scapy. Python 3 support or not does not change that. |
|
@mcpat is a bit right if you ask me. You have been told that python 3 support was 'not a priority' 10 months ago. That was our mistake and isn't the case anymore. We're currently working on it. |
|
This conversation appears to be a waste of everyone's time. I would appreciate it if someone could ping me when this repository's support for Python 3 is nearing stability and I'll look at possible paths for the change at that time. |
|
Hm why exactly is it not helpful? I just proposed a way to clean the current situation while not frustrating the users of the scapy3k API. I mean you decided yourself to package it under python3-scapy even though it was clear back than, that ongoing development and extensions are done in this repo here - and a merge path was and is not established! |
|
#861 has landed! 🎆 |
|
If there can be an RC release, I'm currently releasing the next version of PATHspider (mami-project/pathspider#188) and can test this against that. There is then only one other Debian package with a dependency on python3-scapy and then there should be a clear path to having python3-scapy switch upstream back to this repository. Edit: Ideally this would be today, as that would fit nicely in the release workflow and I can fix any remaining issues, otherwise any Debian upload may have to wait for a PATHspider point release. |
|
Hm ok, do you see any significant changes happening between now and then? I can do my PATHspider testing with a local Debian package based on the current master perhaps? |
|
Cool, thanks. |
|
I just tagged 2.4-rc1, see https://github.com/secdev/scapy/releases We need to merge some more PR before v2.4.0 but this will happen in the next weeks. |
|
Ok, from my testing this is looking good for the requirements of PATHspider. I've not tested dhcpcanon yet though. I've opened phaethon/kamene#231 to look at making this co-installable and a proof of concept patch was working, until I messed it up and broke the patch which will need to be rewritten.
|
|
@guedou Will you wait until v2.4.0 before making a PyPI release or were you also planning to release a "-dev" release onto PyPI? |
|
I did not manage to do it today. I will try to make one tomorrow.
|
|
@johnthagen please try |
|
@guedou I installed 2.4rc2 into a Python 3.6 virtual environment, played around making a couple sample packets, and everything worked! 🥇 |
|
@guedou pressed the "release" button for 2.4.0, so I guess this can be considered fixed :) Thanks a lot to everybody involved in this great project! |
|
Open-source development is a fantastic and fulfilling initiative, but comes with its share of human vices. You can't reasonably ask for everybody to play fairly and without pride. @phaethon made a great contribution to bring Python 3 support to scapy which, at the time, may have been lagging behind – however good their reasons. If @phaethon doesn't want to collaborate with upstream, so be it. It's still his/her right to keep the GitHub repo online. Let's stay respectful of the work accomplished by everybody and gently educate users about the issue, even if that takes time. |
|
Thanks @zopieux for bringing a share of fairness into the discussion. |
|
Well, thanks for keeping the discussion alive. Having a page explaining the state of the various forks is a good idea, as long as we play fair and use the same wording in upstream and forks. I guess the real issue is the branding: scapy3k is not scapy with 3k compatibility (because scapy now has it), it's a fork that has diverged significantly over time. I just saw the phaethon/kamene#231 discussion about changing the module name and I appreciate the decision to implement the change. I hope the secdev people are okay with the @phaethon, would you be okay with a prominent line in your README stating this is a fork of the original scapy project with a link? |
|
@zopieux I am fully supportive on placing objective bilateral linking with information reflecting reality. Probably, we would need to agree on 'what reality is' first, but sure it is possible with a reasonable mediator in between the maintainers of both repos. |
|
@irl Hello ! It might be worth updating the debian issue: scapy3k is now kamene, so I'm guessing a package name like |
|
I filed bugs about this a while ago, waiting on ftp master to process https://bugs.debian.org/898105. Can't really do anything until that happens. |
|
You could always email [email protected] with words of encouragement. |
|
python3-scapy built from secdev sources has now been uploaded. This will require manual approval from the ftp masters as this is a new binary package. |
|
FTR, the list of depedencies could be updated:
|
|
Is scapy-2.4.3 still relying on old pyx version (that is python2 only) for 2D graphs or documentation needs updating? Thanks for the info :) |
|
@pacho2 No, Scapy will work with any PyX version. It uses it only for optional features and will work without it. PyX >= 1.13 supports Python 3 (https://github.com/pyx-project/pyx) but dropped Python 2.7. The doc makes the distinction correctly:
I'll however lock this issue because this is sorta unrelated, and I don't want to keep pinging so many people. Please open a different issue or ask on the gitter if you have additional questions :-) |
Hello,
I'm trying to find out if there are any plans for "the original" scapy to support Python 3 or not.
The only references to real code I found is in #87 (comment) but it seems to me that the plan for scapy rewrite is not moving forward because repo https://bitbucket.org/secdev/scapy3-prototype2 has last commit from 2015-02-25.
Given that current Fedora (and so future of RHEL/CentOS/Scientific Linux) is migrating away from Python 2 (as well as all other Linux distros) this forces me to ask the question again:
What are the plans for Python 3 support in scapy?
Without some form plan, Fedora/RHEL/CentOS will have to implement the same approach as Debian (see #87 (comment)), i.e. use scapy3k instead of scapy.
If this is the case, are there any plans to start merging features between scapy and scapy3k?
Thank you for your time and answers.
The text was updated successfully, but these errors were encountered: