Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lxml 4.7.x results in "A valid SubjectConfirmation was not found on this Response" with ADFS #292

Open
aquatix opened this issue Dec 21, 2021 · 16 comments

Comments

@aquatix
Copy link

aquatix commented Dec 21, 2021

We recently upgraded our lxml from 4.6.4 to 4.7.1 and suddenly there were some SSO issues:

OneLogin works fine, but when a client logs in from an ADFS instance, they get the message that "A valid SubjectConfirmation was not found on this Response".

Moving back to 4.6.4 resolves this issue.

The configuration:

        return {
            'strict': True,
            'debug': True,
            'sp': {
                'entityId': f'{servername}/saml/metadata/{self.logincode}',
                'assertionConsumerService': {
                    'url': f'{servername}/saml/acs/{self.logincode}',
                    'binding': 'urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST',
                },
                'singleLogoutService': {
                    'url': f'{servername}/saml/sls/{self.logincode}',
                    'binding': 'urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect',
                },
                'NameIDFormat': 'urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified',
                'x509cert': '',
                'privateKey': '',
            },
            'idp': {
                'entityId': self.idprovider_entity_id,
                'singleSignOnService': {
                    'url': self.idprovider_url,
                    'binding': 'urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect',
                },
                'singleLogoutService': {
                    'url': self.idprovider_url,  # TODO: add SLO url property
                    'binding': 'urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect',
                },
                'x509cert': self.x509certificate,
            },
            'security':
            {
                'signatureAlgorithm': 'http://www.w3.org/2001/04/xmldsig-more#rsa-sha256',
                'digestAlgorithm': 'http://www.w3.org/2001/04/xmlenc#sha256',
                'requestedAuthnContext': False,
            },
        }
@memocong
Copy link

memocong commented Dec 29, 2021

Encountered the same issue and it only happens when the SAMLResponse is encrypted.
The issue can be mitigated by patching the code:

from onelogin.saml2.xmlparser import tostring, fromstring
self.decrypted_document = fromstring(tostring(self.decrypted_document))

Reproduced the issue with the code in demo-django and dockerfile added (reference this repo)

  1. folder structure
demo-django
├── Dockerfile
├── demo
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   ├── views.py
│   └── wsgi.py
├── docker-compose.yml
├── manage.py
├── requirements.txt
├── saml
│   ├── advanced_settings.json
│   ├── certs
│   │   ├── README
│   │   ├── sp.crt
│   │   └── sp.key
│   └── settings.json
└── templates
    ├── attrs.html
    ├── base.html
    └── index.html
  1. Dockerfile
FROM python:3.8

RUN apt-get update && apt-get install -y \
    libffi-dev libxmlsec1-dev libxmlsec1-openssl xmlsec1

ADD . /code
WORKDIR /code

ADD requirements.txt /requirements.txt
RUN pip install -r /requirements.txt
  1. requirements.txt
Django==3.2
python3-saml
  1. docker-compose.yml
version: '3'

services:
  web:
    restart: always
    build: .
    command: sh -c "python manage.py runserver 0.0.0.0:9000"
    volumes:
      - .:/code
    ports:
      - "9000:9000"
  1. installed packages
Package      Version
------------ -------
asgiref      3.4.1
Django       3.2
isodate      0.6.1
lxml         4.7.1
pip          21.2.4
python3-saml 1.12.0
pytz         2021.3
setuptools   57.5.0
six          1.16.0
sqlparse     0.4.2
wheel        0.37.0
xmlsec       1.3.12

@nijel
Copy link

nijel commented Jan 14, 2022

This issue is also exposed in the python-social-auth testsuite, see python-social-auth/social-core#653

@nijel
Copy link

nijel commented Jan 14, 2022

I've tried workaround mentioned by @memocong in #292 (comment), and it does somewhat improve the situation, but still is not 100% reliable (without the workaround I get nearly 100% fail rate, with the workaround it is down to something like 20%).

Just for the reference, the workaround patch I've tried:

--- .venv/lib/python3.9/site-packages/onelogin/saml2/response.py.orig	2022-01-14 11:36:50.176430928 +0100
+++ .venv/lib/python3.9/site-packages/onelogin/saml2/response.py	2022-01-14 11:36:53.344432690 +0100
@@ -47,6 +47,8 @@
             decrypted_document = deepcopy(self.document)
             self.encrypted = True
             self.decrypted_document = self.__decrypt_assertion(decrypted_document)
+            from onelogin.saml2.xmlparser import tostring, fromstring
+            self.decrypted_document = fromstring(tostring(self.decrypted_document))
 
     def is_valid(self, request_data, request_id=None, raise_exceptions=False):
         """

To me it seems like there is some memory corruption going on inside lxml inside __decrypt_assertion.

Also, sometimes this fails with tons of namespace errors:

namespace error : Namespace prefix saml2 on Attribute is not defined
1.4.1.5923.1.1.1.1" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri"
                                                                               ^
namespace error : Namespace prefix xmlns for xsi on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix xsi for type on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix saml2 on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix xmlns for xsi on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix xsi for type on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix saml2 on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix saml2 on Attribute is not defined
1.4.1.5923.1.1.1.6" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri"
                                                                               ^
namespace error : Namespace prefix xmlns for xsi on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix xsi for type on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix saml2 on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix saml2 on Attribute is not defined
e="urn:oid:2.5.4.4" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri"
                                                                               ^
namespace error : Namespace prefix xmlns for xsi on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix xsi for type on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix saml2 on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix saml2 on Attribute is not defined
1.4.1.5923.1.1.1.9" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri"
                                                                               ^
namespace error : Namespace prefix xmlns for xsi on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix xsi for type on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix saml2 on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix xmlns for xsi on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix xsi for type on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix saml2 on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix saml2 on Attribute is not defined
="urn:oid:2.5.4.42" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri"
                                                                               ^
namespace error : Namespace prefix xmlns for xsi on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix xsi for type on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix saml2 on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix saml2 on Attribute is not defined
1.4.1.5923.1.1.1.7" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri"
                                                                               ^
namespace error : Namespace prefix xmlns for xsi on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix xsi for type on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix saml2 on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix saml2 on Attribute is not defined
e="urn:oid:2.5.4.3" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri"
                                                                               ^
namespace error : Namespace prefix xmlns for xsi on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix xsi for type on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix saml2 on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix saml2 on Attribute is not defined
.4.1.5923.1.1.1.10" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri"
                                                                               ^
namespace error : Namespace prefix saml2 on AttributeValue is not defined
meFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri"><saml2:AttributeValue
                                                                               ^
namespace error : Namespace prefix saml2 on NameID is not defined
ibboleth" SPNameQualifier="https://github.com/omab/python-social-auth/saml-test"
                                                                               ^
namespace error : Namespace prefix saml2 on Attribute is not defined
="urn:oid:2.5.4.20" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri"
                                                                               ^
namespace error : Namespace prefix xmlns for xsi on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix xsi for type on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^
namespace error : Namespace prefix saml2 on AttributeValue is not defined
Value xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string"
                                                                               ^

pitbulk added a commit to SAML-Toolkits/python-saml that referenced this issue Jan 28, 2022
… inside a encrypted element is validated.

See SAML-Toolkits/python3-saml#292
Also reverted dm.xmlsec.binding to 1.3.7
pitbulk added a commit that referenced this issue Jan 28, 2022
… inside a encrypted element is validated. See #292
@mredaelli
Copy link

Has it been determined if this is an lxml regression (as I assume it is), or if this needs to be addressed here?

If the former, does anyone have a link to an issue on lxml's bug tracker?

@MattiasLevlin
Copy link

MattiasLevlin commented May 5, 2022

Encountered the same "SubjectConfirmation" issue in our SSO solution, which utilizes python3-saml. Downgrading lxml from 4.7.0 to 4.6.4 seems to have solved the issue.

https://github.com/CSCfi/fairdata-sso/releases/tag/1.2.1

@AvnerCohen
Copy link

AvnerCohen commented May 19, 2022

My guess is that this was in issue in the 4.7.0, this version is yanked and is already fixed in 4.7.1
The change log has: https://lxml.de/4.7/changes-4.7.1.html
That points to this thread -https://mail.python.org/archives/list/[email protected]/thread/6ZFBHFOVHOS5GFDOAMPCT6HM5HZPWQ4Q/

And has been fixed in 4.7.1 if I get this right.

@nijel
Copy link

nijel commented May 19, 2022

No, it is broken in 4.7.1 and 4.8.0 as well (4.6.5 works fine). The 4.7.0 had an issue that it did not include all files in the release, that's why it's yanked.

@AvnerCohen
Copy link

Oh, I see thanks for the clarification! I'll be be updating my comment to avoid confusing people then..

@mredaelli
Copy link

No, it is broken in 4.7.1 and 4.8.0 as well (4.6.5 works fine). The 4.7.0 had an issue that it did not include all files in the release, that's why it's yanked.

Then let me ask again: do we know if they know they have a bug? Did someone open an issue somewhere?

@AvnerCohen
Copy link

@mredaelli list of open lxml bugs are here - https://bugs.launchpad.net/lxml
I think that to be able to check if the bug is open, we need to better understand what is the root cause here, on the lxml level. I don't think I see in this thread anything like that.

@nijel
Copy link

nijel commented Jun 1, 2022

I've just verified it happens in 4.9.0 as well. But it seems to happen only with the binary wheels, if I build lxml from the source, it works correctly. Can somebody else confirm such behavior? (I'm testing using python-social-auth testsuite, and it does not trigger the bug in 100%, so there is always some estimation). You can install from the source using pip install --force-reinstall --no-binary :all: lxml==4.9.0

Looking at lxml bugs, this one might be the one we're hitting: https://bugs.launchpad.net/lxml/+bug/1960668

nijel added a commit to nijel/docker that referenced this issue Jun 7, 2022
@SilviaAmAm
Copy link

I'm wondering if there is any resolution in sight for this issue? lxml has released version 4.9.1 which resolves a vulnerability issue, but this issue blocks the upgrade of projects using python3-saml 🙁

@AndrewGrossman
Copy link

I'm wondering if there is any resolution in sight for this issue? lxml has released version 4.9.1 which resolves a vulnerability issue, but this issue blocks the upgrade of projects using python3-saml 🙁

+1

nijel added a commit to WeblateOrg/docker that referenced this issue Jul 15, 2022
Recent python3-saml versions block lxml upgrade, which in turn contains
security fixes for  GHSA-wrxv-2j5q-m38w.
The version restriction seems to be caused by binary wheel not
compatible with some distros (see
SAML-Toolkits/python3-saml#292). As we build lxml
from the source, we're not affected by this.

Once python3-saml raises the restriction (for example by
SAML-Toolkits/python3-saml#309), we can switch back
to the latest version.
@bofeng
Copy link

bofeng commented Sep 15, 2022

I am having the same issue here, however, without changing any source code, this solved my issue:

$ pip uninstall lxml
$ pip install --no-binary lxml lxml==4.7.0

@nosnilmot
Copy link
Contributor

I am having the same issue here, however, without changing any source code, this solved my issue:

$ pip uninstall lxml
$ pip install --no-binary lxml lxml==4.7.0

yes, that works, but you would be better to install lxml==4.9.1

https://bugs.launchpad.net/lxml/+bug/1960668 is the canonical upstream lxml bug report. It still isn't clear if this is solvable with code changes in one or more of lxml, xmlsec and libxml2, or if the above is the correct long-term solution to ensure xmlsec and lxml are using the same version of libxml2.

sergei-maertens added a commit to open-formulieren/open-forms that referenced this issue Oct 26, 2022
This is in preparation to upgrading python3-saml and then lxml to their
latest versions.

lxml breaks on some edge cases becaues it was statically built against
a different libxml2 than *other* xml-based tooling using at runtime (
because that's dynamically linked), causing some tree lookups to fail.

* python3-saml issue: SAML-Toolkits/python3-saml#292
* upstream bug: https://bugs.launchpad.net/lxml/+bug/1960668
sergei-maertens added a commit to open-formulieren/open-forms that referenced this issue Oct 26, 2022
This is in preparation to upgrading python3-saml and then lxml to their
latest versions.

lxml breaks on some edge cases becaues it was statically built against
a different libxml2 than *other* xml-based tooling using at runtime (
because that's dynamically linked), causing some tree lookups to fail.

* python3-saml issue: SAML-Toolkits/python3-saml#292
* upstream bug: https://bugs.launchpad.net/lxml/+bug/1960668
nkukard added a commit to nkukard/PowerDNS-Admin that referenced this issue Mar 14, 2023
… built against old libxml2

This fixes PowerDNS-Admin#1442 and closes PowerDNS-Admin#1433.

The issue with python3-saml not working is because the binary lxml wheel is built against a different version of libxml2.

``--no-binary lxml`` can be used to fix this and python3-saml will work.

for references check these:
- SAML-Toolkits/python3-saml#292
- https://bugs.launchpad.net/lxml/+bug/1960668
- open-formulieren/open-forms#2247
nkukard added a commit to nkukard/PowerDNS-Admin that referenced this issue Mar 14, 2023
… built against old libxml2

This fixes PowerDNS-Admin#1442 and closes PowerDNS-Admin#1433.

The issue with python3-saml not working is because the binary lxml wheel is built against a different version of libxml2.

``--no-binary lxml`` can be used to fix this and python3-saml will work.

for references check these:
- SAML-Toolkits/python3-saml#292
- https://bugs.launchpad.net/lxml/+bug/1960668
- open-formulieren/open-forms#2247

```
Tested on:
 - ubuntu:23.04 - WORKING
 - ubuntu:22.10 - WORKING
 - ubuntu:22.04 - WORKING
 - ubuntu:20.04 - WORKING
 - ubuntu:18.04 - NOT WORKING - pip usage error with -r requirements.txt
 - debian:10    - WORKING
 - debian:11    - WORKING
 - alpine:edge  - WORKING
 - alpine:3.17  - WORKING
 - alpine:3.16  - WORKING
 - alpine:3.15  - WORKING
 - alpine:3.14  - WORKING
 - rockylinux:9 - WORKING
 - rockylinux:8 - NOT WORKING - pip usage error with -r requirements.txt
 - almalinux:9  - WORKING
 - almalinux:8  - NOT WORKING - pip usage error with -r requirements.txt
 - fedora:36    - WORKING
 - fedora:37    - WORKING
 - fedora:38    - WORKING
 - fedora:39    - WORKING
```
nkukard added a commit to nkukard/PowerDNS-Admin that referenced this issue Mar 14, 2023
Fixes Python 3.11 incompatibility using ancient lxml binary PyPI built against old libxml2.

This fixes PowerDNS-Admin#1442 and closes PowerDNS-Admin#1433.

The issue with python3-saml not working is because the binary lxml wheel is built against a different version of libxml2.

``--no-binary lxml`` can be used to fix this and python3-saml will work.

for references check these:
- SAML-Toolkits/python3-saml#292
- https://bugs.launchpad.net/lxml/+bug/1960668
- open-formulieren/open-forms#2247

```
Tested on:
 - ubuntu:23.04 - WORKING
 - ubuntu:22.10 - WORKING
 - ubuntu:22.04 - WORKING
 - ubuntu:20.04 - WORKING
 - ubuntu:18.04 - NOT WORKING - pip usage error with -r requirements.txt
 - debian:10    - WORKING
 - debian:11    - WORKING
 - alpine:edge  - WORKING
 - alpine:3.17  - WORKING
 - alpine:3.16  - WORKING
 - alpine:3.15  - WORKING
 - alpine:3.14  - WORKING
 - rockylinux:9 - WORKING
 - rockylinux:8 - NOT WORKING - pip usage error with -r requirements.txt
 - almalinux:9  - WORKING
 - almalinux:8  - NOT WORKING - pip usage error with -r requirements.txt
 - fedora:36    - WORKING
 - fedora:37    - WORKING
 - fedora:38    - WORKING
 - fedora:39    - WORKING
```
lunika added a commit to openfun/marsha that referenced this issue Apr 20, 2023
Release 4.4 of python social auth upgrades the lxml packages and seemd
to be not compatible with our code.
The issue we have is similar to this issue:
SAML-Toolkits/python3-saml#292
In a first time we decided to downgrade python social auth to a known
working version.
lunika added a commit to openfun/marsha that referenced this issue Apr 20, 2023
Release 4.4 of python social auth upgrades the lxml packages and seemd
to be not compatible with our code.
The issue we have is similar to this issue:
SAML-Toolkits/python3-saml#292
In a first time we decided to downgrade python social auth to a known
working version.
lunika added a commit to openfun/marsha that referenced this issue Apr 20, 2023
Release 4.4 of python social auth upgrades the lxml packages and seemd
to be not compatible with our code.
The issue we have is similar to this issue:
SAML-Toolkits/python3-saml#292
In a first time we decided to downgrade python social auth to a known
working version.
@jeffsawatzky
Copy link

For those of you installing python3-saml using a requirements.txt file, try doing something like this:

--no-binary lxml
# other deps
python3-saml
# more deps

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests