Potential Bug: JSONField Serializer Failed to Validate null Json Value in HTML Form

[yitistica]

Summary

It seems that the JSONField serializer in Django Rest Framework (DRF) does not correctly validate null values when the input comes from an HTML form, which also causes inconsistent behaviour between HTML form submissions and API client requests.

to reproduce:

Create a serializer with json_data field and does not allow null value:

from rest_framework import serializers

class FormTestSerializer(serializers.Serializer):
    json_data = serializers.JSONField(allow_null=False)

test:

from django.http import QueryDict

data = QueryDict(mutable=True)
data.update({'json_data': 'null'})
serializer = FormTestSerializer(data=data)
is_valid = serializer.is_valid()  # Returns True, should be False.
validated_data = serializer.validated_data  # {'json_data': None}  correct data

Expected Behavior

The serializer should raise a validation error when allow_null=False and the input is 'null', regardless of the input source (HTML form or API client).

Actual Behavior

• HTML form input: The serializer does not raise a validation error. It returns True for is_valid().
• API client input: The serializer correctly raises a validation error.

Root Cause

data process:

within Serializer.to_internal_value(self, data):

1. retrieves the value, i.e., `primitive_value = JSONField.get_value(self, data)` 
    1.1 checks if it is html input with `html.is_html_input(dictionary)`. 
    1.2 if it is, constructs a JSONString object (defined inside the method), and returns the JSONString object. 
2. loops through fields, and calls `validated_value = Field.run_validation(self, primitive_value)` on each field. 
    2.1 calls parent's `Field.run_validation(self, primitive_value)`. 
        2.1.1 validates if the value is empty with `Field.validate_empty_values(self, primitive_value)` 
            2.1.1. JsonString('null') is not treated as a null / empty value, returns False, and the value. 
        2.1.2 if the value is empty: returns the value. 
        2.1.3 calls `value = JsonField.to_internal_value(data)`. 
            2.1.3.1 JsonString('null') will be converted to None (`json.loads('null')` and return None). 
        2.1.4* runs other validators. 
        2.1.5 returns the value. 
  3. others

During validation (2.1.1), where JsonString('null') is passed to JsonField's parent validate_empty_values(), the method implemented in the parent class does not treat it as a null/empty value.

Possible Fix

1. Add additional validation:

class JSONField(Field):
    # ...
    
    def validate_empty_values(self, data):

        if_empty, data = super().validate_empty_values(data)
        
        if if_empty:
            return if_empty, data
        
        # html form input
        if data == 'null':
            if self.allow_null:
                return (True, None)
            else:
                self.fail('null')
                
        return if_empty, data

Alternatively, 2. convert 'null' in JsonField.get_value(dictionary) method. (I do see some conversion logic implemented in other subclasses of Field regarding empty value).

    def get_value(self, dictionary):
        if html.is_html_input(dictionary) and self.field_name in dictionary:
            # When HTML form input is used, mark up the input
            # as being a JSON string, rather than a JSON primitive.
            
            value = dictionary.get(self.field_name, empty)  # <- added
            if value == 'null':                                                # <- added
                return None                                                   # <- added
            
            class JSONString(str):
                def __new__(cls, value):
                    ret = str.__new__(cls, value)
                    ret.is_json_string = True
                    return ret
            return JSONString(dictionary[self.field_name])
        return dictionary.get(self.field_name, empty)