[red-knot] Literal special form #13874

Glyphack · 2024-10-22T06:58:40Z

Handling Literal type in annotations.

Resolves: #13672

Implementation

Since Literals are not a fully defined type in typeshed. I used a trick to figure out when a special form is a literal.
When we are inferring assignment types I am checking if the type of that assignment was resolved to typing.SpecialForm and the name of the target is Literal if that is the case then I am re creating a new instance type and set the known instance field to KnownInstance:Literal.

Why not defining a new type?

From this issue I learned that we want to resolve members to SpecialMethod class. So if we create a new instance here we can rely on the member resolving in that already exists.

Tests

https://typing.readthedocs.io/en/latest/spec/literal.html#equivalence-of-two-literals
Since the type of the value inside Literal is evaluated as a Literal(LiteralString, LiteralInt, ...) then the equality is only true when types and value are equal.

https://typing.readthedocs.io/en/latest/spec/literal.html#legal-and-illegal-parameterizations

The illegal parameterizations are mostly implemented I'm currently checking the slice expression and the slice type to make sure it's valid.

Not covered:

I did not find an easy way to error on things like Literal["foo".replace("o", "b")] because I cannot fully disable attribute expressions in the Literal since enum members are allowed.
parenthesized Tuples are not allowed. Although pyright allows this in the doc is stated that tuples containing valid literal types are illegal. Tuples are valid in case of Literal["w", "r"] for example.

The union creation with Literals is not working because I saw comments about Union not implemented yet.
https://typing.readthedocs.io/en/latest/spec/literal.html#shortening-unions-of-literals

Summary

Test Plan

github-actions · 2024-10-22T22:27:00Z

`ruff-ecosystem` results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

carljm

Nice work! This task is actually quite a bit harder than I made it sound, I was forgetting some of the complexity in recognizing special forms :) This is a really good initial effort. Let me know if any of the comments below don't make sense or need further clarification.

crates/red_knot_python_semantic/resources/mdtest/literal/literal.md

crates/red_knot_python_semantic/resources/mdtest/unary/instance.md

carljm · 2024-10-22T23:55:32Z

crates/red_knot_python_semantic/src/types.rs

@@ -1130,6 +1132,7 @@ impl<'db> KnownClass {
            Self::ModuleType => "ModuleType",
            Self::FunctionType => "FunctionType",
            Self::NoneType => "NoneType",
+            Self::SpecialForm => "SpecialForm",


Probably this should match the actual name of the symbol in the typing module?

Suggested change

Self::SpecialForm => "SpecialForm",

Self::SpecialForm => "_SpecialForm",

(Un-resolved this comment, because it doesn't look addressed.) Is there a reason this needs to stay "SpecialForm" and not match the actual name in the module?

No sorry I accidentally force pushed.

crates/red_knot_python_semantic/src/types.rs

carljm · 2024-10-22T23:59:45Z

crates/red_knot_python_semantic/src/types/infer.rs

-        let annotation_ty = self.infer_annotation_expression(annotation);
+        let mut annotation_ty = self.infer_annotation_expression(annotation);
+
+        // If the variable is annotation with SpecialForm then create a new class with name of the


Suggested change

// If the variable is annotation with SpecialForm then create a new class with name of the

// If the variable is annotated with SpecialForm then create a new class with name of the

This comment also doesn't look resolved?

crates/red_knot_python_semantic/src/types/infer.rs

carljm · 2024-10-23T00:22:03Z

crates/red_knot_python_semantic/src/types/infer.rs

-                self.infer_subscript_expression(subscript);
-                Type::Todo
-            }
+            ast::Expr::Subscript(subscript) => self.infer_subscript_expression(subscript),


Here is where we need the code to recognize special forms, but we should not be falling back to infer_subscript_expression here (that's for value expressions), instead we should have a dedicated infer_subscript_type_expression method, which should use infer_type_expression on the value and the index, and for now handle only the case where the value is typing.Literal special form, otherwise just return Todo.

(The fact that infer_subscript_expression was previously called here was just an easy placeholder way to ensure we cover all the sub-expressions, until we added proper support for inferring types correctly in type expressions.)

That makes sense. Just one question, should we use infer_type_expression for all indexes?
For example Literal in here is defined with an expression in the index. Others have annotation_expression in their grammar. So here I use infer_type_expression for other things but if it's Literal I use infer_expression.

My reasoning behind it was when the value is True. The True itself should not have any meaning when used alone in the type annotation.

AlexWaygood · 2024-10-27T20:59:04Z

crates/red_knot_python_semantic/src/stdlib.rs

+/// Lookup the type of `symbol` in the `_typeshed` module namespace.
+///
+/// Returns `Unbound` if the `_typeshed` module isn't available for some reason.


Suggested change

/// Lookup the type of `symbol` in the `_typeshed` module namespace.

///

/// Returns `Unbound` if the `_typeshed` module isn't available for some reason.

/// Lookup the type of `symbol` in the `typing` module namespace.

///

/// Returns `Unbound` if the `typing` module isn't available for some reason.

This comment was marked resolved, but it looks like it is still relevant and not addressed yet? I un-resolved it.

IIRC @Glyphack did apply this suggestion using the GitHub web UI... Possibly it was accidentally lost in a force-push following that @Glyphack? :-)

Yes I'm sorry about that it caused a lot of resolved ones to be unresolved again. I will keep the comments open so I check them again before requesing review.

Glyphack · 2024-10-27T21:21:09Z

I applied the comments will spend another day on adding diagnostic messages for https://typing.readthedocs.io/en/latest/spec/literal.html#legal-and-illegal-parameterizations

Glyphack · 2024-10-28T20:46:04Z

Okay I added more parts of the legal and illegal parameters from the spec. Right now we have:

All Literal types I managed to do this through a lot of recursion.
Nested Literals

I think the remaining part is assignability check. Right now the Literals are unwrapped to their inner type I don't think this is the right way, is it? It works in the tests but I'm not sure if Literal types should carry some special flags with themselves.

I did not find an easy way to error on things like Literal["foo".replace("o", "b")] because I cannot fully disable attribute expressions in the Literal since enum members are allowed.
I can do the same I did on nested literals:

Check if it's attribute
Check the type of value and if it's a class and has Enum in bases otherwise emit diagnostic.

Does this sound good?

parenthesized Tuples are not allowed. Although pyright allows this in the doc is stated that tuples containing valid literal types are illegal. Tuples are valid in case of Literal["w", "r"] for example.

Also I'm not correctly joining union when it's possible. I left it as a todo in the tests:

    # TODO: revealed: Literal[1, 2, 3, "foo", 5] | None
    reveal_type(union_var)  # revealed: Literal[1, 2, 3] | Literal["foo"] | Literal[5] | None

Please let me know what do you think.

Glyphack · 2024-10-28T22:31:44Z

crates/red_knot_python_semantic/src/types.rs

@@ -568,58 +575,76 @@ impl<'db> Type<'db> {

            (Type::None, Type::Instance(class_type)) | (Type::Instance(class_type), Type::None) => {


I'm thinking to go over all of the instance of Instance(class_type) and rename to Instance(instance) so it's not misleading.

Yes, agree that we should do this before landing this PR. I did a couple more in the commit I just pushed, but not all of them.

Glyphack · 2024-10-28T22:35:48Z

crates/red_knot_python_semantic/src/types/infer.rs

+        }
+    }
+
+    fn infer_literal_parameter_type(&mut self, parameters: &ast::Expr) -> Type<'db> {


I created this function so I can call this on each parameter inside the [] so if we have Literal[Literal[expr]] it's converted to Literal[expr]

Glyphack · 2024-10-28T22:37:39Z

crates/red_knot_python_semantic/src/types/infer.rs

+            value_ty => {
+                let value_node = value.as_ref();
+                let slice_ty = self.infer_expression(slice);
+                // TODO: currently the logic to get the type of type of a subscript in type


I needed to keep this here because we have a test case that checks we emit unsubscriptable error in type annotations so I kept it here with a todo to not break that test.

I think if you look carefully at those test cases, they all have their own TODO comments saying that they shouldn't emit that unsubscriptable error :)

This PR will eventually need to merge with #13943 so you can look at what @AlexWaygood did there and follow the same approach.

carljm · 2024-10-29T01:22:26Z

Explaining the commit I just pushed:

My intent in suggesting InstanceType was that it would not be a Salsa interned struct, but just a regular struct that would be stored inline in Type, so we wouldn't add an extra layer of database queries. That's why I was discussing the size of Type. I wanted to verify that this really worked, and didn't increase the size of Type, before making the suggestion -- and once I had verified that, I figured I might as well push the changes and not ask you to make them again :)

carljm

Tests are looking really good here! I pushed some changes to the InstanceType implementation (so it's not Salsa-interned), and left some comments on the inference implementation.

carljm · 2024-10-28T23:38:25Z

crates/red_knot_python_semantic/resources/mdtest/literal/literal.md

+a3: Literal[-4]
+a4: Literal["hello world"]
+a5: Literal[b"hello world"]
+a6: Literal["hello world"]


This looks identical to a4?

crates/red_knot_python_semantic/resources/mdtest/literal/literal.md

carljm · 2024-10-28T23:43:03Z

crates/red_knot_python_semantic/src/stdlib.rs

+/// Lookup the type of `symbol` in the `_typeshed` module namespace.
+///
+/// Returns `Unbound` if the `_typeshed` module isn't available for some reason.


This comment was marked resolved, but it looks like it is still relevant and not addressed yet? I un-resolved it.

carljm · 2024-10-29T00:07:29Z

crates/red_knot_python_semantic/src/types.rs

@@ -1130,6 +1132,7 @@ impl<'db> KnownClass {
            Self::ModuleType => "ModuleType",
            Self::FunctionType => "FunctionType",
            Self::NoneType => "NoneType",
+            Self::SpecialForm => "SpecialForm",


(Un-resolved this comment, because it doesn't look addressed.) Is there a reason this needs to stay "SpecialForm" and not match the actual name in the module?

carljm · 2024-10-29T00:08:32Z

crates/red_knot_python_semantic/src/types.rs

+            Self::SpecialForm => {
+                let t = typing_symbol_ty(db, self.as_str());
+                debug_assert!(t.is_unbound(), "special form not found");
+                t
+            }


We don't do this assert for the other special forms, I don't think we need to here either.

Suggested change

Self::SpecialForm => {

let t = typing_symbol_ty(db, self.as_str());

debug_assert!(t.is_unbound(), "special form not found");

t

}

Self::SpecialForm => typing_symbol_ty(db, self.as_str())

carljm · 2024-10-29T01:51:54Z

crates/red_knot_python_semantic/src/types/infer.rs

+            value_ty => {
+                let value_node = value.as_ref();
+                let slice_ty = self.infer_expression(slice);
+                // TODO: currently the logic to get the type of type of a subscript in type


I think if you look carefully at those test cases, they all have their own TODO comments saying that they shouldn't emit that unsubscriptable error :)

This PR will eventually need to merge with #13943 so you can look at what @AlexWaygood did there and follow the same approach.

carljm · 2024-10-29T01:53:41Z

crates/red_knot_python_semantic/src/types/infer.rs

+        // slice_ty is treated as expression because Literal accepts expression
+        // inside the []


This comment seems out of place and probably unnecessary? There's nothing near this comment named slice_ty, and I'm not sure what "treated as expression" means -- the slice of a subscript expression is an expression in the AST, that's just a fact.

carljm · 2024-10-29T01:56:20Z

crates/red_knot_python_semantic/src/types/infer.rs

+                match parameters {
+                    ruff_python_ast::Expr::StringLiteral(_)
+                    | ruff_python_ast::Expr::BytesLiteral(_)
+                    | ruff_python_ast::Expr::BooleanLiteral(_)
+                    // For enum values
+                    | ruff_python_ast::Expr::Attribute(_)
+                    // For Another Literal inside this Literal
+                    | ruff_python_ast::Expr::Subscript(_)
+                    | ruff_python_ast::Expr::NoneLiteral(_) => {}
+                    // for negative numbers
+                    ruff_python_ast::Expr::UnaryOp(ref u) if (u.op == UnaryOp::USub || u.op == UnaryOp::UAdd) && u.operand.is_number_literal_expr() => {}
+                    ruff_python_ast::Expr::NumberLiteral(ref number) if number.value.is_int() => {}
+                    ruff_python_ast::Expr::Tuple(ref t) if !t.parenthesized => {}
+                    _ => {
+                        self.add_diagnostic(
+                            parameters.into(),
+                            "invalid-literal-parameter",
+                            format_args!(
+                                "Type arguments for `Literal` must be None, a literal value (int, bool, str, or bytes), or an enum value",
+                            ),
+                        );
+                        return Type::Unknown;
+                    }
+                };
+
+                let slice_ty = self.infer_literal_parameter_type(parameters);
+
+                match slice_ty {
+                    Type::Never
+                    | Type::Unknown
+                    | Type::Unbound
+                    | Type::Todo
+                    | Type::FunctionLiteral(_)
+                    | Type::ModuleLiteral(_)
+                    | Type::ClassLiteral(_)
+                    | Type::Union(_)
+                    | Type::Intersection(_)
+                    | Type::Any => {
+                        self.add_diagnostic(
+                                    parameters.into(),
+                                    "invalid-literal-parameter",
+                                    format_args!(
+                                        "Type arguments for `Literal` must be None, a literal value (int, bool, str, or bytes), or an enum value",
+                                    ),
+                                );
+                        Type::Unknown
+                    }
+                    Type::Tuple(tuple) => {
+                        let elts = tuple.elements(self.db);
+                        Type::Union(UnionType::new(self.db, elts))
+                    }
+                    ty => ty,
+                }
+            }


I think a better approach here would be for infer_literal_parameter_type to return Option<Type<'db>>, and return None if the literal parameter is not valid, otherwise the right Type. Then you only need to emit the error in one place (if infer_literal_parameter_type returns None), and you don't need these two extra match statements here.

carljm · 2024-10-29T01:59:26Z

crates/red_knot_python_semantic/src/types/infer.rs

+        // the values
+        match parameters {
+            ruff_python_ast::Expr::Subscript(inner_literal_subscript) => {
+                let inner_subscript_value = self.infer_expression(&inner_literal_subscript.value);


Doesn't seem right to use infer_expression here, it should be infer_type_expression -- and then you shouldn't have to repeat the recognition of Literal below, or the invalid-literal-parameter error, infer_type_expression should do all that for you? You just have to verify the type you get back is a literal type.

carljm · 2024-10-29T02:01:38Z

crates/red_knot_python_semantic/src/types/infer.rs

+                }
+                Type::Tuple(TupleType::new(self.db, elts.into_boxed_slice()))
+            }
+            _ => self.infer_expression(parameters),


Rather than matching out in infer_parameterized_known_instance_type_expression on AST forms known not to be valid, I think we should explicitly match here on each AST form known to be valid, and directly return the right type, without relying on self.infer_expression.

Co-authored-by: Carl Meyer <[email protected]>

AlexWaygood

A couple more comments on top of Carl's

AlexWaygood · 2024-10-29T13:56:40Z

crates/red_knot_python_semantic/src/types/infer.rs

+                        let elts = tuple.elements(self.db);
+                        Type::Union(UnionType::new(self.db, elts))


there's a bit of a footgun here in our current design, in that you should never really use UnionType::new() directly, because it doesn't deduplicate the elements in the union. Instead you should always use UnionType::from_elements(), which takes care of all the deduplication for you

AlexWaygood · 2024-10-29T13:58:50Z

crates/red_knot_python_semantic/src/types/infer.rs

+                    self.add_diagnostic(
+                                        parameters.into(),
+                                        "invalid-literal-parameter",
+                                        format_args!(
+                                            "Type arguments for `Literal` must be None, a literal value (int, bool, str, or bytes), or an enum value",
+                                        ),
+                                    );


the formatting here is somewhat skew-whiff (I think cargo fmt is failing to spot it because of the macro, unfortunately)

Suggested change

self.add_diagnostic(

parameters.into(),

"invalid-literal-parameter",

format_args!(

"Type arguments for `Literal` must be None, a literal value (int, bool, str, or bytes), or an enum value",

),

);

self.add_diagnostic(

parameters.into(),

"invalid-literal-parameter",

format_args!(

"Type arguments for `Literal` must be None, a literal value (int, bool, str, or bytes), or an enum value",

),

);

AlexWaygood · 2024-10-29T14:01:09Z

crates/red_knot_python_semantic/src/types/infer.rs

+            ruff_python_ast::Expr::Tuple(t) => {
+                let mut elts = vec![];
+                for elm in &t.elts {
+                    elts.push(self.infer_literal_parameter_type(elm));
+                }
+                Type::Tuple(TupleType::new(self.db, elts.into_boxed_slice()))
+            }


Suggested change

ruff_python_ast::Expr::Tuple(t) => {

let mut elts = vec![];

for elm in &t.elts {

elts.push(self.infer_literal_parameter_type(elm));

}

Type::Tuple(TupleType::new(self.db, elts.into_boxed_slice()))

}

ruff_python_ast::Expr::Tuple(t) => {

let elements: Box<_> = t.iter().map(|elt| self.infer_literal_parameter_type(elt)).collect();

Type::Tuple(TupleType::new(self.db, elements))

}

Co-authored-by: Alex Waygood <[email protected]>

…al.md Co-authored-by: Carl Meyer <[email protected]>

Glyphack force-pushed the literal-type branch 4 times, most recently from 3fe84bd to fbcc66c Compare October 22, 2024 22:13

Glyphack marked this pull request as ready for review October 22, 2024 22:26

Glyphack requested review from carljm, MichaReiser and AlexWaygood as code owners October 22, 2024 22:26

carljm requested changes Oct 23, 2024

View reviewed changes

carljm added the red-knot Multi-file analysis & type inference label Oct 23, 2024

Glyphack force-pushed the literal-type branch 2 times, most recently from bb69a4c to a5a4f7f Compare October 27, 2024 17:07

AlexWaygood reviewed Oct 27, 2024

View reviewed changes

Glyphack added 5 commits October 28, 2024 21:44

Handle Literal special form

3994c5d

Update test comments

2bcd3d9

Create InstanceType value for Type::Instance

2cbc65f

Handle inferring Literal type annotation

2bb37dc

Handle legal and illegal parameterization for literals

93e3358

Glyphack force-pushed the literal-type branch from 3fc52a5 to 93e3358 Compare October 28, 2024 20:45

Glyphack added 2 commits October 28, 2024 21:52

Add todo for producing correct unions with literals

c69d1dc

Cargo clippy fix

ea75875

Glyphack requested review from carljm and AlexWaygood October 28, 2024 21:04

Glyphack changed the title ~~Add Literal special form to types~~ Literal special form Oct 28, 2024

Glyphack changed the title ~~Literal special form~~ [red-knot] Literal special form Oct 28, 2024

Update instance.md

52b6ea8

Glyphack commented Oct 28, 2024

View reviewed changes

Update infer.rs

eea67c0

Glyphack commented Oct 28, 2024

View reviewed changes

don't intern InstanceType

a3cfd30

carljm requested changes Oct 29, 2024

View reviewed changes

Glyphack and others added 3 commits October 29, 2024 09:22

Update crates/red_knot_python_semantic/src/types.rs

68fbb1a

Co-authored-by: Carl Meyer <[email protected]>

Update crates/red_knot_python_semantic/src/types/infer.rs

063d52f

Co-authored-by: Carl Meyer <[email protected]>

Update crates/red_knot_python_semantic/src/types.rs

07698e4

Co-authored-by: Carl Meyer <[email protected]>

AlexWaygood reviewed Oct 29, 2024

View reviewed changes

Glyphack and others added 6 commits October 29, 2024 18:54

Update crates/red_knot_python_semantic/src/stdlib.rs

4ed3007

Co-authored-by: Alex Waygood <[email protected]>

Merge branch 'main' into literal-type

4916142

Update crates/red_knot_python_semantic/resources/mdtest/literal/liter…

aad11c8

…al.md Co-authored-by: Carl Meyer <[email protected]>

Renamed all usages of Instance(class)

1699d2c

Remove duplicate test case

1a7b01a

Replace Instance(class) with Instance(instance)

7b8dd85

	Self::SpecialForm => "SpecialForm",
	Self::SpecialForm => "_SpecialForm",

	// If the variable is annotation with SpecialForm then create a new class with name of the
	// If the variable is annotated with SpecialForm then create a new class with name of the

		@@ -568,58 +575,76 @@ impl<'db> Type<'db> {

		(Type::None, Type::Instance(class_type)) \| (Type::Instance(class_type), Type::None) => {

		// slice_ty is treated as expression because Literal accepts expression
		// inside the []

		let elts = tuple.elements(self.db);
		Type::Union(UnionType::new(self.db, elts))

[red-knot] Literal special form #13874

Are you sure you want to change the base?

[red-knot] Literal special form #13874

Conversation

Glyphack commented Oct 22, 2024 • edited Loading

Implementation

Tests

Summary

Test Plan

github-actions bot commented Oct 22, 2024 • edited Loading

ruff-ecosystem results

Linter (stable)

Linter (preview)

carljm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carljm Oct 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexWaygood Oct 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Glyphack commented Oct 27, 2024 • edited Loading

Glyphack commented Oct 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carljm commented Oct 29, 2024

carljm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexWaygood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Glyphack commented Oct 22, 2024 •

edited

Loading

github-actions bot commented Oct 22, 2024 •

edited

Loading

`ruff-ecosystem` results

carljm Oct 23, 2024 •

edited

Loading

AlexWaygood Oct 29, 2024 •

edited

Loading

Glyphack commented Oct 27, 2024 •

edited

Loading

Glyphack commented Oct 28, 2024 •

edited

Loading