[red-knot] Literal special form #13874

Glyphack · 2024-10-22T06:58:40Z

Handling Literal type in annotations.

Resolves: #13672

Implementation

Since Literals are not a fully defined type in typeshed. I used a trick to figure out when a special form is a literal.
When we are inferring assignment types I am checking if the type of that assignment was resolved to typing.SpecialForm and the name of the target is Literal if that is the case then I am re creating a new instance type and set the known instance field to KnownInstance:Literal.

Why not defining a new type?

From this issue I learned that we want to resolve members to SpecialMethod class. So if we create a new instance here we can rely on the member resolving in that already exists.

Tests

https://typing.readthedocs.io/en/latest/spec/literal.html#equivalence-of-two-literals
Since the type of the value inside Literal is evaluated as a Literal(LiteralString, LiteralInt, ...) then the equality is only true when types and value are equal.

https://typing.readthedocs.io/en/latest/spec/literal.html#legal-and-illegal-parameterizations

The illegal parameterizations are mostly implemented I'm currently checking the slice expression and the slice type to make sure it's valid.

Not covered:

I did not find an easy way to error on things like Literal["foo".replace("o", "b")] because I cannot fully disable attribute expressions in the Literal since enum members are allowed.
parenthesized Tuples are not allowed. Although pyright allows this in the doc is stated that tuples containing valid literal types are illegal. Tuples are valid in case of Literal["w", "r"] for example.

The union creation with Literals is not working because I saw comments about Union not implemented yet.
https://typing.readthedocs.io/en/latest/spec/literal.html#shortening-unions-of-literals

Summary

Test Plan

github-actions · 2024-10-22T22:27:00Z

`ruff-ecosystem` results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

carljm

Nice work! This task is actually quite a bit harder than I made it sound, I was forgetting some of the complexity in recognizing special forms :) This is a really good initial effort. Let me know if any of the comments below don't make sense or need further clarification.

crates/red_knot_python_semantic/resources/mdtest/literal/literal.md

crates/red_knot_python_semantic/resources/mdtest/unary/instance.md

carljm · 2024-10-22T23:55:32Z

crates/red_knot_python_semantic/src/types.rs

@@ -1130,6 +1132,7 @@ impl<'db> KnownClass {
            Self::ModuleType => "ModuleType",
            Self::FunctionType => "FunctionType",
            Self::NoneType => "NoneType",
+            Self::SpecialForm => "SpecialForm",


Probably this should match the actual name of the symbol in the typing module?

Suggested change

Self::SpecialForm => "SpecialForm",

Self::SpecialForm => "_SpecialForm",

(Un-resolved this comment, because it doesn't look addressed.) Is there a reason this needs to stay "SpecialForm" and not match the actual name in the module?

No sorry I accidentally force pushed.

crates/red_knot_python_semantic/src/types.rs

carljm · 2024-10-22T23:59:45Z

crates/red_knot_python_semantic/src/types/infer.rs

-        let annotation_ty = self.infer_annotation_expression(annotation);
+        let mut annotation_ty = self.infer_annotation_expression(annotation);
+
+        // If the variable is annotation with SpecialForm then create a new class with name of the


Suggested change

// If the variable is annotation with SpecialForm then create a new class with name of the

// If the variable is annotated with SpecialForm then create a new class with name of the

This comment also doesn't look resolved?

crates/red_knot_python_semantic/src/types/infer.rs

carljm · 2024-10-23T00:22:03Z

crates/red_knot_python_semantic/src/types/infer.rs

-                self.infer_subscript_expression(subscript);
-                Type::Todo
-            }
+            ast::Expr::Subscript(subscript) => self.infer_subscript_expression(subscript),


Here is where we need the code to recognize special forms, but we should not be falling back to infer_subscript_expression here (that's for value expressions), instead we should have a dedicated infer_subscript_type_expression method, which should use infer_type_expression on the value and the index, and for now handle only the case where the value is typing.Literal special form, otherwise just return Todo.

(The fact that infer_subscript_expression was previously called here was just an easy placeholder way to ensure we cover all the sub-expressions, until we added proper support for inferring types correctly in type expressions.)

That makes sense. Just one question, should we use infer_type_expression for all indexes?
For example Literal in here is defined with an expression in the index. Others have annotation_expression in their grammar. So here I use infer_type_expression for other things but if it's Literal I use infer_expression.

My reasoning behind it was when the value is True. The True itself should not have any meaning when used alone in the type annotation.

AlexWaygood · 2024-10-27T20:59:04Z

crates/red_knot_python_semantic/src/stdlib.rs

+/// Lookup the type of `symbol` in the `_typeshed` module namespace.
+///
+/// Returns `Unbound` if the `_typeshed` module isn't available for some reason.


Suggested change

/// Lookup the type of `symbol` in the `_typeshed` module namespace.

///

/// Returns `Unbound` if the `_typeshed` module isn't available for some reason.

/// Lookup the type of `symbol` in the `typing` module namespace.

///

/// Returns `Unbound` if the `typing` module isn't available for some reason.

This comment was marked resolved, but it looks like it is still relevant and not addressed yet? I un-resolved it.

IIRC @Glyphack did apply this suggestion using the GitHub web UI... Possibly it was accidentally lost in a force-push following that @Glyphack? :-)

Yes I'm sorry about that it caused a lot of resolved ones to be unresolved again. I will keep the comments open so I check them again before requesing review.

Glyphack · 2024-10-27T21:21:09Z

I applied the comments will spend another day on adding diagnostic messages for https://typing.readthedocs.io/en/latest/spec/literal.html#legal-and-illegal-parameterizations

Glyphack · 2024-10-28T20:46:04Z

Okay I added more parts of the legal and illegal parameters from the spec. Right now we have:

All Literal types I managed to do this through a lot of recursion.
Nested Literals

I think the remaining part is assignability check. Right now the Literals are unwrapped to their inner type I don't think this is the right way, is it? It works in the tests but I'm not sure if Literal types should carry some special flags with themselves.

I did not find an easy way to error on things like Literal["foo".replace("o", "b")] because I cannot fully disable attribute expressions in the Literal since enum members are allowed.
I can do the same I did on nested literals:

Check if it's attribute
Check the type of value and if it's a class and has Enum in bases otherwise emit diagnostic.

Does this sound good?

parenthesized Tuples are not allowed. Although pyright allows this in the doc is stated that tuples containing valid literal types are illegal. Tuples are valid in case of Literal["w", "r"] for example.

Also I'm not correctly joining union when it's possible. I left it as a todo in the tests:

    # TODO: revealed: Literal[1, 2, 3, "foo", 5] | None
    reveal_type(union_var)  # revealed: Literal[1, 2, 3] | Literal["foo"] | Literal[5] | None

Please let me know what do you think.

crates/red_knot_python_semantic/src/types.rs

crates/red_knot_python_semantic/src/types/builder.rs

crates/red_knot_python_semantic/src/types/infer.rs

crates/red_knot_python_semantic/src/types.rs

AlexWaygood · 2024-11-04T15:00:13Z

crates/red_knot_python_semantic/src/types.rs

+    pub fn anonymous(class: ClassType<'db>) -> Self {
+        Self { class, known: None }
+    }
+
+    pub fn known(class: ClassType<'db>, known: KnownInstance) -> Self {
+        Self {
+            class,
+            known: Some(known),
+        }
+    }


I think the known method here needs to have a different name, or it conflicts with the name of the field on the class. Rust lets you do that, but I find it pretty confusing to have a method the same name as a field that does quite a different thing 😆

How about new_known() and new_unknown for the two constructors? That would also have the nice property of making it obvious that they are parallel to each other

Hmm, I quite like the names as they are :/ There's no actual conflict; the method is an associated method, so it can't be called on an instance of the struct, only as InstanceType::known, and the field can only be accessed on an instance (self.known). And they are closely related and refer to the same thing, so it's not like we have a terminology clash that's confusing two unrelated concepts. It makes sense to me to have a constructor named known that creates an instance of the struct with its optional known field set to Some(...).

I feel like avoiding this is maybe a Python habit, where it really would be a problem.

Using new_* for alternate constructors feels unwieldy and unusual. The Default trait doesn't define a new_default method, it defines a default method. And I feel that new_unknown is too close to the Unknown type. I prefer "anonymous" over "unknown" as the opposite to a "known" instance or class.

crates/red_knot_python_semantic/src/types.rs

AlexWaygood

Nice, this looks great!! A few nitpicks, but nothing serious. There's a new merge conflict (sorry!) resulting from the recent removal of Type::None in favour of Type::Instance(_typeshed.NoneType), but it shouldn't be too hard to fix, I don't think

crates/red_knot_python_semantic/src/types/infer.rs

crates/red_knot_python_semantic/src/types.rs

crates/red_knot_python_semantic/src/types/infer.rs

Co-authored-by: Alex Waygood <[email protected]>

* main: (39 commits) Also remove trailing comma while fixing C409 and C419 (astral-sh#14097) Re-enable clippy `useless-format` (astral-sh#14095) Derive message formats macro support to string (astral-sh#14093) Avoid cloning `Name` when looking up function and class types (astral-sh#14092) Replace `format!` without parameters with `.to_string()` (astral-sh#14090) [red-knot] Do not panic when encountering string annotations (astral-sh#14091) [red-knot] Add MRO resolution for classes (astral-sh#14027) [red-knot] Remove `Type::None` (astral-sh#14024) Cached inference of all definitions in an unpacking (astral-sh#13979) Update dependency uuid to v11 (astral-sh#14084) Update Rust crate notify to v7 (astral-sh#14083) Update cloudflare/wrangler-action action to v3.11.0 (astral-sh#14080) Update dependency mdformat-mkdocs to v3.1.1 (astral-sh#14081) Update pre-commit dependencies (astral-sh#14082) Update dependency ruff to v0.7.2 (astral-sh#14077) Update NPM Development dependencies (astral-sh#14078) Update Rust crate thiserror to v1.0.67 (astral-sh#14076) Update Rust crate syn to v2.0.87 (astral-sh#14075) Update Rust crate serde to v1.0.214 (astral-sh#14074) Update Rust crate pep440_rs to v0.7.2 (astral-sh#14073) ...

carljm

There were some things about the behavior of this PR that confused me, so I went ahead and merged in main and did the merge conflict resolution so I could explore those things. Since I did the work, going ahead and pushing it (with a few other small changes) to make sure we don't duplicate work.

Now continuing to review the rest...

carljm · 2024-11-05T00:33:33Z

crates/red_knot_python_semantic/resources/mdtest/literal/literal.md

+a1: Literal[26]
+
+def f():
+    reveal_type(a1)  # revealed: @Todo


I merged in main and this changed to Literal[26], which is what it should be. I'm not sure why the merge from main fixed it? Maybe I fixed some logic in merge conflict resolution.

carljm · 2024-11-05T00:40:48Z

crates/red_knot_python_semantic/resources/mdtest/literal/literal.md

+```py
+from typing import _SpecialForm
+
+Literal: _SpecialForm
+a1: Literal[26]
+
+def f():
+    reveal_type(a1)  # revealed: @Todo
+```


I think this test was passing for the wrong reason (for the same reason the below test wrongly had @Todo).

The reason this test was behaving oddly turned out to be pretty subtle. Locally within the same scope, we don't consider an annotated assignment with no RHS to actually define the name (because, well, at runtime it doesn't.) So what was actually happening here on the a1: Literal[26] line is that we failed to find a value for Literal in the same scope, and we looked it up in builtins, and we found it there! Imported from typing.Literal, because builtins.pyi actually does import Literal from typing, and we haven't yet implemented the re-export rules for imports that would tell us to ignore it: #14099 So currently we wrongly treat typing.Literal as a built-in :)

I fixed this test to create its "custom" Literal in a separate type stub and import it from there, which makes the test work as expected.

* main: [red-knot] Implement type narrowing for boolean conditionals (astral-sh#14037)

carljm

Great work!! The fixes needed were minor, and I really want to get this merged now to avoid more conflict resolution needed tomorrow, so I went ahead and made the small updates and am going to go ahead and merge it. If anyone has review comments on the bits that I'm landing unreviewed (or on the InstanceType::known naming question), one of us can put up a small follow-up PR tomorrow to take care of them.

Thank you @Glyphack for all your work on this! It definitely turned out to be more complex than I'd initially realized :)

carljm · 2024-11-05T00:54:08Z

crates/red_knot_python_semantic/src/types.rs

+    pub fn anonymous(class: ClassType<'db>) -> Self {
+        Self { class, known: None }
+    }
+
+    pub fn known(class: ClassType<'db>, known: KnownInstance) -> Self {
+        Self {
+            class,
+            known: Some(known),
+        }
+    }


Hmm, I quite like the names as they are :/ There's no actual conflict; the method is an associated method, so it can't be called on an instance of the struct, only as InstanceType::known, and the field can only be accessed on an instance (self.known). And they are closely related and refer to the same thing, so it's not like we have a terminology clash that's confusing two unrelated concepts. It makes sense to me to have a constructor named known that creates an instance of the struct with its optional known field set to Some(...).

I feel like avoiding this is maybe a Python habit, where it really would be a problem.

Using new_* for alternate constructors feels unwieldy and unusual. The Default trait doesn't define a new_default method, it defines a default method. And I feel that new_unknown is too close to the Unknown type. I prefer "anonymous" over "unknown" as the opposite to a "known" instance or class.

Glyphack · 2024-11-05T04:37:26Z

Thanks for your time and guidance, I appreciate it.

Glyphack force-pushed the literal-type branch 4 times, most recently from 3fe84bd to fbcc66c Compare October 22, 2024 22:13

Glyphack marked this pull request as ready for review October 22, 2024 22:26

Glyphack requested review from carljm, MichaReiser and AlexWaygood as code owners October 22, 2024 22:26

carljm requested changes Oct 23, 2024

View reviewed changes

carljm added the red-knot Multi-file analysis & type inference label Oct 23, 2024

Glyphack force-pushed the literal-type branch 2 times, most recently from bb69a4c to a5a4f7f Compare October 27, 2024 17:07

AlexWaygood reviewed Oct 27, 2024

View reviewed changes

Glyphack added 5 commits October 28, 2024 21:44

Handle Literal special form

3994c5d

Update test comments

2bcd3d9

Create InstanceType value for Type::Instance

2cbc65f

Handle inferring Literal type annotation

2bb37dc

Handle legal and illegal parameterization for literals

93e3358

Glyphack force-pushed the literal-type branch from 3fc52a5 to 93e3358 Compare October 28, 2024 20:45

Glyphack added 2 commits October 28, 2024 21:52

Add todo for producing correct unions with literals

c69d1dc

Cargo clippy fix

ea75875

Glyphack requested review from carljm and AlexWaygood October 28, 2024 21:04

Glyphack changed the title ~~Add Literal special form to types~~ Literal special form Oct 28, 2024

Glyphack changed the title ~~Literal special form~~ [red-knot] Literal special form Oct 28, 2024

Update instance.md

52b6ea8

Glyphack commented Oct 28, 2024

View reviewed changes

crates/red_knot_python_semantic/src/types.rs Outdated Show resolved Hide resolved

Glyphack added 3 commits November 2, 2024 17:41

Unpack class type when matching instance

eb30e9b

Use InstanceType::anonymous method

2b78629

Reformat code

95a7118

Glyphack commented Nov 2, 2024

View reviewed changes

crates/red_knot_python_semantic/src/types/builder.rs Show resolved Hide resolved

Update crates/red_knot_python_semantic/src/types/builder.rs

bb59ec8

Glyphack commented Nov 2, 2024

View reviewed changes

crates/red_knot_python_semantic/src/types/infer.rs Outdated Show resolved Hide resolved

Glyphack added 6 commits November 2, 2024 17:59

Update crates/red_knot_python_semantic/src/types/infer.rs

a675695

Merge branch 'main' into literal-type

1be6c40

Fix

2b6cb4b

Reorder imports

fa37e42

Reformat test

67c1deb

Fix infer type for todo cases

47c9d2b

Glyphack requested review from AlexWaygood and carljm November 2, 2024 18:21

AlexWaygood reviewed Nov 4, 2024

View reviewed changes

Glyphack and others added 4 commits November 4, 2024 10:17

Apply suggestions from code review

9a73ab2

Co-authored-by: Alex Waygood <[email protected]>

fix wrong-Literal test

8880e2e

display Literal differently from _SpecialForm

2d6c97e

carljm reviewed Nov 5, 2024

View reviewed changes

carljm added 2 commits November 4, 2024 16:46

Merge branch 'main' into literal-type

06d550d

* main: [red-knot] Implement type narrowing for boolean conditionals (astral-sh#14037)

improve error handling

1d081dd

carljm approved these changes Nov 5, 2024

View reviewed changes

tighten up handling of nested Literal

cf71e88

carljm enabled auto-merge (squash) November 5, 2024 01:41

carljm merged commit 9dddd73 into astral-sh:main Nov 5, 2024
17 checks passed

Glyphack deleted the literal-type branch November 5, 2024 04:36

AlexWaygood mentioned this pull request Nov 27, 2024

[red-knot] No panic for illegal self-referential f-string annotations #14629

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[red-knot] Literal special form #13874

[red-knot] Literal special form #13874

Glyphack commented Oct 22, 2024 •

edited

Loading

github-actions bot commented Oct 22, 2024 •

edited

Loading

carljm left a comment

carljm Oct 22, 2024

carljm Oct 29, 2024

Glyphack Oct 29, 2024

carljm Oct 22, 2024

carljm Oct 29, 2024

carljm Oct 23, 2024 •

edited

Loading

Glyphack Oct 27, 2024

AlexWaygood Oct 27, 2024

carljm Oct 28, 2024

AlexWaygood Oct 29, 2024 •

edited

Loading

Glyphack Oct 29, 2024

Glyphack commented Oct 27, 2024 •

edited

Loading

Glyphack commented Oct 28, 2024 •

edited

Loading

AlexWaygood Nov 4, 2024 •

edited

Loading

carljm Nov 5, 2024

AlexWaygood left a comment

carljm left a comment

carljm Nov 5, 2024

carljm Nov 5, 2024 •

edited

Loading

carljm left a comment

carljm Nov 5, 2024

Glyphack commented Nov 5, 2024

	Self::SpecialForm => "SpecialForm",
	Self::SpecialForm => "_SpecialForm",

	// If the variable is annotation with SpecialForm then create a new class with name of the
	// If the variable is annotated with SpecialForm then create a new class with name of the

[red-knot] Literal special form #13874

[red-knot] Literal special form #13874

Conversation

Glyphack commented Oct 22, 2024 • edited Loading

Implementation

Tests

Summary

Test Plan

github-actions bot commented Oct 22, 2024 • edited Loading

ruff-ecosystem results

Linter (stable)

Linter (preview)

carljm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carljm Oct 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexWaygood Oct 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Glyphack commented Oct 27, 2024 • edited Loading

Glyphack commented Oct 28, 2024 • edited Loading

AlexWaygood Nov 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexWaygood left a comment

Choose a reason for hiding this comment

carljm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carljm Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

carljm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Glyphack commented Nov 5, 2024

Glyphack commented Oct 22, 2024 •

edited

Loading

github-actions bot commented Oct 22, 2024 •

edited

Loading

`ruff-ecosystem` results

carljm Oct 23, 2024 •

edited

Loading

AlexWaygood Oct 29, 2024 •

edited

Loading

Glyphack commented Oct 27, 2024 •

edited

Loading

Glyphack commented Oct 28, 2024 •

edited

Loading

AlexWaygood Nov 4, 2024 •

edited

Loading

carljm Nov 5, 2024 •

edited

Loading