we may be able to use extra during validation: https://pydantic-docs.helpmanual.io/usage/model_config/ - i need to check if this adds anything to the json schema.

we will also have to explicitly drop "@context"

we will also have to explicitly drop "@context"

can't we add/define it to jsonschema export?

the issue is on the ingestion side for validate/migrate/process

@candleindark please look into how we could make our exported jsonschema more stringent and why pydantic by default does not define "additionalProperties": false or how should we change DandiBaseModel so that everything is more stringent. we might also explicitly add allowing for @context which ATM for some reason is added at dandi-archive level (as we figured out).

ATM we have only 1 object type defined with additional restriction -- that all additional properties of a digest should contain a string value.

❯ grep -4 additionalProper ../schema/releases/0.6.8/asset.json
      "nskey": "schema",
      "title": "File encoding format"
    },
    "digest": {
      "additionalProperties": {
        "type": "string"
      },
      "nskey": "dandi",
      "title": "A map of dandi digests to their values",

And indeed we would likely need to explicitly define @context as available field in json serialization (not sure how in pydantic since as cannot start with @, @candleindark mentions some "alias" feature).

@yarikoptic @satra Below is a demo of including "@context" as a field in the model and making the model not to accept any extra/additional field values.

import json

from pydantic import BaseModel, Field, AnyHttpUrl, ConfigDict, ValidationError
from jsonschema import validate, Draft202012Validator
import jsonschema


class M(BaseModel):
    context: AnyHttpUrl = Field(..., alias="@context")

    model_config = ConfigDict(extra="forbid")


# Validate a model expressed in JSON
m = M.model_validate_json('{"@context": "https://schema.org"}')

# Dump validated model
print(m.model_dump_json(by_alias=True))
'{"@context":"https://schema.org/"}'

# Provide extra field in validation input
try:
    M.model_validate_json('{"@context": "https://schema.org", "e": 42}')
except ValidationError as e:
    print("====================================")
    print(e)
    """
    1 validation error for M
    e
      Extra inputs are not permitted [type=extra_forbidden, input_value=42, input_type=int]
        For further information visit https://errors.pydantic.dev/2.9/v/extra_forbidden
    """

# JSON schema of the model
schema = M.model_json_schema()
with open("json_schema.json", "w") as f:
    json.dump(schema, f, indent=2)

instance = {"@context": "https://schema.org"}
instance_with_invalid_context = {"@context": "invalid context"}
instance_with_extra = {"@context": "https://schema.org", "e": 42}

validate(instance, schema, format_checker=Draft202012Validator.FORMAT_CHECKER)

try:
    validate(
        instance_with_invalid_context,
        schema,
        format_checker=Draft202012Validator.FORMAT_CHECKER,
    )
except jsonschema.exceptions.ValidationError as e:
    print("====================================")
    print(e)
    """
    'invalid context' is not a 'uri'
    Failed validating 'format' in schema['properties']['@context']:
        {'format': 'uri', 'minLength': 1, 'title': '@Context', 'type': 'string'}
    On instance['@context']:
        'invalid context'
    """

try:
    validate(
        instance_with_extra, schema, format_checker=Draft202012Validator.FORMAT_CHECKER
    )
except jsonschema.exceptions.ValidationError as e:
    print("====================================")
    print(e)
    """
    Additional properties are not allowed ('e' was unexpected)
    Failed validating 'additionalProperties' in schema:
        {'additionalProperties': False,
         'properties': {'@context': {'format': 'uri',
                                     'minLength': 1,
                                     'title': '@Context',
                                     'type': 'string'}},
         'required': ['@context'],
         'title': 'M',
         'type': 'object'}
    On instance:
        {'@context': 'https://schema.org', 'e': 42}
    """

The JSON schema of the model is the following.

{
  "additionalProperties": false,
  "properties": {
    "@context": {
      "format": "uri",
      "minLength": 1,
      "title": "@Context",
      "type": "string"
    }
  },
  "required": [
    "@context"
  ],
  "title": "M",
  "type": "object"
}

Please let me know if you have any objection to adding the "@context" field to the Dandiset (and PublishedDandiset) model and forbidding extra field values in the models in this manner.

Questions:

1. Is the "@context" field only needed in the Dandiset and PublishedDandiset models? Is it also needed in the Asset and PublishedAsset models?

Note:
Adding the "@context" to the models may complicate the translation to LinkML, but let's worry about that later.

Very nice. Thanks!

1. Is the "@context" field only needed in the Dandiset and PublishedDandiset models? Is it also needed in the Asset and PublishedAsset models?

yes, AFAIK in all the models if we want to unify it fully with jsonld

dandi@drogon:/tmp$ curl --silent https://dandiarchive.s3.amazonaws.com/dandisets/000003/0.230629.1955/assets.jsonld | jq .[0] | grep @context
  "@context": "https://raw.githubusercontent.com/dandi/schema/master/releases/0.4.4/context.json",

it seems we have somewhat of dichotomy now:

• pydantic model -- no @context in any form
• jsonschema exports: also no @context (besides context.json):

❯ pwd
/home/yoh/proj/dandi/schema/releases/0.6.8
❯ grep -l @context *
context.json
 ❯ 

• then we add @context into models on server,

❯ curl --silent -X 'GET' 'https://api.dandiarchive.org/api/dandisets/000003/versions/draft/' -H 'accept: application/json' | jq . | grep @cont
  "@context": "https://raw.githubusercontent.com/dandi/schema/master/releases/0.6.0/context.json",

❯ curl --silent -X GET 'https://api.dandiarchive.org/api/dandisets/000003/versions/draft/assets/d426ff9a-bab3-446d-8104-e373ae188bd3/' | jq -r '."@context"'
https://raw.githubusercontent.com/dandi/schema/master/releases/0.4.4/context.json

so would be good to unify. Please check if anything else differs between "pure" json model and "jsonld" we are exporting

Please let me know if you have any objection to adding the "@context" field to the Dandiset (and PublishedDandiset) model and forbidding extra field values in the models in this manner.

ideally we would need to check all current models in the dandisets before jumping to such a change while avoid "minor" (which is our "major") version boost and some metadata migrations.

It is unfortunate that our json dumps of models we carry in https://github.com/dandisets/ and not exact copy of the manifests we dump on S3... so for such a check it would be needed to "quickly" get them all from S3 s3://dandiarchive/dandisets/ and test draft versions... the "difficulty" might be is that models might need to be migrated first before checking.

we can go from linkml to jsonld context (the jsonld context is only relevant to instances of data). for a pydantic model, we do want to autogenerate it from linkml, and thus we would rely on the linkml generator. i don't think we want to hand craft pydantic models, perhaps only hand patch them if needed. also we could consider the linkml generator creating an appropriate context file.

in the current schema we generate that context file by hand in a function.

we can go from linkml to jsonld context (the jsonld context is only relevant to instances of data). for a pydantic model, we do want to autogenerate it from linkml, and thus we would rely on the linkml generator. i don't think we want to hand craft pydantic models, perhaps only hand patch them if needed. also we could consider the linkml generator creating an appropriate context file.

in the current schema we generate that context file by hand in a function.

@yarikoptic In this case, may be we don't need to do anything regarding the @context field in the Pydantic models and allow extra field values in the data instances for now? I know that LinkML doesn't allow extra field values in validation of data instances by default.

so, with the goal to make export jsonschema more stringent lets

• make pydantic model with model_config = ConfigDict(extra="forbid")
• in the export to jsonschema, define @context to be permitted

BTW -- I stared to import from S3 , will give you access to all of those to see if anything gets broken by more stringent checks

@candleindark dump of manifests from S3 could be found on drogon:/mnt/backup/dandi/dandiset-manifests-s3cmd . Let me know if you have problem accessing them

so, with the goal to make export jsonschema more stringent lets

• make pydantic model with model_config = ConfigDict(extra="forbid")
• in the export to jsonschema, define @context to be permitted

This will work but only in the jsonschema level. If we do any validation at Pydantic level against data instances that have the "@context" value, then this solution doesn't work.

from pydantic import BaseModel, Field, AnyHttpUrl, ConfigDict, ValidationError


class M0(BaseModel):
    pass
    # model_config["extra"] defaults to "ignore"

class M1(BaseModel):
    context: AnyHttpUrl = Field(..., alias="@context")

    model_config = ConfigDict(extra="forbid")

class M2(BaseModel):
    model_config = ConfigDict(extra="forbid")

data_instance = {"@context": "https://schema.org"}


# Validate without any problem
m0 = M0.model_validate(data_instance)

# Validate without any problem
m1 = M1.model_validate(data_instance)

try:
    m2 = M2.model_validate(data_instance)
except ValidationError as e:
    print(e)
    """
    1 validation error for M2
    @context
      Extra inputs are not permitted [type=extra_forbidden, input_value='https://schema.org', input_type=str]
        For further information visit https://errors.pydantic.dev/2.9/v/extra_forbidden
    """

In other words, if you want the proposed change, then we must make sure all data instances have the "@context" key removed when they are validated against the corresponding Pydantic model. @yarikoptic Do we proceed with this change?

@yarikoptic Never mind about the last post. I think I have a way around the issue.