Entities MUST contain a rel attribute?

[ericelliott]

entities

A collection of related sub-entities. If a sub-entity contains an href value, it should be treated as an embedded link. Clients may choose to optimistically load embedded links. If no href value exists, the sub-entity is an embedded entity representation that contains all the characteristics of a typical entity. One difference is that a sub-entity MUST contain a rel attribute to describe its relationship to the parent entity.

I'm building a resourceful API. Each resource has an index where you can list / search / browse child entities.

The way that it's structured, it's obvious that the relationship of the entities in the index is that they are child records associated with the resource. For example:

/albums
Nevermind - Nirvana
Pretty Hate Machine - Nine Inch Nails
Discovery - Daft Punk

I don't want to come up with a different rel relationship for each kind of resource, or it would turn into an ever-proliferating taxonomy that essentially adds no real value.

Which leads me to my next point: These rels aren't adding any value to my API, but there's a rel associated with every record in the list -- the same rel. It doesn't make any sense to duplicate that rel for every item in the list.

I'd just make these links instead (SHOULD, yay!), except then I can't specify properties to display with each link (a nice feature of entities which lets you call out the interesting features of an entity which might be good for listing entities on a web page).

Any chance we could change MUST to SHOULD here? I'm not sure I understand the rigidity of this requirement.

[kevinswiber]

The idea is that a sub-entity has some kind of relationship worth broadcasting to clients. For collection/item scenarios, I often use "rel": ["item"], per RFC 6573[1] and often don't make up individual link relations that would be specific yet provide little to no value.

Side Note
Just to make sure we're on the same page with terminology, here's how I think about different elements of Siren. In general, a link relation helps a client decide if the relationship should be explored by the user agent (automated or otherwise). Classes help determine characteristics about the entity once a client is there. Properties provide domain specific data. Sub-entities communicate domain-specific relationships. Links are often non-specific to a domain. Actions offer a way to execute queries and state transitions.

The other question is whether or not this could change from MUST to SHOULD. In general, I'm open to relaxing anything that's a MUST or MUST NOT. I'm a little hesitant on this one, however. There is a clear distinction between what constitutes a property value vs. a sub-entity value. I'm concerned that relaxing this requirement will cause those concepts to start to blend. Thoughts?

[1] http://tools.ietf.org/html/rfc6573

[apsoto]

@kevinswiber that 'Side Note' is the BEST explanation so far. Really should put that in the spec somewhere.

[ericelliott]

@kevinswiber My concern is that the "rel": ["item"] relationship is basically implied for every entity returned from an index operation. It is not efficient to require that each item entity declare that it is an item entity when the relationship is clearly implied by virtue of the fact that they're being listed as subordinates of the resource index.

Any resource-aware client can safely assume that without relying on any out-of-band information. It makes more sense to signal to the client that the endpoint is a resource index by including information in the index properties, instead of repeating the same information on every item in the collection.

Here's another idea: Make it possible to assign a rel value to the entities collection itself, implying that the value applies to every entity in the collection. That way you communicate exactly the same information in a much more efficient manner.

I'm not sure what you're getting at with the index properties concern. Properties don't seem to be a good place to put resource children, since they are clearly distinct entities, and not simply properties of the index entity.

[apsoto]

I'm no expert with the rel stuff as it is, but the 'item' is a simple example, and if all sub-entities are of the same type, then it is implied.

I think the rel may be more useful in the case when you have multiple sub-entities that aren't of the same type? Totally made up example below. Not sure if it's what @kevinswiber would have in mind, but how I think about it.

/book/1
/book/1/authors/1
/book/1/authors/2
/book/1/publisher/1
/book/1/sales_data/1-201310
...
/book/1/sales_data/1-201301


{
....
  "entities" : [
    {
      "class" : ["author"],
      "rel" : ['primary'],
      .....
    },
    {
      "class" : ["author"],
      "rel" : ['contributor'],
      .....
    },
    {
      "class" : ["publisher"],
      "rel" : ['something'],
      .....
    },
    {
      "class" : ["sales_data"],
      "rel" : ['current'],
      .....
    },
    {
      "class" : ["sales_data"],
      "rel" : ['last'],
      .....
    }


  ]
}

[kevinswiber]

@dilvie Thank you for your feedback and suggestions. They're helpful for moving the spec forward. Bear with me as I dig a little deeper.

@kevinswiber My concern is that the "rel": ["item"] relationship is basically implied for every entity returned from an index operation. It is not efficient to require that each item entity declare that it is an item entity when the relationship is clearly implied by virtue of the fact that they're being listed as subordinates of the resource index.

I believe this is implied by your application semantics, not by the message itself. Application semantics will differ across API implementations.

Any resource-aware client can safely assume that without relying on any out-of-band information. It makes more sense to signal to the client that the endpoint is a resource index by including information in the index properties, instead of repeating the same information on every item in the collection.

There is out-of-band information here, though. Nothing in the message format declares that a sub-entity MUST be an item in a collection, just that a sub-entity should be related in some domain-applicable way. Are you proposing Siren should include language to declare that unspecified link relations imply a collection/item relationship?

I'm not sure what you're getting at with the index properties concern. Properties don't seem to be a good place to put resource children, since they are clearly distinct entities, and not simply properties of the index entity.

Sometimes there are related arrays of "value objects," which are not URI-addressable. This is all very domain-specific, however. Maybe in a Real Estate domain, an address would be a sub-entity, but in a Contact Management domain, an address (or group of addresses) might be listed under properties. Siren was built to be flexible to both scenarios. Requiring a link relation, IIRC, was a measure to help drive design along these lines.

Perhaps, however, it is not Siren's place to drive design in this way, so I'm open to feedback and use cases. I think the format currently does a pretty good job at staying away from hard restrictions (could be a benefit to some or a flaw to others).

Is your concern about redundancy of data? Payload size? Coding complexity? Does it just "feel" wrong (totally valid argument, by the way)?

Again, please bear with me. These questions are important. :)

[ericelliott]

@kevinswiber My concern is that the "rel": ["item"] relationship is basically implied for every entity returned from an index operation. It is not efficient to require that each item entity declare that it is an item entity when the relationship is clearly implied by virtue of the fact that they're being listed as subordinates of the resource index.
I believe this is implied by your application semantics, not by the message itself. Application semantics will differ across API implementations.

Agreed, but those semantics are shared by every application that uses resourceful routing, which is to say, quite a few applications. All of them could benefit from improved rel handling for collection items.

Are you proposing Siren should include language to declare that unspecified link relations imply a collection/item relationship?

Nope. I'm implying that Siren should include a more efficient way of assigning the same link relationship to all items in the entities collection. When I say that the client can interpret it without out-of-band information, I really mean that Siren can be improved such that no out-of-band information is necessary to make this happen.

Sometimes there are related arrays of "value objects," which are not URI-addressable.

We're talking about distinct entities that explicitly are URI-addressable. I'm talking about the index GET on a resourceful resource. The whole point of it is to point to other entities.

Is your concern about redundancy of data? Payload size? Coding complexity? Does it just "feel" wrong (totally valid argument, by the way)?

All of the above.

You seem to have glossed over:

Here's another idea: Make it possible to assign a rel value to the entities collection itself, implying that the value applies to every entity in the collection. That way you communicate exactly the same information in a much more efficient manner.

That would solve the problem, and ensure that all the entities do indeed get proper rel values assigned -- without repeating the same information over and over in the message bodies.

[kevinswiber]

You seem to have glossed over:

Here's another idea: Make it possible to assign a rel value to the entities collection itself, implying that the value applies to every entity in the collection. That way you communicate exactly the same information in a much more efficient manner.
That would solve the problem, and ensure that all the entities do indeed get proper rel values assigned -- without repeating the same information over and over in the message bodies.

I totally did. Thinking about it now. Thanks. How that would look in the message design? Any ideas?

[ericelliott]

How about an entityAttributes key that essentially means, "the following attributes should be assigned to every entity in the entities collection":

{
   "class":[
      "albums"
   ],
   "entityAttributes":{
      "rel":[
         "item"
      ],
      "class":[
         "album"
      ]
   },
   "entities":[
      {
         "properties":{
            "id": "123",
            "name":"Pretty Hate Machine",
            "artist":"Nine Inch Nails",
            "href":"http://albums.com/albums/123"
         }
      }
   ],
   "links":[
      {
         "rel":[
            "previous"
         ],
         "href":"http://albums.com/albums/offset/2/items/1"
      },
      {
         "rel":[
            "previous"
         ],
         "href":"http://albums.com/albums/offset/1/items/1"
      },
      {
         "rel":[
            "next"
         ],
         "href":"http://albums.com/albums/offset/3/items/1"
      }
   ]
}

Something along those lines...

[apsoto]

I can understand the desire to DRY things up, but I think entityAttributes only helps for index resources when the resources are homogeneous.

I don't think the repetition is a big deal since it's consumed by some parser that can easily handle it and compressed responses make the payload tiny.

[ericelliott]

@apsoto The clutter in the response doesn't just hurt download times (and it does that to some degree, whether the response is compressed or not), it also makes it less browsable by human beings. By eliminating that clutter, the user is more able to focus on the important data.

Also, resourceful APIs are extremely common, and resourceful APIs usually feature index routes like this, where the entities are guaranteed to be homogeneous.

I just finished writing a section about API design in my book, "Programming JavaScript Applications". Here's a relevant excerpt:

Usable

There are two foundational principles of API design that can help improve the usability of your API, and reduce the time it takes for new developers to learn it:

• Focus - The purpose of your API and each API endpoint should be clear, just by interacting with the API.
• Consistency - Each API resource should share the same conventions as the other resources for the same API.

Focus

To focus your API, present only the information your users need, and eliminate clutter. If your application is for music lovers, don't provide links to cooking recipes or abstract art.

It sounds easy, but it goes much deeper than that, and a lack of focus plagues many APIs. For example, many endpoints will list nested objects -- resources with related data embedded directly in the response. That practice can be convenient, but it can also be confusing. Don't overwhelm API users with gigantic records when they may be looking for a small, specific subset.

For example, consider an API that delivers information about music albums:

{
  "id": "chmzq50np0002gfixtr1qp64o",
  "name": "Settle",
  "artist": {
    "id": "chmzq4l480001gfixe8a3nzhm",
    "name": "Disclosure",
    "tourDates": [
      {
        "id": "chmzq45yn0000gfixa0wj6z9j",
        "date": "Sat Oct 19 2013",
        "name": "Treasure Island Music Festival",
        "location": "San Francisco, CA, US",
        "attending": [
          {
            "id": "chmzq7tyj0003gfix0rcylkls",
            "name": "Guy Lawrence"
          },
          {
            "id": "chmzqougy0004gfixuk66lhv4",
            "name": "Howard Lawrence"
          }
        ]
      }
    ]
  },
  "...": "..."
}

If all the user is looking for is a list of album titles and artist names, this is going to be overload. Instead of this, you can focus on the particular information that your user is most likely to need. Of course you should make it easy for them to get more if they want it, too. Just make the default sensible.

Take a look at this version and see if it makes more sense:

{
  "id": "chmzq50np0002gfixtr1qp64o",
  "name": "Settle",
  "artist": "Disclosure",
  "artistId": "chmzq4l480001gfixe8a3nzhm",
  "coverImage": "/covers/medium/zrms5gxr.jpg",
  "year": "2013",
  "genres": [
    "electronic", "house", "garage", "UK garage",
    "future garage"
  ]
}

Instead of inlining a bunch of completely unrelated data about the artist, you get the artist name (which you'll likely need to display in any album listing), and an ID you can use to query more information about the artist if it's needed.

The listing now is focused on the particular resource that's being queried right now: The album. Focus may be the most important principle of good UX, whether you're designing the UI for a shopping cart, or the shopping cart's API.

[apsoto]

I'll agree index resources are typically homogeneous, but I still think that's the only case where the repetition of rel's could be argued as 'clutter'. However, Siren isn't only about index resources.

I also agree that eliminating the clutter allows the user to navigate the API better, but I think that's unfortunately a result of the best tooling still being a browser and a plugin to render json nicely. Better tooling would make that clutter less of an issue. I published a siren-browser tool late last year. Unfortunately I've had no time to enhance it. Pull requests welcome ;)

I'd say, if it doesn't work for you, but you still like everything else, just don't use them. I've yet to see an official Siren client so you're rolling you're own, and can gloss over the MUST HAVE requirement.

That said, @kevinswiber mentioned, he's afraid there would be a blurring of the distinction between a property and sub-entity value. I think it warrants some pro/con discussion about making the rel requirement optional rather than adding some sort of meta-attribute/syntax sugar like 'entityAttributes'. What's an example where NOT having the rel makes the response 'bad'?

[ericelliott]

I'd say, if it doesn't work for you, but you still like everything else, just don't use them. I've yet to see an official Siren client so you're rolling you're own, and can gloss over the MUST HAVE requirement.

While it's probably good for a client to be forgiving (the robustness principle: "be liberal in what you accept from others"), it's probably not a good idea as an api designer to purposely violate the spec (the other half of the robustness principle: "be conservative in what you do").

If you want to ignore MUST as an API designer, sure, that's fine. Just don't call it Siren, because it isn't anymore.

Besides, I see this as a good opportunity to improve the spec for everybody.

he's afraid there would be a blurring of the distinction between a property and sub-entity value.

I believe the MUST requirement for entities to have hrefs keeps those lines drawn nicely. There was a situation before that made me think that entities should not be required to have hrefs, but as it turned out, properties were a better fit.

I'm pretty certain that entities are the right place to list resource collection items with links, though, and truthfully, it might be useful if they could have rel and class values -- I just don't want them repeated in the JSON for every entity in the collection when the data they contain is identical across the whole collection.

What's an example where NOT having the rel makes the response 'bad'?

When you really want the items to have rel and class properties, but you don't want them repeated. NOT having them in the message would require out-of-band information in the client to interpret the message correctly. That would point to a deficiency in the media type.

IMO, the media type should be capable of communicating all necessary information in an efficient manner. This is a case where efficiency can be improved.

[ericelliott]

I have a test implementation started. Resourceful routing with siren+json hypermedia for Express.

Here's how you use it:

var resource = require('siren-resource'),
  index = function index(req, res, next) {
    var href = '/albums',
      idx = map(albums, function (album) {
        return {
          href: href + '/' + album.id,
          properties: {
            name: album.name,
            artist: album.artist
          }
        };
      }),
      sobj = resource.entity(href, {
        title: 'Albums',
        entityAttributes: {
          rel: ['item'],
          class: ['album']
        },
        entities: idx
      });
    res.send(sobj);
  };

resource('/albums', app, {
  routes: {
    index: index
  }
});

Here's what the output looks like:

{
  "title": "Albums",
  "entityAttributes": {
    "rel": [
      "item"
    ],
    "class": [
      "album"
    ]
  },
  "entities": [
    {
      "href": "/albums/chmzq50np0002gfixtr1qp64o",
      "properties": {
        "name": "Settle",
        "artist": "Disclosure"
      }
    }
  ],
  "links": [
    {
      "rel": [
        "self"
      ],
      "href": "/albums"
    }
  ]
}

Note that it also hooks up lots of other routes, defaulting to 404 and 401 error handling for undefined routes. It also handles paging automatically... just pass paging rules in on the options, and it will automatically create prev / next links that look like:

/resource/offset/20/limit/10

It should be pretty trivial to create a client utility to attach the entityAttributes to the entities.

[ryanhiebert]

It makes sense to me that ['item'] would be the default rel, although that's not a strong opinion.

I do wonder if when there is a case that there might be multiple sub-entity rels that they really should be contained in a parent rel. As a concrete example, consider a class person. That would have sub-entities with class parent, but also with class child. It might make sense for there to instead to be a children sub-entity and a parents sub-entity as containers.

I do wonder if there's a case where that technique might cause duplicates, but I can't think of an example that's not contrived at the moment.

[ericelliott]

The more I think about this, the more I'm convinced that MUST is too strong. Relationship to the parent should really be the parent collection's concern. Sub-entities can and most likely should be blissfully unaware that a parent collection even exists. See my proposed Siren error extension for a practical example. I solicited feedback from the community and nobody seemed to think that the sub-entities needed rels.

Requiring sub entities to embed data relating to the parent collection is redundant, wasteful, and could cause tight coupling between children and parent by implication.

rel makes sense for links in the context of a parent document describing the relationship of the linked entity to the parent entity. It does not make sense in the context of a child which may have many parent entities. That mimics the tight coupling in classical inheritance which I have ranted about in detail, elsewhere.

I feel quite strongly about this. If it is not a change we're going to make in Siren, I intend to fork Siren and implement a competing standard. I'm already ignoring this requirement in all my implementations (including the published O'Reilly book, Programming JavaScript Applications, optimistically hoping we can remove this requirement.

I do support entityAttributes in all my implementations which would allow clients to figure out any common relationships in all the entities.

[ericelliott]

#45

[apsoto]

I think an entity should be an addressable resource, but the error case is a bit of a special case, so to use that as the argument for no rels on the sub-entity doesn't seem valid IMO. Also, throwing the fork word around if you don't get your way doesn't help with the discussion. Maybe it makes more sense to work together towards a Siren 2.0 that is not backwards compatible.

However, having the rel on the sub-entity has caused me a little extra work because the sub-entity response JSON is different from the same resource's top-level entity's response JSON.

For example:

/series/:id/episodes/:id
/episodes/:id

In this case, it's the same resource, just the specific episode is also a sub-entity to the series. I'd love to be able to re-use the response JSON for the /episodes/:id when building the series response (and vice versa). However, having the rel attribute on the sub-entity means I can't.

[ericelliott]

Also, throwing the fork word around if you don't get your way doesn't help with the discussion.

I need this change, so if it doesn't happen here, it will happen in a fork. I'm not making threats, just making my intentions clear.

So far I haven't heard any convincing arguments that we MUST include rel attributes in sub-entities. They're already required for sub-entity links, and that makes much more sense.

Why not make the sub-entity requirements the same as a root level entity? =)

having the rel on the sub-entity has caused me a little extra work because the sub-entity response JSON is different from the same resource's top-level entity's response JSON.

Another compelling reason to remove "MUST".

=)

[kevinswiber]

This issue has been hanging around for a while, and I think I'm still having trouble seeing why you would need a sub-entity without a relation value. I am open to seeing it this way, but chances are, I just need a little push. :)

Part of this might be because I often design my inline sub-entities as a subset of their full representation. (Think of a list view representation vs. a complete item representation.) It's a slightly different model, and therefore, I need to have specific code to piece together the representation anyway, often in the parent context. Adding a link relation is no big deal in this case.

Here's my big question: As a client, how do I determine why I should care about the sub-entity? I currently use the link relation for that. If there were no link relation, what information should I inspect?

I think we might be able to get better progress on this with the magic power of voice, video, and screen-sharing. @ericelliott Are you open to discussing this on the Siren Hangout this Friday? We can start piecing together an agenda.

[kevinswiber]

Also, regarding forking... I'm completely open and supportive of anyone looking to fork any work I've touched. I think of that as a positive action. That said, I'm hoping we can foster an open community who feels that Siren is, in fact, a collectively owned project. If we all arrive at an understanding of the change request and there's a legitimate disagreement about how Siren should move forward, I think that would be a good case for forking, and I would offer my full support. Otherwise, I would love to see that energy put into making the Siren specification better for all of us. :)

[ericelliott]

I'm all for improving Siren instead of forking. I'd love to discuss it with you in a hangout. Have we settled on a specific time and date?

I'm also completely open to the idea of having relations for all sub-entities. My problem with it is where those relationships are located. Almost all of my use cases for entities boil down to collections of similar things, like lists of albums, lists of orders, lists of users, lists of x, and in all of those cases, the collection -- not the items themselves -- is responsible for keeping track of the relationship, which all items share.

So my problem is 2-fold:

• Including rel at the sub-item level is redundant and wasteful, because all items share the same rel to the parent.
• We can't just reuse the same representation that the item has at the root level of a response, meaning that we'd have to inject the rel to produce the sub-entity representations, causing us to write special case code, when we could have just used exactly the same code we would if the item was a root entity to produce the sub-entities.

In other words, this requirement forces more software complexity on the API implementation, and creates larger payloads, and it offers absolutely no value that couldn't be achieved with a simpler implementation, such as a shared entityAttributes key in the parent entity.

[gxxcastillo]

I think an entity should be an addressable resource

+1 I feel this distinction is key to differentiating between what is an entity and what is a property.

As a client, how do I determine why I should care about the sub-entity? I currently use the link relation for that. If there were no link relation, what information should I inspect?

What about using the properties object? (I didn't come up with this but remember seeing this mentioned on a thread a while back)

ex:

    {
        properties: {
            name: 'ralph'
            , age: 78
            , profile_image: 'self://x.io/user/1234/image'
            , friends: 'self://x.io/user/1234/friends'
        }

        , entities: [
            { /* "profile_image" entity */ }
            , { /* "friends" collection entity */ }
        ]
    }

We could discuss the proper syntax for the internal reference (e.g. the self://), as well as how to treat linked sub-entities, and any other details in a separate thread but I think you get the idea.

• Retains the relationship information from parent to child
• Allows you to re-use the root representation without having to modify it
• Collections don't need a rel so this would not affect collections.
• Allows for clients to implement hash fragments (e.g. http://x.io/user/1234#profile_image)
• Would resolve this old issue: Add an entity.id #33 :)

Also, wrt @apsoto's comments in #45:

The entities property is currently an array, so it's a breaking change to convert it to an object

This is backward compatible

In addition, current spec requires the rels to be an array. Using the example above means there is only one rel per child entity. I don't think I'd miss that requirement very much, but I only have the perspective of my specific API's problem domain.

I also would not miss that requirement :)

I like treating everything as a collection instead of having the possibility of it being an object or an array

This keeps the entities array an array.

[ericelliott]

Seems somebody beat me to changing this requirement. I don't see it in the current version.

Should we close this, or do we still need to add wording for the default rel we discussed in the hangout?

[ericelliott]

Oops. Looks like I jumped the gun.. now I'm looking at the doc again, I still see the requirement. I'll open a PR.