feat: improve idempotency guideline (zalando#456)

Author: tkrop

chapters/common-headers.adoc

@@ -138,11 +138,12 @@ To expose conflicts between concurrent update operations via {PUT}, {POST},
 or {PATCH}, the `If-Match: <entity-tag>` header can be used to force the server
 to check whether the version of the updated entity is conforming to the
 requested `<entity-tag>`. If no matching entity is found, the operation is
-supposed a to respond with status code 412 - precondition failed.
+supposed a to respond with status code {412} - precondition failed.
 
 Beside other use cases, `If-None-Match: *` can be used in a similar way to
 expose conflicts in resource creation. If any matching entity is found, the
-operation is supposed a to respond with status code 412 - precondition failed.
+operation is supposed a to respond with status code {412} - precondition
+failed.
 
 The {ETag}, {If-Match}, and {If-None-Match} headers can be defined as
 follows in the API definition:
@@ -193,22 +194,22 @@ Please see <<optimistic-locking>> for a detailed discussion and options.
 == {MAY} Consider Using `Idempotency-Key` Header
 
 When creating or updating resources it can be helpful or necessary to prevent
-duplicate execution to ensure idempotent response behavior. Generally, this can 
-be achieved by sending a client specific _unique request key_ – that is not
-part of the resource – using the {Idempotency-Key} header.
+duplicate execution to ensure <<idempotent>> behavior. Generally, this can be
+achieved by sending a client specific _unique request key_ – that is not part
+of the resource – using the {Idempotency-Key} header.
 
 The _unique request key_ is stored temporarily, e.g. for 24 hours, together
 with the response and a request hash (optionally) in a key cache. The service
 can now look up the _unique request key_ in the key cache and serve the
 response from the key cache, instead of re-executing the request, to ensure
-idempotent behavior. Optionally, it can check the request hash for consistency
-before serving the response. If the key is not in the key store, the request
-is executed as usual and the response is stored in the key cache.
+<<idempotent>> behavior. Optionally, it can check the request hash for
+consistency before serving the response. If the key is not in the key store,
+the request is executed as usual and the response is stored in the key cache.
 
 This allows clients to safely retry requests after timeouts etc while receive
 the same response multiple times.
 
-*Note:* To grant a reliable idempotent execution semantic, the resource and
+*Note:* To grant a reliable <<idempotent>> execution semantic, the resource and
 the key cache have to be updated with hard transaction semantics – considering
 all potential pitfalls of failures, timeouts, and concurrent requests in a
 distributed systems. This makes a correct implementation very hard

chapters/http-requests.adoc

@@ -12,11 +12,11 @@ Be compliant with the standardized HTTP method semantics summarized as follows:
 
 {GET} requests are used to *read* either a single or a collection resource.
 
-* {GET} requests for individual resources will usually generate a 404 if the
+* {GET} requests for individual resources will usually generate a {404} if the
 resource does not exist
-* {GET} requests for collection resources may return either 200 (if the
-collection is empty) or 404 (if the collection is missing)
-* {GET} requests must NOT have a request body payload
+* {GET} requests for collection resources may return either {200} (if the
+collection is empty) or {404} (if the collection is missing)
+* {GET} requests must NOT have a request body payload (see <<GET-with-Body>>)
 
 *Note:* {GET} requests on collection resources should provide sufficient
 <<137, filter>> and <<pagination>> mechanisms.
@@ -65,9 +65,9 @@ implicitly creating before updating
 * on successful {PUT} requests, the server will *replace the entire resource*
 addressed by the URL with the representation passed in the payload (subsequent
 reads will deliver the same payload)
-* successful {PUT} requests will usually generate 200 or 204 (if the resource
-was updated – with or without actual content returned), and 201 (if the
-resource was created)
+* successful {PUT} requests will usually generate {200} or {204} (if the
+resource was updated – with or without actual content returned), and {201} (if
+the resource was created)
 
 *Important:* It is best practice to prefer {POST} over {PUT} for creation of
 (at least top-level) resources. This leaves the resource ID under control of
@@ -76,7 +76,7 @@ follows.
 
 *Note:* In the rare cases where {PUT} is although used for resource creation,
 the resource IDs are maintained by the client and passed as a URL path segment.
-Putting the same resource twice is required to be _idempotent_ and to result
+Putting the same resource twice is required to be <<idempotent>> and to result
 in the same single resource instance (see <<149>>).
 
 *Hint:* To prevent unnoticed concurrent updates and duplicate creations when
@@ -95,10 +95,10 @@ identified by the URL"_.
 
 * on a successful {POST} request, the server will create one or multiple new
 resources and provide their URI/URLs in the response
-* successful {POST} requests will usually generate 200 (if resources have
-been updated), 201 (if resources have been created), 202 (if the request was
-accepted but has not been finished yet), and exceptionally 204 with {Location}
-header (if the actual resource is not returned).
+* successful {POST} requests will usually generate {200} (if resources have
+been updated), {201} (if resources have been created), {202} (if the request
+was accepted but has not been finished yet), and exceptionally {204} with
+{Location} header (if the actual resource is not returned).
 
 The semantic for single resource endpoints is best described as _"please
 execute the given well specified request on the resource identified by the
@@ -111,7 +111,7 @@ other methods sufficiently. In such cases, make sure to document the fact that
 *Note:* Resource IDs with respect to {POST} requests are created and maintained
 by server and returned with response payload.
 
-*Hint:* Posting the same resource twice is *not* required to be _idempotent_
+*Hint:* Posting the same resource twice is *not* required to be <<idempotent>>
 (check <<149>>) and may result in multiple resources. However, you <<229>> to
 prevent this.
 
@@ -132,8 +132,8 @@ collection is challenging
 instances
 * on successful {PATCH} requests, the server will update parts of the resource
 addressed by the URL as defined by the change request in the payload
-* successful {PATCH} requests will usually generate 200 or 204 (if resources
-have been updated with or without updated content returned)
+* successful {PATCH} requests will usually generate {200} or {204} (if
+resources have been updated with or without updated content returned)
 
 *Note:* since implementing {PATCH} correctly is a bit tricky, we strongly suggest
 to choose one and only one of the following patterns per endpoint, unless
@@ -159,7 +159,7 @@ http://tools.ietf.org/html/rfc6902[JSON Patch] can shown its full power while
 still showing readable patch requests (see also
 http://erosb.github.io/post/json-patch-vs-merge-patch[JSON patch vs. merge]).
 
-*Note:* Patching the same resource twice is *not* required to be _idempotent_
+*Note:* Patching the same resource twice is *not* required to be <<idempotent>>
 (check <<149>>) and may result in a changing result. However, you <<229>> to
 prevent this.
 
@@ -177,14 +177,15 @@ described as _"please delete the resource identified by the URL"_.
 * {DELETE} requests are usually applied to single resources, not on collection
 resources, as this would imply deleting the entire collection
 * successful {DELETE} requests will usually generate 200 (if the deleted
-resource is returned) or 204 (if no content is returned)
-* failed {DELETE} requests will usually generate 404 (if the resource cannot
-be found) or 410 (if the resource was already deleted before)
+resource is returned) or {204} (if no content is returned)
+* failed {DELETE} requests will usually generate {404} (if the resource cannot
+be found) or {410} (if the resource was already deleted before)
 
 *Important:* After deleting a resource with {DELETE}, a {GET} request on the
-resource is expected to either return 404 (not found) or 410 (gone) depending
-on how the resource is represented after deletion. Under no circumstances the
-resource must be accessible after this operation on its endpoint. 
+resource is expected to either return {404} (not found) or {410} (gone)
+depending on how the resource is represented after deletion. Under no
+circumstances the resource must be accessible after this operation on its
+endpoint.
 
 
 [[head]]
@@ -198,7 +199,7 @@ body.
 
 *Hint:* {HEAD} is particular useful to efficiently lookup whether large
 resources or collection resources have been updated in conjunction with the
-https://tools.ietf.org/html/rfc7232#section-2.3[`ETag`-header].
+{ETag}-header.
 
 [[options]]
 === OPTIONS
@@ -218,10 +219,18 @@ self-describe the full functionality of a resource.
 
 An operation can be...
 
-* idempotent, i.e. operation will have the same effect on the server's state,
-if executed once or multiple times (note: this does not necessarily mean
-returning the same response or status code)
-* safe, i.e. must not have side effects such as state changes
+* [[idempotent, idempotent]]{RFC-idempotent} - the operation has the same
+  _intended effect_ on the server state, independently whether it is executed
+  once or multiple times. *Note:* this does not mean that it is returning the
+  same response or status code.
+* [[safe, safe]]{RFC-safe} - the operation semantic is defined to be read-only,
+  meaning it must not have _intended side effects_, i.e. changes, to the server
+  state.
+
+*Note:* The above definitions, of _intended (side) effect_ allows the server
+to provide additional state changing behavior as logging, accounting, pre-
+fetching, etc. However, these actual effects and state changes, must not be
+intended by the operation so that it can be held accountable.
 
 Method implementations must fulfill the following basic properties:
 
@@ -242,57 +251,75 @@ Method implementations must fulfill the following basic properties:
 == {SHOULD} Consider To Design `POST` and `PATCH` Idempotent
 
 In many cases it is helpful or even necessary to design {POST} and {PATCH}
-idempotent for clients to expose conflicts and prevent duplicate creation or
-lost updates, e.g. if same resources may be created or changed in parallel or
-multiple times. To design an idempotent API owners should consider to apply
-one of the following three patterns.
+<<idempotent>> for clients to expose conflicts and prevent resource duplicate
+(a.k.a. zombie resources) or lost updates, e.g. if same resources may be
+created or changed in parallel or multiple times. To design an <<idempotent>>
+API endpoint owners should consider to apply one of the following three
+patterns.
 
 * A resource specific *conditional key* provided via <<182,`If-Match` header>>
   in the request. The key is in general a meta information of the resource,
   e.g. a _hash_ or _version number_, often stored with it. It allows to detect
-  concurrent creations and updates to ensure idempotent request behavior
-  (see <<182>>).
+  concurrent creations and updates to ensure <<idempotent>> behavior (see
+  <<182>>).
 * A resource specific *secondary key* provided as resource property in the
-  request body. The _secondary key_ is stored permanently in the resource as
-  _alternate_ (or _unique foreign key_, if it links to an another – often
-  parent – resource, e.g. the shopping cart ID is a natural _unique foreign
-  key_ candidate for an order. It allows to ensure idempotent request behavior
-  in case of multiple independent resource creations from different clients
-  (see <<231>>).
+  request body. The _secondary key_ is stored permanently in the resource. It
+  allows to ensure <<idempotent>> behavior by looking up the resource in case
+  of multiple independent resource creations from different clients (see
+  <<231>>).
 * A client specific *idempotency key* provided via {Idempotency-Key} header
   in the request. The key is not part of the resource but stored temporarily
-  pointing to the original response to ensure idempotent response behavior
-  when retrying a request (see <<230>>).
+  pointing to the original response to ensure <<idempotent>> behavior when
+  retrying a request (see <<230>>).
 
 *Note:* While *conditional key* and *secondary key* are focused on handling
 concurrent requests, the *idempotency key* is focused on providing exact same
 responses and can be combined with the other patterns.
 
 To decide, which pattern is suitable for your use case, please consult the
-following table showing the major properties of each pattern. As far as this
-patterns can be applied to {PUT} we added it to allow full comparison:
-
-[,cols="40%,20%,20%,20%",options="header",]
-|========================================================================================
-|                                      | Conditional Key | Secodary Key | Idempotency Key
-| Applicable with                      | {PUT}/{PATCH} | {POST} | {POST}/{PUT}/{PATCH}
-| HTTP Standard                        | {NO}  | {NO}  | {NO}
-| Prevents concurrent creation         | {YES} | {YES} | {NO}
-| Prevents concurrent (lost) updates   | {YES} | {NO}  | {NO} 
-| Supports safe retries                | {YES} | {YES} | {YES}
-| Supports exact same response         | {NO}  | {NO}  | {YES}
-| Can be inspected (by intermediaries) | {YES} | {NO}  | {YES}
-| Usable without additional {GET}
-(before {PUT}/{POST}/{PATCH})          | {YES}  | {YES} | {NO}
-|========================================================================================
+following table showing the major properties of each pattern:
+
+[,cols="46%,18%,18%,18%",options="header",]
+|==================================================================================
+|                               | Conditional Key | Secondary Key | Idempotency Key
+| Applicable with                       | {PATCH} | {POST}  | {POST}/{PATCH}
+| HTTP Standard                         | {YES}   | {NO}    | {NO}
+| Prevents duplicate (zombie) resources | {YES}   | {YES}   | {NO}
+| Prevents concurrent lost updates      | {YES}   | {NO}    | {NO} 
+| Supports safe retries                 | {YES}   | {YES}   | {YES}
+| Supports exact same response          | {NO}    | {NO}    | {YES}
+| Can be inspected (by intermediaries)  | {YES}   | {NO}    | {YES}
+| Usable without previous {GET}         | {NO}    | {YES}   | {YES}
+|==================================================================================
+
+*Note:* The patterns applicable to {PATCH} can be applied in the same way to
+{PUT} and {DELETE} providing the same properties.
 
 If you mainly aim to support safe retries, we suggest to apply <<182,
 conditional key>> and <<231,secondary key>> pattern before the <<230,
 Idempotency Key>> pattern.
 
 [#231]
-== {MAY} Use Secondary Key for Idempotent `POST`
-
+== {Should} Use Secondary Key for Idempotent `POST` Design
+
+The most important pattern to design {POST} <<idempotent>> for creation is to
+introduce a resource specific *secondary key* provided in the request body, to
+eliminate the problem of duplicate (a.k.a zombie) resources.
+
+The secondary key is stored permanently in the resource as _alternate key_
+or _combined key_ (i.e. consisting of multiple properties), that is visible
+when reading the resource. The best and often naturally existing candidate
+is a _unique foreign key_, that points to another resource having _one-on-one_
+relationship with the newly created resource, e.g. a parent process identifier.
+
+A good example here for a secondary key is the shopping cart ID in an order
+resource.
+
+*Note:* When using the secondary key pattern without {Idempotency-Key} all
+subsequent retries should fail with status code {409} - conflict. We suggest
+to avoid {200} here unless you make sure, that the delivered resource is the
+original one implementing a well defined behavior. Using {204} without content
+would be a similar well defined option.
 
 [#154]
 == {SHOULD} Define Collection Format of Query Parameters and Headers