First steps to add encoding capability to outbound traffic #73

nicolas-rempulski · 2014-01-13T23:36:43Z

Need Review
Merge Candidate

Here are first steps of adding encoding to Wisdom as stated in #10

Done :

Add content encoding constants
Add content encoders in the content manager service (GZIP, DEFLATE, IDENTITY)
Add content encoder retrieval in content engine (with encoding)
Add Accepting_Encoding parsing (partially done) + encoding sorting based on order and qValue (lots of TODO in code)
Add content encoding to result processing
Add a configuration key to enable / disable encoding globally
Add annotations to enable / disable encoding for a route / controller
Add size limit to @AllowEncoding and global configuration (code is still rough)
Review Accepting_Encoding parsing as I'm not sure of some behaviors + "*;qValue" is not handled
Don't work on /assets/xxx routes, they have a length of -1 => Not RenderableFile but RenderableStream. Added a configuration key to allow encoding of ALL stream. Still disable with the @DenyEncoding annotation.
Documentation for annotations, default behaviour
Filter on mime types
Stream are not encoded (no size no mime type). RenderableUrl are encoded if encoding.url is set and mime type not in exclude list
Filter on already encoded files (Content_Encoding set and different from identitity)
Unit tests (done for encoders)
Documentation
Encoding is now asynchrone

ToDo :

Check encoding behavior with cache (Compressed each time or the result is cached ?)

Future Work :

Add "compress" support
Add "non-standard" encoding ( http://en.wikipedia.org/wiki/HTTP_compression )
Which "non-standard" encodings to support
Handle compressed inbound traffic

To discuss :

——Default behavior : enabled / disabled—— => enabled
--Configuration key name. Currently : global.encoding, encoding.max.size, encoding.min.size-- encoding.global, encoding.max.size, encoding.min.size, encoding.url
--Should we allow a user to force a given encoding, discarding Accept_Encoding order-- no
--Filter on file extensions ? (http://www.fileinfo.com/filetypes/compressed)-- on mime type
--Default minimum / max size : http://webmasters.stackexchange.com/questions/31750/what-is-recommended-minimum-object-size-for-gzip-performance-benefits-- Ok 10ko - 10Mo

I also made a change to RenderableString, as the Charset was not used to convert the string to bytes. Need reviews. (I think it is linked to #42)

String.getBytes() encode a string using the platform default encoding. In RenderableString, the string was encoded without using the Charset specified or didn't specify the default Charset if no Charset was set. Now, when using RenderableString, it will always specify the charset.

A ContentEncoder encodes an InputStream to a given compression format

- GZIP - DEFLATE - IDENTITY

Default is enabled

EncodingHelp allows to parse the Accept_Encoding header and sort the encoding methods according (mostly) to HTTP/1.1

Result parameter as null was not handled

These annotations allow to override the application configuration regarding content encoding. They can be set both on class & methods

* maxSize is the upper boundary for encoding * minSize is the lower boundary for encoding

…values

* if renderable.length is not in encoding boundaries, abort encoding * shouldEncode now take a renderable as parameter, preparing for filter on extensions

Changing package names to org.wisdom.content.codecs

* Adding AbstractDefInfCodec for Gzip and Deflate * Adding in file documentation

Moving shouldEncode from WisdomHandler to EncodingHelper

* shouldEncode is now in EncodingHelper * parseAcceptEncodingHeader now correctly handle wildcard "*"

* Add default value for encoding.global : true * Add encoding.stream configuration key to allow stream encoding (no size check !) * Add default value for encoding.stream : true

Should encode will not check stream size in encoding.stream is set to true in the configuration

* Adding get functions * Update Configuration Unit Test

* Added a ContentEncodingHelper interface * Provide a ContentEncodingHelperImpl in content-manager as a service * Retrieval through content-engine * Updated UT (was throwing null pointer)

* Add UT and FakeControllers * Add mockito dependency * Modifications to ContentEncodingHelperImpl to satisfy tests

* Configuration keys * URL encoding * MimeType filtering * Cosmetic changes

nicolas-rempulski · 2014-01-23T09:43:49Z

This PR is now ready for review & merge.

Interaction between cache and encoding should be examined.

* add fromAsync parameter to write / finalizeWrite to know is their call is from an async result and need cleanup at the end * Remove cleanup after AsyncResult handling as there could be an async content encoding running * Extract content encoding from process result to handle it in writeResponse * Add a proceedAsyncEncoding function to encode content InputStream asynchronously * FinalizeWrite now check if the call is from an Async function and if keepAlive is not set, cleanup this handler

Conflicts: wisdom-api/src/main/java/org/wisdom/api/bodies/RenderableString.java wisdom-api/src/main/java/org/wisdom/api/utils/KnownMimeTypes.java wisdom-engine/src/main/java/org/wisdom/engine/server/WisdomHandler.java

Conflicts: application-configuration/src/main/java/org/wisdom/configuration/ApplicationConfigurationImpl.java application-configuration/src/test/java/org/wisdom/configuration/ApplicationConfigurationTest.java wisdom-api/src/main/java/org/wisdom/api/configuration/ApplicationConfiguration.java

nicolas-rempulski · 2014-02-27T11:18:29Z

First steps to add encoding capability to outbound traffic

nicolas-rempulski added 30 commits January 14, 2014 00:06

Add ContentEncoder interface

f401002

A ContentEncoder encodes an InputStream to a given compression format

Add content encoders for wisdom

d773dc2

- GZIP - DEFLATE - IDENTITY

Add EncodingNames with constants for encoding methods

c1ab9d3

Add ContentEncoder methods to ContentEngine

08857c7

Add a configuration key to enable / disable encoding globally

a6e0ed7

Default is enabled

Adding EncodingHelper

a696344

EncodingHelp allows to parse the Accept_Encoding header and sort the encoding methods according (mostly) to HTTP/1.1

Add encoding handling to the result processing

2532659

RenderableString.render was not passing UT

062cc9a

Result parameter as null was not handled

Add @AllowEncoding and @DenyEncoding

31b5999

These annotations allow to override the application configuration regarding content encoding. They can be set both on class & methods

Refactor process to decide whether to encode response content or not

13883e4

add size fields on @AllowEncoding annotations

517c9ea

* maxSize is the upper boundary for encoding * minSize is the lower boundary for encoding

Add configuration keys encoding upper and lower boundaries + default …

f885c8f

…values

Adding size computation to shouldEncode

331b2a5

* if renderable.length is not in encoding boundaries, abort encoding * shouldEncode now take a renderable as parameter, preparing for filter on extensions

Refactoring encoders to codecs

10db476

Adding unit tests for codecs

6520fdd

Finalizing encoders to codecs refactoring

5abf514

Changing package names to org.wisdom.content.codecs

Codecs refactoring

edfff79

* Adding AbstractDefInfCodec for Gzip and Deflate * Adding in file documentation

Refactoring shouldEncode

845eccb

Moving shouldEncode from WisdomHandler to EncodingHelper

Refactoring shouldEncode and parseAcceptEncodingHeader

6a77954

* shouldEncode is now in EncodingHelper * parseAcceptEncodingHeader now correctly handle wildcard "*"

Adding documentation for content encoding

9b0641e

Fix documentation

8e8cea6

Add configuration values for encoding

d16c8a3

* Add default value for encoding.global : true * Add encoding.stream configuration key to allow stream encoding (no size check !) * Add default value for encoding.stream : true

Refactor shouldEncode do handle stream

fa7a06a

Should encode will not check stream size in encoding.stream is set to true in the configuration

Add functions to handle Long properties

b5bd947

* Adding get functions * Update Configuration Unit Test

Finalize the refactoring of ContentEncoder to ContentCodec

6585624

Add already compressed MimeTypes

287e1dd

ContentEncodingHelper refactoring

6690ed3

* Added a ContentEncodingHelper interface * Provide a ContentEncodingHelperImpl in content-manager as a service * Retrieval through content-engine * Updated UT (was throwing null pointer)

Add filtering of already encoded content

8543a01

Refactor @AllowEncoding annotation to use long instead of int

e9c02d8

nicolas-rempulski added 2 commits January 23, 2014 10:05

Add ContentEncodingHelperImpl UT

fb7193b

* Add UT and FakeControllers * Add mockito dependency * Modifications to ContentEncodingHelperImpl to satisfy tests

Update documentation to match latest changes

a817c1c

* Configuration keys * URL encoding * MimeType filtering * Cosmetic changes

nicolas-rempulski added 5 commits January 23, 2014 16:14

Refactor AkkaBootstrap to handle multiple Callable types

d047644

return writeResponse results as it can holds Async informations

4d916ae

Merge branch 'refs/heads/master' into encoding-implem

caea421

Conflicts: wisdom-api/src/main/java/org/wisdom/api/bodies/RenderableString.java wisdom-api/src/main/java/org/wisdom/api/utils/KnownMimeTypes.java wisdom-engine/src/main/java/org/wisdom/engine/server/WisdomHandler.java

cescoffier added a commit that referenced this pull request Feb 28, 2014

Merge pull request #73 from wisdom-framework/encoding-implem

d28e7c0

First steps to add encoding capability to outbound traffic

cescoffier merged commit d28e7c0 into master Feb 28, 2014

cescoffier added this to the 0.4.0 milestone Apr 4, 2014

cescoffier deleted the encoding-implem branch August 25, 2014 09:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

First steps to add encoding capability to outbound traffic #73

First steps to add encoding capability to outbound traffic #73

nicolas-rempulski commented Jan 13, 2014

nicolas-rempulski commented Jan 23, 2014

nicolas-rempulski commented Feb 27, 2014

First steps to add encoding capability to outbound traffic #73

First steps to add encoding capability to outbound traffic #73

Conversation

nicolas-rempulski commented Jan 13, 2014

nicolas-rempulski commented Jan 23, 2014

nicolas-rempulski commented Feb 27, 2014