A simple implementation of Windows Azure Storage API for Ruby, inspired by the S3 gems and self experience of dealing with queues. The major goal of the whole gem is to enable ruby developers [like me =)] to leverage Windows Azure Storage features and have another option for cloud storage.
The whole gem is implemented based on Microsoft’s specs from the communication and underlying service description and protocol (REST). The API is for ruby developers built by a ruby developer. I’m trying to follow idioms, patterns and fluent type of doing APIs on Ruby.
This work isn’t related at all with StorageClient Sample shipped with Microsoft SDK and written in .NET, the whole API is based on my own understanding, experience and values of elegance and ruby development.
Full documentation for the gem is available at waz-storage.heroku.com
Well, this is a sum up of the whole experience of writing those gems and getting them to work together to simplify end user experience. Although there’re some breaking changes, it’s pretty backward compatible with existing gems.
-
Fixes an issue with Double Encoding (thanks Phil Ibis!)
The main difference with the original library is the adding of the method “railsetag”. This method add to any blob object an “x_ms_meta_railsetag” header, its value is calculated as the rails etag way (Digest::MD5.hexdigest(file))
By doing so we can easily know if a locale copy of a file is up-to-date on Azure CDN. This is very much used in the “waz-sync” gem which purpose is to sync file between local storage and Azure.
-
Upload from stream, plus XML/URI escaping in various places, and a few other minor fixes
-
Add upload method to container
-
Add put_block_list to support upload
-
Fix message canonicalization of ?comp= parameters for versions prior to 2009-09-19 (seems put_block was broken)
-
Unescape query string parameters when constructing signatures (needed for put_block, since some base64-encoded names include non-URI-friendly characters)
-
XML-escape property values for table entities (needed to insert things containing &, <, etc.)
-
Remove Unicode characters from rakefile (was breaking something on Windows)
-
Add tests for put_block_list and upload
-
Fix tests for content type (typo, = instead of .should ==)
-
Fix tests using RestClient.beautify_headers (expects an array, not a scalar)
-
Update tests to rspec 2.5.0
-
Allow authorization using only SharedAccessSignature
-
Fixed for list_blobs as it wasn’t fully merged to 2009-09-19. Now it works without 403 exceptions [Thanks tomconte]
-
Merged with sriramk fix for loading path issues on some environments
-
Completed Blobs API migration to 2009-09-19, _fully supporting_ what third-party tools does (e.g. Cerebrata) [thanks percent20]
-
Added Syntax’s sugar for ensuring a Queue (get or create)
-
Added support for table service to query, get_one, insert, update, merge and delete entities.
-
Added support for running against the Storage Developement Fabriq shipped with Microsoft SDK.
-
Added signature support for Tables service according to msdn.microsoft.com/en-us/library/dd179428.aspx
-
Added support to enumerate, create, and delete tables on give storage account.
-
Improved support for stacked connection management.
-
When simulating a container using a forward slash starting the blob name it crashed with 404, now it’s fixed.
-
Fixed an issue on the Queues Documentation (thanks @ezequielm)
-
Added naming validation for Containers according to msdn.microsoft.com/en-us/library/dd135715.aspx (thanks @panosjee)
-
Added naming validation for Queues according to msdn.microsoft.com/en-us/library/dd179349.aspx (thanks @panosjee)
-
Added support for Blob snapshots
-
Added support for rest-client 1.3.0. (thanks @jpgd)
-
Fixes a critical bug on URL management that some time prevents messages from being deleted, caused by a missing encoding on the URL parameters.
-
Added new shared key authentication support for 2009-09-19 Version of the Storage API
-
Queues API has been migrated to the 2009-09-19 Version of the Storage API
-
Added a new parameter for listing queues with metadata
-
Added support for DequeueCount on messages being retrieved from the Queue
-
Known Issue: Creating a queue multiple times with same metadata throws 409.
sudo gem install waz-storage –source gemcutter.org
One of the major changes from the waz-queues and waz-blobs APIs was the ability to set a single storage account (99% of the cases) to be used on your code just once. Since Windows Azure Storage Services are all related to a single account that includes Tables, Queues and Blobs, I’ve changed the API in order to make ease that administration by requiring you to set the configuration just once:
require ‘waz-queues’ #=> if you want to use queues require ‘waz-blobs’ #=> if you want to use blobs (or you can include both for using both)
WAZ::Storage::Base.establish_connection!(:account_name => account_name, :access_key => access_key) As you can see you, the way to get started it pretty simple, just include the establish_connection! call on your application bootstrapper, for example if you are doing a sinatra-rb application it can be
configure :production do # here is were you hook up with WAZ Storage Stuff. end
That’s all you need to get started and running your Windows Azure Code on a sinatra application.
Windows Azure Queues are extremely powerful for asyc programming, they are very useful when you don’t need “always consistent” data operations allowing you to enqueue and process on background. It’s important to mention that being WAZ-Queues a buffered message system, you should design for idempotent operations given the fact that there’s no guarantee that a message can be repeated.
The implementation of the Windows Azure Queues REST API available online at msdn.microsoft.com/en-us/library/dd179363.aspx is fully covered here.
Here’s a quick getting started of the usage of the API:
WAZ::Storage::Base.establish_connection!(:account_name => account_name, :access_key => access_key)
# NEW! Now you can do ensure and it will return the queue, first trying to retrieve it # and then if it doesn’t exist, will created it queue = WAZ::Queues::Queue.ensure(‘my-queue’) # excepts that the metadata for the queue changes this method behaves as PUT (create/replace) # remarks: it performs a validation whether metadata changed or not (if changed HTTP 409 conflict) queue = WAZ::Queues::Queue.create(‘my-queue’)
10.times do |m| # enqueue a receives string. Message content can be anything up to 8KB # you can serialize and send anything that serializes to UTF-8 string (JSON, XML, etc) queue.enqueue!(“message##{m}”) end
while(queue.size > 0) do # Since WAZ implements the peek lock pattern we are locking messages (not dequeuing) # it has two parameters how many messages and for how long they are locked messages = queue.lock(10)
puts “dequeued message: #{messages.size}”
# deletes the message from the queue so other clients do not pick it after # visibility time out expires messages.each {|m| m.destroy!} end It’s pretty intuitive, but full documentation (RDoc) is available for the API for further reference.
The blobs implementation inside this gem is fully compliant with the spec available at msdn.microsoft.com/en-us/library/dd135733.aspx. The Windows Azure Blobs REST API isn’t fully covered here (see TODO’s for more information). It’s pretty usable and stable right now, I’ve been doing lot of testing around and it works seamlessly with the current Windows Azure implementation.
require ‘waz-blobs’
WAZ::Storage::Base.establish_connection!(:account_name => account_name, :access_key => access_key)
# creates a container container = WAZ::Blobs::Container.create(‘my-container’)
# stores a blob with custom properties (metadata) blob = container.store(‘my_blob.txt’,
‘this is the content of my blob’, ‘plain/text’, {:x_ms_meta_Custom_Property => “custom_value” })
# return a specific blob from a container blob = container
# retrieves a blob value blob.value
It’s pretty intuitive, but full documentation (RDoc) is available for the API for further reference.
This feature allow to access a blob only with the information of the blob’s name, the container, and the Shared Access Signature. If you don’t have permission to perform an action (for example: list the blobs, create a new blob or something like that) you will receive an exception. If you want more details about this: msdn.microsoft.com/en-us/library/ee395415.aspx
require ‘waz-blobs’ WAZ::Storage::Base.establish_connection!(:account_name=> ‘mysharedblob’, :use_sas_auth_only=> true, :sharedaccesssignature =>“?se=XXXXXX&sr=c&si=XXXXX&sig=XXXXXXXXXXXXXXXXXXXXX”)
container = WAZ::Blobs::Container.new(:name=>‘container’)
blob = container.store(‘myfile.txt’,File.read(‘fileofthecontent.txt’),‘plain/text’)
The feature was only tested storing a blob.
The tables implementation inside this gem is fully compliant with the spec available at msdn.microsoft.com/en-us/library/dd179423.aspx. The Windows Azure Tables REST API is fully covered here.
require ‘waz-storage’ require ‘waz-tables’
WAZ::Storage::Base.establish_connection!(:account_name => account_name, :access_key => access_key)
# creates a new table table = WAZ::Tables::Table.create(‘my-table’)
# list available tables # (returns a maximum of 1,000 items at one time (more details here: msdn.microsoft.com/en-us/library/dd135718.aspx) tables = WAZ::Tables::Table.list
# more tables WAZ::Tables::Table.list(tables.continuation_token)
# get a specific table, returning nil when the specified table is not found my_table = WAZ::Tables::Table.find(‘my-table’)
# table properties my_table.name my_table.url
# delete a table my_table.destroy!
# Entity operations using the Table service
# get the existing service instance service = WAZ::Tables::Table.service_instance
# define a new entity
entity = { :address => 'Mountain View', :age => 23, :amount_due => 200.23, :binary_data => File.open(__FILE__), :customer_code => 'aaaaaaaa-bbbb-cccc-dddd-aaaabbbbcccc', :customer_since => Time.now.utc, :is_active => true, :num_of_orders => 255, :partition_key => 'customer', :row_key => "myRowKey#{rand(2000000).to_s}", :Timestamp => Time.now.utc }
# inserts a new entity service.insert_entity(‘customer_table’, entity)
# retrieves all entities from a table # (returns a maximum of 1,000 items at one time (more details here: msdn.microsoft.com/en-us/library/dd135718.aspx) # Remarks on development storage it retrieves all items instead the first 1,000 entities = service.query(‘customer_table’)
# retrieves more entities providing the obtained continuation_token service.query(‘customer_table’, {:continuation_token => entities.continuation_token} )
# retrieves all records that match with the specified query but only the first fifteen rows service.query(‘customer_table’, {:expression => “(PartitionKey eq ‘customer’) and (Age eq 23)”, :top => 15} )
# get an existing entity by its partion_key and row_key entity = service.get_entity(‘customer_table’, ‘customer’, ‘rowKey1’)
# updates an entity entity = 90 service.update_entity(‘customer_table’, new_entity)
# merges an entity (more details here: msdn.microsoft.com/en-us/library/dd179392.aspx) entity = 20 service.merge_entity(‘customer_table’, new_entity)
# deletes an entity service.delete_entity(‘customer_table’, ‘customer’, ‘rowKey1’)
It’s pretty intuitive, but full documentation (RDoc) is available for the API for further reference.
Sometimes while you are building a web application you may require handling different storage account but contextualized. The sample that comes to my mind is something like a Storage Explorer or Account Monitor for WAZ.
That is why? I’ve added a new way of handling a stack-based contextual connection handling. The usage is pretty simple: WAZ::Storage::Base.establish_connection(options) do container = WAZ::Blobs::Container.find(‘container-name’) blob = container blob.destroy! end
As it is described on the example above, there’s a new way of establishing a connection and use it on a given block. The whole implementation is stack based, and will let you scope your context for some rare cases where you have another account.
Disclaimer: Moving objects across context isn’t contemplated yet, and if you try to do changes among scopes you will get to some wired Windows Azure Errors regarding objects that may not exist.
Windows Azure Storage API works flawlessly from Heroku and custom ruby hosting deployments on EC2, as far as I tested it. You can leverage the storage services without the need of having to write the application on .NET or hosting your application on Windows Azure.
The documentation and implementation exposed here is for the pre-release version and is subject to change on the future.
As far as users start using it, I’ll be building a backlog and probably handling a wish-list of features, but right now I’ve the following TODO’s already enqueued for further releases of the waz-storage API.
-Generate a sample application to better show the usage.
The things listed above do not represent any sort of priority, or the order they are going to be tackled. It’s just a list.
Written by Johnny G. Halife (johnny.halife at me dot com)
contributed by: Ezequiel Morito (twitter.com/ezequielm), Juan Pablo Garcia (twitter.com/jpgd), Steve Marx (twitter.com/smarx), G. Montard (github.com/gmontard)
Released under the MIT License: www.opensource.org/licenses/mit-license.php