Asset Manager Guide

Overview

The asset system is designed to take away many of the micro-management tasks associated with asset handling. These tasks include:

• Asset Discovery
• Asset Identity
• Asset Loading
• Asset Unloading
• Asset Dependencies
• Asset Grouping
• Asset Query
• Asset Meta-data
• Asset Renaming, Relocating and Deleting

Prerequisite Knowledge

The asset system uses the Module manager so a familiarity with that system is recommended for the following document. The main principle to understand is that the module system raises module events. Some of these events are "listened to" by the asset system, namely the "Module Load" and "Module Unload" events. When a module is loaded, this event is received by the asset system and it responds by loading the assets contained within that module. Likewise, when a module is unloaded, the event is received by the asset system which then responds by unloading the assets contained within that module. The specific details of this loading/unloading mechanism can be found below however the main point to understand is that asset loading and unloading is triggered by a respective module being loaded and unloaded.

Additionally, like many other Torque systems, TAML is used as the persistence mechanism so knowledge of this system is recommended. The examples here will typically show TAML in an XML format however it should be clear that these examples could also be in TAMLs binary format but for obvious reasons XML is a better example.

Finally, understanding path references using "expandos" is important.

TorqueScript Bindings

• Asset Manager
• Asset Base
• Asset Query
• Asset Tags

What are assets?

An asset is defined as an instance of a known engine type. This rather abstract definition does however mean that an asset can be almost anything! It is typical to think of assets as "stuff you use in games" such as images, animations, sounds etc. Whilst this is true, an asset could also be a script, a GUI, a player profile etc. As long as the object can be represented by a single instance of an engine type that can be created and destroyed dynamically and can be persisted correctly using Taml then it can be an asset.

The overall architecture of the asset system is based upon an asset being explicitly defined by an asset definition file. An asset definition file is persisted using TAML and it contains a persisted version of the instance of the engine type representing the asset itself.

The asset system maintains a database of available assets. Available assets are those which have been found whilst scanning for asset definition files with each file representing a single asset.

When an asset is added to the database it can then be loaded, unloaded, renamed, deleted, removed etc.

INFO! This document refers to the "Asset System" but in actual fact, this is an engine type named "AssetManager". At engine start-up an instance of this is automatically generated inside the engine itself. This instance is exposed to the scripts by the named object "AssetDatabase". It is named this because in Torque you cannot name an object the same as its type name. The terms "Asset System", "AssetManager" & "AssetDatabase" can be used interchangeably to mean the same thing.

Assets typically contain the following information:

• Information to configure the asset
• References to other files (asset loose files)
• References to other assets (asset dependencies)

An example of information an asset can contain would be specifying how many cells are in an image asset. An example of a reference to a file would be an image asset referring to the bitmap it uses. Finally, an example of a reference to another asset would be an animation asset referring to the image asset it uses as the source of the frames it animates.

Asset Primer

Whilst the majority of this document won't go into the C++ engine details, it is useful to understand the basic hierarchy of any asset engine type.

When generating a new asset type, the only requirement is that the new type is derived from an engine type named AssetBase. This type contains the common functionality for assets including fields and accessors and the internal communication with the asset system. When a new type derived from AssetBase is created, there is no special functionality you must implement.

For instance, if in C++ we were creating a "ScriptAsset" (ignore how its used, it's just an example) then we'd simply use this:

class ScriptAsset : public AssetBase
{
   ...
}

The above is a valid asset that the asset system can accept. For Torque you obviously need to specify the console macros but that's it. At this point it could be persisted as an asset definition and subsequently loaded.

Here's an example of such an asset persisted as an asset definition file:

<ScriptAsset
    AssetName="HighScoreScript" />

As you can see the persisted form of our custom type "ScriptAsset", which has no fields or functionality apart from what it inherits from AssetBase, is practically empty. Indeed, the only mandatory field that any asset must have is the "AssetName" field which is inherited from AssetBase itself.

In theory you could load-up an asset definition yourself using Taml and you will be returned an instance of the engine type within it just like you would any Taml file. You could think of this as you "manually loading" the asset.

In reality however you do not and should not do this, this is the job of the asset system. When you ask the asset system to load an asset, it essentially does just this for you i.e. it loads up the asset definition, retrieves the instance of the asset type and returns it back to you. Of course, there's a lot of internal house-keeping going on as well but at a high-level, the process is simple to understand.

To make the point concrete: an asset definition file is simply an instance of an engine type that is derived from AssetBase persisted using Taml

Declared Assets

So with the basics out of the way, it's important to understand how assets become known to the asset system. As described above, when a module is loaded, the asset system receives an event about the module being loaded at which point it then "scans" for assets. But how does it do this? For performance reasons it was decided that recursively iterating each and every folder within a module for asset definition files was potentially too expensive. Also, it offers little control for the developer who wishes to only scan specific folders for specific files.

To this end, the asset system allows you to configure which folders to scan, whether to scan them recursively and what file-extensions to look for. Rather than using a separate file to define these, the asset system simply looks at the module-definition file for your module. Let's say your module definition file looks like this:

<ModuleDefinition
    ModuleId="TruckToy"
    VersionId="1"
    Description="A monster truck toy."
    Dependencies="ToyAssets=1"
    Type="toy"
    ToyCategoryIndex="5"
    ScriptFile="main.cs"
    CreateFunction="create"
    DestroyFunction="destroy"/>

As you can see, this is the module definition for a "TruckToy". All the details here are for the module system and are not relevant to assets or this document at all. However, the "ModuleDefinition" can have children objects of any type and this can be used to add child objects that other systems can consume.

In this case, we want to add a "DeclaredAssets" child to the module definition like this:

<ModuleDefinition
    ModuleId="TruckToy"
    VersionId="1"
    Description="A monster truck toy."
    Dependencies="ToyAssets=1"
    Type="toy"
    ToyCategoryIndex="5"
    ScriptFile="main.cs"
    CreateFunction="create"
    DestroyFunction="destroy">
        <DeclaredAssets
            Path="assets"
            Extension="asset.taml"
            Recurse="true"/>		
</ModuleDefinition>

As you can see, a child object of the type "DeclaredAssets" has been added. This type has three fields:

• Path - The sub-folder (if any) relative to the module definition from which to search
• Extension - The extension (actually end of filename) of files to include in the search.
• Recurse - Whether to recurse the path and its sub-paths or not.

In the example, the sub-folder relative to the location of the module definition file is searched for any files ending in "asset.taml". All sub-paths are also searched.

This configuration allows you place all your assets anywhere in the "assets" sub-folder but you can organize this however you like. To aid in this, you can add an unlimited quantity of "DeclaredAsset" objects onto the module definition like this:

<ModuleDefinition
    ModuleId="TruckToy"
    VersionId="1"
    Description="A monster truck toy."
    Dependencies="ToyAssets=1"
    Type="toy"
    ToyCategoryIndex="5"
    ScriptFile="main.cs"
    CreateFunction="create"
    DestroyFunction="destroy">
        <DeclaredAssets
            Path="EnemyAssets"
            Extension="asset.taml"
            Recurse="true"/>
        <DeclaredAssets
            Path="GuiAssets"
            Extension="asset.taml"
            Recurse="true"/>
</ModuleDefinition>

While this offers flexibility, it's far easier to have a root folder containing all your assets (for this module) and just create sub-folders however you like. As will be discussed later, the asset system provides ways to group assets logically using the asset meta-data but its common to physically separate assets by type i.e. "Images", "Animations", "Particles", "Audio" etc by simply placing these as sub-folders of the main folder to be searched as specified above.

Asset States

So before we get into some real asset usage, one final and important point that needs to be covered is asset states.

Remember, just because you have an asset defined on disk, doesn't mean the engine automatically knows about it. As you've seen above, you need to ensure that at some point the asset system discovers this asset by scanning a folder that its in. If you don't do this then the engine won't know about it at all.

While this is obvious to some, it's important to clarify the three possible states of any asset. These are:

• Unknown
• Unloaded (Unreferenced)
• Loaded (Referenced)

If you simply create an asset but don't scan it then the asset is essentially unknown. A simple but important principle. The Unloaded state is when you've scanned the asset but nothing is currently referencing it. Loaded is when something is referencing the asset and the asset is loaded. In typical use an asset will go from unloaded to loaded and back again when things reference it.

This loading and unloading is completely automatic however there are several ways to configure the asset system (even on a per-asset basis) on how this happens.

Asset Identification

So far we've covered how to configure your module definition so that the asset system knows where to scan for assets. This has an important implication that was not stated: assets are defined within a module and are essentially owned by the module.

Now, let's look at how you actually define an asset. Let's use a real asset type that the engine uses i.e. "ImageAsset". Here's how one is defined:

<ImageAsset
   AssetName="background"
   ImageFile="sky.png"
/>

Without getting into the details of this particular asset type, you should be able to see immediately that the asset type is "ImageAsset" and that it defines an image file of "sky.png" (loose asset file).

The focus of this example however is the "AssetName" field. As you can see, this asset is named "background". Now you'll be forgiven in thinking that you'll simply use the name "background" alone to identify this asset when you're using it but that's not the case! Fear not, here's why and how indeed you would access this asset.

Let's start with a question. How likely is it that you'll either already have an asset named "background" or that you'll be provided with another module (containing assets) that similarly has an asset named "background"? It's actually quite likely.

Even if it could be argued that it's not likely, it requires effort to ensure that a conflict doesn't happen, especially when collaborating as a team. You can create your own nomenclature to try to reduce this such as adding prefixes and suffixes to try to make it less and less likely for a conflict.

The asset system gets around this by using the principle outlined above: all assets are owned by a module. With this in mind, know that when you refer to every asset by what is know as a asset-Id.

So what is an asset-Id? An asset Id is an automatic concatenation of the module-Id with the asset-Id.

As an example, let's say we have the "background" example above defined in a module Id of "SceneryAssets". This would mean that the asset-Id would be:

"SceneryAssets:background"

This is of the form:

<ModuleId>:<AssetName>

In other words, take the module-Id that the asset is in and the asset name itself and separate with a full colon (:) character.

So how would this actually be used in a real example?

%sprite = new Sprite();
%sprite.Image = "SceneryAssets:background";

So the prefix of the module-Id is mandatory. You can think of this as a scope or module location. If you had another "background" asset in another module then it wouldn't be confused with this one.

It should be obvious however that you cannot have the same asset name in the same module even if they are of different asset types. This shouldn't be a problem however. The primary reason to do this is to allow a team of people working on a module to not have to worry about asset conflicts on other modules they (potentially) have no control over.

It also means that for the future, anyone selling asset packs would do so by simply packaging a module(s) without feature of asset conflicts. Also for the future, these references will be handled by an editor and so will be mostly transparent during the game design process anyway.

Finally, if you were to copy an asset to another module then it automatically has a unique asset-Id without having to change the asset itself.

Asset Usage

So we showed the example of an asset being used like this:

%sprite = new Sprite();
%sprite.Image = "SceneryAssets:background";

... but what did this example actually do?

Well after creating the "Sprite" type, the field "Image" was assigned an asset-Id. Pretty simple stuff but what happens behind the scenes is important in getting a deeper understanding of your assets.

The "Image" field of type "Sprite" type accepts only a single type of asset, in this case an "ImageAsset". When you assign an asset-Id, the asset system will be asked to acquire an asset-Id of "SceneryAssets:background". If it's not available then it fails and nothing is returned therefore the sprite won't render anything. You will get a warning in the log however and you can easily ask for more verbose detail on what went wrong.

So if the asset-Id is valid then the asset system will load it up if it's not already loaded and return the asset itself to the sprite. If the asset was already loaded then the asset systems adds a usage reference to the asset and again returns the asset itself to the sprite.

The asset itself stays loaded and assigned to the sprite and can be assigned to many other objects simultaneously. The asset they each have is in-fact the same object, not copies. The asset will stay loaded until all objects using the asset either have a different assignment or are themselves destroyed.

This loading and unloading of assets is done automatically, you don't need to manage this yourself. Internally, each time an asset is referenced (also know as "acquired") then the asset system either loads the asset or simply retrieves it if already loaded but it always adds increases reference count of the asset. When an asset is no longer referenced either because the object referencing it is destroyed or simply uses another asset then the reference count to that asset is reduced. When the asset count reduces to zero then the asset is unloaded saving memory. This is the default behavior, you can control how asset loading and unload occurs globally or even on a per-asset basis depending on what you wish to achieve.

INFO! When you assign an asset Id to an objects fields, it only accepts one type of asset i.e. a specific engine type such as "ImageAsset", "SoundAsset" etc. Because the asset Id itself doesn't specify the type of the asset, only its identity, it's possible to attempt to assign the wrong type to an object. In this case, the asset system will reject the assignment and nothing will happen (apart from a warning).

Asset System Exposed

The asset system itself is exposed to TorqueScript using a single named object of "AssetDatabase". This object exposes all the functionality provided by the asset system and as such, is fairly complex. For general usage, it's unlikely that you'll need this system at all. For the most part, you'll just assign an asset-Id to an object that wants one and the job is done.

It's worth getting to know at least a few of the features that the asset system directly exposes. For now though, here's how you'd use the asset system. In this example we simply want to get a count of all the currently declared assets (assets that have been scanned and discovered).

Here's how you'd do that:

%count = AssetDatabase.getDeclaredAssetCount();

Advanced Asset Usage

The remainder of this document is for the advanced user of the asset system. It's essential knowledge to those writing asset editors or a general game editor system that wishes to handle an asset pipeline and publishing set-up.

Asset Queries

It's great that assets are loaded when their respective modules are loaded and the same goes for assets being unloaded when their respective modules are unloaded however from an editors point-of-view, knowing what assets are available and the details of each is critical. Thankfully, performing these actions is easy.

The asset system (also know by its name as the "asset database") exposes many methods of which a certain subset are known as the "query" methods. These methods are:

• findAllAssets() - Finds all assets.
• findAssetName() - Finds assets of a particular name, even partial name searches.
• findAssetCategory() - Finds assets with a specific category setting.
• findAssetAutoUnload() - Finds assets with a specific auto-unload setting.
• findAssetInternal() - Finds assets with a specific internal setting.
• findAssetPrivate() - Finds assets with a specific private setting.
• findAssetType() - Finds assets with a specific type (this is the engine type of the asset)
• findAssetDependsOn() - Find assets that the specified asset depends on.
• findAssetIsDependedOn() - Find assets that are depended on by the specified asset.
• findInvalidAssetReferences() - Finds assets that are referenced but are not registered (currently)
• findTaggedAssets() - Finds assets that are tagged with the specified tag(s).
• findAssetLooseFile() - Finds any assets using the specified loose file.

It's beyond the scope of this document to detail each and every aspect of all these query methods although a few will be covered. Besides, once a few have been described, the others merely expose different asset details to query on. Full details of what each does as well as descriptions of the respective method arguments can be found in the script-binding source or generated documentation.

All these methods have one thing in common, they can query the asset database for assets or can query a previous query! The asset system maintains an internal database of all assets currently registered, this is what can be queried. If you have performed a previous query, you can specify that the query can be the source (database) of the new query. This allows you to perform multiple overlapping searches which is very useful in providing a fast way to allow the end-user filter the (potentially) vast number of assets.

The way the query system works warrants more detail however. TorqueScript is a string-based language and is notorious for its bad performance due to heavy use of strings. Obviously, performing queries that return large sets of strings back and forth between the engine and TorqueScript wouldn't scale well and would provide terrible performance.

To combat this, the strategy for performing queries is like this:

• Create a "AssetQuery" type object.
• Call the query method i.e. "findAllAssets()", "findAssetName()" etc passing the asset-query object reference.
• The asset system will populate the asset-query object with the results
• You can enumerate the asset-query results or use it for further queries to refine the results
• Delete the "AssetQuery" object.

This strategy allows many queries to be performed without the overhead of passing strings back and forth. When the final query has been performed, the "AssetQuery" provides a few simply methods to enumerate the results. Each result is simply an asset Id string. The following methods are available on the "AssetQuery" type:

• getCount() -Get the count of results currently in this asset query instance.
• getAsset() -Get the specified result currently in this asset query instance.
• clear() - Clear the results from this asset query instance.
• set() - Set the asset query to be a copy of the results of another asset query instance.

Here's a simple example that can be used to retrieve all the currently registered assets:

// Create query instance.
%query = new AssetQuery();
 
// Find all assets.
AssetDatabase.findAllAssets( %query );
 
// Show all the asset Ids we found.
for( %i = 0; %i < %query.getCount(); %i++ )
{
   echo( %query.getAsset( %i ) );
}
 
// Delete query instance.
%query.delete();

Here's a more complex example that finds all the assets that have the string "Ninja" in them which are of the type "ImageAsset":

// Create query instance.
%query = new AssetQuery();
 
// Find partiallly-named assets.
AssetDatabase.findAssetName( %query, "Ninja", true );
 
// Refine the search to specific asset types.
AssetDatabase.findAssetType( %query, "ImageAsset", true );
 
// Show all the asset Ids we found.
for( %i = 0; %i < %query.getCount(); %i++ )
{
   echo( %query.getAsset( %i ) );
}
 
// Delete query instance.
%query.delete();

Asset Meta-Data

Armed with an asset-Id whether that be just known or via one of the many asset query methods described above you have everything need to perform any action on that asset. You do not need to know anything about the asset apart from its unique asset-Id to work with it.

Like the asset query methods, the asset system exposes methods used to retrieve information about assets. The asset system provides a standard set of meta-data that can optionally be assigned to each asset such as description, category etc. There's also read-only meta-data that the asset system generates such as the assets type, location etc.

All these methods simply require an asset-Id, they are:

• getAssetName() - Get the asset name. Recall that the assets name is the second component of the asset-Id i.e. "MyModule:Ninja" would return "Ninja". This is exposed on the asset as a field "AssetName".
• getAssetDescription() - Gets the asset description. It allows an optional plain-language description of the asset that can be presented to end-users, used in tool-tips etc. The asset system does not actively use this field. This is exposed on the asset as a field "AssetDescription".
• getAssetCategory() - Gets the asset category. It allows you to categorize assets which is especially useful for complex projects. The asset system does not actively use this field. This is exposed on the asset as a field "AssetCategory".
• getAssetType() - Gets the asset engine type i.e. "ImageAsset", "SoundAsset" etc.
• getAssetFilePath() - Gets the file-path (path that includes the file) to the asset definition file.
• getAssetPath() - Gets the path (does not include the file) that the asset definition file is located in.
• getAssetModule() -Gets the module definition object instance that the asset is contained within.
• isAssetInternal() - Gets whether the asset is flagged as internal use only or not. It can be useful in hiding assets used internal-only i.e. in (say) an editor. The asset system does not actively use this field. This is exposed on the asset as a field "AssetInternal".
• isAssetPrivate() - Gets whether the asset is flagged as private use only or not.
• isAssetAutoUnload() - Gets whether the asset is flagged to automatically unload or not when all references to it are released. This was discussed previously and can be used to control each asset individually. Only use this if there is clear evidence that this asset is causing issues. Turning auto-unload off can result in heavy memory usage. This is exposed on the asset as a field "AssetAutoUnload".
• isAssetLoaded() - Gets whether the asset is currently loaded or not.
• isDeclaredAsset() - Gets whether the asset is current declared or not. A "declared" asset is a "known" asset. if it is declared then it can be used. This is useful when wanting to check to see if an asset Id is currently valid or not. Note that the asset Id may be valid but if the respective asset is not registered because the module it is contained within is not loaded then the asset system has no knowledge of it.
• isReferencedAsset() - Gets whether the asset is currently referenced in persisted Taml files or not. This topic has not yet been covered but will be later in this document. The "reference" here refers to a reference in another file such as a persisted "Scene" or other file and not an object in memory.

Here's a basic usage example:

%assetId = "MyModule:Ninja";
 
echo( AssetDatabase.getAssetName(%assetId) );
echo( AssetDatabase.getAssetDescription(%assetId) );
echo( AssetDatabase.getAssetType(%assetId) );

The asset definition exposes this some of this fixed meta-data as fields:

• "AssetName"
• "AssetDescription"

Explicit Acquire and Release

As has been described, assets are automatically loaded (acquired) and unloaded (released) when their respective asset-Id is assigned to an object or objects. Loading of an asset is known as acquiring an asset because asking for an asset doesn't always result in it being loaded as it may already be loaded. Likewise, unloading of an asset is known as releasing an asset because setting an object so that it no longer uses an asset does not always result in the asset being unloaded as it may also currently be used by other objects. As a user, you don't care about that, simply that you want to acquire and release assets.

Nearly all the time, this is done under-the-hood and it is highly discouraged explicitly acquiring and releasing assets however there are some circumstances, primarily inside an asset editor, where this is required. Acquiring and releasing an asset is done, as always, by specifying an asset-Id. The asset system exposes two methods for this:

• acquireAsset() - Get the asset instance of the specified asset-Id. May result in the asset being loaded if it's not already else you'll just get the loaded asset instance.
• releaseAsset() - Release the asset instance of the specified asset-Id. May result in the asset being unloaded if it has no other references and is allow via its auto-unload setting.

These methods, in TorqueScript, give you direct access to the asset instance which is only required if you wish to perform work directly on the asset itself, something which editors obviously do. Beyond that, you do not need to perform these actions explicitly. Also, when you acquire an asset, you must release it when you are finished with it otherwise the asset will continue to be referenced resulting in the asset staying in memory. More importantly, never release an asset unless you are sure you've acquired it yourself. While this won't cause a crash, it could cause the asset to be unloaded and any other exist users of the asset will automatically have their asset unassigned.

Asset Administration

There are several methods exposed by the asset system that provide low-level access, so called "admin" access to assets. These methods, whilst powerful, are more complex to understand. They are only really useful when writing an application that is manipulating assets. These are:

• compileReferencedAssets() -Specifies a module to scan for assets being referenced. This topic will be discussed later in this document.
• addModuleDeclaredAssets() - Specifies a module to scan for asset definitions. This is automatically done when a module is loaded however this allows explicit control to scan for assets. This is for advanced use.
• addDeclaredAsset() - Specifies a module to add a specific asset definition. This is automatically done when a module is loaded however this allows explicit control to discover a specific asset. This can be used by editors when they have generated a new asset definition on disk and wish for it to be discovered in an already active module.
• addPrivateAsset() - Adds a private asset. This topic will be discussed later in this document.
• removeDeclaredAsset() - Removes a specific asset Id from the asset system database. This does not delete the asset definition but simple removes the knowledge of the asset definition from the asset system. This causes the asset to be immediately unloaded resulting in objects referencing the asset to no longer reference it.
• renamedDeclaredAsset() - Rename a declared asset. The in-memory asset name will be updated as well as the asset definition on disk.
• renamedReferencedAsset() - Rename a referenced asset i.e. a reference to a declared asset. All previously compiled referenced assets discovered (using "compileReferencedAssets()" will be renamed on disk.
• deleteAsset() - Deletes a specific asset Id causing it to not only to be removed from the asset database but also for the asset definition to be removed from the disk. This will also (optionally) remove loose asset files (raw files the asset definition refers to) as well as (optionally) deleting assets that depend on this asset.
• purgeAssets() - Typically, when assets are no longer referenced, they are automatically unloaded but as has been shown, setting auto-unload to off can result in assets staying in memory so that if they are subsequently acquired, they no longer need to be loaded. This however means that the assets can stay in memory for long periods of time. This method will purge (unload) assets that are in this state i.e. assets that are loaded but have no references. This can be useful during logical game transitions such as moving to another scene to loading another game. It provides a useful method to purge unused assets at an explicit time.
• refreshAsset() - Synchronizes the in-memory asset with the asset definition file on-disk. Essentially, the in-memory asset is saved to the asset definition file, not the other way around. It also causes a cascade of events to happen including updating any asset dependencies and loose files. In essence, the asset definition and all in-memory asset database references are updated.
• refreshAllAssets() - Performs the same action as "refreshAsset()" however it does so on all currently registered assets. This is an expensive operation and should be rarely used.

External, Internal and Private Assets

An external asset is a standard asset (and most common) that can be assigned to objects as normal. An external asset reference can safely be persisted via objects e.g. as a reference on a sprite being persisted as part of a scene.

An internal asset is essentially an external asset but flagged as internal-use-only. This has no functional difference for the asset system and is only a passive flag as far as it is concerned. It does however allow querying of the asset by this flag so that an editor could (for instance) omit showing to the end-user, assets that are for internal-use-only i.e. hidden assets out of reach.

A private asset however has a fairly large functional difference from an external or internal asset. It is designed primarily for when an asset is needed temporarily but still be allowed to be assigned to objects. A private asset does not have an asset definition on disk and is never persisted as such therefore it can be thought of as "volatile" i.e. it is in-memory only. This is extremely powerful when creating an editor as an asset object instance can be created, configured and added as a private asset directly i.e. you pass the object. There is no need to have an asset definition file and you can start assigning the object immediately.

Finally, a private asset does not need to come up with a unique name or worry itself about naming conflicts simply because the asset system automatically assigns it a temporary name when it is added as a private asset.

For example:

// Generate an anonymous asset.
%asset = new ImageAsset()
{
   ImageFile = "Alien.png";
};
 
// Add my anonymous asset as a private asset.
%assetId = AssetDatabase.addPrivateAsset( %asset );
 
// Assign the private asset to a Sprite.
%sprite = new Sprite();
%sprite.ImageMap = %assetId;
 
// Dump some info to the console.
echo( %asset.getAssetId() );

As you can see, you can generate an asset object instance directly and after adding it as a private asset, start using it immediately. It is also available not only for assigning to objects but also is included in the asset system as any normal asset for all other features such as asset queries (this is why the asset queries allow filtering by external, internal and private assets).

Unlike external or internal assets however, a private asset is not associated (at least externally) with a specific module. This means that it will never be removed from the asset system unloading any modules. It is the developers responsibility to remove private assets!

To remove a private asset, you simply use the standard method used to remove any asset whether it be external, internal or private:

// Remove the asset.
AssetDatabase.removeSingleDeclaredAsset( %myPrivateAsset );

IMPORTANT! You must never persist object instances that have a reference to a private asset. Doing so will work but the subsequent asset reference won't be available when the object instance is next loaded. Even if you were to generate a new private asset next time, you cannot guarantee its asset Id as that is automatically allocated by the asset system when you add the private asset. Doing so will result in an asset warning being emitted to the console. You can spot these easily as private asset Ids are formatted like so e.g. "ImageAsset_10", "SoundAsset_3", "AnimationAsset_2" etc. If you see these in the console then it is because an object has tried to acquire a private asset which is no longer available, probably because it was persisted like this._

As a convenience, you can also acquire a standard asset but ask for a copy of of it as a private asset. This allows you to use the specified asset but then go on to perhaps change it. The new cloned private asset uses the same rules including the fact that you should not persist it as a reference because its asset Id is a special asset Id allocated by the asset system and not the asset Id of the asset it came from.

You can do this using:

// Acquire an asset but acquire it as private asset.
// This temporarily acquires the specified asset, clones it, adds it as a private asset then releases the asset it was cloned from.
%privateAssetId = AssetDatabase.acquireAsset( %assetId, true );

Referenced Assets

The asset system primarily deals with what it terms "Declared" assets. This term refers to the declaration of an asset i.e. the asset definition file on the disk. When the asset system discovers an asset definition file it is discovering a declared asset.

Declared assets however are only part of the story. The obvious reason to have declared assets in the first place is to use them! When you use an asset you typically assign its asset Id to an object instance field that can use it. Typically, these objects are persisted to disk using Taml i.e. Scenes or other collections of objects such as SceneObjects in SimSets. Whatever the contents of the Taml file, many of these objects persist references to asset Ids.

It is these references which are what the asset system is interested in. Why? Well the asset system not only takes on the responsibility of maintaining a database of declared assets, allowing you to work with them in many ways but it also allows you to perform actions such as renaming and destroy them.

It could leave its responsibility right there but the problem with doing so lies at the doorstep of referenced assets. What happens if you rename an asset that is referenced in several other Taml files whether they be Scenes or other content? If you perform that action then the other files immediately become out of date. The worst case is that these files, when loaded, emit warnings to the console informing the user that the specified assets could not be found.

This would be a poor situation to be in but luckily, the asset system was designed to be able to take on the responsibility of updating all files that reference declared assets.

The asset system can do this because it offers the ability to scan Taml files for referenced assets. When it has discovered these referenced assets, it can maintain knowledge about them in its database. Now, if you were to rename an asset i.e. a declared asset then the asset system can automatically update the respective referenced assets in the respective files. The same goes for a more destructive operation of deleting a declared asset. In this case it can remove the asset reference completely.

To allow the asset system to do this you must tell it what files to scan. Having the asset system scan each and every file would be prohibitively expensive therefore it adopts the exact same strategy as was employed when scanning for declared assets. This strategy is to specify, on the module definition, a referenced asset location to search as a child object however, instead of using a "DeclaredAsset" type you use a "ReferencedAsset" type like so:

<ModuleDefinition
    ModuleId="TruckToy"
    VersionId="1"
    Description="A monster truck toy."
    Dependencies="ToyAssets=1"
    Type="toy"
    ToyCategoryIndex="5"
    ScriptFile="main.cs"
    CreateFunction="create"
    DestroyFunction="destroy">
        <ReferencedAssets
            Path="levels"
            Extension="level.taml"
            Recurse="true"/>		
</ModuleDefinition>

In the example, the sub-location of "levels" is searched for all files ending in "level.taml". You can use as many of these are you required depending on your organizational needs. It's more likely that these files are more widespread than asset files so depending on the projects layout, you'll probably use more than a single one of these. It'll also be needed if you want to scan several different types of file.

With this configured, the module contains all the information required by the asset system for scanning for referenced assets. So with that done, when does the asset system use this information? The answer is that it doesn't do it automatically simply because it does not know when it is appropriate to do so. The asset system exposes a single method to perform this action however:

// Compile referenced assets.
AssetDatabase.compileReferencedAssets( %module );

As simple as that. You pass the module definition as the argument and the asset system then looks at the module and retrieves any specified referenced asset manifest. If none is specified then it stops there. If one is found then all the files specified in the asset manifest are scanned. This scanning results in a database of all asset references which it can then use when performing operations such as asset renaming or deleting.

It is important to understand however that whilst the asset system will maintain these asset references when you perform operations on assets, the opposite is not true i.e. if you change an asset reference directly on disk then the asset system has no way to know this. This typically happens in an editor when (say) a scene is persisted to disk or maybe the scene was deleted altogether. Also, a new scene may have been added. The asset system has no way of knowing about these changes!

In these circumstances, it's quite simple to deal with: Simply recompile the referenced assets by calling "compileReferencedAssets()". When you call this method, the asset system completely clears its referenced asset database and performs a scan.

Info! During the initial design, I had the ability to add multiple modules as targets for referenced assets but I decided that initially this was overkill. Currently you can only specify one module as a target for referenced assets however it is relatively simply to modify the API to accommodate multiple module targets for referenced assets.

It should be clear then that failing to add a referenced asset manifest to a module that contains files that referenced assets and failing to call "compileReferencedAssets()" on that module when it is first loaded and during subsequent changes to files that contain referenced assets will result in the asset system being unaware that operations like asset renaming and deleting have external references that need updating. This will result in orphaned asset references in such files, which while not fatal does result in not only warning being emitted to the console but also a poor experience for the end user who needs to update each and every object that references a changed asset. This would be especially frustrating if an asset is renamed.

Assets Referencing Assets

Whilst it's obvious something like (say) a "Sprite" type referencing an asset like an "ImageAsset", it's not immediately obvious that one asset might reference another! If you think about it, an asset using another asset makes complete sense, after all an "AnimationAsset" contains frames in a separate and specific "ImageAsset" therefore the "AnimationAsset" references an "ImageAsset" right inside its asset definition file!

To be absolutely clear here, the reference for such a relationship would be in the form of a field from one asset type containing an asset Id to another identified because it has the "@asset=" field prefix. This states quite clearly that for this asset to "work", it has a mandatory requirement of the specified asset before itself can be acquired.

This kind of reference has a special term known as an asset dependency. It's also handled automatically by the asset system in that you do not need to specify asset definition files to be scanned for asset references inside the reference asset manifest. Indeed, doing so wouldn't make sense because assets can be in multiple modules whereas (currently) referenced assets can only be in a single module.

The good news here is that the asset system automatically looks for these asset dependencies when initially scanning the asset definition file of any asset. Indeed, when it finds asset references it stores these references in the asset database as a dependency of the asset. You can query these dependencies as shown previously.

It's important to understand some underlying details of this dependency system. When an asset definition is scanned and dependencies found, the asset system does not validate the asset dependencies it finds. It doesn't do this because when assets are scanned, the order they are scanned is not a well known order and a dependency could be found for an asset that hasn't yet been scanned and therefore, to the asset system, isn't a declared asset.

The asset system stores the asset Id away as a dependency and an asset can have as many dependencies as it likes. The asset system only uses these dependencies during two periods. The first is when the asset with the dependencies is loaded and the second is when it is unloaded.

With all that "back-story" out of the way, here's the critical part of the asset dependency story: When the asset system is asked to acquire an asset Id, as described before, it first sees if it's already loaded, if so then it simply returns the loaded asset however if it is not then it needs to load the asset. The problem however is that before an asset can be loaded, it must have its dependencies loaded. In the first pass of the asset system, these dependencies were processed recursively but this led to problems. In this asset system generation, the asset system simply loads the asset. By doing this however it will read the asset definition file which itself contains those references. When they are assigned to the asset, the asset fields attempt to acquire the specified asset dependency. If that asset also has a dependency or several others then they are processed to. The net result is that by loading an asset, all its dependencies are acquired. If they are not acquired then the asset simply has empty asset references.

In a real world example this might be loading an animation asset which refers to an image asset which isn't available. In this case the animation would be available but with zero frames. It could be assigned but would not perform any animations.

Info! This load but fail gracefully is the idealistic asset type implementation i.e. loading but failing gracefully and where appropriate, outputting relevant debug information to the console for tracing. It's important that these objects do load because without this, their issues cannot be resolved by an editor without modifying Taml output which is not a good solution. Having an animation load but have no animation frames because it refers to an invalid image asset is easy for an editor to resolve by allowing an end-user to modify the asset it refers to. As soon as it does that, the animation has frames and all objects using it suddenly "come to life" as their animation asset references still refer to the animation.

As can be seen, when an asset is loaded, any dependencies would also be acquired (they may already be loaded) and this happens recursively although in reality, these dependencies are typically simple and shallow. On the opposite side of things, when an asset is released, this also releases the asset dependencies recursively. Any loading or unloading here follows the rules already described, this isn't a special process. The asset acquire and release here are the same calls made by the rest of the asset system

Asset Debugging

The asset system maintains many metrics to help the developer keep track of what the asset system is doing. Some of these are exposed as methods as well as on the SceneWindow debug banner.

Here are the ones exposed as methods:

• getDeclaredAssetCount() - Gets a count of the number of declared assets that have been scanned and are active in the asset database. Many more may exist on the disk however if those modules are not currently loaded then they won't be counted here.
• getReferencedAssetCount() - Gets a count of the number of uniquely referenced assets that have been scanned. In other words, if a 1000 asset references were found but altogether they only referenced 200 asset Ids then this would return 200.
• getLoadedExternalAssetCount() - Gets a count of the number of declared assets flagged as external that are currently loaded. Loaded assets are assets that are actually in use and not just registered.
• getMaxLoadedExternalAssetCount() - Gets a count of the maximum number of declared assets flagged as external. This is essentially the maximum value that "getLoadedExternalAssetCount()" has ever been.
• getLoadedInternalAssetCount() - Gets a count of the number of declared assets flagged as internal that are currently loaded. Loaded assets are assets that are actually in use and not just registered.
• getMaxLoadedInternalAssetCount() - Gets a count of the maximum number of declared assets flagged as internal. This is essentially the maximum value that "getLoadedInternalAssetCount()" has ever been.
• dumpDeclaredAssets() - Dumps to the console a complete list of declared assets complete with all available metrics for each and every one.

The "dumpDeclaredAssets()" warrants some extra information. It dumps the information to the console in such a way that each field is comma-separated which makes it easy to import into a spreadsheet for further analysis. It dumps the following information:

• Asset-Id
• Asset References
• Load Count
• Unload Count
• Auto Unload?
• Loaded?
• Internal?
• Private?
• Asset Type
• Module-Id/Version-Id
• Asset Defintion File

There are some interesting additional fields in the dumped declared assets. Most notable are the "LoadCount" and "UnloadCount". These show how many times the specific asset has been loaded and unloaded i.e. the asset object instance has been read from the asset definition file and created and subsequently destroyed. Very high values for these could indicate asset thrashing where assets are being loaded and unloaded at a high frequency potentially affecting performance. The current number of references is also shown.

INFO! Some of these metrics are also available on the debug overlay that can be turned-on when rendering a scene.

Asset Tags

A common requirement for assets is categorization. Whilst the asset system has a passive field on each asset for a category, this is typically used by the editor system to broadly categorize assets for usage etc. For the user-facing asset experience, it's often required to assign assets to multiple overlapping categories. For this purpose, the asset system exposed asset "tags". An asset tag is simply a string that can be assigned to an asset. Each asset can be assigned multiple tags. The asset system applies no special meaning to these tags, they are purely for the end-user.

**Info! One of the problems to deal with before considering implementing a tags system is where to store such tags. There are several issues here to consider. One of the most important is how to associate tags with assets without modifying the assets themselves. Storing asset tags on the asset definitions themselves is a problem because this would be a global change i.e. it would affect all projects using that asset. **

So the relationship between a tag and an asset must be stored on a per-game-project basis. With this in mind, the asset system uses the strategy we've seen several times now where we associate data with a specific module using a manifest. In this case we don't use an asset manifest but rather a tags manifest. As expected, you specify this by specifying in the module definition the asset tags manifest like so:

<ModuleDefinition
    ModuleId="MyModule"
    VersionId="1"
    Purchased="1"
    Description="My Test Module"
    AssetTagsManifest="^MyModule/AssetTags.Manifest.taml"
 />

With this in place, when the module is loaded, the asset system will automatically load-up the specified asset tags manifest file. If the specified asset tags manifest does not exist, it will automatically be created at the specified location by the asset system therefore there is no need to generate an empty one yourself. Additionally, there can be only one asset tags manifest. As soon as one is loaded, any subsequent attempts to load another manifest (assuming one is specified on another module) will result in a warning and be ignored. Another manifest can only be loaded after the current one has been unloaded which happens when the module it is associated with is itself unloaded.

An example of an asset tags manifest is:

<AssetTagsManifest>
    <AssetTagsManifest.Tags>
        <tag Name="Backgrounds" />
        <tag Name="Terrain" />
        <tag Name="Projectiles" />
        <tag Name="Objects" />
    </AssetTagsManifest.Tags>
    <AssetTagsManifest.Assets>
        <tag
            assetID="{PhysicsLauncherAssets}:PL_DefaultWorldObjectAnim0"
            Name="Objects" />
        <tag
            assetID="{PhysicsLauncherAssets}:PL_GorillaIdleProjectileAnim"
            Name="Projectiles" />
        <tag
            assetID="{PhysicsLauncherAssets}:PL_DefaultWorldObjectAnim2"
            Name="Objects" />
        <tag
            assetID="{PhysicsLauncherAssets}:PL_DefaultWorldObjectAnim1"
            Name="Objects" />
    </AssetTagsManifest.Assets>
</AssetTagsManifest>

As you can see, there are two sections to the manifest. The first is a list of the valid tag names. The second is an association of an asset Id with a tag name.

There are two areas where you access the asset tags. The first is directly on the asset system itself. The second is directly on the asset tags manifest object instance.

Here are the methods available on the asset system:

• saveAssetTags() - Save any currently loaded asset tags manifest. This takes no arguments and simply persists to where it was loaded from. This must happen when a tag changes otherwise changes will be lost. If no asset tags manifest is currently loaded then this method does nothing.
• restoreAssettags() - Replace any currently loaded asset tags manifest with its copy on disk i.e. reload it dropping any changes in memory. This essential acts like an "undo" and works because any changes to asset tags are not automatically persisted back to the asset tags manifest on disk. If no asset tags manifest is currently loaded then this method does nothing.
• getAssetTags() - Gets any currently loaded asset tags manifest object instance. This is the object that you perform actual asset tagging on.

Here are the methods available on the asset tags manifest (retrieved using "getAssetTags() above):

• createTag() - Creates a new tag name. If the tag exists then no action is taken.
• renameTag() - Renames an existing tag updating all references to this tag.
• deleteTag() - Removes an exist tag removing all references to this tag.
• isTag() - Checks to see if a tag name exists or not.
• getTagCount() - Gets the current count of tag names.
• getTag() - Gets a specific tag name by zero-based index.
• getAssetTagCount() - Gets the current count of assigned tag names for a specific asset Id.
• getAssetTag() - Gets a specific tag assignment by zero-based index for a specific asset Id.
• tag() - Tags an asset Id with a specific tag name. An asset Id can be assigned to multiple tag names.
• untag() - Untags an asset Id from a specific tag name. If the asset is not tagged with the specified tag name then no action is taken.
• hasTag() - Checks to see if an asset Id has a specific tag name or not.

Asset Snapshots

Another feature that is designed primarily for asset editors is the ability to take a quick snapshot of an asset so that it can be restored later. The snapshot consists entirely of all the fields exposed by an asset. This feature therefore allows an undo/redo feature for asset editing to be created but there are other potential uses.

The asset snapshot system works by getting and setting snapshot information to and from a type named "AssetSnapshot". This type acts as a container for the snapshot information in very much the same way the "AssetQuery" type acts as a container for asset query information.

The two methods exposed by the asset system to do this are:

• getAssetSnapshot() - Gets a snapshot of the specified asset and places it into the specified asset snapshot object.
• setAssetSnapshot() - Sets the snapshot from the specified asset snapshot and places it into the specified asset.

A simple example of this being used is:

// Create the asset snapshot.
%snapshot = new AssetSnapshot();
 
// Take a snapshot of our asset.
AssetDatabase.getAssetSnapshot( %snapshot, %assetId  );
 
// Modify our asset somehow just as a test.
...
...
...
 
// Restore the asset with the snapshot.
AssetDatabase.setAssetSnapshot( %snapshot, %assetId );
 
// Delete the asset snapshot.
%snapshot.delete();

In-Engine Usage

Most of the document so far has covered access from TorqueScript. This part of the document will cover the common asset system usage from within the C++ engine. This idea here isn't to go into great detail on how the asset system does what it does but rather cover how assets themselves are used by types such as "Sprite", "Scroller", "GuiSpriteCtrl" etc. These are the in-engine users of the asset system and is what a typically C++ developer will do when using the asset system.

As a C++ developer you want to design types that can have a field or fields that can be assigned asset Ids. Given an asset Id you'd like to acquire the asset so you can use it. When the object instance of your type is destroyed (or at a point of your choosing) you'd like to release the asset Id. Not only this but restricting the asset Id to a specific asset type is critical. Finally, you'd like to have these fields persisting using Taml so that the correct asset Ids are referenced, not only that but the asset Ids are prefixed with the correct "@asset=" or "@assetFile=".

Performing all these actions manually can be tricky and the logic of it can become cumbersome, particularly when multiple asset references are being maintained.

The good news is that the asset system can perform all of these actions for you completely automatically!

Smart Asset Pointer

To achieve this, the asset system provides a C++ template "smart-pointer" type:

AssetPtr<T>

This smart asset pointer type allows you to not only maintain a pointer to an asset object instance but perform all the necessary duties described above. Here's an example of one in use:

void main( void )
{
    AssetPtr<ImageAsset> mAvatarImage;
}

In the example above, the asset pointer is set to use an "ImageAsset" type meaning it can only be assigned to that type of asset. The above code does not allocate an ImageAsset, it is only a pointer and occupies only 4 bytes of memory as it is simply a code wrapper around a simple pointer.

The AssetPtr type exposes many useful members, most important of which are the overloaded operators. The first important thing you need to know about this type is how to assign an asset to it. This is can be shown in this example:

void main( void )
{
    // Create the asset pointer.
    AssetPtr<ImageAsset> mAvatarImage;
 
    // Assign an asset Id.
    mAvatarImage = "MyModule:Alien"; 
}

As you can see, assigning an asset is done by assigning the asset Id string to the asset pointer. When this happens, the asset pointer asks the asset system to acquire the specified asset Id. If it fails then no action is taken (apart from a warning being emitted to the console). If it succeeds then the asset is then available via the asset pointer. As discussed previously, you do not need to be concerned directly with asset loading or unloading, the asset system is responsible for that aspect.

Assuming the above example succeeds and the asset is available, the next thing to do is access the asset itself. This can be done with the overloaded "->" operator as in this example:

void main( void )
{
    // Create the asset pointer.
    AssetPtr<ImageAsset> mAvatarImage;
 
    // Assign an asset Id.
    mAvatarImage = "MyModule:Alien";
 
    // Get some information from the asset.
    const U32 frameCount = mAvatarImage->getFrameCount();
}

In this example, the assigned asset is directly accessed via the "->" operator. There is however a problem with this code in that it is potentially unsafe. What happens if the asset assignment fails? In that case, the asset pointer points to nothing i.e. NULL. If that were the case then the final line retrieving the "frameCount" would crash the code accessing a NULL.

What is needed here is some defensive programming. To allow this, the asset pointer provides a set of methods designed to touch the asset pointer contents without needing to touch the asset instance itself. These methods are:

• isNull() - Checks whether the asset pointer currently points to an asset or not. If true then the asset pointer does not point to an asset instance.
• notNull() - Checks whether the asset pointer currently points to an asset or not. If false then the asset pointer does not point to an asset instance.
• clear() - Removes any currently assigned asset from the asset pointer. After this call, "isNull()" will return true. Any assigned asset will be released automatically by the asset system.
• setAssetId() - Assigns an asset Id to the asset pointer. Any existing asset being referenced will be released prior to the new one being assigned.
• getAssetId() - Gets the currently assigned asset Id. If nothing is assigned then an empty string is returned (actually "StringTable->EmptyString").
• isAssetId() - Checks if the currently assigned asset Id matches the specified asset Id. In other words, is the specified assigned Id the one that is assigned. If nothing is assigned then this always returns false.
• getAssetType() - Gets the currently assigned asset type. If nothing is assigned then an empty string is returned (actually "StringTable->EmptyString").

Info! The "getAssetType()" may seem odd when you consider that the asset pointer is always defined with the asset type e.g. "AssetPtr" however, the "AssetPtr" type has a base type of "AssetPtrBase". This type is a pure virtual type that exposes this and other methods. Lower-level engine access can then use this type to abstractly manipulate any asset pointer, including querying the real asset type it can contain.

As an example, here's how to check if an asset is assigned or not:

void main( void )
{
    // Create the asset pointer.
    AssetPtr<ImageAsset> mAvatarImage;
 
    // Assign an asset Id.
    mAvatarImage = "MyModule:Alien";
 
    // Finish if nothing assigned.
    if ( mAvatarImage.isNull() ) return;
 
    // Get some information from the asset.
    const U32 frameCount = mAvatarImage->getFrameCount();
}

In the example, ".isNull()" is used to determine if an asset is assigned or not. The "." (period operator) accesses the asset pointer itself and not any content and is therefore always safe to use. Remember that the "->" operator actually accesses the asset pointer contents (if any) itself.

When the asset pointer object itself goes out of scope i.e. it is destroyed then any asset it references will automatically be released. This applies no matter what scope the asset pointer is whether it be created on the stack or on the heap. Knowing this, it should be clear that in the example above, when the "main" function finishes, the asset pointer goes out of scope and is destroyed therefore releasing any assigned asset automatically.

Info! If any smart asset pointers are assigned to an asset object instance and the object instance is deleted then all of the smart asset pointers are automatically cleared. This is a very powerful feature and ensures that orphaned asset references don't exist. It should be clear however that either deleting or unloaded an asset immediately causes asset references to be removed. If these references are required then the asset should not have been destroyed or unloaded prior to the asset references being persisted. An example of this problem would be unloading a module containing assets when those assets are referenced in a Scene which is currently active. All those references are cleared resulting in objects in the Scene not rendering, playing audio etc. Even worse, if the Scene was saved, these references would not be there.

As the examples already show, assigning an asset Id can be done by a direct string assignment or by the "setAssetId()" method. The direct string assignment is achieved by overloading the "=" (assignment) operator. This operator is overloaded twice like so:

• operator=(assetId)
• operator=(assetPtr)

These overloads allow an assignment of a string containing an asset Id or another asset pointer (of the same asset type). In the case of assigning an asset pointer, it effectively assigns whatever the specified asset pointer contains. If the specified asset pointer references a valid asset then the target asset pointer will also. If the specified asset pointer does not reference an asset then the target asset pointer will not reference an asset either i.e. it will be cleared of any asset reference.

So using an asset pointer allows the developer to access assets easily without any concern regarding the real management of asset access. The final part of the developer story here is not only exposing these asset pointers to TorqueScript but also being able to persist them correctly.

Again, the asset system deals with this automatically. The only thing the developer needs to do is first expose this to TorqueScript using the standard field methods.

With specific aspects removed because they are standard Torque overhead and not relevant here, here's an example:

class Player : public SimObject
{
private:
    AssetPtr<ImageAsset> mAvatarImage;
 
...
...
};
 
void Player::initPersistFields()
{
    // Call parent.
    Parent::initPersistFields();
 
    // Expose our avatar image asset.
    addField( "Avatar", TypeImageAssetPtr, Offset(mAvatarImage, Player), "Player Avatar" );
}

In this example, we define the asset pointer to an "ImageAsset". We then expose a TorqueScript field named "Avatar". The most important aspect here is the field-type. The asset system defines a type for each valid asset. When a new asset type is created by a developer (a fairly rare event), they create a field type for the asset which is simple to do. This allows the asset type to be persisted. The following are the currently available asset field types:

TypeImageAssetPtr - Used for a "AssetPtr<ImageAsset>" type.
TypeAnimationAssetPtr - Used for a "AssetPtr<AnimationAsset>" type.
TypeAudioAssetPtr - Used for a "AssetPtr<AudioAsset>" type.

Using the above example, the following example would be possible:

// Create and configure our player.
%player1 = new Player();
%player1.Avatar = "MyModule:Alien";

The good news here is that with the field exposed, the type is now automatically capable of persisting the asset reference using Taml! The developer is now free to persist the object type knowing that it will work correctly with the asset system. This is made possible because all actions are going through the smart asset pointer.

As an example, here's how we could persist our player:

// Create and configure our player.
%player1 = new Player();
%player1.Avatar = "MyModule:Alien";
 
// Persist our player.
TamlWrite( "PlayerInfo.taml", %player1 );

The above code would produce a Taml file identical to the following:

<Player Avatar="@asset=MyModule:Alien"/>

Notice that "@asset=" has automatically been prefixed to the asset field. As discussed previously, this prefix allows the recognition of an asset reference for the asset manager.

Creating New Asset Types

Although rarely done, it's important to understand how to create new asset types. The asset system was designed to make this as easy as possible reducing the need to make changes in several places in the code. Indeed, a new asset type can be created completely in one location!

To demonstrate this, a simple example is needed. In this case a new type of "FontAsset" will be created. For this example it'll only contain two fields named "FaceName" and "FaceSize".

Here's how the declaration is defined:

#include "console/console.h"
#include "assets/assetBase.h"
 
class FontAsset : public AssetBase
{
private:
    typedef AssetBase   Parent;
     
    StringTableEntry    mFaceName;
    S32                 mFaceSize;
 
public:
    FontAsset() {}
    virtual ~FontAsset() {}
 
    static void initPersistFields();
 
    DECLARE_CONOBJECT(FontAsset);
};

As you can see, the asset is derived from the base type "AssetBase". This provides all the necessary infrastructure for the asset to interact with the asset system. None of this infrastructure needs support from the developer, simply deriving from "AssetBase" is enough.

Now to implement the type. For our example, the only thing that needs implementing is the initialization for persisting our two fields like so:

IMPLEMENT_CONOBJECT(FontAsset);
 
void FontAsset::initPersistFields()
{
    // Call parent.
    Parent::initPersistFields();
 
    // Implement the font fields.
    addField( "FaceName", TypeString, Offset(mFaceName, FontAsset), "The font face name." );
    addField( "FaceSize", TypeS32, Offset(mFaceSize, FontAsset), "The font face size in points." );
}

With that implemented we can now perform the following example in TorqueScript:

%font = new FontAsset();
%font.FaceName = "Arial";
%font.FaceSize = 48;

Now let's create an asset definition file for this type. We could do this in a number of ways, the first is to assign an asset name then persist it using Taml or simply hand-craft the asset definition file. Either way, here's how it would look:

<FontAsset
    AssetName="BigFont"
    FaceName="Arial"
    FaceSize="48"
/>

If the above asset definition file was discovered by the asset system then it would be available as a declared asset under the module it was found under. Assuming it was found under a module of "MyModule" then its asset Id would obviously be "MyModule:BigFont" and could be assigned to types who have fields that can accept such an asset type.

So that's the next step; the examples above show you how to create an asset type containing arbitrary information and have it accepted by the asset system. This amounted to little more than deriving the type from the base type "AssetBase". Now it becomes important to assign the type!

The good news is that the asset will already allow the following code:

void main( void )
{
    AssetPtr<FontAsset> font = "MyModule:BigFont";
}

That's good news indeed. In-fact, the only missing piece is how to create a field that exposes this asset pointer type of "AssetPtr". Here's the code we'd like to write:

class Page: public SimObject
{
private:
    typedef SimObject   Parent;
 
    AssetPtr<FontAsset> mFont;
 
...
...
};
 
void Page::initPersistFields()
{
    // Call parent.
    Parent::initPersistFields();
 
    // Expose our avatar image asset.
    addField( "Font", ??????, Offset(mFont, Page), "The font to use on the page." );
}

In the example above, the "Page" type wants to expose a field named "Font" for the "AssetPtr" type however, as indicated by the "??????", there isn't any console type available for the field. That's the missing piece and fortunately, it's a simple feature to implement.

Here's an example of the declaration for such as type:

#include "console/consoleBaseType.h"
 
DefineConsoleType( TypeFontAssetPtr )

**INFO! This console field type can be implement alongside the asset type code if required so that it stays close to the code it relates to. **

Here's an example of the implementation for such a type:

#include "assets/assetFieldTypes.h"
 
// Implement type.
ConsoleType( fontAssetPtr, TypeFontAssetPtr, sizeof(AssetPtr<FontAsset>), ASSET_ID_FIELD_PREFIX )
 
// Implement getter.
ConsoleGetType( TypeFontAssetPtr )
{
    // Fetch asset Id.
    return (*((AssetPtr<FontAsset>*)dptr)).getAssetId();
}
 
// Implement setter.
ConsoleSetType( TypeFontAssetPtr )
{
    // Was a single argument specified?
    if( argc == 1 )
    {
        // Yes, so fetch field value.
        const char* pFieldValue = argv[0];
 
        // Fetch asset pointer.
        AssetPtr<FontAsset>* pAssetPtr = dynamic_cast<AssetPtr<FontAsset>*>((AssetPtrBase*)(dptr));
 
        // Is the asset pointer the correct type?
        if ( pAssetPtr == NULL )
        {
            // No, so fail.
            Con::warnf( "(TypeFontAssetPtr) - Failed to set asset Id '%d'.", pFieldValue );
            return;
        }
 
        // Set asset.
        pAssetPtr->setAssetId( pFieldValue );
        return;
   }
    // Warn.
    Con::warnf( "(TypeFontAssetPtr) - Cannot set multiple args to a single asset." );
}

Whilst this code may look complex, it's actually performing two simple operations: get and set.

The getter operation simply casts the "void*" coming from Torques field system into the concrete type of "AssetPtr*" i.e. a pointer to our smart asset pointer. It then gets the asset Id from it.

The setter operation is the opposite in that it accepts an assigned asset Id and safely casts the "void*" coming from Torques field system into out smart asset pointer. It then assigns the specified asset Id. The setter is slightly more complex simply because it is ensuring that a warning is emitted if more than a single argument is passed.

With the above code in place, we've defined a console field type of "TypeFontAssetPtr" which we can now used in fields. Here's it being used in our "Page" example from above:

class Page: public SimObject
{
private:
    typedef SimObject   Parent;
 
    AssetPtr<FontAsset> mFont;
 
...
...
};
 
void Page::initPersistFields()
{
    // Call parent.
    Parent::initPersistFields();
 
    // Expose our avatar image asset.
    addField( "Font", TypeFontAssetPtr, Offset(mFont, Page), "The font to use on the page." );
}

With this in place and assuming the example "MyModule:BigFont" asset is available, the following code works:

%page = new Page()
{
    Font = "MyModule:FontAsset";
};

If this type were persisted using Taml, it would appear exactly as this:

<Page Font="@asset=MyModule:FontAsset" />

Asset Customization

When developing custom assets, everything described so far should be enough for basic usage however there are various features that any well behaved asset should try to adhere to. The following sections describe how to gain access to those features.

Asset Initialization

The first feature is that if an asset has to perform work then it should do so when explicitly initialized and not in (say) its constructor. This explicit initialization pattern helps the asset system control how an asset behaves. To adhere to this principle, simply override the "AssetBase" types' "initializeAsset()" protected virtual method like so:

void FontAsset::initializeAsset( void )
{
    // Call parent.
    Parent::initializeAsset();
 
    // Perform some font initialization work here...
}

Another example is the "ImageAsset" type. When this type is initialized it performs the image calculations for cells within the image, it is not done in its constructor.

Asset Refresh

Using the "FontAsset" as an example, imagine that it has been acquired as a declared asset from disk. If the (say) FaceName was then changed on the acquired object then it should be clear that the in-memory representation of the asset is immediately out-of-sync with the on-disk asset definition file. If the asset was saved as the asset definition file then both would be in sync again; this is essentially what refreshing an asset does however there's much more to it.

Having an asset definition file out of sync with an acquired in-memory asset isn't so much of a problem until you consider that those changes are volatile i.e. they would be lost if the application finished. For an editor, this can be bad therefore to keep things in sync after such a change, simply performing the following action would keep things in sync:

// Refresh the asset.
%myAsset.refreshAsset();
 
// Alternately refresh the asset via the asset system directly (same thing).
AssetDatabase.refreshAsset( %myAsset.getAssetId() );

When "refreshAsset()" is called on an asset, it first calls the protected virtual method "onAssetRefresh()" exposed via "AssetBase". This notifies the asset that it is being refreshed. It's here that it can take any action it needs to do. In most cases there's nothing to do but it does allow you to perform any actions necessary should it be required.

Next, the asset system persists the asset object to its asset definition file at which point the disk is then in sync with the asset.

Next, the asset system updates both asset dependencies and asset loose-files for the asset

Next, the asset system updates any "AssetPtr" objects that refer to this asset essentially allowing any object using the "AssetPtr" to be notified that the asset it is using has been updated. This will be described in detail later.

Finally, the asset system notifies any assets that depend on this asset by calling "onAssetRefresh()" on each. This is another reason to implement code in the "onAssetRefresh()" i.e. if the asset depends on another asset, you can assume that might have changed if the asset is refreshed.

As you can see, this chain of events ensures that all parties i.e. the asset itself, assets that depend on the asset and users of the "AssetPtr" type that refers to the asset to know about the asset being refreshed. During the asset refresh, anything could have changed about the asset so it's wise to perform any validation of the existing data or just recalculate it. This isn't something that should be happening during a game (at least not often if it can be avoided) but provides a solid notification platform for an editor.

INFO! Whilst it may seem that the "assetRefresh()" is a one-way-action i.e. it takes what is in memory and persists it to disk, that isn't the only use for it. For instance, if an audio asset was using an audio file, the audio asset could ensure that it reloaded the audio file during its "onAssetRefresh()" callback. The same goes for an image asset reloaded its image. This is why the "onAssetRefresh()" is called first prior to any other actions being taken, so that that asset itself is ready prior to performing the subsequent notifications and updates.

INFO! It may seem odd that the term "refresh" was used as part of the functionality is to update the on-disk asset definition file from the in-memory asset and not the other way around. This is done because the asset system provides no direct functionality to modify assets via the asset definition file on-disk, only directly through the asset object instances which is far more convenient.

INFO! If the asset is a private asset then only its "onAssetRefresh()" is called followed by notifying any "AssetPtr" objects that use it. No dependencies or loose-files are process because a private asset can have neither.

So now that we know what the asset refresh does, how should it be used. Well calling it explicitly should be avoided simply because it's something that the developer has to continually remember to do and they are only human therefore mistakes are made. Omitting a call to refresh the asset can be a hard thing to track down on more complex code bases.

This is why, a well behaved asset should in-fact call this itself! Indeed, the in-engine asset types do this. For instance, if you modify the (say) "ImageFile" on an ImageAsset type, it will immediately call asset refresh to ensure it is in-sync. In other words, when a field is changed, it should automatically call asset refresh. This may seem like overkill however, it is expected that assets are, for the most part, static during a game therefore there is no performance impact in authoring assets like this. The greatest advantage is within an editor which does not need to ensure that the asset is synchronized. An opposite point of view might be that this commits the change to disk so performing an "undo" action is difficult however this is where using an "AssetSnapshot" (described previously) comes into play taking a snapshot of the asset prior to it being modified and potentially restoring it later if the subsequent changes are not required. In short, all changes to an in-memory asset should take instant effect including modifying the asset definition file on-disk.

A final note in the performance aspect of performing an asset refresh when an asset field changes is what happens when an asset is loaded from its asset definition file. Does this mean that as TAML sets each field that an asset refresh occurs? Technically yes, the call happens however the asset system is smart enough to know that the asset is not yet added to the asset system i.e. it has not been initialized and so therefore performs no action. In other words, asset refreshes on assets that are not yet initialized and added to the asset system perform no action.

As discussed above, when an asset is refreshed, any "AssetPtr" objects that currently refer to the asset being refreshed are notified so that the user of the asset pointer knows that the asset has changed but how does this work? This starts by the user of the asset pointer requesting to be notified if the asset is refreshed like so:

class Page : private AssetPtrCallback
{
private:
    AssetPtr<FontAsset> mFont;
    ...
    ...
}
 
Page::Page()
{
    mFont.registerRefreshNotify( this );
}
 
Page::~Page()
{
    // Technically we don't need to do this as the "mFont" will be destroyed anyway.
    mFont.unregisterRefreshNotify( this );
}
 
void Page::onAssetRefreshed( AssetPtrBase* pAssetPtrBase )
{
    // Update our page as the font asset has potentially changed...
}

As can be seen, the "AssetPtr" type exposes two methods of "registerRefreshNotify()" and "unregisterRefreshNotify()". These turn on and off respectively the notification if an assigned asset is refreshed. To be clear, no notification will be raised if the asset pointer is clear i.e. it has no asset assigned and even then, a notification will only be raised if its the assigned asset that has been refreshed.

When a notification occurs, a pointer to "AssetPtrBase*" is passed. This is the underlying base object of the "AssetPtr" being notified.

In the example above, we might update our page because the font asset that was assigned has potentially changed.

Finally, as a real example of asset notification in action, the engine type "AnimationController" works with an "AssetPtr". It registers for refresh notifications on this asset pointer and when a notification comes in, it simply starts playing the asset if it was already playing because it cannot guarantee that the frame position it was on is now still around. If you consider that the "AnimationAsset" started an asset refresh cycle when it changed and that any animation controllers now immediately start playing with the change then you can see how powerful this notification system is. If the "AnimationAsset" had not performed an immediate refresh then the animation controller would not know of this change and attempt to animate using potentially invalid frames. Whilst the animation controller is resilient to this kind of error, with the notification in place, it now doesn't need to be.

Asset File Paths

When an asset refers to a file (see "Asset Loose File Fields" below) it typically uses a file-path that is expanded when reading and collapsed when writing. This expand/collapse is based upon a path relative to the asset definition file itself i.e. a file that resides alongside the asset definition file has no path specification at all. This creates a file-path in a persisted form that typically looks like this:

<AudioAsset
    AssetName="Ring"
    AudioFile="OldPhoneRing.wav"
/>

However, it's temping to use an expando form like this:

<AudioAsset
    AssetName="Ring"
    AudioFile="^MyModule/assets/audio/OldPhoneRing.wav"
/>

The problem with this form is that if you were to move the asset to another module (or location) then you'd need to update the path, particularly the "MyModule" expando. Note also that it's possible to rename a module-Id and in that case, any assets using this form would be broken. Coupling the asset definitions with a specific location and worse, a specific module-Id is considered bad practice!

The asset system provides its own expand-path and collapse-path functions that allow you to convert the file-path into something that does not refer to the module-Id but rather converts to an asset-definition-relative path.

Because these methods are designed to be used by asset-type developers, they are located on the "AssetBase" type:

• expandAssetFilePath()
• collapseAssetFilePath()

When you collapsed a file-path using these methods, they calculate the relative location of the file being referred to, to the location of the asset definition. In other words, you get a path relative to the asset itself.

INFO! You should not use "./" as a prefix even though it does indicate the current directory.

There is however a subtle problem you can run into when using these specialized expand and collapse calls: when to call them. As per the example above, when it is read as an asset, the field "AudioFile" contains the string "OldPhoneRing.wav" however this cannot be used by any of Torques' file-IO calls as its actual location is unknown so we need to expand it using the AssetBase "expandAssetFilePath()" to return a full file-path that can be used by Torques' file-IO calls. Another caution is that this Torque field is exposed publicly so can be accessed by the scripts therefore leaving it in an unexpanded form would be problematic.

This means that we must not only expand the file-path but also replace it with its expanded form. This isn't a problem but it introduces another problem which is that we must ensure that the path is then collapsed prior to it being persisted using Taml. Again, this isn't a problem as will be shown shortly however, when and where it is done in an asset is also very important.

When an asset is "live" i.e. its available for inspection and modification to the scripts, file-paths should be in a fully expanded form for users to use.

So when should we expand the file-path? That's simple, use the "initializeAsset()" method described previously. This method is called after an asset has been fully loaded (created and all fields have been set) and has been added to the asset system (known as being "owned" by the asset system) therefore you should expand file-paths here like so:

AudioAsset:initializeAsset( void )
{
    // Call parent.
    Parent::initializeAsset();
 
    mAudioFile = expandAssetFilePath( mAudioFile );
}

So that expands the path and replaces itself with a form that can be used publicly but what about when the asset is persisted (say) during an asset refresh? Doing so is easy but does involve using Taml callbacks like so:

AudioAsset:onTamlPreWrite( void )
{
    // Call parent.
    Parent::onTamlPreWrite();
 
    mAudioFile = collapseAssetFilePath( mAudioFile );
}
 
AudioAsset:onTamlPostWrite( void )
{
    // Call parent.
    Parent::onTamlPostWrite();
 
    mAudioFile = expandAssetFilePath( mAudioFile );
}

In this example, the audio asset collapses the file-path prior to it being written using Taml and expands it back afterwards. Altogether this may seem like a lot of work but in the end, it's only a one-off few lines of code and provides a huge usability benefit to the asset.

INFO! Private assets or assets not added to the asset system cannot expand or collapse file-paths in this way and the expand and collapse methods know this. If it's not appropriate then an appropriate file-path will be returned.

Specialized Console Asset Field Types

Asset Loose-File Fields

When developing asset types, it's common for these assets to require access to additional files from the disk. If these files are selectable by the end-user then they would obviously be exposed via a Torque field. If this was done then you could simply use a string type like so:

void CustomAsset::initPersistFields()
{
    // Call parent.
    Parent::initPersistFields();
 
    // Implement the fields.
    addField( "MyFile", TypeString, Offset(mMyFile, CustomAsset), "My File" );
}

Whilst this would work, it doesn't support a feature that the asset system provides. When the asset system first encounters an asset definition file it performs a scan on it and retrieves important information from it without actually creating the asset. One piece of information that is retrieved is a list of files that the asset requires. These files are known as loose files.. They are loose in the sense that they are not part of the asset definition file itself i.e. they contain separate data or state that the asset depends on.

Whilst this information may not seem important, it is for certain operations to work completely. For instance, if the asset is deleted using the asset system "deleteAsset()", that method provides the option to delete any associated loose files. If the asset system is not aware of any loose files then it obviously cannot delete them. Additionally, for the future, being able to relocate and pack only the assets used by a game project requires that any additional loose files for assets also be relocated as moving the asset definition file alone is obviously not enough.

The asset system identifies a loose file when it encounters the field prefix of "@assetFile=" in the same way that it identifies an asset reference (described previously) by the prefix "@asset=". With this prefix in place, the asset system associates the specified loose file with the asset for the operations described.

This leads back to the example above. Using a simple type of string doesn't emit the "@assetFile=" prefix as it's a raw string. To get the prefix in place is simple and requires the dedicated console type of "TypeAssetLooseFilePath" to be used instead as per this example:

class CustomAsset : public AssetBase
{
private:
    typedef AssetBase Parent;
 
    StringTableEntry mMyFile;
...
...
};
 
void CustomAsset::initPersistFields()
{
    // Call parent.
    Parent::initPersistFields();
 
    // Implement the fields.
    addField( "MyFile", TypeAssetLooseFilePath, Offset(mMyFile, CustomAsset), "My File" );
}

As is shown, a simple change of the field type is all you need.

Asset Id Fields

In previous examples, when an asset field was used, the types were exposing the smart asset pointer of AssetPtr and as powerful as this is, it's not always appropriate. A typical reason to not use the AssetPtr for the field relates to what the asset pointer does when it is assigned an asset Id.

If you expose an asset pointer as a field, you should be aware that when the object is read from disk and the asset pointer is assigned the appropriate assign Id and the asset is immediately acquired. This isn't always appropriate, especially in the case of Torque GUI types. These types are designed to be in a wake or sleep state. In a sleep state they should use as few resources as possible and upon them waking, they should acquired whatever resources they need. Whilst this isn't a hard rule, it's good practice and the original intent of the GUI system.

To support this, it's clear the the GUI needs to expose the ability to assign an asset Id (or in some cases multiple asset Ids) but when that is assigned, it should not be acquired immediately. This could easily be supported by doing what we originally did when we wanted a loose asset file above i.e. we could just use a string type on the field (TypeString).

Just like before however we want to ensure that because this is an asset reference we get the asset prefix of "@asset=" so that it is recognized as an asset reference by the asset system. Again, just like before we simply need to use a specialized type, in this case "TypeAssetId" like so:

class CustomGui : public GuiControl
{
private:
    typedef GuiControl      Parent;
 
    StringTableEntry        mAudioId;
    AssetPtr<AudioAsset>  mAudio;
...
...
    virtual void onWake();
    virtual void onSleep();
 };
 
void CustomGui::initPersistFields()
{
    // Call parent.
    Parent::initPersistFields();
 
    // Implement the fields.
    addField( "Audio", TypeAssetId, Offset(mAudioId, CustomGui ), "Audio to play" );
}
 
void CustomGui::onWake()
{
    mAudio = mAudioId;
}
 
void CustomGui::onSleep()
{
    mAudio.clear();
}

As can be seen, the field exposes a simple string which can be assigned an asset Id (the asset Id can also be retrieved) however assigning an asset Id does not immediately cause the asset to be acquired. Only when the GUI control wakes is the asset acquire by assigning the asset Id to the asset pointer. Likewise, when the GUI control sleeps, the asset pointer is cleared. This does not affect the persisting of the asset Id as that is stored in the simple string. Finally and more importantly, when the string is persisted, it persists with the correct prefix as shown here:

<CustomGui Audio="@asset=MyModule:Ping" />

Summary

To the end-user, the asset system provides a wealth of benefits not least of which is the ability to simply assign asset Ids and have them "just work". When used correctly by an editor, it can provide a wide range of useful functionality for generating, modifying, querying and deleting of assets and related files. The additional features that allow assets to work seamlessly alongside modules and TAML provide a solid foundation for customized organization for the storage of assets.

The asset system provides a powerful yet simple means for engine developers to utilize assets by allowing them to take direct control of asset references or use the smart asset pointer system. Many administration and debugging features are also available that allow the developer access to information to enable them to trace problems in more complex systems. Finally, higher-order functionality can easily be created on top of the existing asset system to allow more sophisticated asset management for asset manipulation and querying.