Caching and External Sources – ShiVa Engine

Caching and External Sources

2012-11-20

In the past, we have written a lot about Remote XML manipulation and Distant Environments. Using those APIs, you can easily load save games, highscores and configuration data. However, those APIs are limited in a way that you can only load string data, not objects, videos, materials, scripts and so on. This tutorial will show you how to load, cache and use distant resources.

Streaming Web Videos

You can stream videos directly from the internet and show them to your users. Although the name would certainly suggest so, application.playOverlayExternalMovie ( ) – a function used primarily to load films that have been added to the “Additional Files” column in the UAT – is not the correct way to do that. Use the cache API instead.
Building a Big Buck Bunny Web Video Player in ShiVa is really easy:

--note the extension(s)
cache.addFile ( "BigBuckBunnyVideo.ogg", "http://clips.vorwaerts-gmbh.de/big_buck_bunny.ogv" )

Streaming videos must be a Theora Video / Vorbis Audio stream, commonly packaged in an *.ogg or *.ogv container.
You now have to decide whether you want to play the film on a HUD element or on a material texture.

--HUD code, note that extensions are absent
hud.setMovieClip ( hToYourMovieComponent, "BigBuckBunnyVideo" )

The movie will start to play immediately after that if you do not pause the stream explicitly. You can either control the Video using HUD actions…

… or do it completely in code:

local hUser = this.getUser ( )
local hPlayStreamMovie = hud.newAction ( hUser, "PlayStreamMovie" )
hud.beginActionCommand ( hPlayStreamMovie, hud.kCommandTypePlayMovie )
--assuming your Movie Player HUD is named "MoviePlayer" and your movie component "Movie"
hud.pushActionCommandArgument ( hPlayStreamMovie, hud.getComponent ( hUser, "MoviePlayer.Movie" ) )
hud.endActionCommand ( hPlayStreamMovie )
hud.callAction ( hUser, "PlayStreamMovie" )

You should also remove the video from cache when the program finishes, using the onApplicationWillQuit ( ) handler:

--delete downloaded movie from cache
cache.removeFile ( "BigBuckBunnyVideo.ogg" )

Movie Playback on a material texture looks similar:

--load...
shape.overrideMeshSubsetMaterialEffectMap0  ( hYourObject, nMaterialSubsetIndex, "BigBuckBunnyVideo", shape.kMapTypeMovie )
-- ...and playback
shape.playMeshSubsetMaterialEffectMap0Movie ( hYourObject, nMaterialSubsetIndex )

For a working example, please check out the MoviePlayer Sample that comes with every ShiVa installation.
It is also possible to stream videos directly without saving them to your disk. Instead of cache.addFile, simply use cache.addStreamFile:

--stream it!
cache.addStreamFile ( "BigBuckBunnyVideo.ogg", "http://clips.vorwaerts-gmbh.de/big_buck_bunny.ogv" )

The playback code remains the same, but you do not need to cache.removeFile the movie after you are done with it.
Note to all Web Player games creators: The Chrome browser will try to read incoming .ogg files on its own and the download of the video will stop. To avoid this, you just have to rename the remote .ogg stream to something else, like .dat, .blu, whatever you want.

Loading Texture Files

Do you want your gamer community to be able to use custom skins? Do you have a billboard in your scene you want to display ads on, and change those images over time? Thankfully, you do not have to replace the whole STK to make those changes. Textures can be loaded directly from the Internet, like so:

if ( cache.getFileStatus  ( "pictureWeb.tga" ) < 0 ) then
    cache.addFile ( "pictureWeb.tga", "http://www.stonetrip.com/developer/dump/picture.tga" )
end

This works well for JPG and TGA images. DDS produces mixed results and anything else like PNG or BMP will throw you a “resource not referenced” error message. DDS, PVR, ETC are supported on GPUs that can handle those compressed file formats. To stay truly cross-platform, we recommend going with JPG and TGA.
You can now display those textures on your objects. This is very similar to the way we used the ogg video.

--note: no extension again
if ( cache.getFileStatus  ( "pictureWeb.tga" ) == 1 ) then
    shape.overrideMeshSubsetMaterialEffectMap0  ( hYourObject, nMaterialSubsetIndex, "pictureWeb", shape.kMapTypeTexture )
    --now leave this loop/state either by entering another state...
    --this.null()
    --...or by setting a control variable:
    --this.bTexturesPreloaded(true)
end

Make sure that the whole file is loaded (query cache.getFileStatus until it equals 1) before you use the new image, otherwise you will not see any changes in your scene and the function will silently fail.
This function should run in a loop, preferably in its own state. As soon as the texture is cached, cache.getFileStatus will always return 1, so make sure you exit the loop after the “== 1” branch has run exactly once. If you use states, switch to another state, and if you do everything in onEnterFrame, use a member control variable and encapsulate the whole preloading code into one if-branch this.bTexturesPreloaded() == false.

Caching and running remote Game STKs

TheHunt sample game that comes with ShiVa is split into 2 games. One is called st_loading and contains the airplane scene as well as the caching code for the hunt game STK which lies remotely on the StoneTrip servers, the other one is theHunt STK itself. Essentially, running theHunt on the web, you are executing a game within a game.
Interestingly, you do not need to use the cache.add API to load remote game STKs. Loading and executing is done with the system API, system.install in particular. 

local sFileToLoadURI = "http://....."
local nProgress = system.getInstallationStatus ( sFileToLoadURI )
if ( nProgress < 0 )
then
       system.install ( sFileToLoadURI )
-- [...]

Still, it can be beneficial to download (cache) the file first, save it to disk, and then run it. We will approach this issue in the last chapter of this tutorial, “Saving Cached Resources”.
If you rely on system.install, you can query the current download progress can using system.getInstallationStatus. As soon as it returns 1, the pack is ready. Finally, launch the new game from the STK using system.launch. Note how the pack string is a generic variable that could either be a local or a remote resource.

-- [...]
elseif ( nProgress == 1 )
then
      system.launch ( sFileToLoadURI, "" )
end

You should note that the Install System is not available when running your game inside the editor, which makes it impossible to test it without a proper export first.
For more a more in-depth code example, you should go through the st_loading sample inside ShiVa as well as look at the corresponding doc pages. Some time ago, we have also written a specific tutorial about launching STK games from within games.

Loading remote Resource STKs

Not all resources can be loaded as separate files. While it is possible to load single images, models for instance cannot be loaded as individual files. To load resources other than textures, movies, XML end environment data, you have to pack them int STK archives and cache them into shiva.
Resource STKs should not contain games. Create them by right-clicking on your desired item and choose “Add to Export”. After you have added everything you want to group into one pack, click “Export” in that window and choose “STK” as export option.

Caching resource STKs is similar to caching textures or videos. For simplicity reasons, I have unloaded the caching function into its own userAI state:

--------------------------------------------------------------------------------
function Main.loadResources_onLoop ( )
--------------------------------------------------------------------------------
    local nProgress = cache.getFileStatus  ( "ResourcePack.stk" )
    log.message ( "STK loading onLoop progress: "..nProgress )
    if ( nProgress < 0) then
        cache.addFile ( "ResourcePack.stk", "http://www.stonetrip.com/developer/dump/ResourcePack.stk" ) --remote file
        -- or if you want, a local file:
        --cache.addFile ( "ResourcePack.stk", "file://C:/path/to/your/ResourcePack.stk" )
    end
    if (nProgress == 1) then
        --load a HUD that was in the resource pack
        hud.newTemplateInstance ( this.getUser ( ), "ResourcePack/iPhoneJoypadEmulation", "JoyPad" )
        this.null ( ) -- go to another state and leave the loop
    end
--------------------------------------------------------------------------------
end
--------------------------------------------------------------------------------

Every resource in the pack can now be accessed through its PackName/ResourceName handle.

Saving Cached Resources

External STKs can grow fairly big. Though an option exists not to clear the cache every time the engine stops (see image below), it is more elegant to save your cached files to your local storage device.

You can permanently save a cached file to disk using cache.sendFile.

if ( cache.getFileStatus  ( "package.stk" ) == 1 )
then
    cache.sendFile ( "package.stk", "file://"..application.getPackDirectory ( ).."/package.stk" )
end

Check for the local STK first before downloading a new one.

local tFiletable = table.newInstance ( )
system.findFiles ( tFiletable, "file://"..application.getPackDirectory ( ).."/", "*.stk" )
--or wherever you decided to save your cached STKs
if table.contains ( tFiletable, "package.stk" ) then
    this.sFileToLoadURI ( "file://"..application.getPackDirectory ( ).."/package.stk" )
else
    this.sFileToLoadURI ( "http://www.stonetrip.com/content/techdemo/package.stk" )
end

The rest of the caching code stays the same, the only difference being the fact that you cache from a local disk rather than a remote host. this method works with both system.install ( local or remote URI ) and cache.add ( local or remote URI ).
However, you should check if there is a newer version of the STK on the internet. cache.getFileStatus offers this functionality. When you use cache.getFileStatus and set application.setOption ( application.kOptionNetworkStreams, x ) with x > 0 (default x == 1), the engine has to check on the server if a newer version of the file is available.