Since its inception, the rendering technology behind Voxel Quest has undergone 3 major revisions.  The first was isometric chunk rendering (every voxel computed at a pixel scale), and you can find an explanation of how that worked here.  The next revision added perspective, and meshed those voxels with a point cloud.  The current and final revision only renders visible surface points, and it procedurally generates these in real time. Arguably the isometric rendering looked the best of all the methods, but I also spent over a year refining it.  The current revision is only 3-4 months in so give it time to mature. :)

The fact that VQ has undergone three tech revisions over two years probably seems a bit ridiculous, and maybe it is.  Something like this would normally kill a game.  That said, the point here is not just to make a game (plenty of people are doing that already), but to make a unique engine, and that could not happen in a vacuum. All I know is that I am finally happy with where the engine is at in terms of performance and flexibility, and I couldn't have gotten here without knowing everything I've learned since the start.

So the most common question I get, of course, is how does this stuff work? It is actually simpler than it might seem.

At a high level it is based on ray tracing, specifically a method known as Signed Distance Field ray marching (SDF ray marching).  But that is only part of what is going on.

Vanilla ray marching simply steps along a ray at fixed intervals until it hits something.  This is costly with a small step size, and erroneous with a large step size.  SDF ray marching computes the distance from the nearest objects, and moves along the ray no more than the smallest distance from any of these objects.

(Image credit: http://www.polygonpi.com/)

This is very fast when the ray actually hits the object, but considerably slower if the ray "just misses" the object as shown above.  The interesting thing about SDF marching is that it works hand in hand with boolean geometry or solid modeling.  Inigo Quilez has a great primer on distance functions and operations located here.  You can also find many great live examples of SDF ray marching at shadertoy.com.

The problem with naive SDF is that every ray at every pixel is considering every object in the scene to be rendered.  This becomes insanely expensive if you have many objects in the scene.  For example, imagine I have 1920x1080 pixels to render, which amounts to about 2 million rays to cast.  Each ray is calculating the distance from 1000 objects, 60 times per second.  Lets imagine each ray travels 100 steps on average, and requires 100 instructions to make its calculations.  You are looking at 1.2 quadrillion instructions per second, if I did my math right.

The trick I use is aggressively reducing the number of objects each ray considers, using very straightforward, naive cell hashing.

Here we have a ray (purple) intersecting 3 objects.  The ray gradually marches through each cell, from position B3 to N13.  If the cell is empty, it does nothing.  If any objects are within the cell, it adds them to the list of objects which it should compute.  It does not store duplicate object references.  Using math, it determines exactly whether or not it hit a given object - but this is not a final result.  If an object has a bumpy surface, or has holes in its surface, its possible a ray can hit the object but miss it when it renders the surface in detail.  All of these objects are gathered up and then a traditional SDF pass is used to determine the details and exact hits.  For some further clarity, cells can contain multiple references.  Cell H8 contains references to both the green rectangle and red circle, but cell M11 only contains 1 reference (the orange pentagon).  The way in which you implement the cell data is up to you, but I use OpenGL's samplerBuffer (which is like a texture but functions more like an array).

In order to step through cells efficiently, I recommend using a supercover line algorithm, such that each cell is only considered once as the ray travels through it (image credit: http://lifc.univ-fcomte.fr/home/~ededu/projects/bresenham/).

You will want to toy with cell sizes to hit the sweet spot: big enough so that one cell usually fits an entire object, but small enough such that there are not too many objects within a given cell.

Because VQ uses only one type of uber geometry (rounded boxes with super-ellipsoid-like corners), it can compute exact hits for every single type of primitive only using one set of instructions (from cones to cylinders to spheres to rounded boxes to you name it), eliminating the need for branching in that area.  So it narrows it down to geometry that the ray hits exactly.

But it goes further than that.  How many objects is a ray likely to miss?  As it turns out, you can usually get away with only considering the first 4 to 8 objects and you will rarely get errors.  In the illustration above I have just shown 3 objects, but imagine if that box is filled with hundreds of objects.  Now, for each ray, I only need to consider the first couple of objects that get hit instead of hundreds of objects for every ray.  As I have shown in the past, you can render millions of objects using this method and there is not really any slow down - it tends to run at a pretty constant speed.  You can optionally improve performance further through use of the flyweight pattern or templates.  If I am using an object that will be repeated many times (for example, a hallway) - I need only store the specifications for this object once, and then instance of it is just a reference with a bit of data like position and orientation.

Not every object is traditional geometry though.  The terrain is composed of smooth chunks of earth, which is produced by sampling a volume texture with bilinear filtering.  As it turns out, vanilla SDF won't work that great trying to compute distances based on textures.  The trick I have found is to allow the SDF algorithm to overstep into a solid area, and then step backwards along the ray (correcting itself), back and forth a few times until it gets close to the surface.  There are probably better ways of analytically computing distance from a bilinear filtered point, but this works for now.

This is not a full update, just a recent peek at some improvements and performance.  In my next full update I will release a video.  If you are curious to know more about the planned development timeline, see the bottom of this post.

A constant question I get is something along the lines of "what is performance like?"  The truthful answer was - I did not know! It looked like it was getting at least 60 FPS.  Accurately measuring FPS can be tricky in OpenGL without profiling software, and I have not yet gotten to the point where I am using that (I don't want to get sucked into tweaking the heck out of things to profile this early on).  If you think it is as simple as doing a timer between frames, that is not true.  See here for further details on why getting the FPS can be tricky.

In order to perform these frame tests, I did it the old fashioned way - render a large amount of frames without any sort of timer constraints, and measure the amount of time it takes (in this case, I am measuring how long it takes to render 10,000 frames).

Some VERY important notes :D

1. These performance figures are still largely unoptimized. I suspect in the long run I can get at least 2x performance.
   
2. These performance figures are largely IRRELEVANT for isometric mode (which the game will still support) - in isometric mode, it need only render the scene once every so often as the camera moves larger distances, or when an environmental object is changed/destroyed.  You could quite easily render an isometric scene at 1080p without much lag.
   
3. I have made a conscious decision to favor scene complexity over resolution (in fact, I think it is a bit ridiculous to play a game at 4k resolution these days when it only magnifies the inefficiencies in texture resolution and polygon count).  I am intentionally trying to make the game look good at old resolutions similar to Mode 13h (or equivalent for widescreen).  I loved the look of Doom, Hexen/Heretic, Ultima Underworld, and so forth.
   
4. Although these shots are untextured, the texturing is not a very expensive operation (I'll explain more about this trick soon!).
   
5. The date at which I targeted "going gold" is still 3 years away (although preliminary releases will be much sooner than that.  During this time hardware will become faster and cheaper.  The GTX 980 which I am rendering on now should cost $100-$200 if not less by then and about 50 percent of the users on Steam should have some as fast or faster than that, extrapolating from current hardware surveys.  I am targeting to get it running well on a GTX 960 or equivalent, which should already be feasible but I need to test.
6. The interesting thing is that the object count makes little difference.  In some cases, rendering 10,000 objects is faster than rendering 1,000 objects (due to occlusion).  There is no limit on how many objects you can render - you can render a million objects with little change in performance.  By the way these objects (which are an extension of superellipsoids) can render any classic piece of geometry like cones, spheres, diamonds, triangular prisms or ramps, and more - all using one instruction set.  Below this I have included a couple further updates from the Voxel Quest twitter account regarding improvements and further tests with this rendering.
7. (Edit) Also note that the point here is not so much to compete pixel for pixel with a polygon engine.  I can do things that could simply never be done with polygon engines, and that realistically requires a sacrifice to both resolution and performance.  These are SOLID objects that can be updated with completely different geometry, volumetric textures, and whatever in realtime.  While rendering one object might be slower in my engine, you must account for the fact that I can render millions of objects faster than a polygon engine could.  In other words, my engine is resolution-dependent, not polygon-dependent.

The new rendering methods now support any amount of overlapping geometry (previously, each piece of geometry was constrained to one instance within its own cell.  There is still more work to do before I do a full writeup, as the process is still changing slightly and there are more steps to finish.  I am also doing a huge amount of work just to manage dynamically placing objects in the scene, saving your creations, and a lot of stuff along those lines, which I will also talk about in the next update.

I'm not going to type a ton about the development status of the game - I've already done that in the forums if you are curious (check recent threads).  But let me just provide a brief summary of recent issues:

• I am finishing the tech side of things this month, and all focus will be on gameplay once that is done (FOR REAL).
• I am shipping a minimally viable prototype game ASAP that will be a slight deviation from the long term goals of VQ.  I am still planning to achieve my long term goals, but that will be down the road.  The first goal is to get something fun into the hands of backers and ensure that I don't get stuck on hard problems in the short run.
• I am in the middle of raising funds to complete the game.  I am currently considering money from acquaintances I know and angels. I have enough funds in place to carry me for a good while longer, it is mostly a matter of choosing who to take money from at this point (and fingers crossed that at least one or more of these deals actually goes through, as getting the signed check is not always the same as getting the "ok").  My primary goal is to not compromise my vision or freedom in this process, which seems viable based on who has offered so far.
• There have been many reasons for the delays thus far, largely involving tech revisions, and being adamant about shipping a "fun" game.  I have intentionally not given a hard release date because I realistically don't know when that will be, but I am striving to both get a tech demo out in the short term, get the source ready to distribute, and to get a minimally viable and fun game out.

Edit: I am soon going to a writeup on the rendering. I wrote this post in haste so have not gotten a chance to write much yet but in as few words as possible the new rendering is much faster because it only calculates visible surfaces instead of volumes.  It is based on cell marching and distance field marching.  More info to come soon!

Since the last update:

• Implemented an entirely new ray-based rendering method which includes such benefits as...
• No more chunk loading.  All scene data appears instantly
• No more refreshing of chunks. Changes can occur in realtime, at 60+ FPS.
• Vastly Increased performance and draw distance.
• "Infinite detail" (limited only by floating point precision and algorithmic complexity).
• Ability to dynamically animate any surface or object.
• Successfully integrated with much of the old pipeline which optionally includes various screen-space effects, material rendering, and so forth.
• I also implemented complex fluid dynamics, which include such features as...
• Constant fluid volume (impossible to destroy matter (unless intended)).  Fluids maintain correct proximity and do not lose or gain water.
• Water pressure.  Fluids will balance out (i.e. fill a U-bend evenly) and simulate correct pressure and distribution.
• Free of most artifacts but still needs work.  Particularly notable is that the water does not jitter around as it does in many constant fluid simulations.

I have been fairly busy with "real life" issues, which seems to be the norm these days, but I have managed to get quite a bit done regardless.

Below is one of the earlier tests I put together - rendering terrain based on VQ's generated map.

Added in bilinear filtering to smooth out the water (sorry for all the twitter embeds, I am lazy):

And one more demo of the smooth water:

Since the last update:

• Explosions (temporary effect, needs improvement); Sound effects are also all placeholders by the way
• More intelligent chunk refreshing that reduces redundant chunk refreshes (useful for destruction among other things).
• Rendering spheres of arbitrary size in screen space with real depth, normals, clipping, etc
• Projectiles
• Camera Shake (on jump, on objects hitting ground, fades with distance from camera); it is exaggerated right now
• Zooming in and out to first person mode
• Walking around with collision, realtime smooth movement that can revert to unit space for turn-based combat and so forth.
• Physics (gravity, dampening, rigid body physics with adjustable bounciness/hardness, friction, etc).  Currently operates on axis-aligned boxes (even when boxes are rotated on the Z axis)
• All movement-related effects are now timer-based (rather than frame-based), although you can also force a certain time interval to prevent stutter/frame skipping.
• Jumping
• Lantern placement fixed (raised up inside of houses, no more lantern clusters for more efficient lighting)
• Volumetric textures optimized for lower voxel resolutions (larger shingles, bricks, and boards)
• Better character selection/deselection with dotted line selection effect; now distinguishes between the active entity and the target entity
• Better sprite rendering with no clipping or depth sorting errors
• Labeled all 1600+ items and entities (even though they are just placeholder objects)
• Added context sensitive options to the drop down menu based on which targets (if any) are selected
• Quick experiment with trees: upped the number of limbs to produce dramatically larger trees (as seen below)
• Improved, less naive chunk picking (for rendering) - now uses a flood fill to pick chunks, instead of a bounding cube.
• Many other small improvements I am probably forgetting to list :)

I am still working on many aspects of the AI but only a day or two per week right now - I am trying to not get stuck in any one area as there are many tasks left to cover.  I think this most recent set of changes opens up a lot more modding potential (in particular, for realtime games).  I think it would be pretty easy to make a platformer (like Mario N64), an artillery game (like Scorched Earth or Worms), or many other options (at least, easier than it would have been with strictly unit/turn-based movement).

I came in with the intention of doing videos weekly but as it turns out monthly makes more sense - there are some weeks where not a lot happens visually.  It also saves me time.  Nonetheless I advise you to follow the @voxelquest (or @gavanw) account on Twitter or my Youtube channel for smaller updates in between.

Also, I posted this a bit late for April Fool's, but it is worth reposting I think.  Since I added in first person view, I obviously need the obligatory hand swinging some weapon.  (In this case it is one of the weapons from Hexen):

In spite of this being a joke, I really do dig this style (as did Twitter, apparently) and I would like to perhaps have something similar ultimately.

Hello! It has been over a month since the last update.  I apologize for the delay, but as I have mentioned elsewhere there really was not much to look at as the majority of my work was doing IO in a text-based console.  I have been hard at work on the AI, and now have the logical portion of it mostly complete.  It is capable of making deductions as I have described prior in the forums and elsewhere.  This is a full "scripting" language with compiler, although for the time being it is parsed using rules from JSON which makes it somewhat unwieldy but I just wanted to get a parser working as fast as possible.  It contains many features, and is both a subset and extension of the language Prolog, albeit with different syntax and grammar.

I use quotes on "scripting" language because it is not writing AI scripts like you might see in other games.  Rather it is just specifying a knowledge base with various rules and relations.  You can then pose queries to this knowledge base in order to make deductions and determine whether or not things are true or false.  Additionally you can do more advanced proofs and solve statements for a given set of variables, finding the variables which result in a true set of facts.  If you are interested in learning more about this, I recommend learning Prolog - there is a good free book I used here for reference.

So why not just use an existing solution like Prolog? Well, first of all Prolog is relatively bloated.  My language fits within under 1000 lines of code if you ignore the JSON parsing (less minus comments/whitespace).  It is compiled to integer symbols so it avoids string manipulation and comparison.  I only need a subset of Prolog's functionality, so it would be wasteful to include the entire thing.  Moreover, Prolog is not sufficient on its own.  I must carefully modify the deduction algorithms to support things like score maximization, and thus I must have a full understanding of the code involved.  This would be much harder if I went in and tried to modify an existing Prolog implementation, and more error-prone due to the larger breadth Prolog has over my subset language.

The ultimate goal of this is to build an AI that can make logical deductions and perform surprising, emergent behavior based on things that are not explicitly scripted.  The term I have been throwing around, as have some other games, is a "Game of Thrones" simulator - something capable of weaving together a complex plot in which every NPC has their own set of motivations.

I also added in some billboarding to show sprites in the placeholder boxes, which makes it much easier to figure out what a given object is.  Also reorganized all the items and added in placeholder NPCs and monsters.  Art is by 7soul and available for purchase in most game dev asset stores.

Also, as a quick (dumb) experiment, I messed with voxel warping.  Just 2 lines of code, could have some interesting uses for spell effects and such.

See full update here.

Happy New Year!

It has been a while since the last update, I know. As I may have mentioned prior, aside from the holidays, I have been just trying to get something interesting to show, which is hard when the majority of my work the past few weeks has been on relatively boring back-end stuff.

Here is a video - sorry I was tired when I made it!

Since the last video update, we have:

- Walking around, with collision and auto-climbing/descent of stairs/hills (the power of technology!!!)
- Camera that follows the character, orients itself in your travel direction, and more
- Saving and loading of character poses
- Rendered (fully voxel-based / procedural) character has replaced the placeholder box object.
- Ability to assign materials to each character limb, or other objects
- Material editor has been migrated from the web client. For those who have not seen it, the web client looked like this. The web client had many problems, including performance (slowing down the main client), inability to effectively manage data over a websocket, requiring a web server to run (i.e. a localhost), being generally unwieldy, and more.
- Much more advanced GUI features, data binding, and data serialization. The GUI now supports multiple features to keep everything DRY (Don't Repeat Yourself), keep data and presentation separate, and more. This is important because many times it is not a 1:1 mapping of data to UI - for example the color picker has 6+ components which map to only 3 data values.
- Several more optimizations, in particular some UI optimizations that make it load faster and make it more responsive.
- Many more small fixes and improvements.

In the coming week I am going to work on inventory management, picking up items and dropping them, and rendering additional NPCs and making them walk around.

As usual feel free to ask me anything.

This week I did a pretty huge cleanup on the code (I deleted thousands of lines while retaining the same functionality and more). I also added in collision detection, a context menu (right click menu), and some functions for spawning/removing/editing entities - mostly for the purposes of quickly testing stuff but also useful for the eventual sandbox.

Making the collision work was not exactly trivial but it is a lot easier now that voxels are passed to the CPU and meshed. It looks at all the voxels within a given cell and determines (based on a given threshold) whether or not that cell is mostly solid, air, water, etc. More granular collisions can be done in screenspace, but this is necessary for offscreen or hidden collision testing. This coming week I will be working on camera movement (following an NPC or trackballing around it), in addition to registering collision and behaving accordingly (i.e. climbing stairs automatically if you collide with a step, etc). I will also try to work on managing movement and collision when other NPCs are present.

Also, if you enjoyed the classic Master of Magic, you might want to check out my friend Aaron's game Worlds of Magic. He is running for Indie Game of the Year (in the top 100 now), so throw him a vote if you are feeling generous:

http://www.indiedb.com/games/worlds-of-magic

Also, David Sahlin, who I have known for almost a decade (since I worked on Genesis) sent me his ideas for a mod for VQ, drawing inspiration from the likes of Pokemon, Spore, etc - I think it is a pretty interesting idea and I would love to see someone make a mod like this:

https://prezi.com/8dzstjlitnoi/vq-mod-mythical-creatures/