Hi everyone.
Phew!
It has been about 6 months since my last major update. In that time I got married and had my first child (it's a girl!). But aside from real life I've been neck-deep in voxels.
So what has changed? A lot - a huge amount. There are plenty of visual changes but also tons of things under the hood - not so much (premature) optimization, but really a restructuring of a good portion of the engine to make it easier to work with.
One small, but major change is the support for voxel super-sampling. Everything you are viewing has been rendered to bitmaps that are 1/4 to 1/8th as large as the bitmaps used for chunks in the previous video. This reduction in buffers for the chunks has led to a massive decrease in the memory needed to store chunks - since chunks are stored volumetrically, every reduction of 2 decreases memory by a factor of about 8 (2*2*2). So a reduction of 4 leads to 4*4*4 or 64 times less memory for chunks.
Phew!
It has been about 6 months since my last major update. In that time I got married and had my first child (it's a girl!). But aside from real life I've been neck-deep in voxels.
So what has changed? A lot - a huge amount. There are plenty of visual changes but also tons of things under the hood - not so much (premature) optimization, but really a restructuring of a good portion of the engine to make it easier to work with.
One small, but major change is the support for voxel super-sampling. Everything you are viewing has been rendered to bitmaps that are 1/4 to 1/8th as large as the bitmaps used for chunks in the previous video. This reduction in buffers for the chunks has led to a massive decrease in the memory needed to store chunks - since chunks are stored volumetrically, every reduction of 2 decreases memory by a factor of about 8 (2*2*2). So a reduction of 4 leads to 4*4*4 or 64 times less memory for chunks.
In the image above, I am using 1/8 chunk buffer resolution (8*8*8 or 512 voxels per pixel!). You would think such super sampling to be excessive, but it makes all the difference visually and ensures that no voxels get dropped (as you may remember in the last video, lowering the voxel resolution led to holes in the plaster and defects in other thin areas). It is also using 1/2 screen buffer resolution (1/2 1080p, stretched to 1080p). Lowering by a factor of 8 also reduces the number of ray casts and normal calculations needed by a factor of 64. Total memory usage is only 472 megabytes and it has rendered about 46 billion voxels without unloading anything (you may have notice how quickly chunks unloaded in the old video). I have a 3 GB video card but I've capped mem usage at 2560 MB to reserve space for the operating system (Windows seems to use usually only a few hundred MB unless a browser is open).
The TL;DR here is that it can now store massive areas in memory without needing to unload anything, without any major drop in visuals. There is still tons of room for "low-hanging" or trivial optimizations like better front-to back loading.
Another thing that has changed is the way the world is represented. The old terrain system was based on a heightmap. The new one is based on macroscopic chunks (you can see each cliff or bluff here is sort of cubic, but rounded off to look more natural. All this data is pulled from a volumetric buffer and makes it incredibly easy to calculate things like dynamic water. Right now each unit in this volumetric set is 8x8x8 meters large, but it can be adjusted. This is big enough to carve out sections of cave or dungeon and ensure that any type of monster will fit in there without hitting the ceiling or walls, making pathfinding more trivial. Speaking of which, I have added in pathfinding support as well (you can see this at 5:13 to 5:22 in the Kickstarter video). Chunks do not need to be rendered to perform pathfinding, as shown (first image below).Ensuring that the world is constructed properly and all roads are reachable when you have multiple levels, and ensuring that stairs are spaced enough to not be too steep, and dealing with a million other factors, was a big challenge (see second image below). :) I even tried to implement winding staircases to address the issue of small space within towers, but I pulled that feature for now (third image below).
I've also added in camera tilt - this was mostly to be able to easily see around walls, seen at 7:44 in the KS video.
The TL;DR here is that it can now store massive areas in memory without needing to unload anything, without any major drop in visuals. There is still tons of room for "low-hanging" or trivial optimizations like better front-to back loading.
Another thing that has changed is the way the world is represented. The old terrain system was based on a heightmap. The new one is based on macroscopic chunks (you can see each cliff or bluff here is sort of cubic, but rounded off to look more natural. All this data is pulled from a volumetric buffer and makes it incredibly easy to calculate things like dynamic water. Right now each unit in this volumetric set is 8x8x8 meters large, but it can be adjusted. This is big enough to carve out sections of cave or dungeon and ensure that any type of monster will fit in there without hitting the ceiling or walls, making pathfinding more trivial. Speaking of which, I have added in pathfinding support as well (you can see this at 5:13 to 5:22 in the Kickstarter video). Chunks do not need to be rendered to perform pathfinding, as shown (first image below).Ensuring that the world is constructed properly and all roads are reachable when you have multiple levels, and ensuring that stairs are spaced enough to not be too steep, and dealing with a million other factors, was a big challenge (see second image below). :) I even tried to implement winding staircases to address the issue of small space within towers, but I pulled that feature for now (third image below).
I've also added in camera tilt - this was mostly to be able to easily see around walls, seen at 7:44 in the KS video.
The UI is another major change. The UI is tightly bound to the data storage, although the presentation layer and the data storage layer are completely different structures - its not MVC though, a paradigm I hate. I have built so many UI's I'm sick of it, but I must say I've gotten the hang of it. The VQ UI system (let's just call it VQUI for now) is built to address many of the headaches I've had in the past with UIs, namely:
Writing your own UI would seem like a bad idea in most circumstances, but I really evaluated every solution and most of them honestly suck. I would have loved to use browser-based tech like in the old VQ editor, but that turned out to be a really, really bad idea for multiple reasons I won't delve into (hey, I make mistakes like anyone else :) ). On the bright side, it was a good practice run for what not to do.
One really powerful thing in the UI is that you can autobind shader variables to sliders, as seen in the video (the part where the hue is adjusted). This works by using a shader preprocessor that I wrote, which already was in there for automating other tasks. You only have to surround a variable name with "@" symbols, and this will autobind it to a slider after you refresh the shaders in the program. You don't even need to declare the variable, it will do that for you, so if you quickly want to fiddle with a value, you just throw in one of these special variables.
This also has an important performance side effect - you can bake these values into the shader after you have changed them, so that you don't need to pass in a uniform (the preprocessor just declares a constant value and sticks it in the generated program). None of this effects the code while you are writing (it modifies the shader in memory, not the file you are working on). The impact has been huge - I can now tweak values without having to constantly save and refresh the shader.
So, you can see that I've faced many challenges - the devil is in the details as they say. There were a million other improvements and fixes to the engine, but I can't cover everything - too much other work to do now! I do have most of the pieces in place to actually start building a real, genuine game though - enough of this engine nonsense. :)
- Manual layout: having to manually specify every (x,y) coordinate, margin, etc and make sure things are aligned properly and do not overlap is a major pain. Many games actually still use manual layout. VQUI automatically lays out things, just as if you were writing HTML (it uses JSON though).
- Binding every single event properly to each button or UI element (i.e. when you click this button, execute this action). VQ Automatically binds events and lets you grab UI elements with string selectors to get their values and other properties.
- Recompiling the whole program for each little GUI change, only to realize some button is one pixel off. VQUI can be hotloaded.
- Properly binding data to the UI (say, for example you have a list of inventory items; you only want to store the item id and quantity for each NPC, but when you look at an item in game it brings up an advanced sheet of properties and perhaps some dialogue options). VQUI supports templates applied to classes of children nodes, even recursively (i.e. a factory).
- Saving data that is changed by the player, only to have the data structure or layout change and break a save file. The data structure in VQUI is "unbreakable" - it doesn't matter if you change the schema of data, it is all stored in a linear fashion with unique ids for every property. This can be inefficient, but the sanity gains outweigh the performance loss (which is trivial). Even if you destroy some data key, it will retain it for future loads so nothing breaks in between changes (and you can optionally clean unused data keys on a save).
- Balancing between performance and having a GUI that is dynamic and easy to modify in realtime.
- Keeping everything DRY (Don't Repeat Yourself - i.e. minimize copy-paste code, usually via use of templates or CSS-like rules).
Writing your own UI would seem like a bad idea in most circumstances, but I really evaluated every solution and most of them honestly suck. I would have loved to use browser-based tech like in the old VQ editor, but that turned out to be a really, really bad idea for multiple reasons I won't delve into (hey, I make mistakes like anyone else :) ). On the bright side, it was a good practice run for what not to do.
One really powerful thing in the UI is that you can autobind shader variables to sliders, as seen in the video (the part where the hue is adjusted). This works by using a shader preprocessor that I wrote, which already was in there for automating other tasks. You only have to surround a variable name with "@" symbols, and this will autobind it to a slider after you refresh the shaders in the program. You don't even need to declare the variable, it will do that for you, so if you quickly want to fiddle with a value, you just throw in one of these special variables.
This also has an important performance side effect - you can bake these values into the shader after you have changed them, so that you don't need to pass in a uniform (the preprocessor just declares a constant value and sticks it in the generated program). None of this effects the code while you are writing (it modifies the shader in memory, not the file you are working on). The impact has been huge - I can now tweak values without having to constantly save and refresh the shader.
So, you can see that I've faced many challenges - the devil is in the details as they say. There were a million other improvements and fixes to the engine, but I can't cover everything - too much other work to do now! I do have most of the pieces in place to actually start building a real, genuine game though - enough of this engine nonsense. :)
RSS Feed