Per-block data storage

[woodspeople]

Have been thinking about and discussing this.

Benefits of block-type mineral content:
- all blocks can have water content AND mineral content
- visible ore grades fun to find (oooh, high-grade iron ore!)
Detriments:
- clutters up block types; we already have a lot of cool things with meaningful distinctions (thanks to metouto); doubling or tripling those to add grades might make the plenty less fun and more tedious

Benefits of one resource value:
- neat hidden value for which player could build device or acquire magical ability to see overlaid values (still possible to say ooooh, high-grade iron ore, but only if you advance)
- not really necessary to have water everywhere (come to think of it I had patches of water in my proof of concept for trees, not everywhere water)
Detriments:
- can never have two amounts of anything in a block (maybe doesn't matter)
- mineral values not obvious (but more interesting challenge?)

Another possibility: combine the two. Accept no water+resource as a limitation, but gain the ability to broaden the resource range by having (in some cases, not all, because it is not the only way) block types that bump up the meaning of the range value. Like, for a high-grade ore the 0-16 doesn't really mean that, it means 17-32. You would just need a "resource range modifier" thing in the block definition. Or, you could have a "swamped ore" in which the 0-16 is both water AND ore. Again a boolean in the block definition, which compared to block storage is free. So you could do a lot by messing with resource values in the block type configurations, and in the modules.

 

[Cervator]

Yeah I believe the combination of the "secondary" counter + block type can give us a LOT of options, especially since we can make processes by which the secondary counter causes the block type itself to change.

One of the most intriguing bits of potential in taking this system a long way is observing how the generated terrain will change if affected by a semi-realistic weather system - something as simple as rain refilling rivers, non-infinite liquid blocks, liquids that can flow to pick new paths if they become available, swamps appearing in spots that don't get enough sun, blocks that build up more and more water content over time until they turn into an aquifer, and so on...

And the best part is that it builds on the existing "state" nibble, so that doesn't even add anything extra to the memory demands. That one variable is incredibly underused since currently it is only set when a liquid is present.

In fact, I wonder if the two light nibbles can be reused in blocks that are unaffected by light instead of adding a whole new byte to do my crazy block integrity thing. Depends a bit on whether that value is ever set in plain air, or only on lit blocks adjacent to air. Would be perfect if light only matters in air blocks, then we can fit so much data in solid blocks...

 

[overdhose]

yeah might be quite some potential there... and I love the idea of growing my own swamps.. I can't wait for that one.

 

[eleazzaar]

I had hoped to have trees DEPLETE minerals in an area. However, this COULD be done by converting blocks from IDs with high mineral content (for whatever that tree species considers mineral content) to blocks with low mineral content.

That sounds more satisfactory to me: Trees (or other plants) when growing have a chance of converting "fertile soil" blocks into "soil" blocks. "Soil" blocks don't produce as much (or as quick) growth, and in turn can be converted by plants into "poor soil" blocks -- which support much less growth. Or something like that.

As Cervator mentioned earlier, with block types, you can see the difference between the types of soil, and thus can interact with the system, by intelligently applying fertilizer.

 

[woodspeople]

In fact, I wonder if the two light nibbles can be reused in blocks that are unaffected by light instead of adding a whole new byte to do my crazy block integrity thing. Depends a bit on whether that value is ever set in plain air, or only on lit blocks adjacent to air. Would be perfect if light only matters in air blocks, then we can fit so much data in solid blocks...

For my system to work I only need one per-block light index; the other is per vertical column (i.e., sunlight at the top of the world). Is there some other reason to need two light indexes per block?

Agree about reuse of light index position(s) for blocks unaffected by light - as long as the light index can be reinstated if and when the block is suddenly exposed to light (blocks above it removed by player). Also missed what "my crazy block integrity thing" is: how broken a block is?

 

[eleazzaar]

One of the most intriguing bits of potential in taking this system a long way is observing how the generated terrain will change if affected by a semi-realistic weather system - something as simple as rain refilling rivers, non-infinite liquid blocks, liquids that can flow to pick new paths if they become available, swamps appearing in spots that don't get enough sun, blocks that build up more and more water content over time until they turn into an aquifer, and so on...

As cool as a semi-realistic hydraulic cycle would be there are some huge obstacles. Unless you keep the whole world active and running in memory, you could have a long river dry up for instance, because the mountain where it originates is not in active memory, and thus not getting any rain.

A couple of other potential (and mutually exclusive) uses for a secondary "counter".

• stores damage to solid blocks. If you half mine an obsidian block it stays half mined forever, or until you finish it off. Multiple explosions damage would be cumulative
• stores the stress or weight on a block for some sort of physics simulation

 

[Immortius]

For my system to work I only need one per-block light index; the other is per vertical column (i.e., sunlight at the top of the world). Is there some other reason to need two light indexes per block?

It's used for rendering. We need a light index per block so we know how lit things should be in that block, and we need a sunlight index so we know how well lit things should be varying with the time of day/night. Also sunlight isn't just done in vertical columns, it can propagate under ledges and into caves (3D world after all).

Agree about reuse of light index position(s) for blocks unaffected by light - as long as the light index can be reinstated if and when the block is suddenly exposed to light (blocks above it removed by player). Also missed what "my crazy block integrity thing" is: how broken a block is?

Yeah. The tricky thing is that the amount of data you need a block to have is dependent on the maximum number of things that block may have at a time. So you can't really share, say, light and dampness, because a block can both be lit and damp at the same time. You could share water and magma levels, because a block won't have both. Basically the only time you can reuse data space is where you have mutually exclusive properties.

For really sparse values like damage using entities to store the data is better. Damage doesn't have to be temporary even through entities, although I added in some regen so that the world didn't get filled with ugly cracked blocks.

 

[Cervator]

Here's one thread on my Block Dynamics concept: http://forum.movingblocks.net/threads/block-dynamics.214/

There are a couple issues and some other threads on it too. In a nutshell: Block Integrity is how well supported blocks are, based on some "bottom of the world" type effect, then as you go up and expose more and more to the air, carve caves, blow up stuff, etc the integrity value might be go low enough to physics-enable a block to allow it to fall. Would solve floating blocks and unrealistic overhangs while allowing explosions to remove support enough where you could start a landslide. It also leads to some interesting secondary effects in non-supportable blocks (water, aquifers, lava, etc) like pressure.

On the hydraulic cycle - yeah, quite tricky, but I figure if you set blocks currently at the "edge" of the world as temporary infinite sources maybe you could emulate it to a degree.

Immortius - do you think light calculations could work entirely off light values present in air blocks? As in, the air block above a dirt block is highly lit -> that one side of the dirt block is rendered with its appropriate light level, yet the dirt block itself does not store a light value? That would be a huge boon to the mutually exclusive setup as we'd have an entire byte for solid blocks for "free" - main trick I see is light level underwater, but I figure you could just use one nibble there rather than both. Dunno if custom shape blocks would break that idea.

 

[Immortius]

It sort of works that way (lighting is done by sampling surrounding blocks), but translucent blocks (like grass, leaves, torches, water) and luminescent blocks store light values too. Although it can possibly be skipped for luminescent blocks. Still need both light nibbles for translucent blocks, because we have two types of lights and they are not mutually exclusive. But theoretically non-translucent, non-luminescent blocks could store something else. Would need to update the renderer and light propagation systems to understand this of course.

This is ignoring some more complex lighting calculations we may need in the future to handle things like doors properly - where individual sides of a block may be translucent or not. But that shouldn't impact things too much.

 

[Cervator]

Hmm, I think that could work. Pure glass structures might be tricky if they can't hold integrity values per block, but stuff like doors probably shouldn't provide integrity anyway. And there's probably some sort of workaround for glass. Billboard plants shouldn't provide integrity either. Leaves - could be another tweak, or we could have branches or the growth simulator deal with it.

 

[Immortius]

Well, remember that things like thin columns, branches, and any sort of sub-block structures are translucent for this purpose.

 

[Cervator]

Generally I think that'll work OK - structural support columns might be tricky. Maybe they could simply be set to infinite strength being that they're specifically for support. Anything normal on top would still slowly drop in strength. Could possibly do some column-specific check and only allow a set height based on material. Could be something specific to artificial structures (that in theory would be the only things ever built out of custom shaped blocks, and rare on the grand scale of things)

There'd be some sketchy areas like that for sure. Question is whether the lower (well, same as now rather than increased) memory footprint is worth the trouble. I'm just eager to see the actual functionality it can power

 

[dei]

As cool as a semi-realistic hydraulic cycle would be there are some huge obstacles. Unless you keep the whole world active and running in memory, you could have a long river dry up for instance, because the mountain where it originates is not in active memory, and thus not getting any rain.

This problem will come up in multiplayer anyway so if we solve it now it's probably simpler than afterwards:
If we don't want to simulate the whole world on the server during runtime (which can be reeeeeeeaaally huge, see the minecraft servers which waste memory like it was as cheap as HDD-Space) and can't rely on the client-cpu/memory to prevent cheating, we have to make all our simulations deterministic.

Every simulation would be a function of time and the surrounding world in its state before simulation, so basically matrix-transformations which could eventually even be calculated by GPU. So when a client encounters a new Chunk not in local memory the server "fast forwards" the Chunk using these deterministic simulation-functions and sends the new Chunk(s) to the client. The same thing you could do on the client in singleplayer. (I came to the idea because Unigine Engine handles it's physics simulation this way, so I think it must be a good way doing it)

So we would only need to save a timer per Chunk, when it was last updated and a world-timer. The simulation can then be calculated based on the difference. If the chunks currently in memory could use the same simulation-technique would maintainability-wise be great (my head's to full at the moment to think about that too).

This way the server even doesn't need to to do all calculations, only to validate the clients simulation as soon as the client is interacting with the world (and only would need to transfer a checksum instead of all blocks).

To bridge Chunk-Edges we would have to pass a remainder-matrix (like in bitwise arithmetics) and be able to aproximate this remainder without simulating all the blocks of the chunk, but I think this should be possible too.

 

[Cervator]

I was wondering about the time simulation angle, especially since you could likely approximate some effects over large time spans, such as tree growth or rain. Not sure if those examples make sense, but just for the thought - if on average it rains 10% of the time, then rather than simulate one time unit after another to catch up, randomizing when it was raining, simply take the total time units passed, divide by 10, and apply that amount of rain effect.

Of course, if the chunks were unloaded they also didn't drain/soak up water, so don't want to overwhelm with liquid. Some sort of balance would have to be found.

Dei - if you're familiar with that sort of functionality, want to take it a step further with some outright design and maybe even implementation?

 

[eleazzaar]

It would be great if plant growth could be simulated for the time when a chunk is inactive

On the hydraulic cycle - yeah, quite tricky, but I figure if you set blocks currently at the "edge" of the world as temporary infinite sources maybe you could emulate it to a degree.

If you have volumetric water, ever having any infinite sources, even temporarily could be extremely dangerous.

Dump a bucket of water on the top of a mountain and log off. Some one else at the right (or wrong) distance from that could get the visible world filled from that bucket.

What happens at the down-stream end? Either water destroys itself by flowing into an unloaded chunk, and thus oceans and lakes would drain out when they go past the edge, or water stops at the edge of loaded chunks, and with a little cleverness, or unluckiness you get "land" chunks filled to a high level with water-- waiting to come crashing down on the next person who comes along.

Somewhere i read a list of reason why something like liquid cubed could never work on an infinite world, that was rather convincing-- sorry can't find it, but it includes stuff like the above.

I'm not saying that we need to make exactly the same compromises with water that Minecraft did. I am saying serious compromises will need to be made with reality.

Having thought about this before, i think you could get closer to reality without breaking by making two kinds of water:

1) infinite water

• occurs only at sea level or below
• probably include all rivers (flat, like mine craft rivers)
• essentially "the ocean"
• if you collect a bucket of it, it will pour out as "finite" water instead.
• what happens when you connect a cavern to infinite water?

2) finite water

• occurs in small lakes above sea-level, and in small pockets underground
• maintains its volume
• when it flows into infinite water is destroyed/consumed

So it's not really a full hydrological cycle, and "infinite" water rivers and oceans serve as sinks for any local overflows of finite water. If you do rain and run-off-- it would be a local event similar to a ditch filling with water during a rain -- but nobody is confused if a ditch is sometimes dry, and oceans/rivers don't dry up no matter what happens upstream, since they don't actually flow, and aren't actually fed by rain. Waterfalls: these "spring source blocks" couldn't be moved with a bucket, and probably should only spawn in the same chuck with some infinite water, so they wouldn't have a long course over which to be glitchy before they are consumed by the sea.
I'm not sure exactly what you have in mind, but this would probably be compatible with having dirt and certain stone blocks contain moisture.

 

[Cervator]

That sounds good too, there would be some parts like the ocean I indeed wouldn't want to simulate / would count as infinite absorbers. The bucket on top of a mountain + logoff or go out of range is a good thought example. I had figured that a chunk that borders (river type) water at the edge of the loaded chunks would "maintain current state", but if that state is in flux that gets tricky.

And matching a desire for a more complex water system vs. the constraints of reality and effort available is certainly also an ever-present downer...

 

[eleazzaar]

...And matching a desire for a more complex water system vs. the constraints of reality and effort available is certainly also an ever-present downer...

Limitations force you to make choices. The way i look at it, having to grapple with the constraints of practicality is what makes game design interesting and challenging.

 

[dei]

Dei - if you're familiar with that sort of functionality, want to take it a step further with some outright design and maybe even implementation?

I'll begin some design for it after having included my sparseTeraArray code. Have to look at simulations anyway to fully understand the needs for a unified TeraArrayInterface, so I'll do that next...

 

[Cervator]

Thanks for that Liquid Cubed link, by the way! Just took a little time to check it out along with their newest project, Blockade Runner - spiffy stuff

 

[Cervator]

Bump - posted https://github.com/MovingBlocks/Terasology/issues/257 to see about getting @dei's SparseChunkArray added sometime with some sort of performance stats included

(Along with several other chunk/block related issues for milestone 7, which is a ways out)