Tweaking Chunk Storage

[Immortius]

Hm... Well, i think the WorldView is a good starting point that can be refined... For example, in my opinion, the chunks should not bother about encapsulation at all. There should be no get/set methods for data on the chunks themselves. They only need to have getters to retrieve the arrays. The views can implement encapsulation. If we distinguish between read/write and read-only access we can also refine the locking mechanism into a read/write lock.

Sounds good to me. Should be noted we don't lock on reads at all at the moment, but we probably need it with the new sparse arrays I guess (ReentrantReadWriteLock would work nicely, obviously).

I was thinking about some kind of "virtual height limit", so the existing terrain generators and light propagation wouldn't have to be changed right now. (Would be easier for me... ) Lets say we have chunks of size 16x16x16. I would then simply generate 16 chunks vertically stacked and have the terrain and light algos work on them through a world view. The algos wouldn't have to know about the internal chunk size, they would just continue to work as always. The virtual height limit could be made configurable through the "create world" dialog.

The LocalChunkProvider would be aware of the virtual height limit and create the chunks accordingly. Above and below the virtual height limit it would just create chunks on demand. Above the limit the chunks would just be filled with air, below they would be filled with stone.

Hmm. I would suggest if you're going to do that... Keep the chunk size the same as it is, and modify the world generator to produce stone/air chunks for non-zero chunk Y positions. Then you won't have to change the main world generation algorithms at all. Light propagation would still need to be updated. This also avoids futzing around with world views, which are not supported by many of the world generation algorithms at the moment anyway.

Ultimately I think the target chunk size should be 16x128x16 - this would work well with the proposed sunlight algorithm.

 

[Panserbjoern]

Hi

I've implemented protobuf support for TeraArrays and Chunks.

Enocoding and decoding of TeraArrays:

• TeraArrays.getInstance().encode(TeraArray)
• TeraArrays.getInstance().decode(ChunksProtobuf.TeraArray)

Encoding and decoding of Chunks:

• Chunks.getInstance().encode(Chunk)
• Chunks.getInstance().decode(ChunksProtobuf.Chunk)

Further i've added a new chunk store which internally uses protobuf to store chunks. I haven't made any benchmarks yet, but i get the feeling that the new chunk store is faster than the old one. And it definitly yields slightly better compression for chunks and entire worlds.

I think, existing worlds which were created using the old chunk store should still work! They just keep using the old chunk store when you load them. At least, it works on my machine. I hope, it works on yours too, i don't have very old worlds to test it against...

I can add a conversion mechanism so your old worlds can switch to the new chunk store.

Maybe someone can test, wether old worlds still work? Don't forget to backup your old worlds!


Pull request:
https://github.com/MovingBlocks/Terasology/pull/409

Can someone look at the pull request? There are some old commits showing up which already were merged... Maybe i did something wrong with my first rebase???

 

[Cervator]

I put some Git comments on GitHub, but I'm still not particularly good at diagnosing Git quirks

I don't keep worlds around long and wouldn't worry much about backwards compatibility until we hit Alpha at the earliest.

Curiously, how do chunk dimensions impact sparse/compressed data efficiency? And would tall chunks make it harder to summarize distant chunk data to display very distant landscape in a fast/minimal fashion?

 

[Panserbjoern]

I have closed the old pull request and created a new one: https://github.com/MovingBlocks/Terasology/pull/417

I have created a new branch called "chunk-storage", which is based on the latest develop branch, and cherry picked my changes into it. I hope this works and is acceptable for you guys.

Immortius, let me know what you think about the chunk serialization code. Is it ok, or do you need any changes, extensions, ... ?

Cervator, well, chunk dimensions do affect sparse arrays. Currently, the sparse arrays are designed for chunks which are thin but tall. Essentially, sparse arrays divide a chunk into a stack of 256 horizontal "plates" of 16x16 blocks. Every plate is an array. If a plate is composed entirely of one single value, its array is not allocated. This leads to some overhead because we have in the worst case 258 arrays. Every array requires a reference, which takes 4 bytes, plus an object header of 8 bytes, plus the length of the array, wich takes another 4 bytes. Therefore, we have an overhead of 16 bytes per array. Plus 256 bytes to memorize the values of the compressed rows. If you reduce the height of the chunk to, for example, 128 blocks, the overhead grows compared to the actual data. If you increase the height of the chunk, the overhead is reduced, compared to the actual data.

In the class TeraStandardDeflator you will find three constants called DEFLATE_MINIMUM_16BIT (8BIT / 4BIT respectivly). These constants tell the deflators how many rows have to be compressed to actually reduce memory consumption. Thats necessary because of the overhead mentioned above. These values are currently statically calculated for a chunk height of 256.

I think, it could be worth investigating alternative sparse array implementations for different chunk dimensions. It also might be worth to implement some kind of runlength encoding for plates. Because, currently, if only one block in a plate differs from the rest, the whole array for the plate has to be allocated.

Actually, i think, it is not necessary to summarize chunk data for distant rendering. This is something the tesselator should do. Essentially, the tesselator should compute a mesh with reduced resolution for distant chunks. The mesh can then be stored permanently and only needs to be recomputed if the chunk was changed.

begla is already working on that: https://github.com/MovingBlocks/Terasology/issues/319

 

[begla]

I've not started working on that specific feature to be honest. But it's something I'm most interested to look into!

 

[Panserbjoern]

Well, i should have looked at the date... But i think, its the best way to go.

Maybe it would make sense, though, to have some kind of chunk summaries for mini maps.

 

[Immortius]

I think just having an block count in each chunk would be quite helpful. If you know the local area (chunk) contains 50 grass, 120 dirt and 3 stone, you can make some nice decisions about music, mobs to spawn and probably other things beside.

 

[Esereja]

tought we need to have way of geting info of one blocks: biome, heat.... and stuff. Even if such data wouldn't exsist in so small detail level. just saying my opinion.

 

[Panserbjoern]

Immortius, i can implement a way to query such infos. I think it would make sense to compute them on demand, though, instead of updating them on change. Guava offers the perfect data structure for that purpose!

 

[Immortius]

Sounds good to me.

 

[Panserbjoern]

Damn! I've spilled coffee over my notebook. I think my SSD is lost and all my data too... And my backup is over a month old... I have to redo my most recent changes. Will probably take a while until i have either repaired my notebook or bought a new one.

 

[begla]

Ouch! Hope there is at least a slight chance left that you'll be able to recover your data.

 

[Panserbjoern]

Thanks. There is probably no hope, though... My SSD is so busted, even the BIOS hangs when i plug it into my old notebook. There is a reason, after all, why one should always maintain an up to date backup... And the graphics card smells burned... But i'll try it anyway.

I think, in the worst case, i will buy one of those:
http://h20386.www2.hp.com/GermanyStore/Merch/List.aspx?sel=PBNB

The spilled one was a HP 8710w, over 5 years old.

 

[Skaldarnar]

Have you thought about a ThinkPad? They may outlive even a coffee accident

 

[Panserbjoern]

The ThinkPads biggest display seems to be 15.6", but i want 17" at least...

 

[Skaldarnar]

Hmm, they had a 17" in the W-series once, but it seems to be gone... Then maybe HP is a good alternative ...

 

[Panserbjoern]

Looks like everyone is reducing display size, or at least resolution. My broken one had 17" with 1920x1200, now the highest res is 1920x1080... I will sure miss those 120 extra lines. Grmpf. At least they have anti glare.

 

[overdhose]

there's always the option of an external monitor... if you can live with that solution.

 

[Panserbjoern]

I already use a second display. Ubuntu has excellent support for multiple displays.

 

[Esereja]

hp's tend to break often(statistic for repairmans), my opinion. Dell's is best you can get at momment(cost's a lot tought). I am sorry for acident.