Design New conceptual layer: sector (plus musings on multi-world/node)

Discussion in 'Suggestions' started by Cervator, Dec 21, 2015.

  1. Cervator

    Cervator Project Lead and Community Wizard Staff Member

    And now for something completely different!

    While the focus on v1.0.0 right now is good I wanted to spend a moment looking to the time beyond. I picked a topic close to my heart that has come up again and again and might make up one of the big features of v2.0.0. Let me start with a simple scope progression:

    Block - Chunk - Sector - (simulation) - Cluster (multi-node) - World - Universe (multi-world) - Metaverse (inter-server)

    Right now we just have the three in bold: blocks, the chunks that contain them, and the world that contains chunks. I think but am not sure if @msteiger started something sector-like for placing cities, please chime in if that's the case, I vaguely recall it coming up.

    Currently we have no collection type object between the world and the chunks in it. We can store world and chunk level entities (I think) and deal with "bigger" stuff than plain blocks that way, but we don't have a good way to handle multi-chunk objects or things that need to simulate something without being attached to live chunks. If Cities does sector-level stuff I'm guessing it would use a world entity of some sort?

    Over time we've discussed concepts that need something more, and even have had independent code for a few, like a climate simulator @Nym Traveel built ages ago. PlaTec is another similar example. Both just produce disconnected artifacts like height maps we'd then import into the game as a one-time event, their output remain static forever.

    I'd like to see (and correct me if I'm way off-based architecturally here) us expand the setup to support sectors or something like it (different name?). Goal would be to have a place to store and process sector-local data without needing a single chunk loaded, along with some more stuff I'll cover further down.

    A simple example would be cities that grow and connect to each other via trade routes. Better and more trade routes lead to faster growth, that happens even with no players around. On a new server/world creation you'd initialize a 3x3x3 grid of sectors, but for ease I'll pretend we're 2D and just imagine a flat 3x3 area. In a new phase of world gen that could be tied to @msteiger's new world preview in the UI you would:
    1. Create the sector storage and add the minimal amount of needed sectors
    2. Run some basic noise-based placement of the "city eggs" we've talked of in the past - each a single entity that lives in the sector storage with some basic generated information about the cities (location, name, size, etc)
    3. Run another basic step downstream of that (like how current facets can be chained) to let each city connect to a set/random number of neighbors. Cities already has roads like that.
      1. If a city is in an outlying sector near enough to the void outside, our normal approach with chunks would be to create the neighbors enough to start generating stuff, In this case we don't have much actual concrete content so can probably just say city x has y connections to void sector z and when that generates make the connections formal.
    4. Optionally simulate city development and trade routes in an accelerated way before starting the game (think Dwarf Fortress' world generation step with visual progress)
      1. Sector-level features would get the same per-game-loop chance to do whatever they want, although probably more rarely (every minute? Every 5? Every hour?) and they'd normally only have access to sector-level data on themselves and neighbors (and world if needed). During accelerated simulation it would run as fast as allowed to speed up progress.
    5. After the game starts you begin to generate appropriate chunks including turning the city eggs into actual cities made of plots, buildings, etc. The actual city generation will attach to the earlier sector-level data to check for needed stats like size of city.
      1. If you opted for an accelerated start the cities will have grown while in the egg state. The original starting state of city eggs was based on pure noise but now we've added a small layer of augmented data on top.
    6. As the game starts running normally the simulation of city growth continues to run in the sector layer, applying changes at the chunk and block layer (spawning new buildings when appropriate, sending out caravans) only if those chunks are loaded. If they're not the change is just done in sector data and queued for block-rasterization next time a player gets near
    7. When a player gets near enough to a sector void you:
      1. Create it in storage
      2. Do initial city egg placement based on original noise
      3. Hook up trade routes, checking with neighbor sectors that already said they expected x routes to the new sector
      4. Run the accelerated simulation based on world age, taking into account any queued events in neighbor sections (city x sent y caravans via trade route z with a large amount of red-dye based goods so those could be particularly common in the new city)
    Similar examples could be made for climate simulation - how does the temperature vary and impact crop growth through the seasons in some location? What if the player builds a giant mountain that blocks the warm wind from the equator? You can't have simulations take the changing world into account without somewhere to store that level of data. River simulation and weather are two more.

    But wait - we pretty much could do that already with world entities! We totally could. I think. It is just code and some storage somewhere above chunks. So why sectors?

    Long-term for aging servers and larger player counts we're going to hit performance issues from the sheer number of entities and things to track. Storing stuff at the world level is like global variables - use them sparingly!

    The real reason for sectors is helping split apart entity storage to where we can have entirely independent areas. If two players are 200 kilometers apart there's no reason they need to know about each other for local data. Just chat and some other limited stuff like that. Heck they might as well be in different worlds. Oh!

    I expect this would be a stepping stone to both multi-world and multi-node (wondered about those above in the scope statement?) as well as a new use of @Florian's Context architecture tweaks. I don't know how large sectors should be, but most likely they shouldn't all be independent from their neighbors. We might actually want to make that determination dynamically based on active player population and location.

    EVE Online does something similar with their server nodes. How many solar systems one node serves changes dynamically based on player population. If a large battle commences the affected node can throw off outlying systems that aren't involved to a different node and focus all resources on the battle. Our sectors could be akin to their solar systems, which admittedly are better isolated from each other (connected by stargates) than our large pieces of contiguous world. Maybe more accurately we'd group sectors together in clusters, like EVE has solar systems in constellations, and one of our nodes would run one or more clusters.

    Another theorized need I figure is that sectors wouldn't be adjacent with no overlap like chunks are currently, they would likely have a border a few chunks thick that both neighbor sectors would contain. That eases processing when entities (like players or creatures) are right on the edge between borders. Rather than switching what sector data is available each time a player crosses the line you'd stay in the one sector until you've walked a few chunks in. The system would need to be able to figure out what side is primary for some related entities.

    I'm less sure about that though, since a fight between a few players and/or creatures right on an edge might not need sector data. Not sure what would. But if you are on a cluster edge where two different server nodes are serving each side there might be some need to have all the intense data/processing happen primarily on just one node. But in that case maybe you would've just kept track of the sectors better and not split them between nodes in the first place?

    In any case we won't need multiple server nodes for quite some time. But I wanted to bring it up for future planning anyway since we still haven't split out gestalt-entity and could keep this in mind on that journey.

    Multi-world is maybe more interesting and less complicated. There you'd have a truer comparison to EVE as two whole worlds would have a nice separation like two solar systems in EVE. With some form of travel you'd be able to move between them, but you don't need both in the same entity store / Context. Something as simple as splitting two entity stores might be a huge win for taking advantage of threading on many-core servers.

    So in summary:
    • Sectors would contain chunks, multiple sectors per world. Sectors can be used for data storage beyond chunks without making excessive world-level "globals" to allow for better world gen and simulation. Sector level data could be processed without any chunks live/loaded
      • Sectors would likely be of a set size like 32x16x32 chunks (to take chunk height into account), but maybe other options are worthwhile for optimization reasons (even 2D sectors if you just care about the main world surface for one particular simulation type?). Maybe you don't need a lot of layers high up in the sky etc. Unsure here.
    • World simulation could occur in an accelerated fashion during world setup/preview plus when a new sector is needed (it can catch up to current world time). Large scale systems like climate simulation would have a home independent of chunks and could be consulted by local systems for details as the world changes. Although in some cases like a finite world (flat "planet") you might want a climate sim at world level, then deal with impacts on biome at the sector level.
    • A Cluster would be a dynamic selection of sectors that could be isolated from other clusters as far as entity storage / other storage / Context is concerned. They could run in parallel as long as they stay in time sync. Keeps down storage sizes and allows for better multi-threading on a single server. On saving / exiting you'd probably still "just" write stuff at a sector or even chunk level.
    • A Node would be a single server machine that can be part of a multi-node game server. A node can run one or more clusters independently of other nodes, excepting some minimal overhead like staying in time sync, handling chat, logins, etc. You'd likely have a single master node or even a controller node of some sort.
    • A World is simply all the sectors making up a planet, a dimension, or some other geographically separated concept. You could still have infinite worlds that keep growing forever as each would have its own coordinate system, chunk storage, etc. Current "global" scope, nothing bigger exists at the moment.
    • A Universe would be the next "global" up from world in a future where we support multi-world. Could probably have more layers like solar system, but that doesn't seem important yet.
    • A Metaverse is just silly, but if somebody really wanted inter-server teleportation they could probably make it work this way. It came up at some point.
    Forgive me for my almost certain architectural missteps or poor memory above. I'm sure I got some details wrong, so please help me correct them!

    At current I'm only really thinking about getting started brainstorming and maybe working on the sectors and simulation over the coming months (may make this another GSOC candidate). Something else to fill in those boring idle moments between working hard on v1.0.0 issues and opening presents! :)
    • Like Like x 1
    • Optimistic Optimistic x 1
  2. Cervator

    Cervator Project Lead and Community Wizard Staff Member

    Addendum: Surfaces

    A missing piece in the above is what to do about things that care specifically about one layer (like sea level), but not so much about other layers.

    At present IIRC sea level is the only "surface" we declare and refer to, although I think we theoretically could declare other surfaces like a major cave level or flying islands. But sea level and any such other surface would be global if my thinking is right (and you don't use math tricks to cover a checkerboard pattern or something). With sectors you could declare a surface local to one or more sectors. So rather than a cave level being *everywhere* you could base its presence on noise or whatever applied to sectors.

    A global surface still would likely make sense for some things like a climate simulator for the main sea level surface. And maybe you'd want a magma sea at a certain level underground for the whole world. You could choose between global and local. Not that a global layer of sky islands would necessarily be thick, they could be super sparse and not appear to be connected, but they'd likely still either be at one layer or generated generically all the way up. Instead you could just define multiple local sky island layers if you so chose to mix it up a bit.

    I figure that would have some beneficial potential for pathfinding. I think @synopia already has a concept of floors in block-based pathfinding, it would probably be nice to cache known paths to get from one surface to another.
  3. chessandgo

    chessandgo Member

    Chunk size - if chunks are smaller:

    • less data is sent until its needed. This can help the networking aspect, and works as a simple anti-cheat for xray. You can xray if we haven't even sent you the blocks!
    • easier, and more verbose metadata. fewer blocks means that its easier to analyze chunks in the background to obtain metadata. Become even more useful when all grouped together. (which may be useful for stuff like if the server should or should not send this chunk, or it should be sent to the rendering engine.)
    • Maybe help new terrain generation. Less choppy loading as chunks are smaller, chunks are delivered to you faster, and in staller bursts of computing power.
    • Might help with chunk recall? (more metadata/structured data to go off of)
    • Maybe increase storage and undo optimization
  4. manu3d

    manu3d Pixel Forge Artisan

    Chunk size is tied to performance. Current chunks are 32 x 64 x 32 blocks = 65,356 blocks or 2^16 blocks, the length of a short int. This is used for fast indexing, to quickly jump from the memory address of one chunk to another. Making them the length of a char (2^8 = 256 blocks) is likely too fine-grained for performance and going the next level up (2^32 = 4'294'967'296 blocks, the length of a long int) is likely to be too big for current hardware. At least some of the issues you are describing @chessandgo can be addressed by this proposal. I.e. the stuttered loading you report is likely a fairly constant-speed loading of chunks that are out of sight.
    • Informative Informative x 1
    Last edited: Nov 23, 2016
  5. Cervator

    Cervator Project Lead and Community Wizard Staff Member

    So I noticed an update to Improbable's SpatialOS which had them launch their service into public Alpha via collaboration with Google. So think massive elastic cloud availability, maybe interesting hooks with AI, and so on, including machine learning.

    In short: SpatialOS is the server side stuff for an MMO. They handle the multi-node hosting for massive worlds and you just code against their API to process workloads (players connecting and doing stuff in a world). Several notable games are already being based on it and with Google throwing their weight behind it (after some 20 million of earlier VC money) that's saying something.

    What made it stick out and got me to post this here is a few striking similarities
    • SpatialOS uses an entity-component system. Huh! Looks oddly familiar. Entities, Components, Component Updates (Events) and Workers (Systems). Interesting!
    • Their architecture primer (scroll down on the front page till you hit "Architecture") looks exactly like how I would visualize Sectors, complete with some degree of overlap possible
    • They have a Java API which again looks weirdly familiar. Sure the class names differ and you're doing callbacks instead of catching events, but conceptually the resemblance is remarkable to me
    • One of the game snippets shown in their primer looks like the exact same kind of space arcade shooter as Destination Sol
    Should we totes get onboard and build Terasology the MMO real quick? Well, no. It'll eventually be a commercial service (not open source, at least not yet) and there are still plenty of differences with them looking to be targeting Unity so far with more work to support other stuff coming. But it looks incredibly interesting to me at a nerdy level, like the kind of thing I would dive into and learn about if I actually had such a thing as free time. It could be a very cool thing to examine and get ideas from.

    And hey, who knows. Maybe at some point in the future when they progress into Beta and beyond, and we've gotten Gestalt extracted and polished some more ... maybe somebody nerdy could find some time to think about how stuff might link together :)

    (Edit: There are even EntityTemplates - prefabs! And some JSON. Okokok I'm stopping, more important stuff to do right now)
    • Like Like x 2
    Last edited: Dec 16, 2016
  6. Skaldarnar

    Skaldarnar Badges badges badges badges mushroom mushroom! Staff Member

    We get it, we should work on sectors :p

    The scope of SpatialOS is way broader than what we are aiming at, using the same fundamentals in form of the ES. We are doing something right if there is that much potential in terms of scalability. The implementation of sectors could very well become a potential GSOC task for the summer - we have lots of ideas and concepts floating around, and a skilled student can definitely come up with a good design here.
    In the far future, hooking up different Terasology worlds to a single universe where players can move between them sounds awesome. But let's start with our very own multi-worlds ;)
    • Like Like x 1
    • Agree Agree x 1

Share This Page