Render DAG Enhancement - GSoC 2019

dave2s

New Member
Contributor
Initial text:
I'm including my proposal to keep the order in things.

Read only: Overleaf (LaTeX editor) link
Edit/comment rights - requires login (e.g. with google acc, comment by clicking review on top and selecting source code - comment button pops up): Overleaf (LaTeX editor) link

This proposal is centered about continuation of render graph topic. Proposing a way to express dependencies among nodes as drafted by some earlier (than me) issues and maybe deal with some blockers for future dag-module exposé. As a second part I'd like to take on mkienenb's PR for render graph overlay (ingame) and I give some ideas there. As a strech, some nodes might be provided which could leverage these changes.
Edit: This project has been accepted for GSOC 2019. yeet :gooey:
Some basic information:
 
Last edited:

dave2s

New Member
Contributor
Week 1 summary

During the first week I've been drafting an extra layer of connections which are attributed to nodes on the AbstractNode level.
This graph shows which parts of the DAG I've begun to edit, compared to reference dag, during the first week. Everything should be compilable and runnable, although I'm not verifying at this stage.
2536
Some of the basic units in the render graph I've been working with for future reference are

WorldRendererImpl(c); RenderGraph(c)
Node(i)->AbstractNode(ac)->SpecificNode(c);
(new) DependencyConnection(ac)->FboConnection(c)
;
(new) NewAbstractNode(ac, surrogate for AbstractNode)

What is possible at this stage: Connecting FboConnections without using centralized system.

How does this work at the moment: Given nodeA and nodeB you
  1. Add output to nodeA - in its constructor/setDependencies method - addOutputFboConnection(1,nodeA);
  2. After nodeB's creation - explicitly* connect nodeA's output to nodeB's input in WorldRendererImpl -nodeB.connectFbo(1, nodeA.getOutputFboConnection(1));
  3. Inside nodeB's setDependencies() method - use this connection however -
    this.getInputFboData(id) / this.getOutputFboData(id) / this.getInputFboConnection()...
*not happy with this yet
These changes seem to so far create the same output and the game runs.
On the other hand, there's a lot of things to consider and this specific approach is still far from its final shape, if something like this is going to be adopted at all (so far no other direction has been spoken).

Next on:
  • Auto matching in RenderGraph - I'm going try and remove the need to explicitly connect dependency connections by hand trough WorldRendererImpl. Since now you connect dependencies and the nodes order separately, which seems rather redundant so far. This includes setting input connections without data inside nodes. So you basically don't care what you are connected to. This should provide for needed modularity.
  • Still considering every little change that has and hasn't been done. Every change has to be proven needed or it's gonna go away. So far everything is easily upgradable to a different approach.
Further on:
  • When it becomes clearer whether we will continue this way, I need to review legacy code a remove centralized dependency storing alltogether.
PR: DAG DependencyConnection draft #3680

Current problems
:
So far making changes to the basic classes requires spreading the changes troughout majority of classes which makes a messy PR.
Trying to maintain top down perspective is a little problematic while spending time trying to make things work right away. It seems as though it might almost be better to just work with a broken code for a bit.

Blockers: Nothing in particular.
 
Last edited:

dave2s

New Member
Contributor
Week 2 summary

During the second week I had tried to come up with ideas around auto-matching of connections but so far it's a dead end. Other than that I've added some attributes to connections to remember who they belong to and whom they're connected with. I added some error checks to connecting mehods.

Some of the basic units in the render graph I've been working with for future reference are

WorldRendererImpl(c); RenderGraph(c)
Node(i)->AbstractNode(ac)->SpecificNode(c);
(new) DependencyConnection(ac)->FboConnection(c)
;
(new) NewAbstractNode(ac, surrogate for AbstractNode)

Additions:
  1. DependencyConnection:
    Java:
    private SimpleUri parentNode; private SimpleUri connectedNode;
    getParentNode(); setParentNode();
    getConnectedNode(); setConnectedNode();
  2. NewAbstractNode: changed addConection hiearchy of methods to return true on success, false otherwise. connectFbo then checks for existing connections as well as connections which have already been read from, to prevent buffer competition. Next on this-I'm considering adding a connectWithCopy as well as generic types to implement these higher level methods.
  3. Whole dag refactor for class and variable names: ...FBOs... -> ...Fbo...
Next on:
  • API vs Whitelist - I'm going to try to tamper with the dag setup from a module and see where it goes.
  • Discuss whether to deal with lastUpdated/stale gbuffer pair passing the same way as with auxiliar buffer passing.
Further on:
  • Make abstract level methods generic.
  • Cherry-pick FBO.copyToFbo() from a private branch to enable connectWithCopy or something of the sort. Need to discuss whether to add scheduler methods for this.
  • When it becomes clearer whether we will continue this way, I need to review legacy code a remove centralized dependency storing alltogether.
  • Can't forget Test module. Made some changes there to compile during week 1, did not test.
PR: Updated DAG DependencyConnection draft #3680

Current problems
:
Can't figure out what to do with the main buffer pair...do I implicitly pass it? explicitly pass it? do I keep them centralized?

Blockers: Nothing in particular.
 
Last edited:

dave2s

New Member
Contributor
Week 3 summary

During the third week I drafted an API for modules to use after the engine DAG setup and a module based DAG change, which reconnects some dependencies to bypass a node and it works. One setback is that currently there're some redundant state change calls (like binding a different texture to the same variable twice) which is caused by repetitive call of bulky node.setDependencies() call, which happens when a node's input is being reconnected. TODO improve this with some more logical update, or at least postpone these calls (AND therefore RenderGraph construction too) until module based changed have been made too so the calls are made after the dag is set up.

This step depends on whether we want to support mid game DAG. If so, the node.setDependency() method would have to be split and moved under methods which create each dependency. These would have to be split into methods enabling setting connections based on what we want to do with them. Example:
node1.connectFbo[Texture](input id, node2.getOutputFbo(id)); Inside connectFbo we would make a call to add stateChange into queue to send a texture to some local program. This approach however is kind of restricting and it would be hard to cover every option. I don't see a point in enabling this since any mid-game changes are handled inside node's process() and propertyChange(PropertyChangedEvent) methods.

On a related note: So far all dependency calls don't co-operate with the RenderGraph itself. This will have to be added soon.
Why do we need this: Tampering with dependencies does not yet take into account how the nodes are connected in the render graph. But only nodes connected in the RenderGraph are run and only RenderGraph sets the order the nodes are processed.
What does this look like: dependency connecting methods (probably on the level of the abstract nodes) will have to check whether they are the first connection to be added / last to be removed and create/remove RenderGraph connections.
Some of the important files:

dagTestingModule github repo. This module extends BaseComponentSystem and uses initialise() phase to be run. Happens after injection, so we can inject context and use it to obtain instance of renderDagApi, which was created during World creation phase (before init, see StateLoading.java's init[Client/Host] methods).
(new) RenderDagApiInterface(i)->RenderDagApi(c);
(m) DependencyConnection(ac)->FboConnection(c)
;
(m) NewNode(i)->NewAbstractNode(ac);
minor tweaks here and there, WorldRendererImpl(c), RenderGraph(c).


Next on:
  • Create a module demonstrating basic render dag api usage, including module based node which is inserted into the dag. (Due June 24)
  • Postpone setDependency calls after the rendering modules have been initialized. Also render dag construction would have to be postponed.
  • Add some API capability.
  • Integrate with RenderDag operations. (!!!)
Further on:
  • Split rendering nodes into 2 sets - 1.essential features, 2.advanced features. Move the groups into modules. Feature set #1 will be mandatory. Feature set #2 will be optional.
  • To do read only Fbo. Cherry-pick FBO.copyToFbo() from a private branch to enable connectWithCopy or something of the sort. Need to discuss whether to add scheduler methods for this.
PR: Updated DAG DependencyConnection draft #3680

Current problems
:
Redundant calls of setting dependencies for entire node based on only one small change. - Not good
Can't figure out what to do with the main buffer pair...do I implicitly pass it? explicitly pass it? do I keep them centralized?

Blockers: Nothing in particular.
 
Last edited:

dave2s

New Member
Contributor
Week 4 summary

During the fourth week I took out the previously introduced tint node and made it module based. The module now inserts a Tint node between finalPostProcessingNode and OutputToScreenNode effectively turning the final image reddish. Added a areConnected(node, node) query to RenderGraph to see if 2 nodes are connected when sorting dependencies. Added findNode(), resetDesiredStateChanges() overloaded methods to RenderGraphApi and Nodes, fixed toString() for connections, added isDependentOn() method to query a node for dependency on another node - used for automatic connecting/disconnecting nodes in the RenderGraph upon reconnection of dependencies. Updated connectFbo() and reconnectInputFboToOutput() renderDagApi/Connection methods to use this. Had to implement somewhat a @API wrapper for java.beans.PropertyChangeListener so that the module-based nodes are not denied access during runtime. Did not test the events' functionality.

Further I updated ShaderManagerLwjgl to allow for adding shaders from within different modules than engine. Just overloaded the addShader() method with another argument. Updated RenderDagApi interface with addShader() and shaderManager injection.

Added new ModuleRenderingSystem extending BaseComponentSystem, which is now extended by the rendering module instead to hide the rendering module's initialization.

dagTestingModule github repo. This module's main class now extends ModuleRenderingSystem which extends BaseComponentSystem. Adds a tint shader as well as TintNode.
(new) ModuleRenderingSystem(ac)<-BaseComponentSystem(ac)
(m) RenderDagApiInterface(i)->RenderDagApi(c);
(m) DependencyConnection(ac)->FboConnection(c)
;
(m)NewAbstractNode(ac);
(m) RenderGraph(c).


Next on:
  • Filling Missing/Incomplete Javadocs
  • removing redundant or unused things from ; getting back to node's setup - getting rid of previous FBO storing, updating all nodes to this system.
Further on:
  • Split rendering nodes into 2 sets - 1.essential features, 2.advanced features. Move the groups into modules. Feature set #1 will be mandatory. Feature set #2 will be optional.
  • To do read only Fbo. Cherry-pick FBO.copyToFbo() from a private branch to enable connectWithCopy or something of the sort. Need to discuss whether to add scheduler methods for this.
PR: Updated DAG DependencyConnection draft #3680
DagTestingModule:
dagTestingModule github repo

Current problems:
  • Java beans property change listener/event inaccessible in modules..workaround not tested, compiles though.
  • So far a few workarounds like storing RenderGraph and Context instances in nodes themselves.
  • Rendering module needs to pass its class to its super so we can store Name of the module for use in creating Uris. I only tried getting classes name which returns name of the class implementing the abstract module rendering class, but that is not sufficient because we need the full path including the module's package to find the class this way. There might be a better way, though it's not that important now. At least I could assert non-null on the ModuleRenderingSystem's protected providingModule attribute somewhere so it's forced for now. In addShader at least, that's where it's needed. We don't need to do that when creating Nodes because they are created within the module, unlike calling addShader().
Blockers: Nothing in particular.
 
Last edited:

dave2s

New Member
Contributor
Week of 1st evaluation summary (5)

During the 1st evaluation week (5) I whitelisted java.beans in ExternalApi.Java for module-based Nodes to be able to use PropertyChangeListener/Events. Removed tint_frag.glsl and tint_vert.glsl from the engine (module based, testing purposes only). Next I removed RenderDagApi.java and RenderDagApiInterface.java and merged its unique code to RenderGraph.java. Then I replaced RenderDagApi with RenderGraph in ModuleRenderingSystem and dagTestingModule's TintMyOutput.java.

List of relevant files/repos:
dagTestingModule github repo
(m) RenderGraph(c)
(deleted) RenderDagApiInterface(i)->RenderDagApi(c)
(m) ExternalApiWhitelist.java
(m) NewAbstractNode(ac) -
comparing Nodes by SimpleUri
(m) WorldRendererImpl.java - remove RenderDagApi relevant code and add RenderGraph to context.
(m) ModuleRenderingSystem(ac) swapped RenderGraphApi for RenderGraph itself.
(no change) DependencyConnection(ac)->FboConnection(c)

Next on:
  • Enable BufferPair's passing by current system
  • Get rid of StateChanges which are Dependent on passing data and replace them by direct gl calls in Node's process() method.
  • Rework rendergraph's map of nodes to reflect dependency connections. Get rid of explicit node connecting.
  • Move all rendering Nodes to BasicRendering module.
  • Provide the changes ? Test first
Further on:
  • Split rendering nodes into 2 sets - 1.essential features, 2.advanced features. Move the groups into modules. Feature set #1 will be mandatory. Feature set #2 will be optional.
  • To do read only Fbo. Cherry-pick FBO.copyToFbo() from a private branch to enable connectWithCopy or something of the sort.
PR: Updated DAG DependencyConnection draft #3680
DagTestingModule:
dagTestingModule github repo

Current problems:
  • Nothing major yet, but I expect problems during the next 2 weeks regarding discussed reworks (mentioned in Next on).
Blockers: Nothing in particular.
 
Last edited:

dave2s

New Member
Contributor
Week (5)6 summary

During the fifth week
I moved all nodes from engine module's package ...dag.nodes to the same package in a new module BasicRendering. A few little hacks I left behind to make some static deps working. The renderGraph node connecting was taken from WorldRendererImpl.java and is now happening BasicRenderingModule.java, which contains needed parts taken out from the world renderer (so far only light camera getter from shadowmap node). Transfered connecting responsibility from nodes to RenderGraph.

List of relevant files/repos:
dagTestingModule github repo
BasicRendering module github repo

Next on:
  • RenderGraph polish. It's now a little messy but works. We've been discussing late building, node's state changes to be fetched later in the process somehow and because there's more one usually needs to do around these, it seems more lucid to have a post-init kind of method which can be called by the rendergraph on build for each node. Regarding automatic connecting...so far nodes are connected when connecting a dependency and have not been yet connected...this might need to be changed to correspond to the legacy rendergraph connecting
  • Figure out how drafted BufferPair connection fits into dependency connecting. There's a lot of tiny details that we discuss on slack.
  • Rework rendergraph's map of nodes to reflect dependency connections. Get rid of explicit node connecting (wip).
  • Provide the changes ? Test first
Further on:
  • Split rendering nodes into 2 sets - 1.essential features, 2.advanced features. Move the groups into modules. Feature set #1 will be mandatory. Feature set #2 will be optional.
  • To do read only Fbo. Cherry-pick FBO.copyToFbo() from a private branch to enable connectWithCopy or something of the sort.
  • Provide reusable nodes, this will require redoing correct uri naming of the dependencies together with module name, so far we use IDs which translate to a simple name which after reading does not hint what it's used for neither what module it's from.
PR: Updated DAG DependencyConnection draft #3680
BasicRenderingModule: BasicRendering module github repo

Current problems
:
  • ToDos RefractiveReflectiveBlocksNodeProxy hack to set water parameters from module. Submersible camera in engine uses RenderHelper to read these and calc stuff.
  • GLSLShader.java uses SSAO kernel elements and SSAO noise size which I hardcoded for the time being, since the values are now BasicRendering module-based and GLSLShader is engine-based.
Blockers: I need to figure out how exactly to pass buffer pairs..whether each node should implicitly pass it, how a node collection idea fits into this. This is blocking me from having the entire node base using the new system.
 
Last edited:

dave2s

New Member
Contributor
Week (6)7 summary (2 weeks pre this post)

During the sixth week I were reconnecting all nodes now in BasicRendering trough my dependency connecting methods. Main trouble was with bufferPair connecting/creation, all buffers must be subscribed to receive updates, so I still let them be created by Fbo managers, but these might either be made thiner or deprecated alltogether (a new subscriber system might be needed, or the entire dag would have to be rebuilt with windows resizing and some other changes, which is obviously a bad idea). Created BufferPair and RunOrder connections in adition to Fbo connections. BufferPair can be copied with swap. I basically reconnected almost everything this week trough Fbo and BufferPair connections, which acts as a run order connecting on the Graph level. Tweaked some getName/toStrings for these connections, removed temporary hack where I needed rendergraph in nodes. Various responsibilities are now more logically spread across classes.

List of relevant files/repos:
Basically all commits between 8-15.July including
BasicRendering module github repo
DAG DependencyConnection draft #3680

Next on:
  • Create RunOrder connections to replace explicit renderGraph.connect() call in rendering modules. It should happen after all modules are initialized.
  • Maybe Dummy Nodes + findAka and a new nodeAka field for arbitrary node names
  • Advanced effects to AdvancedRendering
  • Ordering of rendering modules (NUI style?) without ordering the AKA has half the usability (choosing the strict approach of finding a Node by its exact nodeUri is useless if the renderingModules' initialiazion order is not specified)
PR: Updated DAG DependencyConnection draft #3680
BasicRenderingModule: BasicRendering module github repo

Current problems
:
  • Had some problems with output but solved that by creating the main buffer pair with manager so it's subscribed to receive updates again.
Blockers:
 

dave2s

New Member
Contributor
Week (7)8 summary (didn't number eval week earlier..should've)

During the seventh/eight week I introduced RunOrder type of DependencyConnection to provide a way to replace explicit usage of renderGraph.connect() in rendering modules in order to make sure the run order of certain nodes is correct even though there is no direct materialistic (fbo/bufferpair) connection (not sure if really needed). We inited a AdvancedRenderingRepo, i somewhat successfully moved ssao from BasicRendering to AdvancedRendering, drafted dummy nodes, introduced many many changes to connecting/reconnecting connection removing and replacing methods for Nodes and RenderGraph, expanded DependencyConnections' connectedConnection to HashMap of connectedConnections, which required some updating of existing code. I updated all nodes' constructors to create empty output connections which are filled with data with the late call of setDependency which is called after an ordered list has been created by the renderGraph. This method then is called by engine based ModuleRendering.java which was updated during previous week to again be used by rendering modules as their parent class. This class now has a preBegin() phase which requests renderGraph build from worldRenderer. During this phase all nodes from all modules get an ordered call of their postinit method which has all statechangerequest calls as well as data reading/setting calls on dependencies.

List of relevant files/repos:
Basically all commits between 16-22.July
BasicRendering module github repo
AdvancedRendering module github repo
DAG DependencyConnection draft #3680

Next on:
  • Maybe finish and use Dummy Nodes
  • Rest of advanced effects to AdvancedRendering
  • Ordering of rendering modules (NUI style?) without ordering the AKA has half the usability (choosing the strict approach of finding a Node by its exact nodeUri is useless if the renderingModules' initialiazion order is not specified)
Further on:
  • Create reusable compositing nodes and figure the naming logic
  • Effect replacement test
  • Throw out what we don't need
  • Test and verify what we do need.
PR: Updated DAG DependencyConnection draft #3680
BasicRenderingModule: BasicRendering PR
AdvancedRenderingModule: AdvancedRendering PR

Current problems
:
  • The transfer of advanced effects still requires the basic rendering module nodes to be aware of advanced effects which are taken out. Until the feature sets are separated by a shaderland sword wielding warrior. Then the way AdvancedRendering connects these nodes will have to be updated accordingly.
  • I'm tired all the time and regularly fall asleep on the job. Srsly random hours Vampcat's style're not working for me at all.
Blockers: eh? gooey?
 

dave2s

New Member
Contributor
Week 9 summary

During the ninth week
I moved Shadowmap, bloom and lightshaft nodes to AdvancedRendering module in addition to haze and ssao. At this stage the basic rendering nodes that use these still expect them, until they are further updated outside the scope of this gsoc, when dealing with splitting shader/nodes into more atomic units. I then updated node constructors with source module they were created in (I used this for shaders first, last month) as opposed to the previous state where the node's class file source module was reported (used in node's uri creation). I started drafting a solution to rendering module initialization ordering while coming up with some basic ui to tweak this in game. So far I came with an idea and some code of priority (positive int) for each module and sorting them. This is probably supposed to happen after modules are picked in game, before the game is loaded. Priorities will have some default values set in code, both implicitly or explicitly. These values should be later tweakable in game, which should effectively provide a way of reordering rendering modules.
Lastly I also added a new debugging command runnable from the game's console to enable rerouting certain node's output to certain node's input (FBO connections), so that you can for example reroute any node's FBO output to the final outputToScreenNode and see how it looks again. The former debugging command which used FBO uris to fetch them from centralized storage is now deprecated and does not work.

Picture of SSAO output redirected to outputToScreenNode in runtime

Picture of SSAO output redirected to outputToScreenNode in runtime

List of relevant files/repos:
Basically all commits between 22-28 July
BasicRendering module github repo
AdvancedRendering module github repo
DAG DependencyConnection draft #3680

Next on (preferably during the week of this post):
  • Ordering of rendering modules (+NUI)
  • Patch up rendergraph.replace() and provide an example of module replacing certain node, like skyNode, with a custom copy of sky node. This is a quick test, the only problem there was before was with transfering
  • Create reusable compositing nodes and figure the naming logic
  • I might transfer blur to advanced rendering as it is kind of standalone in node pipeline.
Further
  • Throw out what we don't need
  • Test and verify what we do need.
PR: Updated DAG DependencyConnection draft #3680
BasicRenderingModule: BasicRendering PR
AdvancedRenderingModule: AdvancedRendering PR

Current problems
:
  • I can't figure out a way of getting from Environment->modules/module ids to the module.someClass of which i expect certain parent at this stage at least. When creating a game I need to find a reasonable way to get to the module's files
Blockers:
 
Last edited:

dave2s

New Member
Contributor
Week 10th + 11th summary (29.7.-8.8.2019)

During the tenth and most of eleventh week
I started drafting a UI for configuring initialization priority for rendering modules. I introduced an attribute for expressing priority (1 highest priority), a new RenderingModuleSettingScreen .java/.ui, a new classes for managing rendering module's classes - RenderingModuleManager, RenderingModuleRegistry, ModuleRenderingSubsystem. Creating new subsystem manager and registry was need because of different contexts of game creating ang game loading/running phases. Subsystem manager and registry are created with the subsystem creation in terasologyEngine. Then the registry is filled either in UI of in WorldRendererImpl if it hasn't been already. (One thing to fix later when using the screen during runtime, would be to add a code snippet during initialization of the registry to check whether it's been already initialized and act accordignly.)
The rendering module's classes were then accomodated and no longer extend a BaseComponentSystem but are initialized sooner, in chosen order, by worldRendererImpl. This means that they are now free of Component system. The managers are obtainable from root context always.

This screen could be later extended by other screens for displaying the DAG setup and providing a way to dynamically change the settings. Not sure how this would be used for persistent changes though and is probably no longer an optional goal of this gsoc.


This feature took quite some time to implement also because I had trouble figuring out how to get the list of classes of the rendering modules. We finally settled for reflection and getting subclasses of ModuleRendering, which is also why the subsystem is named ModuleRenderingSubsystem. These names and functionalities are subject to change as the ModuleRendering is a subject of replacement in the future by extension of ModuleManagement alltogether. Also these renderingModule managing classes should be easier to merge later into the main module management classes as they mimick them a little.
I had also trouble understanding the UI and game creation process. Multiple contexts not many people knew how exactly worked, screens creating screens without game-creation-phase hooks and probably some more.
2542
(11th week image courtesy - working prototype - appearance subject to change)

How the UI works
The text window displays an ordered list of rendering modules' classes base module name (This must be updated as more classes can be called of one module) as they're going to be initialized. In the parentheses you can see its current priority. The priority gets updated with the slider on the right, but the order of the modules is calculated only after hitting the Update button. The dropdown combo-box lets you select the module initPriority of which you want to change.
updated Rendering Modules' Config:
2543
2544

List of relevant files/repos:
I finished this durint 11th week, not many updates during 10th week.
BasicRendering module github repo
AdvancedRendering module github repo
DAG DependencyConnection draft #3680

Next on (preferably during the week of this post):
  • Patch up rendergraph.replace() and provide an example of module replacing certain node, like skyNode, with a custom copy of sky node. This is a quick test, the only problem there was before was with transfering connections.
  • Create reusable compositing nodes and figure the naming logic
  • I might transfer blur to advanced rendering as it is kind of standalone in node pipeline.
Further
  • Throw out what we don't need
  • Test and verify what we do need.
PR: Updated DAG DependencyConnection draft #3680
BasicRenderingModule: BasicRendering PR
AdvancedRenderingModule: AdvancedRendering PR

Current problems
:

Blockers:
 
Last edited:
Top