Suggested Telemetry System (collect, analyze and report)

Discussion in 'Suggestions' started by GabrielXia, Mar 11, 2017.

  1. GabrielXia

    GabrielXia New Member

    Related to this GSoC item : Issue reporting improvement
    And maybe related to the close item : Improve visibility of missing resource scenarios and general user reporting
    Basic Idea
    Our Telemetry System idea is to collect a bunch of players' telemetry recording (based on users' option), such as exception, os, version, video card, etc. This information will be gathered in some kind of telemetry storage where some analysis jobs are doing. Their results might help us better improve our Terasology :gooey:

    Here are some similar project :
    I think at first this telemetry system might specifically concentrate on Crash &Issue Reporting, lots of further developments could be made.

    Steal @Cervator's example :)
    1. A bunch of game clients encountered a bug with specific log "Error X, Y, Z occurred". We use the telemetry system and find out "Hey this says that a bug with log entry soandso happened x times, we should look at that". Or a bug happened in a specific OS several times, then we will say "hey there is a bug specific in this OS, we should subit an issue on Github"
    2. If we get a bunch of error report caused by the user enabling 200 different modules, this might suggest to us that we need disclaim to not enable 200 modules.
    Data collected
    • OS
    • Video card
    • Game version
    • Game mode (Singleplayer, modules enable, world generator, etc.)
    • log (especially warning, error)
    • Memory
    Potential telemetry framework
    • Snap. Just ran their example code, looks good. Although it seems to be a server metric tracker, I've a first contact with snap team, they said that it's doable and we need to write a plugin if we need some specific hooks. They are nice, too ;) I'm not familiar with snap framework yet, I need to do some small samples as soon as possible !
    • Google Analysis seems potential. It even has API for unity games, though I guess we don't use unity. Atom metric uses Google Analysis. There are also some similar on open source projects like piwik, but they are more for a web or a mobile application.
    • Thanks to @Rostyslav Zatserkovnyi the potential framework snowplow
    • Need some more suggestion here :sneaky: Thanks
    Need to do
    Lots of things in the issue should be more specific :
    • Issue automatically reporting, how the new system fit to game/CR
    • What will be the telemetry storage ? Where will be it runs on ?
    • What specific analysis job will be done in this system ?
    Future development
    Put some future development here for the motivation :thumbsup: The feedback from users is important !!! It should be made easy or even automatic, and safe. It'll help our game easier to develop and funnier. I think this telemetry system could do a lot of funny stuff :
    • We can get what kind of box users like the most, the most popular tools, their favorite module, etc.
    • The command the users like the most, then we might get ideas of what kinds of new command will be
    • Too much :rofl: , probably need to focus more on the specific stuff to do than the future

    Appreciate any feedback :ajsmug:
    • Informative Informative x 2
    Last edited: Mar 12, 2017
  2. Cervator

    Cervator Project Lead and Community Wizard Staff Member

    I got really excited about Snowplow after realizing we have a "man on the inside" so to say, hehe. I love it when work and hobby can coincide like that. And it looks really well put together, not that I've had time to try it out.

    Google Analytics and maybe Fabric (which Google bought out) has potential and probably would tie really well to Android (maybe first for Destination Sol, then one day for Terasology) but may be overly web or mobile centric. Plus probably there'll be some paranoid devs that'll prefer something more independent. Maybe I'm a bit too worried about conspiracy theories but both devs and users can get really mad about opt-in / opt-out tracking/telemetry.

    We can provide some server infrastructure for testing out whatever, just let me know.

    Try to keep in mind that it would be nice to be able to reuse as much of this as possible for more than just Terasology. Destination Sol and our other tools at first, maybe other games in the future. That's sort of a secondary consideration, I think for GSOC the focus should be on Terasology, but it is a good thing to architect for early.

    I don't know if our CrashReporter / IssueReporter would itself be replaced by some component in an existing framework. Probably it would just report to a different back-end. I don't know the frameworks well enough yet.
    • Like Like x 1
  3. Skaldarnar

    Skaldarnar Badges badges badges badges mushroom mushroom! Staff Member

    Probably very detailed analytics of resources (RAM, CPU, GPU usage) and framerate could also be interesting. Viewing graphs relating chunk processing to memory usage to framerate, etc. This could help us to focus on optimizations that have an impact for players. Also, loading times (of assets, world generation, ...) might be interesting. There are so many potential things to look at, we just have to start somewhere :)
    • Like Like x 1
    • Agree Agree x 1
  4. GabrielXia

    GabrielXia New Member

    Hey :)
    I did a simple test of snowplow tracker with the server set up in cloud, it worked. It's in my Github account repository, I wrote some docs there, thanks for any feedback :) Your guys' cool suggestion is very important to me. Any suggestion such as, what the next steps will be, whether is cool or not and so on will be important to me ~

    Here is some ideas about implementing the tracker in terasology :

    Two goals in mind :
    • The telemetry system should change the code in terasology as little as possible
    • This system could also be used for terasology mobile version as well as DestinationSol
    Implementation :
    • There will be an emitter which sends event to server, using @in Emitter emitter inject to other classes
    • The trackers will be used in several classes :
      • There might be a MetricSystem who extends BaseComponentSystem implements UpdateSubscriberSystem. While initializing it could track some basic information such as os, video card, etc. While updating it could track information like memory usage, framerate, etc.
      • The other specific event might be tracked in specific class
      • ??? Have few ideas about how to track exceptions. Maybe we could read log file, in that case, it might be good to write some code outside the engine
    • The other work will have to be done in the server
    A new idea about future development :
    The user might also want to see some data graph, I think with the telemetry system established, it might be easy to do this locally. E.g. we can set up a local host, and users can see these data graph in the browser

    • Like Like x 2
    • Informative Informative x 1
  5. Skaldarnar

    Skaldarnar Badges badges badges badges mushroom mushroom! Staff Member

    In any case, you have to make sure that it can be disabled completely if user's don't want to track such information. Some people are really sensible when it comes to tracking user data, and we should specify very clearly what kind of data we might observe, where it is sent, etc.

    I think we already have a metric system of sorts which is responsible for the on-screen debug info. Afaik you can also hook up to that system, or rather add new debug screens to the overlay. Maybe (parts of) that system can be re-used/extended for the purpose of the telemetry system?
    • Like Like x 2
    • Agree Agree x 1

Share This Page