News Realms Remove Ads - Go Premium! Sign In or Register

Bluetracker

Tracks Blizzard employees across various accounts.

Been investigating what metrics we can use to better determine what expansions are successful and which ones could be improved. Feedback is always going to be part of that process, but unbias feedback is harder than you'd think to acquire.

Been investigating what metrics we can use to better determine what expansions are successful and which ones could be improved. Feedback is always going to be part of that process, but unbias feedback is harder than you'd think to acquire.

  • Iksar
    Posted 4 years, 5 months ago (Source)
    Been investigating what metrics we can use to better determine what expansions are successful and which ones could be improved. Feedback is always going to be part of that process, but unbias feedback is harder than you'd think to acquire.

    • Iksar
      Posted 4 years, 5 months ago (Source)
      From a teamwide level, engagement metrics are easier to identify. I like looking at MAU over time sliced by broad player types. Broad player types are usually something like new, win-back, casual, and core users.

      • Iksar
        Posted 4 years, 5 months ago (Source)
        Overall MAU will tell you what is happening to the player population over time, and slicing by player type will tell you where the population increases/decreases are happening. A decline in new users will lead you to totally different conclusions than a decline in core users.

        • Iksar
          Posted 4 years, 5 months ago (Source)
          You can further slice to users by platform and by region, but from a general game health pov for a 7+ year established game, WAU or MAU graphs over the last 3-6 months should tell you a clear story.

          • Iksar
            Posted 4 years, 5 months ago (Source)
            BUT.... What about for your team, or the particular thing you work on? In this case, expansion design. MAU is not a great indicator of expansion success because the game is loaded with other variables that affect that metric.

            • Iksar
              Posted 4 years, 5 months ago (Source)
              Session length? Day 7-15-30 retention? Probably not. The population of players is always changing. If we lost a great deal of new players the previous expansion, things like session length and retention will go up because the playerbase is now more core-user heavy.

              • Iksar
                Posted 4 years, 5 months ago (Source)
                Business teams have been using revenue metrics to identify problems and solutions for many years but designers across the industry tend to have an allergic reaction to using KPIs to better understand problems.

                • Iksar
                  Posted 4 years, 5 months ago (Source)
                  I say, embrace the data! Feedback will always be there for you, but using feedback as your north star metric will alienate all the players who aren't loud enough to speak up.

                  • Iksar
                    Posted 4 years, 5 months ago (Source)
                    For any other devs out there, are there any particular metrics you like to look at for sub-teams you find helpful? Do you share these metrics with your team or only at leadership levels? Do you ever define success or failure through a metric, and if not, how do you define it?

                    • janfindl
                      Posted 4 years, 5 months ago (Source)
                      @IksarHS To me there’s two goals of the design. 1. Meeting the principles of the game’s fundamentals - gameplay diversity, low polarisation, challenging decisions, engaging theme. 2. The community engagement - MAUs, social metrics, revenue.

                      • Iksar
                        Posted 4 years, 5 months ago (Source)
                        @janfindl Honest question: If players are having fun, playing at a high rate, and monetizing at a high level (all from goal #2)….. why does goal #1 matter?

                        • RegisKillbin
                          Posted 4 years, 5 months ago (Source)
                          @IksarHS @janfindl This isn't a perspective specific to game dev, but if I dig deep into my brain back to my I/O psych consulting days, we often saw orgs fall into the trap of making the outcome metrics the goals instead of well... the outcomes. 1/

                          • RegisKillbin
                            Posted 4 years, 5 months ago (Source)
                            @IksarHS @janfindl Which can be fine if things align perfectly, but you also get into situations where when you reach specifically for outcomes instead of the actual core tenets of your org, you find gaming the measures is a great way to represent success while actually losing something important.

                            • RegisKillbin
                              Posted 4 years, 5 months ago (Source)
                              @IksarHS @janfindl You see this cited a lot with standardized tests in schools. Teachers teach kids to beat the tests, not to actually internalize how to learn. We focus on memorization because it tests well, not actual deep learning.

                              • RegisKillbin
                                Posted 4 years, 5 months ago (Source)
                                @IksarHS @janfindl You see this in reviews too. People try to coax good review scores instead of building a better product to naturally earn those reviews.

                                • RegisKillbin
                                  Posted 4 years, 5 months ago (Source)
                                  @IksarHS @janfindl Now of course if you think your outcome metrics are REALLY good at measuring what you think they are, this might be hard to game for success. That's awesome.

                                  • RegisKillbin
                                    Posted 4 years, 5 months ago (Source)
                                    @IksarHS @janfindl In gaming sometimes though (maybe not in Hearthstone, I sure don't know the data), my casual eye has seen stuff to encourage logins for the sake of logins, not to get actual deep engagement. That feels like a way to simulate engagement for metrics' sake.

                                    • RegisKillbin
                                      Posted 4 years, 5 months ago (Source)
                                      @IksarHS @janfindl So I guess my outsider questions and once-professional insight would be to take a close look at those KPIs. Are players playing because they are having fun or because they're stuck to a rewards loop? Is revenue up for all groups or in aggregate? Those sorta things.

                                      • RegisKillbin
                                        Posted 4 years, 5 months ago (Source)
                                        @IksarHS @janfindl Those aren't loaded questions by the way, just how I'd think about it. I know the HS team has tons of more experienced data folks than I ever dreamed to be, just wanted to get some thoughts down. Thanks for your time whoever read.

                                        • RegisKillbin
                                          Posted 4 years, 5 months ago (Source)
                                          @IksarHS @janfindl Oh and in the effort of self reflection, I try to avoid using metrics as the goal. Because I can force high CTRs with clickbait, and watch times by burying the lede. That makes for good short term results. But am I creating a community? Am I building loyalty?

                                          • RegisKillbin
                                            Posted 4 years, 5 months ago (Source)
                                            @IksarHS @janfindl So for me, my goals are to make sure every video has a big memorable moment, serves some educational value, or facilitates discussion in some way. If I aim for views, I can cheat it. If I aim for quality, the views should come.

                                            • Iksar
                                              Posted 4 years, 5 months ago (Source)
                                              @RegisKillbin @janfindl I respect the opinions but I don't agree with this mindset. This essentially means setting your own subjective goals and judging your success based on whether you think you met them. I don't think this is a healthy way to judge success if what you are creating is for others.

                                              • Iksar
                                                Posted 4 years, 5 months ago (Source)
                                                @RegisKillbin @janfindl Fwiw I think if you are innovating or exploring new ideas, you should be much less focused on metrics. Innovation often comes with slow adoption, and that’s fine.

                                              • RegisKillbin
                                                Posted 4 years, 5 months ago (Source)
                                                @IksarHS @janfindl As for the bigger discussion, I am curious if we pull this thinking back to internal goals and iteration pre-release, when there are no external metrics to guide you. Aren't subjective goals set and success measured by your own expecations for quality?

                                                • RegisKillbin
                                                  Posted 4 years, 5 months ago (Source)
                                                  @IksarHS @janfindl Perhaps I am clouding my logic by an "artistic aire" but I guess part of me believes that creatives need the vision and subjective goals to create something great, and pushing towards those should be the aim. Maybe too easy to say without a 100+ person team relying on results.

                                                  • Iksar
                                                    Posted 4 years, 5 months ago (Source)
                                                    @RegisKillbin @janfindl The presence of analytics does not have to mean the absence of creativity. You can make a KPI your goal and still reach it through creativity. You can still be proud of your work when you don’t reach your goal.

                                                    • Iksar
                                                      Posted 4 years, 5 months ago (Source)
                                                      @RegisKillbin @janfindl What I don’t think we should do is decide for ourselves what makes a great expansion, make them, then therefore deem all expansions great because they met our personal creative goals.

                                              • RegisKillbin
                                                Posted 4 years, 5 months ago (Source)
                                                @IksarHS @janfindl I am not sure if this was a response to the "me personally" part or the thread in general, but for me specifically, I have the luxury/horror of actually BEING the product in many ways. So I almost have to be my own guiding light, to maintain my own identity.

                                                • RegisKillbin
                                                  Posted 4 years, 5 months ago (Source)
                                                  @IksarHS @janfindl To give a really obvious example here, I know I could get 5x more views by being mad all the time and riling up angry gamers. The few times I've dipped towards outrage, the views come, in droves. Is that success? From a growth and revenue standpoint, yes, certainly. For me, eh.

                                                  • Iksar
                                                    Posted 4 years, 5 months ago (Source)
                                                    @RegisKillbin @janfindl I think in that case we’re moving the goalposts. If success was driven by my own personal desire for happiness I’d probably be turning hearthstone into a farming rogue like.

                                                    • Iksar
                                                      Posted 4 years, 5 months ago (Source)
                                                      @RegisKillbin @janfindl I’m not an influencer I imagine it to be incredibly difficult. Even in that example, though, it sounds like there still might be a long term kpi goal for viewership.

                                                      • Iksar
                                                        Posted 4 years, 5 months ago (Source)
                                                        @RegisKillbin @janfindl But maybe that short term growth path isn’t right for you because it‘a either not sustainable long term or you just wouldn’t enjoy it anymore.

                    • aqword2
                      Posted 4 years, 5 months ago (Source)
                      @IksarHS I'm glad you asked! I live and breathe analytics and I live and breathe Hearthstone. So this question is right up my alley. (1/9)

                      • aqword2
                        Posted 4 years, 5 months ago (Source)
                        @IksarHS Some of the best advice I've ever heard for metrics is to use ratios, not just numbers. Monthly active users is a number, and tends to just go up over time, so it doesn't give a clear signal when things are getting better or worse. (2/9)

                        • aqword2
                          Posted 4 years, 5 months ago (Source)
                          @IksarHS Examples of ratios would be the percent of people active now who were active at the same time in the previous expansion cycle. Or the ratio of people who logged in twice to those who logged in only once. These give clearer indicators of usage over time. (3/9)

                          • aqword2
                            Posted 4 years, 5 months ago (Source)
                            @IksarHS (It's worth noting that any metrics based on the number of users tend to indicate usage rather than engagement. And usage is largely influenced by marketing. Engagement metrics are better for measuring product design, which seems to be what you're interested in.) (4/9)

                            • aqword2
                              Posted 4 years, 5 months ago (Source)
                              @IksarHS You mention session length, which could work well as an engagement metric if you benchmark it across a consistent subset of users. (5/9)

                              • aqword2
                                Posted 4 years, 5 months ago (Source)
                                @IksarHS If you identify a session duration below which people tend not to return and above which they're likely to return, then a good KPI is the percent of sessions longer than that duration. (6/9)

                                • aqword2
                                  Posted 4 years, 5 months ago (Source)
                                  @IksarHS Other good metrics for engagement could be the number of games per session (which is a way of looking at session duration that controls for the duration of individual games changing over time) or the percent of sessions with monetary conversions or the average number of...

                                  • aqword2
                                    Posted 4 years, 5 months ago (Source)
                                    @IksarHS different decks played by each user per month or average number of games per deck. (7/9)

                                    • aqword2
                                      Posted 4 years, 5 months ago (Source)
                                      @IksarHS I work in a non-game web context, so how we use KPIs might be a bit different, but yes, we include leadership in defining which KPIs to track (and then later use KPIs as a regular channel to communicate updates to leadership). (8/9)

                                      • aqword2
                                        Posted 4 years, 5 months ago (Source)
                                        @IksarHS We set goal KPIs and then benchmark against them to determine if/when we need to intervene in the strategic direction of a product, plotting the KPIs on a timeline annotated with major feature releases. (9/9)

                                        • Iksar
                                          Posted 4 years, 5 months ago (Source)
                                          @aqword2 Thanks for the insight! I think we’re very aligned :)

                                          • Iksar
                                            Posted 4 years, 5 months ago (Source)
                                            @aqword2 [some engagement metric] on a consistent subset of users has been a path I’ve been on lately. Seems like at the very least we could do a better job determining success for that group.

                    • charlieplante80
                      Posted 4 years, 5 months ago (Source)
                      @IksarHS Interesting thread. As a marketing professional my business focuses on helping businesses deal with these exact types of issues. Creating quantifying metrics in aggregate data that free flows can be daunting! Usually it’s finding subtle commonalities that string it together!

                      • Iksar
                        Posted 4 years, 5 months ago (Source)
                        @charlieplante80 Agree, it’s hard to have a nuanced conversation on Twitter but i’m certainly not saying we’ll find one magic metric that will completely define our success. Can we find one that will help us improve over time, though? I think the answer is yes.

                  • LorindaGames
                    Posted 4 years, 5 months ago (Source)
                    @IksarHS What people are doing is definitely a better way to assess what is going on than what people are saying. That's not just applicable to gaming or business, but all of life.

                    • Iksar
                      Posted 4 years, 5 months ago (Source)
                      @LorindaGames You’d be surprised how much of the game industry isn’t aligned on that

                      • Iksar
                        Posted 4 years, 5 months ago (Source)
                        @LorindaGames Game industry reminds me of this movie, moneyball. 95% of the baseball scouted players based on feel and intuition and 5% based on advanced metrics. One way was far superior, but because the changeover would result in people of power losing their jobs, there was mass resistance

                        • LorindaGames
                          Posted 4 years, 5 months ago (Source)
                          @IksarHS Moneyball is one of the best books ever written. I used to large numbers of sports simulations, and much of it was derived from the stuff that is highlighted there. It is so good because it's such an extreme example and because of the people stories involved, makes it accessible

                          • Iksar
                            Posted 4 years, 5 months ago (Source)
                            @LorindaGames Advanced sports metrics used to be a passion of mine. Interesting to see them more broadly applied today in complex sports like football and basketball. Still lots of room for innovation.

                  • derajn
                    Posted 4 years, 5 months ago (Source)
                    @IksarHS I do say embrace the data, but data needs context even when segmented, and that context isn't always available in data, so feedback is one of those ways to narrow in on the relevant context.

                    • Iksar
                      Posted 4 years, 5 months ago (Source)
                      @derajn Slippery slope. Using context to adjust how you think about data is too often used as a way to ignore things you don’t like and accept things you already believe.
Tweet