Analysis of twitter's group chat feature

Why Twitter launched this feature?

​Twitter’s mission is to give everyone the power to create and share ideas and information instantly, without barriers. Twitter had been doing well giving users the power to create and share publicly, and the new Group DM feature could promote more private conversations among users.
 
There are several ways of using Group DM:

• Personal use:
  1. People of similar interests can collect and share tweets and information with each other privately, easily, and quickly.
  2. Introduction: people can add someone to the Group DM that the other existing users in the chat are not currently following. Through encouraging users’ contacts to invite new people to the group chat users will expand their network of trusted peers on Twitter.
• Business use:
  1. Team discussion: people in an organization can collect and share tweets and information with their colleagues privately, easily, and quickly.
  2. Manage multiple Twitter accounts: Some companies would have multiple Twitter accounts, or each child company of the same parent company has a Twitter account. Group DM gives the team managing digital media strategy the abilities to communicate important information in an easy and quick way.

 
Based on above usage, Group DM can promote more tweets created and shared privately and publicly, which forms a virtuous loop and makes Twitter more self-perpetuating. Below is a chart to illustrate:

​​To determine if this is a successful feature or not, they need to have both successful signals and failure signals. For this feature, the most important criteria to define the success is how much it can promote users to tweet more publicly and privately. However, this feature might promote private conversation at the cost of public tweet, or makes users feel spammed. These should be important considerations when evaluating if the feature can provide the expected results.

• Signal of success:
  1. Users tweet publicly more frequently
  2. Users send private messages more frequently
• Signal of failure:
  1. Users cannot complete the Group DMs smoothly
  2. Users feel spammed by Group DMs and become less  active on Group DMs and Twitter in general, or change the settings from open/public to very private
  3. Users send private messages more frequently but tweet less publicly

 
Based on the signals, they can set up metrics and conduct A/B testing to compare these metrics in control group and feature group. To see how this feature affects users of different cohorts, it’s helpful to track new users and existing users separately, and track public tweets and private messages separately:

• Engagement:
  1. Number of users (new and existing respectively) tweeting publicly per day
  2. Number of public tweets per user (new and existing respectively) per day
  3. Percentage of DAU (new and existing respectively) tweeting publicly
  4. Number of users (new and existing respectively) sending messages per day
  5. Number of messages sent by per user (new and existing respectively) per day
  6. Percentage of DAU (new and existing respectively) sending messages
• Retention:
  1. Follow above metrics and see how the cohort performance changes over time, e.g. in one week, in two weeks, in three weeks, in four weeks
• Task success:
  1. Time taken to initiated a Group DM
  2. Time taken to reply a Group DM
  3. Percentage of complete (user clicks “Share via Direct Message” and finishes the task) Group DM

 
To be able to ship the Group DM feature, Twitter needs to make sure that the metrics from Feature Group outperforms the metrics from Control Group, so that signals of success are obvious for Feature Group, while the signals of failure are not significant enough to indicate a concern.

What if the feature fails?

In case the feature had failed, I would identify the type of issues, conduct analysis and research to figure out the reasons, and provide solutions for rectification. Below is detailed explanation of each step.
 
There could be different reasons why the Group DM fails:

• Users do not use Group DM actively:
  1. Users don’t realize Group DM
  2. Users couldn’t find Group DM easily
  3. Users don’t know how to use Group DM
  4. Users cannot complete the Group DM smoothly
  5. Users do not like Group DM
  6. Users feel spammed by Group DM and become less  active on Group DMs and Twitter in general, or change the settings from open/public to very private
• Users use Group DM actively, but using Group DM cannibalize public communication on Twitter:
  1. Users share more privately but less publicly
  2. Users visit Twitter more often but are more passive while on the site

 
What the failure would tell me:

• There might be code issues:
  1. Unexpected results prevent users from using the feature normally
• There might be design issues:
  1. Current Group DM’s design is not intuitive enough
     • Users might not realize Group DM
     • Users might not be able to find Group DM
     • Users might not understand how to use Group DM feature
     • Users might not find Group DM feature valuable
• There might be strategic issues:
  1. Turn out this feature cannot promote more active public or private sharing of information among users
     • Neither public nor private sharing is promoted
     • One of the two is promoted at the cost of the other
  2. Users even feel annoyed by this feature

 
To solve the problem, we can find ways to tackle each possible issue:

• Code issues:
  1. Investigate the issue:
     • Look into metrics based on cohorts of different times (two months ago, last month, current month), and to figure out if this is an one time issue or all time issue
     • Review the product specs to see if there is any miscommunication between PM and engineering team
     • Review the code
  2. Solve the issue:
     • If it’s a product specs issue:
     • Re-write product specs and communicate with engineering team in person to make sure that everyone is on the same page
     • If it’s a bug:
     • Fix the bugs
     • Design testing cases (one based on normal user story, and two to three based on edgy stories) and test
• Design issues:
  1. Investigate the issue:
     • Look into metrics to understand how users behave
     • Invite some users and observe how users use it
  2. Solve the issue:
     • If it’s users couldn’t realize/find/understand the feature:
     • Re-design and conduct a heuristic evaluation among different teams before making the real change
     • If it’s users don’t think it valuable:
     • Refer below strategic issues section for detailed explanation
• Strategic issues:
  1. Investigate the issue:
     • Understand the competition landscape:
     • Social media competes on users’ attention and time, so finding problems cannot be restricted to internal only, especially for strategic level issue.
     • Competition is more than other social media, e.g. Facebook, Snapchat, Instagram. Since Group DM has instant message function, so apps such as Whatsapp, Facebook Messenger, LinkedIn in-app message, GroupMe, GChat, Google Hangout, and Skype, are all good analysis targets.
     • Meanwhile, we can also enlarge the competition analysis to include business communication tools such as Yammer, Slack, and HipChat.
     • Last but not least, any traditional way of sharing message that occupies the time of users and forms its own channel is worth studying, e.g. email, in person or virtual meetings.
     • Create a survey:
     • To understand how users feel about Group DMs (benefits and troubles) and how much control they want to have regarding which message they want to receive and want others to receive.
     • The survey better be sent via email or when users log into Twitter the next time, instead of when users click Group DMs, as users who still click Group DM are not the ones who don’t like Group DM so don’t even bother to click it. However, understanding users who don’t like Group DM and don’t click it is critical to get to know the reasons why Group DM fails.
  2. Solve the issue:
     • If the hypothesis is that the re-design can improve the metrics, then we can re-design the feature, A/B test the new design, and review the results.
     • If the re-design cannot be further justified, then we could consider killing the feature.

What additional feature can Twitter build if the Group Chat is successful, and why build it?

The successful outcome confirms that Group DM can promote users to share more actively on Twitter publicly and privately.
 
Right now, when three dots>>Share via Group DM under each Tweet, or Messages on Top Nav Bar is clicked, a modal will be popped up for group chat. In current design, when users are chatting privately in Group DM, users cannot view public tweets; when users are viewing public tweets, they cannot send private message.

• These two actions—browsing public tweets and chatting privately, can actually become complementary to each other:
• Users might want to tell their friends privately about a tweet, and while waiting for their friends’ reply, users want to keep browsing other tweets.
• Users might want to browse more interesting tweets to share in the group chat while sharing with their friends privately

 
To make browsing public tweets and chatting privately really complementary, we can give Group DM its own section on the Home page, instead of showing up in a modal or a separate Message page.
 
Below is my proposal of the change:

• Add a Messages section at the bottom of right side bar. Putting it at the bottom won’t affect current display of “Trends” and “Who to follow”, so as not to affect the way users adding more followers and discovering interesting topics. Though at the bottom, it is still very easy to access and use, compared to previously in a modal or a separate page.

• If there are more than 5 messages, then a scroll bar will show up at the right side. In this way, the Home page won’t be dominated by a long list of group chat. In case a user has a long list of chat groups, scroll bar can let him/her review the list easily.
  

• If create new message icon or any existing conversation is clicked, then the Group DMs modal will be popped up with below modifications:
  • Only show the modal at the bottom of the page, instead of occupying the whole page. Once a message is sent, users’ attention can be directed back to the Home page, while keeping track of the status of their conversations through the right side bar.
  • Add title section to the modal so that users can create a name for this group chat and refer back to it easily in the future.
  • Make adding users, typing information, and sending all in one step, instead of in separate steps, so that users can share more easily and quickly. To make them in one step, I move GIF and picture icon next to emoji, right below the typing box.
    
  • Add a Cancel button next to Send button so that users don’t need to move mouse up to “x” to cancel the action. If user really needs to share, the Cancel button won’t make them quit. However, if user really doesn’t feel the need to share now, the Cancel button would provide more convenience to user. In this way, it can also make sure that the information shared by users is of high quality, and can in turn engage users much more.

I assume this way can promote more public tweets while chatting privately, and more private chat while tweeting publicly, and in turn promote more sharing among users both publicly and privately on Twitter. However, this feature might promote public tweets at the cost of private conversation. These should be important considerations when evaluating if the feature can provide the expected results.

• Signal of success:

1. Users tweet publicly more frequently
2. Users send private messages more frequently

• Signal of failure:

1. Users cannot complete the Group DMs smoothly
2. Users tweet publicly more but send private messages less

 
The metrics to evaluate the success would be similar to what we have when first introducing the Group DM feature:

• Engagement:

1. Number of users (new and existing respectively) tweeting publicly per day
2. Number of public tweets per user (new and existing respectively) per day
3. Percentage of DAU (new and existing respectively) tweeting publicly
4. Number of users (new and existing respectively) sending messages per day
5. Number of messages sent by per user (new and existing respectively) per day
6. Percentage of DAU (new and existing respectively) sending messages

• Retention:

1. Follow above metrics and see how the cohort performance changes over time, e.g. in one week, in two weeks, in three weeks, in four weeks

• Task success:

1. Time taken to initiated a Group DM
2. Time taken to reply a Group DM
3. Percentage of complete (user clicks “Share via Direct Message” and finishes the task) Group DM