Run Techniques

Cohort Analysis

An analytical technique that focuses on analysing the behaviour of a group of users over time, thereby surfacing insights about the experiences of those users, and what the organisation can do to improve those experiences. It is especially useful in benchmarking user engagement for a new service or feature or user journey refresh of an existing service.
This technique is only applicable where the average user interaction rate with the service is regular, e.g. at least monthly.
A typical trigger to use this technique may be the launch of a new service, new feature in an existing service, or major redesign of a key user journey.

There are 3 key anchors to employing this technique: Define the cohort group.
Define the cohort group - the trigger event to start tracking - e.g. deploy new web site, new user journey - and the attributes for inclusion - e.g. existing users, use of mobile channel.
Define the time box, the lagging period, that you want to run your analysis for. A typical value would be one month.
Define the period on which to run the analysis. This should be long enough to identify any trends from the new deployment. A typical value would be 3 months.
Configure the tracking on the web analytics tool in time for the deployment of a key new feature or service.
Use the web analytics reporting dashboard to visualise the performance of each cohort group- weekly, monthly etc.
Identify unexpected trends emerging and prioritise these for more indepth analysis.
In depth analysis may be undertaken via the data analytics tools or qualitatively via user interviews to identify potential root causes.

Logging and monitoring are two different processes that work together to provide a range of data points that help track the health and performance of your infrastructure. Monitoring uses application metrics to measure availability and manage performance. Logging creates a record of log events generated from applications, devices, or web servers that serves as a detailed record of occurrences within a system.

This is usually performed by a DevOps person or overarching application support team

Make sure that your monitoring tool supports your application's programming language to ensure compatibility and ease of use.
Log the right data - Log data needs to tell a succinct but complete story. Helpful log data generally includes actionable items, and includes information such as a timestamp, user IDs, session IDs, and resource-usage metrics.
Collecting a full range of applicable data enhances the information obtained from your monitoring tool.
Streamline your data by making it easier to search, index, and store be ensuring that it is structured, offering a more complete view as to what happened, and can provide your monitoring tool with unique identifiers such as which account ID experienced the error.
Identify application and system trends by applying statistical analysis to system events. Use the historical information provided by log data to determine averages that will make it easier to definitively identify anomalies, or to group event types in a way that allows for accurate comparisons.

(Reference ITIL 4 framework)