Web analytics for the New Zealand government
Index
- Introduction
- What: Understanding the basics
- Terminology, definitions
- Why: Evaluation and reporting
- Selecting success criteria
- Regular monitoring
- Government website functions
- Problems/opportunities
- Reporting
- How: Techniques and tools
- Techniques (the minimum)
- Qualitative methods (the next step)
- Experimentation and testing (if you’re keen)
- Recommended reading
- Relevant legislation
- References
- Appendices
Introduction
"Web analytics is the assessment of a variety of data, including Web traffic, Web-based transactions, Web server performance, usability studies, user submitted information and related sources to help create a generalized understanding of the visitor experience online.”
Web Analytics Demystified: A Marketer’s Guide to Understanding How Your Web Site Affects Your Business, Eric T Peterson, 2004
A website is built with a purpose in mind – you need to know how it’s doing in order to measure its success. A programme of regular evaluation will not only determine whether a website is delivering value, but also provide a basis for proposing development, support and improvement.
This document will outline the basics, describe how to set up a programme, put this in a New Zealand government perspective, and introduce you to some techniques and tools.
Unless otherwise stated, analytics terms are used as defined in the Web Analytics Association Web Analytics Definitions.
What: Understanding the basics
The following definitions have been taken from the Web Analytics Association Web Analytics Definitions. For more details on what the terms mean, and what you need to know about using them, please refer to the latest version of the document (PDF, 111 KB) on their website.
Metrics
When talking about measurement in web analytics, the following terms will come up.
A count is the most basic unit of measure; a single number. This covers anything you measure that gives you a numerical value.
From counts, we move on to ratios; a count divided by a count, for example, page views per visit.
A KPI, or Key Performance Indicator, relates to the goals and strategy of your organisation, and can be either a count or a ratio.
The basics
For what defines use of your website, these are several basic terms.
A page is most obviously a web page, but may also include downloads, PDFs, and media files. Your analytics tool is likely to allow you to define what is covered by “page”.
Once you’ve defined what a page is, you can measure page views; the number of times a page was looked at. Status code pages, such as “404 page not found”, should not be included in page views. Check your analytics tool to see what gets counted.
A user viewing one or more pages on your website is a visit or session. The session ends either when the user leaves the site, or times out (often after 30 minutes, but it’s likely your tool will allow you to set this).
A unique visitor is an individual, taking out other “visitors” such as spiders and robots, who views one or more pages of your website. A unique visitor is always associated with a time period, eg, x unique visitors per day/week/month. You will need to clarify how your tool tracks and measures a unique visitor – any site with login will have an accurate count, otherwise it’s usually done with a cookie. Also related to this are:
- new visitor – a first-time visitor to your website. When compared with return visitor counts, useful for looking at user affinity for the site, or how use changes with familiarity.
- repeat visitor – a unique visitor, with more than one visit in the time period.
- return visitor – a unique visitor, who has visited in a previous reporting period.
The visit
Where the user went, how they got there, and how they behaved once they got there.
As already established, users of your site view pages – in the context of the visit, these can be further broken down to:
- entry page - the first page of a user’s visit.
- landing page - usually associated with a marketing campaign, you may direct users to this page(s) via online ads or specific keywords.
- exit page – the last page of your website that was viewed before the visit ended.
Page views per visit is the number of page views (in a reporting period) divided by the number of visits in that same period.
The visit duration is usually calculated by subtracting the timestamp for the user entry onto your website, from their exit timestamp.
To determine how people found your website, you’ll look at the referrer. This will be the URL of the page the user came from, contained in a field when the request for the page is sent. In the case of the user entering the URL directly, or using their bookmarks, you may get a “no referrer” or empty value. A useful breakdown of referrer types may include:
- internal referrer – a referrer URL from within your own website. Care needs to be taken defining this, as it may include eg, external transaction engines.
- external referrer – a referrer URL from outside of your website.
- search referrer – a URL which indicates that the referral was the result of a search function, eg, Google. It may include the term which generated the search.
Another measure usually associated with advertising activities, click-through is the number of times a user clicked on a link.
The click-through rate or ratio is the number of click-throughs for a specific link divided by the number of times the link was viewed.
Page exit ratio refers to the number of exits from a page, divided by the total number of views of that page.
A visit that consists of one page – the entry and exit pages are the same – regardless of the number of times the page was viewed, is a single-page visit.
Where the visit is a view of one page, once, it is a single page view visit, or a bounce.
The bounce rate is the single page view visits divided by entry pages.
Why: Evaluation and reporting
The information gathered from web analytics activities is only useful if it is monitored regularly and subjected to further analysis. A number of activities need to occur as part of a website evaluation programme:
- Select success criteria by which to evaluate the performance of the site
- Select other metrics to monitor to support business as usual activities
- Obtain the tools required for regular monitoring and analysis
- Monitor and analyse to identify areas for incremental change (can be fixed by business as usual activities using existing resource) and major change (a specific risk or opportunity requiring additional resource)
- Report to stakeholders such as business units and senior management on performance against success criteria
Selecting success criteria
Success criteria or key performance indicators (KPIs) are metrics that relate directly to how the site is delivering against organisational goals.Many websites suffer from poorly defined or no success criteria, making it impossible to prove whether the website is delivering value. KPIs should answer the question 'why is it that we spend all this money on website.govt.nz?'. If you don't have a current web strategy or agreed success criteria for your website, step one is to go back to your organisation's statement of intent to identify the areas where your website could be expected to contribute.
Good success criteria have the following attributes:
- They are meaningful to the organisation and clearly indicate progress (or lack of progress) in delivering on organisational goals
- They are reliable and can be accurately measured over time
- They are regularly reported and always accompanied by additional analysis that explains how any changes relate to action taken (or not taken) on the site
KPIs are designed to be reported to senior management; they should be focussed on overall site performance, and may include information that is important to key stakeholders in your organisation. They should not include all of the additional detailed information the web team will want to record for the purpose of running the site on a day to day basis.
The following section provides samples of success criteria - select or adapt a selection that will provide measurable indicators of progress in the areas where your website contributes to your organisational goals.
Government website functions
While every site is different, there are often common elements (ie, transactions, social media, etc) which have a particular type of delivery and therefore lend themselves to corresponding types of measurement. We have identified six core site functions and provided some suggested success criteria for each of them - this list is not comprehensive and only is intended as a starting point. Most New Zealand government sites are hybrids of one or more of these types. Don't feel that your success criteria have to account for everything that happens on your website. When it comes to reporting, identify what really matters for your organisation and focus on communicating that.
| Function | Attributes | Suggested Measurements |
|---|---|---|
|
All sites |
There is one success criteria that can safely be applied across all types of websites. |
Website satisfaction - website satisfaction surveys can be used for benchmarking user satisfaction across sites and to measure website improvement over time. Web surveys can also help you identify when your site needs radical improvement. |
|
Advice Example(s): Sorted |
The goal is to provide easy access to the appropriate information, eg,
|
|
|
Community Example(s): |
Provides the ability to participate online, eg,
|
Success is dependent on the level of user engagement over time, and having a solid audience of regular users. Evaluate user engagement in relation to the size of the target audience and the rate of visitors to contributors (around 1-10% of visitors are likely to be contributors - see references).
|
|
Corporate
|
Provides information about the organisation, including:
|
It is unlikely that this area will need specific success criteria for reporting to senior management. However, you might have information that should be reported to a subset of your stakeholders, eg,
|
|
Directory Example(s): |
Provides extensive links to other websites (aka 'portals'). |
The measure of a successful directory is the amount of traffic it directs to other sites.
|
|
Reference Example(s):
|
Provides access to government data for researchers, including:
|
|
|
Transaction Example(s):
|
Provides the ability to complete transactions with government online:
|
Success will be determined by completion of the process by a user.
|
What is meaningful may change over time; don’t be scared to change what you measure, but be careful not to lose historical data.
Regular monitoring
The web team will also need to monitor other metrics in order to make and evaluate incremental improvements to the site (eg, search data will provide information to support content management activity).
There is no point in collecting web analytics data if it does not result in action to improve your website. There is an overwhelming amount of information available, so it is essential to be selective. Allocate time for the web team to perform ad hoc analysis on aspects of your site, looking for trends and problems. Consider compiling a website research plan that outlines a systematic approach to analysing each part of your site in turn.
Some suggested areas to monitor include:
| Monitor | Why | Useful for site functions |
|---|---|---|
|
Changes in most popular pages/sections |
|
All |
|
Impact of promotional activities |
|
All (where/when promotional activities are being run) |
|
Changes in search behaviour
|
|
All |
|
Task completion |
|
Advice Transaction |
|
Site overlay |
|
All |
|
Downloads
|
|
Advice Corporate Reference |
|
Site tools
|
|
Advice |
|
Drop-off points |
|
All |
|
Logins/sign-ups |
|
Community |
|
Referrals |
|
All |
|
Browser types |
|
All |
|
“No match” search results |
|
All |
|
Segmentation of searchers - on which page visitors used what search term |
|
All |
|
Click-through
|
|
All |
|
Customer feedback
|
|
All |
Problems/opportunities
If the data you get from your regular monitoring indicates a problem, what happens next? Don’t panic, this could be an opportunity…
Further analysis of the problem areas will need to occur:
- define the problem: design? technical? content?
- look at search behaviour
- natural user cycle: if your regular monitoring has been going long enough, you may see a “natural user cycle” – does it always happen at this time of year?
- review wider: market trends, other government sites (netratings), web stats provider
Further research will be vital to support a business case for change and/or resource. Look at the sections on qualitative methods and experimentation and testing for more information.
Reporting
Your website success criteria will be the basis of reporting to management on the success of the site. Stakeholders (business units and senior management) will generally require two types of information:
- regular reports providing evidence of the site's contribution towards organisational goals (success criteria)
- information about risks and opportunities relating to your site's contribution towards organisational goals (memos, business cases, etc).
And may also include:
- suggested actions/improvements
- evaluation of the outcome/outputs of a new or re-designed website/section
- reporting on the success of specific promotional activities (for agencies doing public campaigns)
- information for auditors (eg, financial year summary requested; capex/opex for website).
Stakeholders will not thank you for providing more information than required. They can also not be expected to make informed decisions from raw data or to understand the difference between 'hits' and 'return visitors' unless you tell them. It is the role of the web team to select meaningful data to report and to provide a compelling picture of how well the site is contributing to business objectives (for inspiration see references 2 and 3).
You may need to download data from your web analytics package (or other sources) into spreadsheeting software in order to create the graphs you need. You should make sure your reports include the following:
- Context - in order to show trends you need to graph data over time. Select the appropriate frequency (eg, monthly) and term (eg, annual) of data to ensure that trends are shown in context (is that spike in the graph special or is it something seasonal you see at this time every year?)
- Commentary - any trends (positive or negative) or abnormalities in the data should be explained in real terms (eg, the spike in new visitors is due to the $150,000 advertising campaign we ran this month and represents an ad spend of $1.50 per new visitor).
- Caveats - web analytics can be notoriously unreliable. If you suspect any issues with the data (eg, your repeat visitors have doubled but you can't see any supporting evidence) ensure that this is prominently noted in your report, to avoid embarrassing data misinterpretations.
- Clarity - an attractive, easy to understand report is a report that gets noticed. If you're not going to get to present your report in person then it needs to be able to convey the message without your assistance.
How: Techniques and tools
Techniques (the minimum)
The data for your regular monitoring will come mainly from the following sources:
| What & how | Advantages | But keep in mind |
|---|---|---|
| Server-based Web logs : the most common source of data about website usage. Created by the website servers, they record information each time a page is requested. An analytics tool is then (usually) used to produce reports.
Used by: NetTracker, AWStats |
|
|
|
Client-based Tags : a method commonly used by web analytics vendors. Data is collected by use of a piece of JavaScript code in each of the website’s pages – when the page is requested, the JavaScript runs, and sends back information. Used by: Google Analytics, Urchin, Neilsen |
|
|
There are other techniques that you may come across, including:
- Packet-sniffing – involves software or hardware that all traffic to and from your server passes through first.
- Server/URL redirection – allows you to log which link visitors left by.
Above all, be aware that there is no one ideal technique – each data collection method is best used in conjunction with another. This is also true of web analytics tools, where any one tool is unlikely to meet all your requirements. And there are other benefits to having more than one tool; the ability to compare trends, knowing when one is broken. Just keep in mind that for each tool the data definitions/capture methods will differ, so the data won’t match.
Qualitative methods (the next step)
Your programme of regular analysis and reporting will help you to gauge the performance of your website, and highlight areas of interest or concern. Further research will then be needed to see if there are problems, where and what they may be. The lifecycle of any website will also include further development. The following methods take you beyond the regular analysis.
Surveys
Surveys can collect data from a large number of users, reasonably quickly, and for a relatively low cost. They can be used to gauge user experience across an entire site, or just specific pages/functions. Surveys complement the data gathered by quantitative methods, and open-ended questions provide insight into user thinking and behaviour.
Done regularly, benchmarking surveys can be a good source of continuous data, revealing trends and changes. Information gathered will help you to understand use and perception of the website, and can be used to influence future development and direction.
Surveys can also be used to further research specific areas, identified by your regular analysis, helping to pinpoint where a problem is, or why it is a problem.
The purpose of the survey – benchmarking or problem-identification – should be clear, and, as with setting KPIs, the survey questions should reflect the core business of the website. A short, web-based survey will capture customer feedback while they are on the site and the experience is still fresh in the user’s mind. The survey may appear automatically when triggered by a certain action, or users may be invited to provide feedback.
User testing
User testing is helpful for looking at user interface and workflow – you’ll get a picture of what a user does and why. It will provide great feedback, but generally only evaluate a small number of users, and with a relatively large investment of time.
User testing is good for quickly and easily identifying the most obvious problems around usability, design and task completion. This is useful both when you want to find where in a process a problem is occurring, and to test new processes and functionality following changes to a website.
- Set tasks for the testers - these should reflect what the website is there to do. Create scenarios for the user that recreate this purpose (find a document, complete a transaction, etc) and provide you with a measure for success.
- Testers - there is a lot of scope here, depending on your resources: internal testers, who can compare their experience with what they know about how the process should work; external testers, who can be observed, as well as providing feedback on their experience; expert testers, who may concentrate on particular aspects, eg, usability.
- Environment – testers could perform tasks in a controlled environment, allowing you to be specific about what is being tested, and how. Testers in their “usual” environment will indicate the effects of eg, software/hardware limitations.
Experimentation and testing (if you’re keen)
If you would like to go beyond your regular analytics programme and user testing - to be able to actively determine user preferences, rather than waiting to analyse data from your analytics tools - then you could look at experimentation and testing. The methodologies include:
- A/B Testing – test more than one version of a web page. Each version (and a control) should be tested against a goal/goals – once again, this goal should reflect the purpose of your website. Best for testing a few, simple variables at a time. You may be able to use internal skills and resources to set this up.
- Multivariate Testing – like A/B testing, but testing different combinations of sections of a page. This can be useful for websites where users follow a fairly structured process. You will most likely need external resources (a vendor, or package) for this.
- Experience Testing – bigger and better still, this allows you to change the entire experience of a user. Site platform allowing, you create multiple versions of how a user sees or interacts with a particular experience on the website, and then see which had the best response. Experience testing is a commitment, in time and resources, but will result in some very rich data being collected.
Recommended reading
- Avinash Kaushik. Occam's razor
- Avinash Kaushik. Web analytics: an hour a day. 2007, Wiley Publishing.
- Steve Krug. Don’t Make Me Think: A Common Sense Approach to Web Usability. 2000, Que.
- Web analytics toolkit. 2006, Victorian Government
- Web analytics definitions. 2007, Web Analytics Association. (PDF, 111KB)
- Website usage monitoring and evaluation. 2004, Australian Government Information Management Office.
- What is website evaluation/web metrics? 2008, Web Content Managers Advisory Council.
Relevant legislation and guidance
- Public Finance Act – indemnity clause, eg, Google Analytics
- Public Records Act – at which stage do you need to record?
- Privacy Act – what information you can collect, and when
- Government Use of Offshore Information and Communication Technologies (ICT) Service Providers - advice for government agencies on managing the risks around the use of offshore ICT service providers. Intended to assist agencies to meet their obligations under existing procurement policies, international obligations, the Public Finance Act, the Privacy Act and the Public Records Act.
References
- Web analytics definitions. 2007, Web Analytics Association. (PDF, 111KB)
- Avinash Kaushik. Six rules for creating a data driven boss. Occam's Razor 24/10/07
- Avinash Kaushik. The action dashboard. Occam's Razor 30/04/08
- Bradley Horowitz. Creators, synthesizers and consumers. Elatable 17/02/06
