VIRTUAL LEARNING ENVIRONMENT USER TESTING PROJECT

Commissioned by TechDis and conducted by the Royal National College for the Blind in Hereford.

By Shirley Evans, RNCB Hereford, Allan Sutherland, TechDis, and RNC Task Force, RNCB Hereford

January 2003

CONTENTS
1. Executive Summary
2. Abstract
3. Introduction
4. Assistive Technology
5. Qualitative Study
6. Quantitative Study
7. Key Issues and Recommendations
8. Conclusion
9. Future Work
10. References

EXECUTIVE SUMMARY

This study was commissioned by TechDis and carried out by the Royal National College for the Blind in Hereford. The purpose of the study was to increase understanding of how to make a virtual learning environment (VLE) accessible to students with disabilities and in particular those who have a visual impairment. The report highlights practical advice and tips which could be used by tutors, learning technologists and anyone who supports visually impaired learners in an on-line environment.  The information will be disseminated to the JISC community in the form of a report, presentations and advice notes.

Previous studies have focused on accessibility and issues specific to particular VLEs such as WebCT and Blackboard. Nielsen's study (2001) found that someone using no assistive technology was 6 times more likely to complete a given internet-based task than someone using a screen-reader. VLEs potentially pose more problems due to their complexity and the expectation that learning may take place. Accessibility statements provided by VLE vendors indicate that they are committed to improving accessibility and usability issues. Blackboard and WebCT (both US vendors) have made great progress and UK vendors (including COSE, FD Learning Environment and LearnWise) have also made recent strides in this area.

The Study

Qualitative study

This involved five learners with various disabilities - three with no sight (one with dyslexia and one with a hearing impairment), two visually impaired (one with a hearing impairment and one a with motor impairment).  Learners were part of an existing transitional skills group using Blackboard. After a period of six weeks learners moved to WebCT where the group continued for another six weeks. Data was collected using questionnaires, observation and analysis of videotapes.

Quantitative study

This involved six students - three with no sight, one visually impaired and two with no visual impairment. Responses of new users to WebCT and Blackboard were explored using a set of five tasks (accessing content, using a discussion board, creating a simple home page, answering an on-line quiz, and uploading and submitting an assignment). Timed tasks were filmed and this was analysed as to how much time was spent Accessing, Using or Doing. Learners were also given a short subjective survey.

Outcomes

All the learners involved in this study enjoyed using the VLEs. The results of the satisfaction ratings may indicate that although learners in general met many difficulties they felt confident in the technology, not frustrated, and satisfied with their execution of the tasks. 

Learners using no assistive technology or using magnification spent significantly more time, two to three times, Doing than the other learners using a screen reader. In most cases the learners using a screen reader spent as much time Using and as much time Accessing as the learners using no assistive technology took Doing. Additionally, the learners using assistive technology in most cases would have had difficulty completing the tasks without prompting. The learners using no assistive technology and the learner using magnification needed relatively few prompts, if any.

Competent navigation of the World Wide Web may not indicate that a learner can engage effectively in a Virtual Learning Environment. Learner competencies may be overestimated in some cases both by tutors and by the learners themselves. Assessment, induction and support strategies need to be developed to address these issues. Failure to do so could have a range of results from a reduced quality of the learning experience to total despondency, loss of confidence and reluctance to engage in any further learning experiences.

The potential for effective and innovative learning experiences is immense. E-learning has the potential to enable learners with particular needs to engage in learning on a level playing field. However, until both accessibility and usability issues are resolved, visually impaired learners will continue to be disadvantaged in terms of cognitive overload and time and energy input, resulting in a poorer learning experience than otherwise.

ABSTRACT

E-learning may have the potential to enable learners with particular needs to engage in learning on a level playing field. Nielsen's usability study (2001) found that it was six times more difficult for someone using a screen reader to access the Internet than someone using no assistive technology. A virtual learning environment may be more multi-dimensional (and possibly more interactive) than a web-site which suggests that it may be more difficult to use. The additional cognitive overload caused by accessibility and usability issues may impinge on the learning process.

There are two parts to this research.  The qualitative study looks at an existing course where transitional skills have been delivered via a virtual learning environment (Blackboard). Learners transferred part-way through the course to a different virtual learning environment (WebCT). The quantitative study explores responses of new users to WebCT and Blackboard using an agreed set of five tasks. Both aspects highlight the need for individualised assessment, induction and support.

Keywords: Assistive technology, accessibility, usability, e-learning, learning, virtual learning environments

INTRODUCTION

Purpose

The purpose of this study was to increase understanding of how to make a virtual learning environment (VLE) accessible to disabled students and in particular those who have a visual impairment. This information will be disseminated  to the JISC community in the form of a report, presentations and advice notes. It is hoped that the information will mean that accessibility is designed into the deployment of VLEs as online learning tools and that individual assessment, induction and support  are implemented.

Virtual Learning Environments

In the context of this study, a VLE is an on-line learning environment. These are web pages where students can engage in learning activities and/or interact with tutors and peers. There can be facilities that include 'controlled' delivery of materials, chat, discussion boards/conferencing, on-line assignments such as multiple choice tests, tracking and home pages etc. Two of the most  widely used VLEs, Blackboard (Version 5 - Level 1) and WebCT (Version 3.6 - Standard Edition) are used in this study. It should be noted that other versions and products may vary in features and accessibility and usability issues.

Background

The ILT/ICT (Information and Learning/Information and Communications Technology) Task Force was set up in September 2000. One of its main foci has been upon assessing virtual learning environments, originally with a view to purchasing one for use at RNC, both for residential and distance learners. It was important that a VLE is found which is both accessible and usable.

The Task Force has been using Blackboard in the College to deliver transitional skills. Please see the section on the Qualitative Study below for further description and Evans (2002) for a description of the pilot study. It is this and the subsequent work by others that have informed this study.

Related studies and reports

Doyle (2001) has produced some guidelines on how to make your module accessible in Blackboard 5.5. These guidelines include a quick tip guide which describes what to do and why you need to do it. Areas covered include:-

· A quick tip guide e.g. use solid background colour; avoid use of more than 10 links per page.
· How to upload accessible materials e.g. some screen readers are not able to read tables; for assistive software users 'click here' on an image map may not be accessible.
· Advanced information on Blackboard 5.5 and accessibility issues e.g.  Blackboard has a frame structure; some learners may use text-based or older screen readers to access online materials; frames are very difficult/impossible to navigate through.
· Further improvements in version 6.0 due out in 2002 - Blackboard have stated that they are currently addressing the accessibility problems in Version 6.0.

Pearson and Koppi's study (2001) set out to investigate means of making higher education (HE) inclusive and accessible to people with disabilities through the use of educational technology. The guidelines were produced as the result of an evaluation of WebCT in practice at the University of New South Wales (UNSW), Australia. The methodologies included an evaluation of the courses by the researcher, discussions with and survey of designers, and interviews with disabled students and student evaluation of the accessibility of on-line courses. The evaluation was based on W3C Guidelines, the Bobby website and courseware design guidelines. These guidelines covered such areas as:- use of graphics and other visual items, use of screen readers and their limitations, tables, colours and pdf and other read-only file formats.

Coyne and Nielsen (2001) have produced a set of methodology guidelines for testing websites and intranets with users of assistive technology. These guidelines are based on previous studies they conducted to gather data and insights for the report 'Beyond Alt Text: Making the Web Easy to Use for Users with Disabilities' (2001). In this study it was found that those using no assistive technology were 6 + 3 times more likely to complete a set of timed Internet tasks than those using screen readers and screen magnification respectively. The guidelines cover ethical issues as trust and consent forms, practical issues such as the use of videoing and methodological issues including the use of satisfaction ratings.

Most of these aspects were not particularly pertinent to the RNCB Hereford study as the participants were learners at the College and the researchers were familiar with this field of study. The most relevant section was on measurements which included information on click errors, subjective satisfaction ratings, timing of tasks and task end-points.

Additionally, TechDis commissioned a report (2002) which outlines research undertaken to investigate the accessibility of VLEs. The VLEs, which include Blackboard and WebCT, were selected because they were part of the Joint Information Systems Committee (JISC) interoperability pilot study. A questionnaire was issued to VLE vendors which covered areas such as:-

· Whether the vendor provides an accessibility statement.
· Whether advice on accessibility is provided by the vendor.
· Whether there is specific accessibility advice on uploading content to the VLE.

A range of guidelines and checklists are available to aid accessibility and to ensure that VLEs are inclusive rather than exclusive. The research suggests that US vendors (Blackboard and WebCT) have made strong commitments to improving the accessibility of their products in response to the legislative requirements of Section 508. However, UK vendors (including COSE, FD Learning Environment and Granada LearnWise) have also made recent strides towards accessibility as can be seen in their corporate policies. They are currently working on programming issues and guidance to users.

This research highlights the need for providers to address accessibility issues in light of current UK legislation and for institutions to consider the accessibility of a VLE as a contributing factor when purchasing and implementing a VLE. It  is essential that institutions which have already purchased a VLE continue to work with vendors to ensure that accessibility for all users remains a high priority.

ASSISTIVE TECHNOLOGY

What is assistive technology?

Assistive technology is defined in the glossary section of the TechDis website as: 'Information and communications technology designed to compensate for impairment and/or limitations in disabled users.' The assistive technologies used in this study were screen readers and screen magnifiers.

Screen readers

A screen reader is a software program that allows a blind person to read text on the screen and identify some graphics such as  buttons on a toolbar or icons on the Desktop. The user hears the information through  a speech synthesiser or the computer's sound card.  A screen reader also allows the user to control the computer using the keyboard rather than the mouse. Many of the keyboard commands that a blind person uses are the same keyboard commands that a sighted person can use on their PC such as Windows standard keystrokes. For example the File menu can be opened by pressing Alt-F.  A blind person also uses special keyboard commands that are created by the screen reader. For example, most screen readers offer a keystroke to read the information in the title bar or in the status bar. Web pages can be navigated by using the tab key to move around links on a page or between pages. Pressing the enter key will activate a link. The cursor keys can be used to read pages and also navigate . There is usually a links list facility - a shortcut which will bring up a list of links on a page where the cursor keys are used to move up and down the list. The list can sometimes be sorted into alphabetical order and to show only recently visited links. Usually pressing the first letter of a link within a list will move the cursor to that link.

Great skill is needed to navigate the World Wide Web using a screen reader and keystrokes. An effective user will typically use a mixture of tabbing around links, cursoring up and down the page and bringing up a links list (if available). Users may 'scan' a page by listening to the contents.

Assistive technology used in this study

Two different screen readers were used, Jaws 3.7 and Supernova 5.02. Additionally the magnification (X2) facility in Supernova 5.01 was used. These are the two screen readers most widely used in the United Kingdom.

For further information and an extensive database of assistive technologies as well as case studies please visit the TechDis web site - http://www.techdis.ac.uk

QUALITATIVE STUDY

Methodology

Prior to this study the group of learners at RNCB Hereford had already been engaged in using Blackboard for reading and posting messages on the discussion board, accessing content and creating a simple home page. The focus was on the discussion board as this was the main delivery method for the transitional skills which were the primary focus of the learning experience. The Electronic Soap Group was set up as a response to a request to find an innovative way to deliver transitional skills (see Evans (2002) which describes the pilot in detail). Transitional skills at RNCB are defined as 'Skills that enable learners to manage continuous change and personal development.' In this discussion board such subjects as teenage pregnancy, alcohol and drugs were discussed in the context of a popular soap. Learners involved were all aged between 16 and 19 and had a very high level of IT skills, both in general terms and with the assistive technology needed.

Learner  Gender      Assistive Tech      Amount of useful sight          Other disabilities
A               M              Jaws                Totally blind                      Hearing impairment and mobility problems
B               M              Jaws                Totally blind                      Dyslexic
C               M              Jaws                Totally blind                      None
D               M              None                Substantial useful sight      Motor- uses keystrokes generally
E                F              None                Substantial useful sight      Hearing impairment
       
   
Additional feedback was gleaned from a tutor who uses a screen reader and has a very high level of IT competence.

Initial Questionnaire

The group members were given questionnaires at the beginning of the course to establish:-

· Their assistive technology needs,
· How long they had been using a computer,
· What they used the computer for and how frequently,
· Frequency of use of email, the Internet, Chat and Discussion Boards,
· What they expected to get from the Electronic Soap Group.

Blackboard Questionnaire

This questionnaire was emailed to learners at the end of the pilot and covered the following areas :-

· Logging in,
· Navigation,
· Specific questions about features e.g. discussion board,
· Preferences between face-to-face and on-line discussion,
· What they had learned about Blackboard or about the discussion topic,
· Support.

WebCT Questionnaire

This was identical to the Blackboard questionnaire and emailed to learners when they had finished using WebCT and returned to Blackboard.

Additional Evidence
 
Evidence was collected on videotape, from observation notes, screen shots and the discussion board. Semi-structured interviews were also carried out as part of another study. These interviews covered social background and use of information technology as well as a discussion of navigational strategies. This latter information is of relevance here as it has formed the basis of a framework in respect of navigational skills using assistive technology and a virtual learning environment.

Analysis

Questionnaires

Initial Questionnaire

All of the learners had been regularly using a computer for at least three years. They used it for course work and used the Internet for email, chat, games, research and news. All the learners used email at least twice a week but some were accessing email four times a day. The three Jaws users had only been using the Internet for six months to one year.  This was very surprising considering their general level of expertise. Amazingly up to six months before the pilot Learner C had been using sighted users to read web pages. None of the learners had used a VLE before and only one of them had used a discussion board, although all of them were frequent users of Chat facilities. It is clear that the three that had only recently been introduced to the Internet may have been deprived of a significant part of life, learning and leisure for a number of years. All of them expected to learn about social issues and none of them mentioned learning about the technology.

 
Blackboard Questionnaire

The responses of only four participants are recorded here as Learner D joined the study at the beginning of the WebCT phase.

Question              
Did you have problems logging in?
Learner A - No
Learner B - No
Learner C - Yes
Learner E - No
Comments - Problem with forms mode in Jaws-possibly a network problem

Question
Did you have problems with navigation? 
Learner A - No 
Learner B - No 
Learner C- No - kept a familiar location in mind and used a links list 
Learner E - No 
Comments - None

Question
Did you enjoy using the discussion board? 
Learner A - Yes
Learner B - Yes
Learner C- Yes
Learner E - Yes
Comments - None

Question
Did you prefer face to face or the discussion board? 
Learner A - Mix
Learner B - Mix
Learner C-  Mix
Learner E - Discussion board - has a hearing impairment 
Comments - None

Question
Did you have problems with creating a home page? 
Learner A - No
Learner B - No
Learner C - No
Learner E - No
Comments - None

Question
Did you have any problems with content? 
Learner A - No
Learner B - No
Learner C-  No
Learner E - No
Comments - None

Question
What do you feel you have learned? 
Learner A - Use a discussion board 
Learner B - Use a discussion board 
Learner C - Use a discussion board and has learned about issues and other points of view 
Learner E - Use a discussion board and has learned about issues and other points of view 
Comments - None

Question
Would you have liked a longer introduction? Learner A - Yes
Learner B - No
Learner C - No
Learner E - No
Comments - Learner A started later than the others and may have missed out on some aspects.

Question
Was support, on-line and face-to-face, satisfactory? 
Learner A - Yes
Learner B - Yes
Learner C- Yes
Learner E - Yes
Comments - On-line support was only used on three occasions-there was a problem logging in - see above

 

WebCT Questionnaire

Question 
Did you have problems logging in? 
Learner A - Yes 
Learner B - Yes 
Learner C - Yes 
Learner E -  Yes 
Comments - Password problems and learners required to add themselves to a course - down to the tutor. No problems after initial session

Question
Did you have problems with navigation? 
Learner A - Unmeaningful links, names and graphics 
Learner B - No 
Learner C - Used links list 
Learner E -  Used the back button 
Comments - Navigation was difficult but no-one could pinpoint why.

Question
Did you enjoy using the discussion board?  
Learner A - No - not very well organised 
Learner B - No -found it difficult to use 
Learner C - Not very well organised - but likes all discussion boards
Learner E -  Difficult to find new messages - OK but preferred Blackboard 
Comments - Set up differently to Blackboard - one forum with several threads

Question
Did you prefer face to face or the discussion board?    
Learner A - Mix
Learner B - Mix 
Learner C - Mix
Learner E - Discussion board - has a hearing impairment
Comments - None

Question
Did you have problems with creating a home page?    
Learner A - Did not do this activity 
Learner B - No
Learner C - Nightmare
Learner E -  Straightforward
Comments - Generally found it  was easy to create a page but not many options - boring

Question
Did you have any problems with content?   
Learner A - No
Learner B - Fairly difficult 
Learner C - Yes- had not been shown enough how to access it
Learner E -  No
Comments - None

Question
What do you feel you have learned? 
Learner A - Use a VLE 
Learner B - Use a VLE 
Learner C - Use a VLE 
Learner E - Use a VLE 
Comments - No one mentioned the transitional curriculum

Question
Would you have liked a longer introduction?    
Learner A - Yes
Learner B - Yes 
Learner C - Yes
Learner E - No
Comments - None

Question
Was support on-line and face-to-face satisfactory?   
Learner A - Yes 
Learner B - Yes
Learner C - Yes
Learner E -  Yes but often difficult to get help in f2f situation 
Comments - Two experienced tutors in the room were sometimes not enough

The learners appear to have experienced more problems with WebCT than with Blackboard. The most likely reason for this is that they had got used to using Blackboard and found it difficult to move to another VLE. Additionally it is possible that insufficient induction into WebCT had been given and it had been assumed that familiarity with one VLE would mean less induction would be necessary with another VLE.  This raises a question relating to transferability of skills between VLEs.

Additional  Evidence

Several on-line and classroom-based discussions took place in respect of use of discussion boards and the use of face-to-face sessions (and the evidence for this is set out above, with all of the participants but one preferring a mix). The discussions centred around preferences between Blackboard and WebCT. It was clear from the classroom-based discussions that the learners using screen readers preferred Blackboard but the learners using no assistive technology preferred WebCT. It is important to note that learners had got used to some aspects of Blackboard. The situation may have been the same if they had moved from WebCT to Blackboard. It may not be advisable to move from one learning environment to another and if there is a need then it should not be assumed that no induction is necessary. The learners returned to Blackboard after a six week period using WebCT. Some of the comments from the discussion board are set out below.

Some quotes from discussions  about the use of online discussions:

"When I am in an ordinary group discussion, I cannot always hear what others are saying and I don't always like to ask them to repeat what they have said.  With Blackboard I don't miss anything."

"Well, in my opinion there are advantages and disadvantages of the teacher talking to you all the while.  It helps the information to sink in if a teacher is babbling away to you for half an hour or so, then giving you work to complete to see how much you really understand about the subject in question. I love this discussion board.  It gives everyone a chance to say what they think whenever and however they want.  It presents a challenge.  How can I express myself in a message? If I am angry how do I convey this??....... I can't wait to read other peoples views."

"I like using computers, face-to-face is ok but it can become boring over a long period of time. I also feel that I can express myself more freely in a forum."

The type of  experience, set out below, was quite common:-

A lot of concentration is required to work within the VLE. This was exemplified on one occasion where Learner B had logged in, navigated to the discussion board and the particular week, read a message and then started to reply. This took a considerable amount of time. As B started the message and was entering the subject B was distracted momentarily. This meant B did not go into forms mode and failed to enter a subject although B thought this had occurred. B then proceeded to post the reply which was about 100 words long. When B went to post the message an error message came up saying that there was no subject heading. When B pressed Enter and a Microsoft web site appeared, Blackboard had closed down and B had to start again from the beginning!

QUANTITATIVE STUDY

Methodology

This study explored responses of new users to WebCT and Blackboard using a set of five tasks  -
· Accessing content - internal content arranged in serial mode rather than in folders
· Reading and posting messages on the discussion board
· Creating a simple home page
· Answering an on-line quiz - multiple choice test
· Uploading and sending an assignment (WebCT)/using the digital drop box (Blackboard)

Each learner followed the same scheme of work, lesson plans and induction, and performed the same tasks. A scheme of work was developed as the 'course' would be delivered by different tutors. The course was devised to be delivered over 6 weeks, or in 6 separate one hour sessions with each session covering a different feature (accessing content, discussion board, creating a home page, uploading an assignment, reading and posting a message and doing a multiple choice quiz). After learners had completed the scheme of work they were timed and filmed carrying out the tasks. Timed tasks were filmed using a digital recorder and this was transferred to videotape. Notes were taken at the time but with one person filming it was not always easy to do this.

Learners were given an initial questionnaire which addressed their experience with IT and in particular discussion boards, email and chat. The tasks were piloted with the two sighted users to ensure that there were no bugs and to identify any problem areas. A short subjective survey using a 1 to 7 Likert scale satisfaction rating was also used and administered after each task. Learners were asked whether or not they agreed that they were satisfied with the way they carried out the task; that they were confident in the technology and whether they felt frustrated with the technology.

Learners

The table below sets out the learners, their gender, any assistive technology used and the relevant VLE. All these learners had a high level of IT skill and were volunteers giving up free time to participate.

 
Learner     Gender     Assistive Technology          VLE
G                  F              None                          Blackboard
H                  M             Jaws                           WebCT
I                   M             Supernova 5.02            Blackboard
J                   F              None                          WebCT
K                  F              Jaws                           WebCT
L                   M             Jaws                           Blackboard
M                  M             Magnification               WebCT

Analytical Framework

It is important to differentiate between usability and accessibility. According to Frontend (2001), 'Usability focuses on making software, websites and on-line applications and services easy for people to use. Accessibility may be defined as making them equally easy for everyone to use, including people who may use assistive technologies such as screen readers etc.'

Tasks were timed every ten seconds and a judgement was made as to whether the learner was Doing, Using or Accessing.  For the purpose of this study, Doing, Using and Accessing are defined as follows:-

Doing - is concerned with task performing which may include: reading instructions, reading and posting a message, uploading a file, opening Word, thinking and learning.

Using -  involves navigating around the VLE. i.e. moving between pages and around pages.

Accessing - is concerned with getting to the information. For this study this will involve assistive technology i.e. a screen reader or magnification. If someone is using no assistive technology then the time spent viewing the page, looking where to go next would be considered to be accessing. For someone using magnification this will be time spent scrolling or moving around the page on the screen to view information. If someone is using a screen reader then accessing will be the time spent listening to find out what is on the page. This would not be the same as reading learning materials or task instructions on the page.

There is always the possibility of misinterpretation of what a  learner is doing at a given point in time, in this case 10 second intervals. It may appear that, for example, a learner using no assistive technology spends little time navigating as this is momentary and may not be captured with the 10 second interval method. This could be resolved by working with the learner using the video tape or the learner thinking out loud whilst they are carrying out the task. However the latter may result in cognitive overload as the learner would have to be analysing whether they were Accessing, Using or Doing at the same time as carrying out the task and using assistive technology. If someone is using a screen reader they will be using the tab key and/or a links list and/or a cursor to do this. If they are using the cursor then it is difficult to see whether they are Using or Doing. If the learner is using magnification then sometimes they will follow the path around with the pointer. Otherwise, as with someone using no assistive technology it may be difficult to discern what they are doing at a given point in time.  However, in practice it was relatively easy to differentiate between Accessing, Using and Doing.

A form, with column headings - Interval, Actual Activity , Accessing/Using/Doing and Comments/Notes including errors, was used to record information taken from the videotapes. A 10 second bleeper was used and at these intervals it was noted down whether the learner was Doing, Using or Accessing. Additionally a note was made on what they were actually engaged in  e.g. reading a message, inputting a web address. Any comments were added particularly in respect of whether there was a user or system error, or whether the learner had to be prompted.

The process can be very time consuming and it is essential that the researcher doing the analysis was present during the videoing. It is recommended that these two tasks should be done by the same person and ideally the latter task be carried out by two researchers. Learners may feel under pressure doing this sort of 'test,' and great care was taken to explain that it was the software that was being tested and not them personally.    

Analysis

Satisfaction ratings

These are examined below in terms of ratings by task and by learner.

Figure 1 is a graph showing that out of a rating of maximum value 35, all users gave ratings in terms of Confidence in the technology, Lack of Frustration with the technology and Satisfaction in the way they performed the task, between the values of 20 and 35. The scores for Confidence ranged from 23 (Learner H) to 35 (Learners I and M). The scores for Lack of Frustration ranged from 20 (Learner H) to 34 (Learner M). The scores for Satisfaction ranged form 21 (Learner J) to 34 (Learner M). The mean rating across all three measures was highest for Learner M (34.7) and lowest for Learner L (23.3). Three Learners each rated Confidence and Satisfaction most highly, only Learner G gave Lack of Frustration the highest rating.
    
    
Figure 2 is a graph showing the Learners' ratings in terms of Confidence in the technology, Lack of Frustration with the technology and Satisfaction in the way they performed the task, in respect of the Content, Home Page, Upload and Assignment, Online Quiz, and Reading and Posting Messages. The graph shows that out of a maximum rating value of 49, the Learners gave Confidence ratings ranging from 38 (Upload and Assignment) to 43 (Reading and Posting Messages), Lack of Frustration ratings ranged from 27 (Upload and Assignment) to 45 Reading and Posting Messages). Satisfaction ratings ranged from 32 (Home Page) to 44 (Online Quiz). The aspect scoring most highly across the three ratings was Reading and Posting Messages (43.7), whereas the lowest mean of 32.7 was for Upload and Assignment.

Table showing individual learner satisfaction with specific tasks; overall satisfaction for each learner; combined satisfaction with each task. Table highlights areas of most and least satisfaction.

Learner - G 
Content - Least satisfied 
Home Page - Most satisfied 
Upload an Assignment - Least satisfied  
On-line quiz -
Read and Post a message - Most satisfied 
Comments - summary for each learner - G only made one error (which was rectified) - not 100% satisfied with any aspects

Learner - H 
Content - Fairly neutral but H navigated well around the last two tasks  
Home Page - . 
Upload an Assignment - Least satisfied - completely lost confidence after doing this and had to be prompted
On-line quiz - Not confident with this as had experienced difficulties - but had found a work-around to radio button not staying selected 
Read and Post a message - Most satisfied 
Comments - summary for each learner - H had the lowest ratings overall and this was due to difficulties in uploading an assignment which was the second task - lost confidence.

Learner - I
Content - Most satisfied 
Home Page - Least satisfied   
Upload an Assignment -
On-line quiz -
Read and Post a message - Most satisfied 
Comments - summary for each learner -  I was generally over-confident

Learner - J   
Content -
Home Page - 
Upload an Assignment - Least satisfied - next to lowest satisfaction rating overall  
On-line quiz -
Read and Post a message - Most satisfied 
Comments - summary for each learner - Confident in technology but frustrated with it and not satisfied with the way they carried out tasks

Learner - K  
Content -
Home Page - Least satisfied  
Upload an Assignment -
On-line quiz - Most satisfied  
Read and Post a message -
Comments - summary for each learner - K was reasonably satisfied in most respects - would not have been able to complete any of the tasks without prompting

Learner - L 
Content - Fairly frustrated with this 
Home Page - Fairly frustrated with this - not intuitive even if you can see it. 
Upload an Assignment - Least satisfied - lowest satisfaction rating overall 
On-line quiz - Most satisfied  
Read and Post a message -
Comments - summary for each learner - L was generally dissatisfied and gave the lowest rating overall, although L is very competent

Learner - M 
Content - Very satisfied 
Home Page - Very satisfied 
Upload an Assignment - Very satisfied 
On-line quiz - Very satisfied 
Read and Post a message - Very satisfied 
Comments - summary for each learner - M was satisfied with all aspects - worked very methodically - may have been aided by X2 magnification

Comments - summary for each task 
Content - Second lowest score - possibly learners were not given enough induction. One of the greatest areas of difficulty. 
Home Page - This had only been done once before - not a task that would be repeated often - hence low score. Sighted learners had higher scores. 
Upload an Assignment - This task caused the greatest amount of frustration - quite complicated and very difficult if no visual clues. Not possible to browse for files. 
On-line quiz - Learners were quite satisfied with this - it was difficult to do but there is a set pattern. Possibly easier in WebCT as new Window opens for each question. 
Read and Post a message - Learners were most satisfied with this - they did it every week. 

There seemed to be no significant differences between the two VLEs, however as described below, creating a home page and uploading an assignment in WebCT were particularly difficult areas as these activities are not intuitive if you cannot see the screen.

 Figure 3 is a screen shot showing the complexity of WebCT assignment submission. The image shows the WebCT navigation menu on the left of the screen, navigation and location links across the top, buttons to access student files and submit assignments, and instructions for the task itself.

 
Figure 4 is a screen shot showing the Blackboard RNC web page, with a navigation column on the left hand side and an uncompleted form for locating a file. The Browse button for locating files does not work with screen readers, therefore users had to remember the full file path for every file they wished to access.

Navigation in internal content may be easier in WebCT due to the structure of the links and the facility to hide/show the navigation bar (which enables more effective accessing).
 
Figure 5 is a screen shot showing the organisation of content in WebCT. The navigation options are on the left hand side and across the top of the screen. There is a 'Hide Navigation' option which may speed progress for users with screen readers. A 'New Mail' alert box is visible on the screen - this may interfere with screen reading.

 
Figure 6 is a screen shot showing an RNC content page in Blackboard. A navigation menu is on the left hand side in addition to two separate horizontal navigational structures across the top. The page content is reduced to approximately 60% of the page. If there is a lot of information on the page, additional navigational instruction may be necessary.

Generally learners were most dissatisfied with the home page and file upload facilities followed by accessing the content. The main exception was the difficulty encountered by H and the problem with Jaws and the multiple-choice quiz.

 
Figure 7 is a screen shot showing a multiple choice quiz in WebCT. The quiz has opened in a new window. The quiz window shows at the top the total number of questions and the 'Finish' and 'Help' buttons, at the right hand side a graphic indicating which questions have been answered, and the first question and answers in the main part of the window.

 
Figure 8 is a screen shot showing a multiple choice quiz in Blackboard. The navigation features are on the left hand side and across the top of the screen. The questions and answers are shown in the main part of the screen. No new window has opened. The questions are arranged on one page in a table. this may be difficult to use with a screen reader as screen readers read tables in a linear fashion row-by-row. If the table has not been constructed with this in mind it may make little sense with a screen reader.

Learners were in general satisfied, however only two of the learners (one fully sighted and one visually impaired) were able to complete all the tasks without prompting.

 
Figure 9 is a screen shot of WebCT with 2x magnification. The navigation menu now fills the left hand half of the screen and the associated graphic icons are partially displayed on the right hand side. This magnification may assist visually impaired users.

Video tape analysis

Accessing, Using and Task Performing
Figure 10 is a graph showing the number of ten-second intervals, over all tasks, by each Learner spent Doing, Accessing and Using. Learner K spent 172 10-second periods Doing, 84 periods Using and 138 Accessing, these were all the highest values for each type of activity. Learner G spent 60 periods Using, 9 periods Using and none Accessing, these were all the lowest values for each type of activity.

As mentioned previously, all participants were experienced in the use of IT and with the assistive technology. Clearly there are some differences in the levels of skills and the implications of this are discussed in general terms under 4.3.2.2 below. These differences will have some effect on the results but it is not the purpose of this report to compare learners' skills.

Learner         Assistive Technology          VLE                           Doing %             Using %            Accessing %
G                 None                              Blackboard                    86                    14                       0
J                  None                              WebCT                         72                    28                       0
M                 Magnification                   WebCT                         76                   16                        8
K                 Jaws                               WebCT                         43                    22                       35
L                 Jaws                               WebCT                         33                    36                       31
H                 Jaws                               Blackboard                   42                     25                       33
I                 Supernova 5.02                 Blackboard                    40                     19                      41

The table above shows learners, their assistive technology and the VLEs they were using as well as the time as a percentage, spent Doing, Using and Accessing

All the learners using screen readers spent more time overall on the task than those using no assistive technology. Only J came close to the other scores, but this was because J spent minimal time on writing answers. It is difficult to make comparisons between VLEs and assistive technologies due to the different amount of time the learners spent on various tasks i.e. spending more time reading and posting messages/writing longer messages. Learner G (using no assistive technology) and Learner L (using Jaws) may be compared up to a point as they spent similar amounts of time actually doing the task and were operating at a similar level of IT expertise. As can be seen from the above table, L spent only 33% of the total time on Doing compared to G's 86%.

In general terms it can be seen that the learners using no assistive technology or magnification spent significantly more time, two to three times, doing than the other learners using a screen reader. In most cases the learners using a screen reader spent as much time Using and as much time Accessing as the learners using no assistive technology took Doing. Additionally, the learners using assistive technology in most cases would have had difficulty completing the tasks without prompting. The learners using no assistive technology and the learner using magnification needed relatively few prompts, if any.

Navigational issues

It was found during the course of the study and analysis of the video tapes that learners were not navigating around the VLE effectively and efficiently. Three categories/levels of navigational expertise for learners using screen readers were developed by the team:-

Level 1 - Linear - uses links to navigate between and within pages. May lack confidence. Cannot solve accessibility and usability problems. May only be engaged in passive learning.

Level 2 - Linear/exploratory - uses a mixture of links and cursor to navigate within pages. Can sometimes resolve problems. Tends to stick to set patterns. Is reasonably confident. May sometimes be engaged in active learning.

Level 3 - Exploratory - uses a mixture of links and cursor to navigate. Can usually solve problems and will experiment with different strategies. Is very confident. Will be engaged in active learning where appropriate.

The use of the terms 'linear' and 'exploratory' are different to the terms 'serial' and 'parallel' as used by Craven (2002) in her study of navigational steps taken by sighted and visually impaired users visiting library databases and web pages. In Craven's study 'parallel' relates to steps taken within a page and 'serial' relates to steps taken between pages. In this study 'linear' and 'exploratory' relate to navigational strategies rather than steps.

The level may vary depending on what the learner needs to do. If they are using the Internet and frequently visiting the same web pages then they need to operate at least at Level 1. However if they are to use a virtual learning environment effectively then they will need to be operating at Level 2/3. A learner may appear to be able to navigate effectively and may indeed think they are doing so. The level at which they navigate may depend on the way they have been taught/trained. Since web pages in general are difficult for someone using a screen reader to access and read there may be a need to start at Level 1. It can be demotivating if learners have to spend a lot of time using and accessing before they get to any 'real' information. Some learners will be able to progress through the levels on their own whilst some may need additional support and training to progress. Level 2 may be sufficient for a learner to access web pages but there are additional skills necessary to use a VLE effectively. Therefore careful assessment of skills is necessary and specific training and/or intervention given accordingly to enable the learner to move towards Level 3. This may be particularly pertinent if the learner is working in isolation.

KEY ISSUES AND RECOMMENDATIONS

Many of the accessibility and usability problems that arose were specific to the VLEs used in this study. The TechDis VLE report (2002) indicates that some of these issues are being addressed as new versions are released. The aim is to provide some generic advice that tutors, learning technologists and others who support learners with a visual impairment can note and, where appropriate, take action. The key issues have been used as a basis for some advice notes in the form of postcards for dissemination to educational establishments. The contents of these postcards are collated at Annex A. Some general issues are set out below under the section headings used for the advice notes.

General issues/Introduction

This section identifies the purpose of advice notes, who they are aimed at and the main issues: -
· VLEs are complex and a high level of navigational skill is necessary for effective engagement. Navigation may be particularly difficult if the learner is using a screen reader.
· A VLE may not be intuitive for a learner using a screen reader
· Time - a learner using a screen reader may need significantly more time to complete tasks. In the study it was found that a learner using a screen reader was spending up to a third of total time in the VLE accessing and navigating. Individualised induction, on-going assessment of navigational skills and timely support are of paramount importance.

Accessing

This section is concerned with types of assistive technology and some of the problems that might occur

 
Using

This section identifies some navigational issues: -
This involves moving around the page and between pages either using a links list, keystrokes and/or the cursor key

· Navigation will be time consuming when using a screen reader
· Hide the navigation bar (where possible) when reading content
· Navigation method may vary depending on which area of the VLE is being accessed and which VLE is used - use of tab key and cursor keys
· Navigation will vary depending on type of assistive technology being used
· Buttons sometimes have different labels from Alt text - this can cause problems when writing instructions
· Links may be named illogically or there may be duplications
· Encourage the use of bookmark facilities (where available)

Content

This section addresses both internal and external content. Internal content is generally under the control of the tutor and the advice notes in this respect are based on the 7 precepts of accessibility and usability to be found on the TechDis website. (www.techdis.ac.uk/seven)

External content may be outside the control of the tutor but it is still up to them to choose content which is accessible.

Learner Issues

This section is concerned with assessment of skills, induction and support of learners. In general terms it says that: -

· Induction may need to be highly individualised and particular care needs to be taken in this area.

· Assessment of skills and accurate and timely assessment are of paramount importance. This will feed in to the induction and will ensure appropriate support.

· Support needs to be on-going and timely

· Navigational expertise- any learner will need to have a certain level of expertise in using the Internet/World Wide Web in order to engage effectively with a VLE. Learners should be encouraged to develop sound skills where they can solve problems and feel confident in their use of the technology

Specific Features

This section sets out some tips/advice on specific features such as discussion boards, on-line assessment and uploading an assignment. This section is not exhaustive but gives some general pointers. Other features such as Chat and email are not addressed, but some of the problems that occur may be similar to those of discussion boards. Some VLEs will have particular problems with specific features which are outside the control of support staff etc.

Useful websites

· Blackboard website http://products.blackboard.com/cp/bb5/access/index.cgi Includes screen reader tutorial, FAQs and link to WebAim on-line accessibility course

· IMS Guidelines for Developing Accessible Learning Applications http://www.imsproject.org/accessibility/accwpv0p6/imsacc_wpv0p6.htm Includes information on assistive technology, disabilities and on-line learning in general

· TechDis http://www.techdis.ac.uk Advice and brokerage on all aspects of technology and disability, plus a comprehensive database of assistive technology, an 'experts' database, various reports and case studies as well as up-to-date information on SENDA.

· Web Accessibility Initiative http://www.w3.org/WAI/WebCT web site http://www.webct.com/products/viewpage?name=products_accessibility Includes FAQs, information on section 508 and training resources

Related studies and guidelines

· Coyne, K., and Nielsen, J., (2001), Beyond ALT Text: Making the Web Easy to Use for Users with Disabilities, NN/g http://www.NNgroup.com/reports/accessibility

· Doyle, C., (2001), Making your module accessible, http://www.uwic.ac.uk/ltsu/5min_guide_module_accessible.htm last accessed 18th September 2002

· Evans, S., (2002), Using a Virtual Learning Environment to Deliver Transitional Skills at the Royal National College for the Blind: Reflections on the Pilot Study, British Journal of Visual Impairment, 2002 20:3

· Pearson, E., and Koppi, T.,(2001), 'Guidelines for Accessible On-line Courses,' EDTeC

· TechDis Report (2002), Towards Accessible Virtual Learning Environments http://www.techdis.ac.uk/resources/VLE001.html

CONCLUSION

Previous studies have focused on accessibility and usability issues within the VLE as well as practical tips and guidelines for tutors in respect of content. Many of the issues highlighted within these studies were encountered in this study. Some of the issues and problems are specific to a VLE, such as the multiple choice quiz usability problem in Blackboard, and many issues are generic such as the non-reading of the Browse button in the file upload. This study goes further than other studies in respect of understanding how to make a VLE accessible to students with disabilities and in particular those who have a visual impairment in that it  considers assessment, induction and support issues.

It has not been possible to fully compare the two VLEs involved. Aspects that were omitted include, for example, use by visually impaired tutors and features such as the virtual classrooms, use of on-line Help, different types of on-line questions, email and calendar functions. Participants would need to be using the VLEs over a greater period of time than in this study and this may mean that they are more familiar with the features. Clearly the amount of time spent Using and Accessing should then be reduced, particularly where there is frequent repetition of tasks. However, insofar  as the study went, it was found that learners encountered similar problems with both VLEs. There were few accessibility problems that were a direct result of  the VLE. Accessing content and taking a multiple choice quiz may be more usable in WebCT, whilst creating a home page, uploading an assignment and using the discussion board may be more usable in Blackboard.

Analysing tasks by looking at how much time was spent on Doing, Using and Accessing has shown that learners using a screen reader are spending up to three time longer Accessing and Using than sighted users. Sighted users are much less likely to need prompting/help than learners using a screen reader since a VLE may not be intuitive for someone with no sight. Magnification (X2) may aid user navigation within the VLE since this lends  itself well to focusing on relevant parts of the pages. However additional time will be spent Accessing i.e. scrolling. Clearly the complexity of the tasks will mean that it is not possible to aim for an ideal ratio between these aspects. The amount of time and effort involved with efficient use of the assistive technology to find out what is on a page/site will reduce the amount of time involved in navigating. Efficient navigation may reduce the amount of time spent on Accessing. The analysis of Doing, Using and Accessing may be used as a benchmark. This could be between learners but more usefully as a benchmark for a learner in particular or for a learner using different applications or types of content. It should be noted that effective engagement in the VLE may not necessarily mean a reduction in time spent Accessing and Using.

Competent navigation of the World Wide Web may not indicate that a learner can engage effectively in a Virtual Learning Environment. Learner competencies may be overestimated in some cases both by tutors and by the learners themselves. Assessment, induction and support strategies need to be developed to address these issues. Failure to do so could result in a reduced quality of the learning experience, possibly leading  to total despondency, loss of confidence and reluctance to engage in any further learning experiences. This would indeed be a tragedy since all the learners involved in this study enjoyed using the VLEs, and the results of the satisfaction ratings may indicate that although learners in general met many difficulties they felt confident in the technology, not frustrated, and satisfied with their execution of the tasks. The potential for effective and innovative learning experiences are immense. E-learning has the potential to enable learners with particular needs to engage in learning on a level playing field. However, until accessibility and usability issues are resolved, visually impaired learners will continue to be disadvantaged in terms of cognitive overload,  time and energy resulting in a poorer learning experience than otherwise.

FUTURE WORK

Related areas of study that will be included in future work include: -

· How learners have learned to use the Internet and whether and how they continue to learn.
· How application of the navigational levels framework works in practice - are skills improved?
· Accessibility and usability of the VLEs for visually impaired tutors.
· Further work on Accessing, Using and Doing with a focus on the effect on learning. This could be coupled with an adapted satisfaction rating which will relate to the learning experience. This will need to be carried out with students who are more experienced with the VLE. This could also draw in aspects of learning styles.
· Focusing on specific features of the VLE.

REFERENCES
· Coyne, K., and Nielsen, J., (2001), Beyond ALT Text: Making the Web Easy to Use for Users with Disabilities, NN/g, http://www.NNgroup.com/reports/accessibility (to download report)

· Coyne, K.P., and Nielsen, J., (2001),'How to Conduct Usability Evaluations for Accessibility,' NN/g

· Craven, J., (2002), Yet to be Published (28th November 2002)

· Doyle, C., (2001), Making your module accessible, http://www.uwic.ac.uk/ltsu/5min_guide_module_accessible.htm, last accessed 22nd September 2002

· Evans, S., (2002), 'Using a Virtual Learning Environment to deliver transitional skills at the Royal National College for the Blind,' British Journal of Visual Impairment, Vol.20, No.3

· Frontend, (2000), .Frontend (May 2001), 'Why can't people use eLearning - Why the eLearning sector needs to learn about usability,' http://infocentre.frontend.com/downloads/Why_people_can't_use_eLearning.pdf, last accessed 28th November 2002

· Pearson, E., and Koppi, T.,(2001), 'Guidelines for Accessible On-line Courses,' EDTeC

· TechDis Report (2002), Towards Accessible Virtual Learning Environments http://www.techdis.ac.uk/resources/VLE001.htm