Use of a Mobile Phone to Stream Context Specific Audio Information to Students with Visual Impairment

JISC TechDisHEAT2 Project Report 29/11/2007

Alasdair G Thin, a.g.thin@hw.ac.uk

School of Life Sciences , Heriot-Watt University , Riccarton, Edinburgh , EH14 4AS

Introduction

Overview

In a science practical class a major proportion of the information a student is required to assimilate is communicated via visual media (e.g. written instructions, notes). This presents significant barriers to participation by students with visual impairment. The aim of this project was to investigate a way to overcome these barriers by building on technology developed and widely used in Japan (1) involving the tagging of items with mobile camera phone readable 2-dimensional (2D) bar codes to enable the viewer to interact in some way (e.g. contact number, SMS text message or URL) without relying on user input (2). More specifically this project was intended to test as a proof of concept, the use of a 2D barcodes attached to objects that a student is interacting with and through decoding of the URL contained in the code, a request is made to a streaming media server to play an audio file pertaining to a given object. This is shown below in the following schema:

Photographs of the system in use are shown in Figures 1-4.

Development of 2D Barcodes

The first barcodes developed were linear in nature and initially the coding schemes only represented numbers. The drive to encode more information led to the development of matrix representations which are now commonly know as 2D barcodes (3). Whilst there are a number of different coding schemas, the Data Matrix 2D barcode schema is a public-domain standard and is particularly efficient in terms of space usage (4). Faced with the problem of limited data entry on the small mobile phone keypad and significant likelihood of errors, 2D barcodes offer an alternative means to enter data rapidly and accurately.

Ubiquity Mobile Phones

Mobiles phones as a consumer product have been very rapidly adopted and phones that incorporate a web browser and camera are now commonplace. Their wide ranging functionality and pervasiveness in Japanese society has lead to the development of what is been termed “Keitai Culture” (5). The Nokia N80 mobile phone (6) used in this project has inbuilt WiFi networking and streaming media capabilities in addition to a 3 megapixel camera, making it possible to receive audio streams over WiFi and not incur mobile data network charges. The phone came onto the UK market in April 2006 and such functionality is likely to soon become widespread due to the high rate of phone replacement, especially amongst teenagers and young adults. Rather than compelling students to switch off their mobile phones before the enter they class, phones with the level of functionality described should be viewed as a useful mobile learning platform, particular in settings where space and/or access to desktop computers is limited.

Concept

The primary concept behind this project is the “Internet of Things” (7) in which networks and networked devices are omnipresent. A great deal is being made of the ability to remotely monitor large numbers of items by means of electronic tags (e.g. RFID tags). However, it is not clear how a human user can easily access information about a particular object that they are currently viewing. The utopia of ubiquitous information is therefore in essence a problem of object identification (4). 2D barcodes can provide a way for the user to direct the flow of machine readable information simply by directing the field of view of a handheld camera and so the access of information is under the control of the user. Such an “Internet of Things” coupled with the ability to provide information in alternative media formats (e.g. audio) ubiquitously on request has the potential to be enabling for students with special educational needs. It is also opens up the possibility of fostering an inquisitive approach to learning for all students.

Development

The process of development of the prototype system and the final system configuration adopted is described in this section. A number of issues arose in the integration of the various technologies that necessitated a number of changes. The final solution is shown below both schematically and illustrated with photographs (Figures 1-4) and will be detailed in the following paragraphs.

Audio Streaming Function of Mobile Phone

Whilst the mobile phone (Nokia N80) is able to receive and playback streaming audio files by way of an internet radio application, this functionality is not integrated with the phone web browser (S60 3rd Edition). In order to add any new radio stations (or other streaming audio services) the URLs have to be manually configured via a plain text file on a desktop computer and then transferred to the phone. As an alternative, the possibility of encoding the audio files in Flash format was investigated as the phone has a cut down version of Adobe Flash Player called Flash Lite (version 1.1). However, this too is not capable of playing streamed audio and is not fully integrated within the phone browser. Fortunately, the web browser does have a browser audio plug-in (8) which can play background sound embedded in a web page. The limitation of this approach is that the entire audio file must be downloaded before playback starts.

Audio Solution

The approach adopted therefore was compose audio descriptions, which due to the lack of streaming capability, were necessarily limited to 30-60seconds duration. The audio object description files were produced from text using a commercially available desktop text-to-speech conversion package (TextAloud (9)). The resultant mp3 files were no more than 200kB in size and resulted in a delay of around 3-4seconds download before the audio started.

2D Barcodes

The label printer utilised in the project (Seiko Instruments Smart Label Printer 450) is able to produce Data Matrix encoded 2D barcodes from a URL via the print driver software. Whilst this is a particularly efficient coding schema, the resultant barcodes produced were very small due to an inability to control the pixel size. Since the ability to decode a 2D barcode depends on the size of the pixels and the resolution of the camera, it was found to be difficult to hold the camera sufficiently steady to reliably capture the codes. It is likely that the envisage use of the codes generated by the printer would involve the use of a dedicated optical scanner with a short focal length. An alternative 2D barcode coding schema was sought. The Quick Response schema is a patented specification, however whilst the term “QR code” is trademarked by Denso Wave, inc., the patent right is not exercised (10). In order to enhance recognition by mobile phones, the codes have relatively large target marks on three of the four corners (see revised schema above for example). Sample QR codes were easily and rapidly recognised, even when the camera was held in one hand and the target object with the labelled code in the other. An online QR code generator tool (11) which gave considerable control over the sizing of the code was used to generate the codes.

Mobile Phone 2D Barcode Decoder Applications

Three separate barcode readers were tested (12,13,14). The Quick Mark reader (see Figure 3) was found to be best suited to particular application in that the camera zoom could be easily adjusted and set to the focal distance of this particular application (around 20cm).

Portable Media Server

The portable media server (QNAP TS-109 Pro (15), Figure 1) simply required the insertion of a hard disk which snapped into place inside the casing. Installation of the software drivers and utilities was straightforward on a desktop computer running Microsoft WindowsXP. The media server features include an Apache Web server with PHP support and a MySQL database server with the web based phpMyAdmin management interface. A number of virtual network drives are created during installation for the various features of the media server including one for the streaming of audio files. However, as discussed above, this particular feature could not be utilised.

User Interface

The mobile phone offered two user programmable “Hot Keys” and these were both set as shortcuts to the 2D barcode reader application for ease of use. It had been originally intended to utilise access keys within the phone browser to control the playback of the audio files. However, such functionality is not available in the current browser version. This meant that repeated presses of the browser exit key were required to close the browser and return to the barcode reader application each time the user wished to scan a new code.

Evaluation

The compact size of the media server and the wireless access point (Figure 1) mean that they are readily portable. This means that the system can be operated as a standalone wireless network without requiring any network infrastructure. The lack of browser integration of the various functions on mobile phone meant resulted in the absence of playback control of the audio files. One additional negative factor was that in order to connect to the wireless network confirmation from the user was required for both the action of initiating a connection and the choice of network. Presumably this is intended to make the user aware that a connection is being made and that they could incur mobile data or WiFi access charges.

The setup was first tested with a student with a visual impairment using a collection of models of human bones and joints labelled with 2D barcodes. The student was very positive about the availability of information in audio format and its immediate availability on demand. In addition, they commented that they could see how such technology would enable them to become a more independent learner. However, they found it difficult to operate the mobile phone due to the repeated prompts for conformation. This latter finding was not unexpected by the author.

The setup was also demonstrated to a class of 50 students who were currently studying anatomy whilst they were working in small groups during a practical class. They were impressed with the functionality that the system offered but there were only limited comments on its potential as a learning tool. There was also an opportunity to demonstrate the setup in turn to small groups of students who had taken the anatomy module the previous year. Not only were they very impressed with the setup, but were enthused by the potential to enhance learning. Some even ventured to look up more information about 2Dbarcodes on the internet and check the availability of barcode readers for their particular models of mobile phones. On reflection, it is likely that this difference is due to this latter cohort of students being further on the way to becoming autonomous learners.

This project was successful in that is has demonstrated a practical working solution of the original concept and the utility of the solution to be an enabling technology for students with visual impairment so that they can become more independent learners. However, the state of the mobile phone browser development at the time the mobile phone was brought to market (April 2006) limits it as a platform for students with a visual impairment due to the repeated confirmation prompts presented to the user. This is unfortunate; however the next version of the Nokia mobile browser (16) incorporates a new feature called “Widgets” which has the potential to significantly enhance the functionality of the browser. However, it is unclear whether or not this will resolve all the issues highlighted.

Literally as this project was being concluded, a low cost ultra mobile personal computer (UMPC) has been launched by Asus worldwide (17). It features a 7’’ screen, solid state disk storage, WiFi and an inbuilt webcam and is only 23x17cm in size and weighs less than 1kg. It is an exiting development that has tremendous potential for use as an educational platform (18) and appears to offer similar web browser capabilities as a desktop pc and therefore should permit the inclusion of the required accessible control features without constraints. It is therefore suggested that further development of this application should evaluate the nature and utility of this new UMPC device.