A Bilingual Bimodal Reading And Writing Tool For Sign Language Users

Nedelina Ivanovaa, Olle Eriksenb

The Communication Centre for the Deaf and Hard-of-Hearing, Reykjavk, Grenssvegur 9, 108 Reykjavk, Iceland. E-mail: nedelina@shh.is b Mller-Trndelag Kompetansesenter, Postboks 175 Heimdal, 7473 Trondheim, Norway. E-mail: olle.eriksen@statped.no Abstract
Sign language is used by many people who were born deaf or who became deaf early in life as their first and/or preferred language. There is no writing system for sign languages; texts are signed on video. As a consequence, texts in sign language are hard to navigate, search and annotate. The BiBiKit project is an easy to use authoring kit which is being developed and enables students, teachers, and virtually everyone to write and read bilingual bimodal texts and thereby creating electronic productions, which link text to sign language video. The main purpose of the project is to develop software that enables the user to link text to video, at the word, phrase and/or sentence level. The software will be developed for sign language and vice versa, but can be used to easily link text to any video: e.g. to add annotations, captions, or navigation points. The three guiding principles are: Software that is 1) stable, 2) easy to use, and 3) foolproof. A web based platform will be developed so the software is available whenever and wherever. Keywords: tools; part of speech tagging; sign language
a

1. Introduction
Many people who were born deaf or who became deaf early in life use sign language as their first and/or preferred language. The number of deaf sign language users is relatively small and in most EU countries, deaf learners are integrated in mainstream education. Each school, each teacher who has a deaf student in his/her class struggles to find solutions to circumvent the reading/writing problems of the deaf learner: This applies equally to primary school, secondary school, vocational training, university, formal and informal adult education. There is no writing system for sign languages; texts are signed on video. As a consequence, texts in sign language are hard to navigate, search and annotate. Often, they are difficult to 'read', because all you see is a frontal signer, comparable to traditional frontal teaching. Although researchers have promised text-to-sign conversion for over a decade, results are very limited. Conversion at word-sign level is not sufficient; each sign language has its own grammar, so text has to be translated, not just converted. There is some existing 'Text-Video linking' software; 3 'types' of linking or linked products were found: Programs developed for sign language research: many functions, but often difficult to use: ELAN, ILEX, SignStream and Berslab; Programs developed for education: Easy to use, but most are outdated, and many no longer work: Link-it, SignSmith, TegnBehandler and TegnSpiller; Programs for website developers: JoomLink, SignLinkStudion 2.0 and Videotext web; There are also linked productions that are made 'by hand', there is no editor, and often each 'link' starts a new video. As said above, there is no software which is so

user friendly, that young children can use it to write their own bilingual stories. There is no software which is so easy to use, that even the children's grandparents can use it to look at the stories of their grandchildren, or to write their own stories. There is no software which is so flexible, that not only children and grandparents, but also students, teachers, authors, and publishers will want to use it.

2. The BiBiKit Software

BiBiKit is an acronym for Bimodal Bilingual Kit for reading and writing tool for sign language users. The name sign refers as follow to: 1. The right handshape refers to: two languages, or two modalities. Also to: using your eyes to read, and to watch video. 2. The movement refers to 'translation'. 3. The left-handshape is for ' page', or 'tablet'. The BiBiKit is an easy to use authoring kit which is being developed and enables students, teachers, and virtually everyone to write and read bilingual bimodal texts and thereby creating electronic productions, which link text to sign language video. The main purpose of the project is to develop software that enables the user to link text to video, at the word, phrase and/or sentence level. Each word, each sentence in the text can then be linked to a sequence in the video. Video can be a translation of the text signed by the author, signed by a sign language interpreter or teacher, or even imported video of a signing 'avatar'. Documents can be saved in various formats. Readers can read text and click on a word or sentence to see the linked video. The software will be developed for sign language and vice versa, but can be used to easily link text to any video: e.g. to add annotations, captions, or navigation points. The actual work will be done in an Editor and the ready product viewed in a Player. The three

3774

guiding principles are: Software that is 1) stable, 2) easy to use, and 3) foolproof.

2.1 How to make it happen?

The BiBiKit is a 2 year project; the starting date was January 2011. In the first year, the software is developed. In the second year, the software will be tested by using it in different contexts. In the BiBiKit consortium, 6 organizations from 6 EU countries work together. To involve more future BiBiKit users, colleagues and members of other EU consortia are asked for input. They are called 'associated partners'. Each organization has 4 associated partners from different EU countries or 30 in the whole, from 12 EU countries. Associated partners are very important because they help to describe the functionality of the software. The associated partners have given feedback on what they want to use BiBiKit for, and what their requirements and/or wishes are. The different prototypes are shown to them and they asked for feedback: Is this what they want? The BiBiKit will be validated in over 30 educational settings in 12 countries, by deaf and hearing writers and readers. Pros

Cons

Conclusion

2.2 The BiBiKit: target group

The target group is very large, and very diverse. Users can be 8 years old, or 88 years old. Users can be children, parents, teachers, students, researchers and publishers. The wishes of these users are very diverse, too. They go from very easy, basic functionalities only to very flexible, as many functionalities as possible. For instance: Basic users want one text window, one video window, and an easy way to link text to video. Researchers want more text windows, more video windows, and the option to link video to video, to link to sign libraries, maybe even to use SignWriting or HamNoSys. As for BiBiKit, was decided to start at the easy end of the spectrum. User requirements and software specifications are written in cooperation with the associated partners. Each partner focuses on the user needs of a specific target group/educational context: Deaf children, deaf students in higher education, sign language interpreter students, sign language teachers, formal and informal adult learning and e-learning.

Web application Run on a webserver and accessed with web browser Can be used on PC/Mac/iPad and other units with a web browser; No installation; Central maintenance and updates; Maintenance of server; Storage for large video files; Slow response and upload of video files; Test video response and upload times; Possible to edit on webserver and download the result (no storage on server)?

Local application AIR application run on a local PC/Mac Local video (user responsibility); Fast response and no upload of video files; Can be used without Internet connection; Limited to use on PC/Mac (no iPad); Program must be installed; Limited choice of video formats; Test AIRapplication; Check possibilities for video conversion;

Table 1: Platform for the Editor HTML5 Pros Can be used on PC/Mac/iPad and other units with a web browser; Can be run from local disk, CD/DVD and web server; Might be differences in layout and functionality for different web browsers; No common video format for all web browsers; Test different web browsers (IE, Safari, Chrome, Mozilla, Opera);

Cons

Conclusion

Table 2: Platform for the Player It was decided to develop a local platform for the Editor and the Player as to test different web browsers. The application platform As said above, for BiBiKit the first priorities are: Stable: The software should not crash and should work 2 or even 5 years from now. Easy to use: Children should be able to use the software, but also their grandparents. Users can use BiBiKit right away, they do not have to go to a training course first, or read a thick manual. Foolproof: Even when the user makes a mistake (he/she clicks on the wrong button, or he/she forget to click on a button), the programme should not crash; he/she should not lose his/her work. When the user makes a mistake, he/she should always be able to correct it.

2.3 The BiBiKit: technichal details

After closer look at the available programming options, was concluded that the best solution would be either to make a local application (Adobe Air), or a web application. Some pros and cons for the two options for the Editor and the Player are listed in Table 1 and Table 2 bellow:

3775

BiBiKit will in some ways resemble PowerPoint: It is an editor or an authoring system, that authors can use to create their own content, that others can read and look at. The unique feature of BiBiKit is that authors can link text and pictures to specific sections of video on the same page and vice versa. Because users want to be able to read productions on desktop computers, but also on tablet PCs and even smart phones, it was decided that the output, the linked productions, can be read (but not edited) with standard browsers like the Internet Explorer and Safari. In the working process was dedicated that no other software, i.e. player, will be needed to read the BiBiKit productions (see 3.2). When exported, the project can be run in a HTML5 compatible browser. The BiBiKit editor The main feature of the BiBiKit Editor is that it can be used to create multimedia productions: electronic productions with links between text, pictures, and video. This is a very useful feature, not just for sign language users but for many other target groups, and for a diversity of productions. There are a number of programs which can be uses to integrate text, pictures and video, but the important feature for BiBiKit is that the user can make links to parts or sections of video clips and not only to the whole clip.

Figure 2: Linked text in colour Changes that the users can make are as follow: add text, correct links, add new video and new picture, both as background and as a clarifying part of the document . So teachers can correct stories made by children. Students can open assignments made by teachers. Authors can work on stories, together. 3.1.1 Feedback of users Having linked sign language and written language it is easy to preview the narrative in a playing mode. In the programme it is easy to make bilingual learning materials for deaf students. The tool is especially valuable in teaching languages to deaf learners. The programme can also be used for making teaching materials in sign language for hearing persons. The users found the interface simple and userfriendly. They liked that sign language is the source language and the important aspect of a continued availability of the learning material and the possibility for repetition, e.g. the student may look at and examine the written text and the sign language text repeatedly, back and forth and at such speed as he/she wishes or needs and can compare the languages involved as often as he/she wishes all in the now. The parents of deaf children have very different sign language skills, that is why teachers found it useful to be able to create learning materials and tasks for different skill levels. The possibility to respond to parents' work, i.e. teachers can communicate about the tasks with the parents was evaluated as very important and original. The basic framework offers great potential for sign language teaching and learning; linking text and signing as a teaching tool makes sense. Students can even write their own version of the signed feedback as a learning opportunity and compare their version with the tutors. The recommendations regard mostly the technical part, see below. The most important of them is that the student can easily make his own materials and that the interface will not get complex. The development team should keep it simple! 3.1.2 Technical results Prototype 1 was made primarily to test the functionality and the user interface. It turned out that users found it difficult 1) to understand how to link text, picture to different video files in the same video frame and 2) the questions about the video format. Screen buttons/ controls need to be clearer and more human/computer

3. BiBiKit: Work In Progress

The program team made two prototypes which users got to try and the users gave feedback on them.

3.1 Prototype 1
Prototype 1 was developed to test the basic functionalities, and to help deciding some of the difficult questions. Prototype 1 is now finished. It cannot be used yet to save documents. The lay-out was not decided and it may still changes: Different buttons, or buttons in different locations. Prototype 1 really is only the first prototype. Prototype 1 is the result of a selection of wishes from all wish-lists- using the three priorities as guideline. Prototype 1 was developed to test these basic functionalities, and to help deciding some of the difficult questions. The basic functionalities are writing a text in a text box, linking it to a video sequence, import bakground for the project (see Figure 1) and the linked text changes colour while the video is played (see Figure 2):

Figure 1: Background

3776

interaction is needed. Video loading time is too long and the video file types are limited. BiBiKit should work on computers with standard configuration. The users found it also difficult to upload video clips. The users commented on what they needed but it was missing in Prototype 1. They want to be able to have multiple (at least 2) video-frames on each page, side by side, at the same time and to use the Editor online so that students and teachers can work on the same project(s), without having to send large files back and forth; e.g. the video file should always be attached when viewing the ready product. A possibility to write directly into the text window with a pen (touch screen) should be an option if the application is locally. The users found it also necessary to be able to insert only audio files so it is possible to follow the story in sign language without inserting a textual description and to save the audio files as to be able to insert subtitles. The commercial publishers requested the option to lock their productions to protect them from unauthorized re-use of text, pictures and videos. They stressed the importance of producing BiBiKit projects for multiple platforms: desktop (Windows and Mac), tablets (iPad and Android), and smartphones (all platforms). The main challenge is still the video navigation and the different video formats. It is not so easy for users to understand that their video files will not work because the video is in a wrong format. This problem must be solved.

Figure 5: BiBiKit Player The functions of Prototype 2 is very similar to the first prototype apart from the possibility to save (and open) projects. All files the user needs for a project are stored in a separate project folder with a respective name. If the user decides to start a new project and to use a video file which has been used in the previous project, this video file is saved again in the folder of the new project. 3.2.1 Feedback of users Partners and associated partners produced their own materials in Prototype 2. They found Prototype 2 nice looking and easy to use in general. It is not longer an imagined software and it is possible to try it. Many recommandations given for Prototype 1 have been taken into account. The users underlined again that they want BiBiKit on the Internet, not local. BiBiKit should be accesible wherever and whenever. 3.2.2 Technical results We had some of the same challenges in Prototype 2 as in Prototype 1: The main challenge still remains the video navigation and the different video formats. The software is to slow even for using it local. The SAVE function in the Editor does not work as it should: If the user works for a short time and closes the program, the written text is gone. If he/she works for a longer time, e.g. 20 min., the written text is there when he/she opens the saved project. It is not possible to forward fast. The software crashes but when working slowly, e.g. forward video, it does not crash. It is not so easy to correct a mistake after the user has opened a wrong video file for a project. New folders are created automatically with too many files inside each time the user makes mistake starting a new project. Every video file which is been opened is saved each time in a new folder, even though the wrong video file has been opened. The user has to delete each folder manually. This is not easy for children and people with limited computer skills. Two videos in one frame are not still an option.

3.2 Prototype 2
Prototype 2 is a better working version of Prototype 1 where the user can produce, save and open his own documents. Prototype 2 has an Editor (Figure 3 and 4) and a Player (Figure 5):

Figure 3: BiBiKit Editor

4. Conclusion And New Strategy

The software team developed two prototypes as local Air applications (made in Flex) that will run on Widows and Mac PCs. The AIR applications accepted only a limited number of video file formats. Apparently there were

Figure 4: BiBiKit Editor

3777

more problems on Mac than on Windows, but the main problem is the limited choice of video formats. The prototypes worked fine for us (the developers) as we knew which video file formats to use. In real life testing the users had problems understanding and accepting the limitations regarding video file formats, and lacked the knowledge of how to convert the video into acceptable formats. During the last year Apple has managed to kill Flash, and there has been a shift towards web applications and HTML5. Unfortunately HTML5 is not mature yet, and it had probably been better if the BiBiKit project had been started later. From the perspective of the software developers, it should have been waited 2-3 years, to see how the battle between Flash and Apple is decided, to wait for the maturation of HTML5, and to see if and how web- and cloud-based services will become the standard. Our target groups, however, cannot wait. They say that theyve been waiting for something like BiBiKit, for much too long already! We will develop a web application, an Editor, with basic functionality. The advanage will be that the user can use any video format, i.e. the video files are converted to the standard HTML5 video formats. To solve this problem we will now make a web application that will do the video format conversion. We will also try to make it possible to run the web application locally o n the users own PC because the problem of setting up and maintaining a server is still there.

5. Acknowledgements
The BiBiKit team is grateful for the financial support from the European Commission, Project Number: 511588-LLP-1-2010-1-NO-KA3-KA3MP.

6. References
Berslab. http://www.ru.nl/publish/pages/610183/ slcn4_poster_ebbinghaus.pdf. BiBiKit. http://www.bibikit.eu. ELAN. http://www.lat-mpi.eu/tools/elan/. HamNoSys. http://www.sign-lang.unihamburg. de/projects/hamnosys.html. ILEX. http://www.sign-lang.uni-hamburg.de/ilex. JoomLink. http://www.equalaccess.nl/ ?option=com_signon&task=list_one&a_id=1. Link-it.http://www.spsm.se/Startpage/Educational materials/Deaf-and-Hard-of-Hearing/Products/ Link-it. SignLinkStudio 2.0. http://www.signlinkstudio.com. SignSmith. http://www.shh.is/FileLib/skjalasafn/ Guidelines_for_teachers_Oct_2003.pdf. SignStream.http://www.bu.edu/asllrp/signstream/ index.html. SignWriting. http://signwriting.org. TegnBehandler. http://www.acm.no/tegnbehandler. TegnSpiller. http://www.acm.no/materiell/support/ tegnspiller/tegnspiller_lesmeg.htm. Videotext.web. http://www.oegsbarrierefrei.at.

3778