1. Introduction
This paper introduces one of the activities of ICER (Innovation Centre for Educational Resources) in Kyushu University Library of Kyushu University, Japan. ICER [1] was established in April, 2011 and its mission is to promote the higher education based on instructional design, the development of educational materials using recent ICT (Information Communication Technologies), and e-Learning by recoding lectures, editing and releasing those videos as SPOCs or MOOCs in the university.
Efficient education will be achieved with attractive educational materials that would be realized by the most recent ICT like 3D CG (Computer Graphics) and VR (Virtual Reality) because current students sometimes called video game generation are used to operate such contents. However, there is the problem that the development of such contents requires technological knowledge and programming skills but usual teachers do not have such knowledge and skills. Teachers need any tools to make it easier to develop attractive educational materials that employ the most recent ICT. So, we have already proposed a framework for the development of web-based interactive 3D educational materials [2,3]. The framework should support the most recent web technologies, i.e., HTML5 (Hyper Text Markup Language) and various JavaScript Libraries including WebGL (3D Graphics Library for JavaScript). In general, interactive 3D contents include various types of media data like texts, audio, videos and 3D geometry data with their animation data. The framework should also support these media data. There are some story-based contents to follow an instructional design. Also, the framework should accept such story information. Based on these requirements, we decided to adopt Excel data for defining story-based state transitions of media data like Gantt chart, one of the visualization methods. As a result, the preparation of such Excel data is the all what teachers need to do. Although, unfortunately, the current framework cannot directly accept such Excel data, it can accept a simple JavaScript file that defines a story called ‘Story Definition File’. This paper describes its details. As a case study, we developed one material of Japanese History education with the proposed framework. Because of the web-based contents, those are available on various platforms such as iPhone, iPad, and Android tablets besides standard desktop PCs. This is significant for BYOD (Bring Your Own Device) classes and Mobile Learning.
For developing attractive educational materials, such materials need multimedia contents like images and videos related to the teaching target. It is difficult for common teachers to make such images and videos. If the multimedia contents can be retrieved from the Internet, the development cost can be reduced. To do so, we focused on Linked Data because Linked Data is the location information of required data on the Internet [4]. Then, we have already proposed an authoring tool of educational materials based on Linked Data [5] and integrated the authoring tool into the first development framework [6,7]. In this paper, we also introduce the integrated development framework based on Lined Data and its educational materials of two teaching targets, i.e., IoT security education and Japanese History education.
Recently, several companies produce 360VR cameras by reasonable prices and it has become easier to develop VR contents of real (physical) scenes using 360VR camera images/videos than ever. So, we also focused on the use of 360VR images/videos in educational materials. We have already proposed another educational materials development framework that supports 360VR images/videos [8,9] and integrated it with the authoring tool based on Linked Data [10]. Finally, we introduce this integrated development framework based on Linked Data and its example educational material for IoT security education.
The remainder of this paper is organized as follows: next Sec. 2 describes several related works. In Sec. 3, we explain the details of the framework for the development of web-based interactive 3D educational materials. its design and functional components. We also introduce its educational material example. In Sec. 4, we introduce the authoring tool based on Linked Data and the integrated development frameworks with it. We also show its two educational material examples. In Sec. 5, we explain the development framework that support 360VR image/videos and the integrated development framework with the authoring tool. We also show its educational material example. Finally, in Sec.6, we conclude the paper and discuss about some future works.
2. Related Works
2.1 Development environments
There are many development systems and tools for 3D contents. Some of them are commercial products like 3D Studio Max, Maya, and so on. Although these products can be used only for creating 3D CG images or 3D CG animation movies, usually, these cannot be used for creating interactive contents. As a development system for 3D interactive contents, there is IntelligentBox, a constructive visual software development system for 3D graphics applications [11,12]. This system seems very useful because there have been many applications practically developed using it. However, it cannot be used for creating web-based contents. Although there is the web-version of IntelligentBox [13-15], it cannot be used for creating story-based contents. IntelligentBox and IntelligentPad [16] are regarded as 3D meme media and standard meme media, respectively, and also Webble [17] is regarded as the latest version of IntelligentPad. With Webble, it is possible to create web-based interactive contents through simple operations for authoring and of course, possible to render 3D graphics assets. However, it does not have functionalities of IntelligentBox.
There are some electronic publication formats like EPUB [18], EduPub [19], iBooks [20] and their authoring tools. Of course, these contents are used as educational materials. Although recently these have become available on the Web, they basically do not support 3D graphics except iBooks. iBooks supports rendering functionality of a 3D scene and control functionality of its viewpoint. However, storybased contents cannot be created using it.
From the above situation, for creating web-based interactive 3D educational materials, we need any dedicated toolkit systems. The most popular one is Unity, one of the game engines [21]. Also, Unity supports creating web-contents. Practically, using Unity, we have developed 3D educational materials [22,23] for the medical course students of our university because Unity is a very powerful tool that supports a lot of functionalities. However, the use of Unity requires any programming knowledge and skills of the operations for it. Therefore, it is impossible for standard end-users like teachers to use Unity. As a result, we have proposed the framework described in this paper.
2.2 Linked data
Berners-Lee coined the term Linked Data describing a set of best practices for publishing and connecting structured data on the Web [24,25]. These practices are the foundation of the evolution of the Web of Documents to the Web of Data, a global data space connecting data from a multitude of different domains. To make part of a single global data space, Berners-Lee proposed several (technical) rules for publishing data on the Web that have become known as Linked Data Principles [24]:
- Use URIs as names for things.
- Use HTTP URIs so that people can look up those names.
- When someone looks up a URI, provide useful information, using the standards (RDF, SPARQL).
- Include links to other URIs, so that they can discover more things.
In summary, Linked Data is about data integration [26]. In the e-Learning, such data can be shared as learning objects (LO) or learning entities of any kind, respectively. Dicheva identifies three generations of web-based educational systems [27]. The systems of the first generation provide a central entry-point for accessing learning materials and online course, e.g., LMS (Learning Management System) and educational portals. The systems of the second generation employ Web and AI technologies to support intelligently personalization and adaption. Such systems are called educational adaptive hypermedia systems. The third generation of web-based educational systems is a class of ontology-aware software, using and enabling Semantic Web standards and technologies to grant scalability, reusability and interoperability of educational material that is distributed over the Web. Therefore, Linked Data technologies raise high expectations with respect to providing solutions in a field like e-Learning.
Resource Description Framework (RDF) is a language for representing information (metadata) about resources in the World Wide Web. The resources are not limited to web pages but can also include things that can be identified on web. [28] RDF data is written in XML format, machine readable, and can be output by machines. However, it is not designed for being displayed to people or designed to be displayed on the web, but computers. By using RDF, information of IoT security on the web can be integrated. RDF can describe the relationship between resources, but it has ambiguity like XML. The RDF Schema (RDFS) is required to reduce the limitation [29].
RDF Schema is RDF's vocabulary description language. RDFS is a standard language to create ontology in RDF. The ontology will define the vocabulary of RDF documents in a specific domain and manage the hierarchical relationship between terms. The ontology also has rules for reasoning of hidden information in RDF. For retrieving rich and accurate query results, advanced retrieval schemes using both metadata and ontology are considered [30].
3. A Framework for the Development of Web-Based Interactive 3D Educational Materials
Our targets are web-based interactive 3D educational materials. Such materials should have any story. Each story is realized by several 3D scenes consisting of several architecture objects like buildings and houses, and several moving characters like humans who have their own shape model and animation data. Such 3D assets data should be prepared. Next, material creators need to define a story as one JavaScript file called ‘Story Definition File’. In our proposed framework, the requirements for creating a material are 3D assets data and ‘Story Definition File’.
Figure 1 shows functional components of the proposed framework, main components (Main.html) and sub components (AnimationCharacter.js). The main components include function related to architecture objects and functions related to AnimationCharacter objects represented for moving characters. The sub components consist of the constructor of new AnimationCharacter class and its member functions. Besides main and sub components, our framework employs Three.js [31], one of the WebGL based 3D Graphics Libraries as subsidiary components.
When developing a story-based interactive 3D educational material with the proposed framework, a teacher needs to prepare one story definition file and 3D assets. The following subsections describe those details separately.
3.1 Sub components of framework
3.1.1 New class for moving characters
function AnimationCharacter( name )
{
-
this.name; // unique name of moving character
this.color = 0xFFFFFF; // color
this.daes; // set of animation data
this.animatio //
this.num_data = 0;
this.cur_data = -1;
this.locations = new Array(255);
this.num_locations = 0;
this.cur_location = -1;
this.camera = null;
this.camera_height = 10.0;
this.walk_no = 0;
// internal
this.next_position = new THREE.Vector3( 0.0, 0.0, 100.0 );
this.speed = new THREE.Vector3( 0.0, 0.0, 0.0 );
this.rotation = 0.0;
This is a constructor of AnimationCharacter object represented for a moving character and its member variables.
3.1.2 Member functions of AnimationCharacter object
init(), current_position(), …..draw_path(), and so on.
These are defined in the JavaScript source file named AnimationCharacter.js. Therefore, the main JavaScript source file should always include AnimationCharacter.js file as shown in Figure 1.
3.2 Main components of framework
The main components of the framework are defined in a certain HTML file, i.e., Main.html in Figure 1 and it includes various types of JavaScript functions. List of the JavaScript functions and their functionalities are described below.
3.2.1 Function related to architecture objects
- function ObjMtlLoad( objname, mtlname, b, pos, rot, scal );
Example
ObjMtlLoad( 'jinnoza_s3.obj', 'jinnoza_s3.mtl', false, [135, 1.34, -130], [0.0, 90.0, 0.0], [0.5, 0.5, 0.5] );
This is defined using functions in MTLLoader.js and OBJMTLLoader.js of Three.js libraries. This function reads 3D model data in the obj file format with its material file. This function is used for loading architecture objects defined as obj format files. Here, objname is the filename of a obj format file to be loaded, mtlname is the material file name of the obj format file, b specifies one of the shading modes like flat shading or smooth shading, pos means x, y, z values of the initial position, rot means x, y, z rotation angles of the initial orientation, scal means x, y, z ratios for the initial size.
3.2.2 Functions related to AnimationCharacters
- var AnimChars = new Array(AnimCharsInfo.length);
For (var i = 0; i < AnimCharsInfo.length; i++)
{
AnimChars[i] = new AnimationCharacter();
AnimChars[i].init( AnimCharsInfo[i][0], AnimCharsInfo[i][1],
AnimCharsInfo[i][2] );
}
Example
var AnimCharsInfo = [ ['daijin', 0xFF0000, 48], ['daiben', 0x0000FF, 4], ['kanjin', 0x80FFFF, 3], ['shi', 0x00EE00, 27], ['ben', 0x00FFFF, 3], …, ['konoe', 0xFF8080, 1], ['enza', 0xFFFFFF, 1], ['enza2', 0xFFFFFF, 1], ['enza3', 0xFFFFFF, 1] ];
This part initializes an array of AnimationCharacter objects, its variable name is ‘AnimChars’, according to the information defined in the variable ‘AnimCharsInfo’ whose each element specifies an name of the character, its color and the number of its animation data.
In this example case, the number of animation data for ‘daijin’ is 48 so that the following DaeLoad(…) function should be called 48 times.
- function DaeLoad( charname, no, daename, b /*1 or 0 = repeat or not*/, pos, rot, scal );
Example
var character_size = 5.0;
DaeLoad( 'daijin', 0, '1_01_daijin_walk.dae', true, [10.0, 0.0, 100.0],
[0.0, 0.0, 0.0], [character_size, character_size, character_size] );
Animation data of each AnimationCharacter object is defined as dae file. Dae is the extension of a Collada file. Collada file format was proposed by Sony Computer Entertainment America in 2004 and it is used for 3D models and their animation data. Here, charname like ‘daijin’, ‘daiben’ specifies one of AnimationCharacter objects. no means the position of the animation array data prepared in AnimationCharacter object. daename is the dae file name. b is Boolean that specifies the included animation data is repeatedly played or not, e.g., a walking animation should be repeatedly played. pos means x, y, z values of the initial position, rot means x, y, z rotation angles of the initial orientation, and scal means x, y, z ratios for the initial size.
- function AddLocation( charname, x, y, z );
Example
var default_height = 0.0;
AddLocation( 'daijin', 240.0, default_height, -140.0 );
AnimationCharacter object can move freely from one location to another location in a 3D scene according to the story. In one story, each AnimationCharacter object has one moving path as a set of sequential location data. This function is used to add locations to the moving path one by one. Here, charname like ‘daijin’, ‘daiben’ specifies one of AnimationCharacter objects. x, y and z are x-, y- and z-coordinate values of a location to be added to the path.
- function DrawPath( charname, col, thi, hei );
Example
var default_height = 0.0;
DrawPath( 'daijin', 0xff0000, 5, drawpath_height + 0.0 );
This function draws the moving path of AnimationCharacter object specified by charname. Here, col, thi and hei specify the color, the thickness and the height of lines of the path, respectively.
3.3 Story definition file and 3D assets files
A story consists of several animations of moving characters. There are two execution modes, i.e., step mode and task mode. One step is the minimum unit of animation play. In step mode, the animation play is executed one step by one step. On the other hand, one task consists of several steps. In task mode, the animation play is executed one task by one task. The definition of each step is one of the following and those meanings are commented;
[‘daijin’, ‘next’] /* ‘daijin’ move to next location */
[‘daijin’, ‘1’] /* ‘daijin’ plays its animation data of No. 1 as its next
action*/
The definition of each task is just an integer number that means how many steps will be executed from the current in the task.
3.3.1 Variables for defining a story
A story is defined as a set of several steps and a content creator needs to define it as a certain variable called step_mode_steps. Furthermore, for the task mode, the creator also needs to define tasks as a set of several integer numbers that can be defined by a certain variable called task_mode_steps.
Example
var step_mode_steps = [ [‘daijin’, ‘next’], [‘daijin’, ‘1’], …];
var task_mode_steps = [ 1, 5, 1, … ];
There is another variable to define text data of a story. This variable called string_contents is an array data of text strings.
Example
var string_contents = [ ‘daijin moves to next location‘, ‘daijin sits down‘, … ];
3.3.2 Initial function to define a story
function InitStory()
{
-
ObjMtlLoad(…);
….
DaeLoad(… );
…..
AddLocation(… );
…..
DrawPath(… );
This function is always called once inside the main components file to activate the above variables and to load 3D assets data of architecture objects and animation data of moving characters.
3.3 3D assets necessitated for defining a story
For architecture objects like buildings and houses, a content creator like a teacher needs to prepare those data as obj format files. Furthermore, for moving characters, the creator needs to also prepare those animation data as dae format files.
3.4 Case Study
We have been developing several educational materials with the proposed framework. In this subsection, as a case study, we introduce one of them that is ‘Kanso’ material for Japanese history education.
3.4.1 ‘Kanso’ material of Japanese history education
‘Kanso’ is one of the ceremonies taken in ‘Gosyo’. ‘Kanso’ includes various types of manners of ‘Tennou’ and ‘Daijin, etc. In Japanese history education, students need to learn such manners by reading old documents about ‘Kanso’. However, it is very difficult to understand the manners from the old documents because such documents written in an ancient calligraphy. Using this interactive 3D educational material of ‘Kanso’ shown in Figure 2, it becomes easy to understand.
As shown in Figure 2, there are three houses as architecture objects. The middle one is ‘Shishinden’, the left one is ‘Seiryoden’, and the right one is ‘Jinnoza’. These architecture objects are prepared as three obj format files with their material files.
Figure 3 shows two different scenes at ‘Jinnoza’ and at ‘Seiryoden’. As moving characters of ‘Kanso’, there are seven characters, i.e., ‘Tennou’, ‘Daijin’, ‘Shi’, etc. Each moving characters are implemented as instances of AnimationCharacter class in JavaScript. AnimationCharacter has its several animation data prepared as dae format files. For this material, we prepared around 100 dae format files. To reduce the cost to prepare these data files, we employ 3D human pictograms for the shape data of moving characters. Since their colors are different, it is possible to recognize each character. For example, the color of ‘Tennou’ is yellow, ‘Daijin’ is red and so on. There are seven poly-lines in a different color shown in Figure 3. Each is the moving path of each corresponding character. Its color is the same as the pictogram.
At the upper and center position of the two scenes, texts data are displayed in white color. These texts explain the behaviours of the moving characters. By reading the texts, the students can understand the manners in ‘Gosyo’ more deeply with looking at the manners as 3D CG animations.
4. Integrated Development Frameworks
In this section, we introduce integrated development frameworks for educational materials based on Linked Data in the two cases of Japanese history education and IoT security education. We integrated the knowledge and concepts of teaching targets, Japanese history education and IoT security education, based on Linked Data. As the same time, we built up the RDF store for storing designed RDF data based on Apache Jena. Thereby, the integration and retrieval of the information of each teaching target which is designed based on RDF by SPARQL are implemented.
4.1 Integrated development framework for Japanese history education
This subsection introduces an integrated development framework for the educational material of Japanese history.
4.1.1 Design of RDF store for Japanese history education
There are three steps in designing the Linked Data for Japanese history education. In the first step, we designed the schema of RDF store. An example of RDF store schema is shown in Figure 4. As the figure shows, we designed a new prefix for Japanese history knowledge according to the RDF schema document. In the second step, an excel file of Japanese history knowledge is created according to the schema. In the third step, the excel file of Japanese history knowledge is converted to turtle format or RDF format by using OpenRefine [32], a free, open source, power tool for working with messy data.
4.1.2 Implementation of RDF store for Japanese history education
There are three steps for the implementation of RDF store of Japanese history education. In the first step, RDF store is built up based on Apache Jena, which is capable to store the integrated data as shown in Figure 5. In the second step, the exported Turtle File of Japanese history knowledge is stored into the RDF store as shown in Figure 6. In the third step, the retrieval of the information of Japanese history through virtuoso [33] (a server works as RDF store) by using the query language is carried out as shown in Figure 7.
4.1.3 Authoring tool for the educational material of Japanese history
Figure 8 shows the overview of our authoring tool and Figure 9 shows the screen image of our prototype system of an authoring tool [7]. By specifying a keyword ‘kyuchu’ (Imperial court) as the target of an educational material, the system retrieves text data, image data and video data related to the keyword from RDF stores. There are several candidate data for each of text, image and video so the user can choose one of them by the operation on [back], [next] buttons. In this way, by using the authoring tool, educational material developers can edit the contents of the material regarding Japanese history knowledge managed in RDF stores easily and efficiently.
After achieving the authoring of Japanese history educational material, such an educational material is supposed to be shown as readable format like ePub/EduPub that facilitates the knowledge of Japanese history to be learned on the Web.
4.1.4 Integrated development framework for Japan history education
Figure 10 shows a screenshot of the interactive web 3D educational material for ‘Kanso’ integrated with the authoring tool functionality executed on a PC browser. If students click the mouse on any moving characters or architecture objects, its detail information will appear on the browser window as shown in the left part of the figure similarly to Figure 9. In this way, the students can learn ‘Kanso’ ceremony more deeply than the case of Figure 2 and 3.
4.2 Integrated development framework for IoT security education
This subsection introduces an integrated development framework for the educational material of IoT security.
4.2.1 Design of RDF store for IoT security information
There are three steps in designing the Linked Data for IoT security we are following. In the first step, we designed the schema of RDF store. An example of RDF store schema is shown in Figure 11. As the figure shows, we designed a new prefix for IoT security, because there is no existed prefix for IoT security field according to the RDF schema document. Meanwhile, IoT security information contains the contents of Table 1 that are thought important as IoT threats information.
In the second step, an excel file of IoT security is created according to the schema. In the third step, as Figure 12 shows, the excel file of IoT security is converted to turtle format or RDF format by using OpenRefine. Figure 13 is the Turtle files of IoT security saved as RDF format.
4.2.2 Implementation of RDF store for IoT security information
There are three steps for the implementation of RDF store of IoT security. In the first step, RDF store is built up based on Apache Jena. In the second step, the exported Turtle File of IoT security is stored into the RDF store as shown in Figure 14. In the third step, the retrieval of the information of IoT security through virtuoso by using the query language is carried out as shown in Figure 15.
4.2.3 An authoring tool for the educational material of IoT security
Figure16 shows the screen image of the authoring tool [5]. By specifying a keyword ‘SmartMeter’ (IoT device name) as the target of an educational material, the system retrieves text data, image data and video data related to the keyword from RDF stores. There are several candidate data for each of text, image and chart so the user can choose one of them by the operation on [back], [next] buttons. In this way, by using the authoring tool, educational material developers can edit the contents of the material regarding IoT security knowledge managed in RDF stores easily and efficiently.
4.2.4 Integrated development Framework for IoT security education
Figure 17 shows a screenshot of the e-learning material as 3D CG for IoT security education integrated with the authoring tool functionality executed on a PC browser. Since our current target is Smart Building/ Smart Home, there are several IoT devices located inside the 3D model of a room as shown in the figure, e.g., a mobile phone, a room light, an air-conditioner, TV, PC, and so on, those are connected to the Internet. The purpose of this e-learning material is teaching the contents of Table 1 to students. Using this material, firstly, a student will read the explanation about IoT. Next, after understanding the meaning of IoT, he/she will start to look for any IoT devices in the room through the mouse operations interactively. If he/she find an IoT device and click the mouse on it, its detail information according to the contents of Table 1 will appear on the browser window as shown in the left part of the figure similarly to Figure 16. In this way, the students will be able to learn IoT threats. For this, we newly added an object picking functionality to the framework.
5. Integrated Development Framework with the Authoring Tool Based on Linked Data that Supports 360VR Images/ Videos
Similarly to the e-learning material for IoT security education shown in Figure 17, we have been developing an e-learning material for IoT security education that supports 360VR images/videos [10] using the database of IoT threats information realized as Linked Data and the web-based viewer of 360VR images/videos [8,9].
Figure 18 shows a screenshot of the e-learning material that is a certain part of 360VR image for IoT security education executed on a PC browser. Since our current target is smart home/building, there are several IoT devices located inside the 360VR image of a room as shown in the figure, e.g., a smart meter, a smart speaker, PC, and so on, those are connected to the Internet. The purpose of this e-learning material is teaching students the information about IoT devices and their threats shown in Table 1. Using this material, a student can change his/her view-direction freely for finding out any IoT devices in the real (physical) room through the mouse operations interactively. If the student finds an IoT device and change his/her view-direction onto the device, its detail information derived from the contents of Table 1 will appear on the browser window as shown in the left part of the figure similarly to Figure 16 and Figure 17. In this way, the students can learn IoT threats. For this, we newly added an object picking functionality by changing the view-direction to the web-based 360VR image/video viewer.
The following is the fundamental mechanism of the web-based 360VR image/video viewer [8,9] realized using WebGL. Originally, there are example JavaScript programs of Three.js that can display 360VR images/videos so that we extended those programs to make them more convenient. For displaying 360VR images/videos, Equirectangular projection images/videos should be prepared as shown in the upper right part of Figure 19. It is possible to generate such a projection image/video by transforming from the image/video captured with a special camera called 360VR camera of multiple fish-eye lenses, e.g., RICHO Theta of two fish-eye lenses and its fisheye image shown in the upper left part and lower part of the figure, respectively. Usually, companies of such 360VR cameras provide dedicated software for the transforming. After that, by the texture mapping of such projection image/video frame onto the surface inside of a sphere model located in a 3D CG space realized by WebGL and Three.js, we obtain 360VR image or one frame of 360VR video shown in the lower right part of the figure. Furthermore, we integrated this web-based 360VR image/video viewer with the functionality of displaying the text explanation, the image and the video of a target IoT device as shown in Figure 18.
6. Conclusions
In this paper, we introduced the integrated development frameworks based on Linked Data for Web-based interactive educational materials. These are the achievement results of our centre called ICER. Firstly, we introduced the framework for the development of web-based interactive 3D educational materials and its educational material example of Japanese history education. Secondly, we introduced the authoring tool based on Linked Data for web-based educational materials and the integrated development framework based on Lined Data and its educational materials of two teaching targets, i.e., IoT security education and Japanese history education. Furthermore, we introduced the integrated development framework based on Linked Data that support 360VR images/videos and its example educational material for IoT security education.
As future work, we will complete the database implemented as Linked Data of IoT threat information as soon as possible. After that, we will finish the development of the web-based interactive 3D material and the web-based material supporting 360VR images/ videos for IoT security education. We will do the education practice of IoT security education. Then, we will evaluate educational efficiency of the developed materials.
Competing Interests
The authors declare that they have no competing interests.
Acknowledgments
This research was mainly supported by Strategic International Research Cooperative Program, Japan Science and Technology Agency (JST) regarding "Security in the Internet of Things Space", and partially supported by JSPS KAKENHI Grant Number JP17H00773. In addition, this work has been achieved collaboratively with the members of Japanese History Study Laboratory in Kyushu University. We appreciate their advices and supports given for this work.