Flexible implementation of advanced neural network capabilities under Scilab
- For a platform such as Scilab, providing a fast and reliable implementation of ANN is of high priority. Users should be able to build and activate a network in just a few commands. Several implementations of ANN exist in various programming languages. The solution is to interface Scilab with one of those existing implementations. This project proposal exposes the details of a solution based upon the SNNS library, and discusses the evolution of such an ANN toolbox.
1. Student time line
See here for the Google Summer of Code planning.
2. Work time line
- Starting a minimal toolbox.
You can have a look at what has been done in the SciFANN toolbox (how to send and get pointers to structure in / from scilab).
- Selecting a "minimal" subset of SNNS functions. The functions necessary to set up a neural network structure, to initialize the weights, to train and to validate. Begin a first "minima" toolbox (doc too).
- Gathering data sets. Start to write unit tests and demos.
- Adding some more details in the toolbox:
- Changing the activation function
- cascade learning
- recurrent neural nets
- Make these compiles under windows too.
3. Project description
So far, an interface between FANN 2.1.0beta and Scilab has been programmed under the name scifann. FANN being a powerful neural network library coded in C, it was a good idea to reuse it in order to enhance Scilab with ANN capabilities. Even though FANN provides a strong platform for neural networks, it is limited when it comes to network customization. Therefore, another neural network solution would benefit from using a more flexible library. This is the goal of the solution I propose.
3.2 Solution for flexibility
Several open source ANN libraries exist, available in various languages. A good resource to find such libraries can be found at http://www.google.com/Top/Computers/Artificial_Intelligence/Neural_Networks/Software/. The most flexible library created so far is the Stuttgart Neural Network Simulator (SNNS). Using this library, it is possible to customize every single part of a network, and to get a fast computation of the results. This library is available as an open source project, but has not been updated since 2001. Nevertheless, the available documentation allows to quickly understand the details of the underlying architecture. Figure 1 is a view of the SNNS graphical interface. This interface won't be implemented in the toolbox, but permits to see how powerful the SNNS library is.
Figure 1: View of the SNNS graphical interface
3.3 Specifications of SNNS
The documentation available for SNNS is amazing. Every single detail of the application is covered, allowing users and developer to play with it without any additional help. I have been through the code to see how the network was implemented. From what I have seen, I can make a few remarks.
To begin with, the whole code uses global variables. This means that when calling the functions to manage the network, these functions deal with the global variables to update the state of the network. That implies that only one network can be used at a time, which constitutes a limitation. This also means that since Scilab is loading and unloading C shared object each time a call is made to an interfacing function, the state of the network will be lost between two function calls. Hopefully, it is possible to use static variables of Scilab to store and retrieve global variables of an interface. This solution has already been used in the mysql toolbox of Scilab for instance.
Then, the neural network activation code itself is sequential. Therefore, by using SNNS, Scilab won't be able to take advantage of clusters and multi-core computers to fasten the computation process.
Finally, most of the customization part is made by the passing of function pointers. These pointers are in C, and therefore not manageable at the Scilab level. The solution for this is to use helper functions, that would simply take strings instead of function pointers, in order to specify the customization to apply. For instance, instead of passing a C function pointer to use the sigmoid activation function, one would just pass the string "ac_sigmoid" to a Scilab macro, which would take care of making the function pointer
3.4 From Scilab's perspective
The Scilab SNNS toolbox will not interface the whole SNNS code, since most of it is not of any interest for Scilab. What needs to be interfaced is mostly contained into the kernel. In the documentation of SNNS, the interesting features are grouped in the Chapters 14 and 15, which represent basically the interface offered by the public interface given in "kr_ui.h". Interfacing all the functions from "kr_ui.h" will be enough for a Scilab user to have access to reliable ANN capabilities. Figure 2 shows a diagram of the solution I propose in order to build an ANN toolbox based on SNNS.
Figure 2: Solution for interfacing SNNS to Scilab.
Due to the fact that this toolbox would be based on several libraries, special care has to be given to how these libraries are handled when the package is built. To be sure that the implementation of such a toolbox would be feasible, I made a few trials to see if a function from SNNS could be interfaced to Scilab. Based on the documentation page regarding toolboxes on Scilab's website, along with the the "toolbox_skeleton" example, I have been able to realize an interfacing under Linux/Ubuntu.
The first step of this implementation has been to extract the kernel from the SNNS package, and to compile it. This done, I made some trials with the given example files, and I have been able to have networks running without the SNNS graphical interface: the kernel was working. Then, I just needed to interface the kernel with Scilab.
To perform this interfacing, I encountered a few issues, all related to library linking. First, this is a fact that Scilab likes shared objects. My first trials consisted in trying to link static libraries ".a", which turned out to be a mistake. From then, I have only built shared objects ".so", which happened to be linkable by the Scilab building process. Second, SNNS uses the lexer flex to parse some data files. Therefore, it is necessary to include this library in the compiling process. Unfortunately, the default shared object provided by the flex package is a fake: this shared object file simply includes the static library file, which makes fail the compiling process. The solution has been to compile specific functions from the flex library into a shared object, and use this shared object during the compiling process of the interface. Everything worked out to be fine, and I managed to have a function of the SNNS library running under Scilab. Figure 3 is a screenshot of the function call.
Figure 3: Screenshot of the first SNNS function mapped to Scilab.
3.6 Smart interfacing
Instead of using the function names of SNNS in Scilab, an ANN interface needs to be designed for Scilab. This interface would wrap calls to the SNNS 4.3 library. That way, if the decision is taken, for some later version, to switch from the SNNS library to another one, the Scilab calls will not change. Using this approach will ensure retro-compatibility of all the code produced by the users before the underlying library would change. Scilab users won't face any obsolete or non-supported function problems.
To permit for such a mechanism to take place, the interface needs to be decoupled from the SNNS library. This simply occur in the interfacing files of the sci_gateway directory. As stated above, this would imply for Scilab to have its own interface, independently of the internals of the SNNS library. To design this interface, several libraries should be reviewed, to extract common features.
3.7 Multi-platform support
It would be interesting to have this toolbox working on several operating systems other than Unix, like for instance the Windows platforms. If the main part of the implementation and testing will occur under Linux, the toolbox would also benefit from being ported under Windows, to allow Windows users to experience such a great tool. A Mac version could also be done, but is not as important as a Windows version.
3.8 Back to the future
With a new toolbox management system coming in Scilab version 5.2, the current ANN toolbox needs to be compliant with the requirements of the new system.
Furthermore, changes will be applied upon version 6.0 regarding the way macro parameters are retrieved in the interface functions. As a consequence, I will avoid the use of functions such as GetRhsVar, and prefer functions like GetRhsVarFromPtr, which will be easier to refactor the day Scilab 6.0 will be release. Basically, a good practice will be to use only what is in "stack3.h", since this is the base of the interfacing functions that will be developed in Scilab 6.0.