SurveyAnywhere: A Field data collection and Synchronization system for portable devices

Amiangshu Shekhar Bosu asbosu@crimson.ua.edu Md. Ashfakul Islam mislam@crimson.ua.edu

Introduction
Field data is important for any kind of research. Marketing researchers collect data from people to analyze market and make their decisions about next marketing move. These decisions may run the company to entirely different direction. Software engineering researchers are interviewing designers, developers, quality tester and other stake-holders to gather their experiences, views and other data to analyze them from different software engineering perspectives. Biological researchers go to various natural inhabitations to collect their required data, pictures of various wild lives. Researchers in each fields depends on field survey data to continue their research. Even different organizations like an university runs survey among students to know about their opinions regarding different issues ( e.g. course curriculum, course structure, instructor evaluation, transportation system, dining facilities, and library services ) and make better decisions for students. At present surveys are very common. Lot of organizations and projects do not have required manpower to conduct a survey. Outsourcing a survey is very popular in almost each sector. Lot of organizations out-source their surveys on behalf of their clients. Clients design their survey, target their population and make all decision related to convey those surveys. Outsourced company make plan for execution, choose appropriate manpower and distribute among workload among those surveyors. Surveyors go to field and communicate with or examine their surveying subjects, fill all data collection forms and gather those data for the client. Portable devices are getting smarter with time. Laptops, PDAs, Net-books, tablets, and smart phones are becoming richer in sense of computational power, storage and network resources. Especially smart phones are becoming very popular in these days. Smart phones can serve a variety of service to their users besides regular phone services. Different kinds of apps are developed to serve different needs of users. So, smart-phones can be used very conveniently to collect data for a survey. One thing a surveyor needs to use their smart phones for data collection is an appropriate application, which is designed and built for smart phones for that particular survey. But design and characteristic of data item in two different surveys are not same. So surveyors need to install different applications for different surveys, which is very non-user friendly especially for the outsourced surveyor companies. They need a customized application for surveying. Moreover if they want to change some form on their survey, they has to rebuild the application and redistribute. They also need a nice and structured framework for these data collection process because uploading collected data directly from the portable devices is time consuming and infeasible sometimes.

We have designed and developed a flexible framework and an android application to manage survey design, survey distribution, data collection, and data synchronize between database server and smart phones.

System Architecture:
The system can be modeled in two parts: Server system and Client system. Researcher or owner of a survey can create their data collection forms through the web interface easily. They can manage access to their forms and collected data in database server. On the other hand surveyor can login to system through android application and access to desired survey forms and upload data to server after completion of a survey.

Web Server
Rest Service Web Frontend

Database Server

Figure 1: System Architecture

Server system
Server system will have the data storage. It will have a web-based front-end that will enable administrator to define the data collection form. He can specify the field names, data types, attributes etc. Based on those definitions the system will customize its REST based web service. The server system will run on a cloud platform (SAAS- Software as A Service). The system will use database to store the text data and storage (e.g. S3) for storing binary data (e.g. pictures, videos, sounds). Cloud platform will enable the application to be scalable. And the users will not need to worry about data backup. The system will also allow the users to download the files easily and export the data in different formats (e.g. csv, xml)

Client systems
The client systems will run on different portable devices. Our initial target is to develop a client application for Android devices. The application will connect to the server and retrieve the form attributes from the server. Using that definition the application will render the form to the user. The users will fill the form using collected data and submit the form to the server. The users can attach different types of files (e.g. videos, pictures, sound) when they upload the data. As we are separating the two sub-components of our system using a defined interface (REST based web service), the sub-components can be updated independently. And we can extend the scope of the system by creating clients for other types of devices.

System Implementation
We have implemented the system according to our design. A web based system was implemented to serve as Server system. And an android application was built to serve client system. PHP was used to implement web front end and MySQL was used as back end database server. Kohana framework (http://kohanaframework.org ) was used for system. Android SDK was used to develop android application. JSON was used for exchanging data between android application and the server. An administrator controls the web front end system. Password based user login was implemented. An administrator can create new users for survey or delete users. He can also design the data collection form. A nice UI was developed for data collection form building. He can add five types of field for data collection form: Text box, Checkbox, Dropdown and Password box, Location data field. He can also mention must require data fields in the forms. He is able see the android preview of the form and reorganizes fields according his choice. After saving a form into database, a new table is created in MySQL database automatically corresponding to that form. When surveyor submits survey data, it will be stored to that table of the database. Administrator can also manage user access to forms: grant access or revoke access.

A password based login system was implemented in client side. After login, surveyors as user of client system can see available form list and description of forms. Available form list will be downloaded from server based on user. When user selects one form from the list, fields of the form will be downloaded from database and it will be built dynamically by android application. User can also able to change their password.

Figure 2: Database diagram of the system

Using the system

The system is divided into three components. 1. Web based frontend for administrative users 2. Android application to access survey forms for mobile users 3. REST web service to manage interaction between Database and Android application

1. Web based frontend for administrative users

The web based frontend allows administrative users to: 1. Create add/edit/delete new mobile users (users that log in from portable devices and access forms) 2. Create add/edit/delete survey forms 3. Provide / Revoke access to a survey form to a mobile forms Users login to the web-frontend using a public url. Our web frontend was hosted on Amazon Web Service (AWS). AWS is cloud based and allows to dynamically scale the application. The URL of our application was: http://ec2-107-22-45-207.compute-1.amazonaws.com/index.php (Figure. 3). The home page contains brief motivation for the application and a login form for administrative users.

Figure 3: Home page and login form for administrative users

a) Administrative User homepage After the administrative user has successfully logged in, the system shows a welcome message (Figure. 4) and shows list of available actions to the user in top menu. Available actions are: 1. Form management: Edit / Delete/ Share existing survey forms 2. Create new form: Create a new survey form 3. User management: Add / Edit /Delete mobile users

4. Change password: Change password for the administrative user

Figure 4: Homepage screen for administrative users after log in

b) User Management
The user management section allows to create Add / Edit /Delete new mobile users. On the landing page of User management section there is a link to access the form to Create a new user (Figure. 5).

Figure 5: Landing page of User management section

The create new user form has 5 fields (Figure. 6). To create a new user the First name, Last name, Email address and Password need to be filled. The confirm password field prevents accidental mistyping of password.

Figure 6: Form to Create new user

As all the fields are required, there is Javascript-based validation implemented in the form. When any type of validation is failed the system provides user proper feedback. It does not show any annoying popup alert. Rather it highlights the field with red color. It also shows the user why the validation was failed on the right side of the field (Figure. 7).

Figure 7: Dynamic validation and feedback to the user in new user creation form

After the user is created successfully, the user is redirected to the user management page. The new user is listed in the user list of User management page (Figure. 8 ). There is link to Edit / Delete user on the right side of each user name on user list. Clicking Edit / Delete link will allow the admin user to Edit/ Delete that user.

Figure 8: New user is listed in the user list of user management page

For example, if the admin user clicks on the delete link the system will ask the admin user to confirm, whether he really wants to delete that user (Figure. 9). This confirmation prevents accidental deletion of a mobile user. If admin clicks Ok on the confirmation screen the user will be deleted.

Figure 9: Confirmation dialog to delete a user

After the user is created the system is redirected to the landing page of User management section. The deleted user is no longer listed in the userlist. Admin can now Add/ Edit/ Delete more users. For example, he might want to edit a user (Figure 10).

Figure 10: Editing a user The edit user form is similar to the Create new user form. Similar validation is also implemented in this form (Figure. 11).

Figure 11: Form to edit an existing user After the user information is saved to the database. The form redirects to user listing page. It now shows that user with modified information (Figure .12).

Figure 12: User list with edited user information

c) Create new form User can access the section clicking on the Create New Form link from the top menu (Figure 10). The form has two panels. On the left side user enters the name, description of the survey form. He can dynamically add new fields to the form using the dropdown menu (Figure 13). On the right side there is preview panel. Clicking on the Preview on Android button the user can see an approximate preview of what the form might look like on an android device.

Figure 13: Section to create a new survey form

The create new form section is the main part of the application. This form is very dynamic and we provided special attention to usability in while designing this form. In any type of survey basically there are three types of questions (Figure. 13). 1. Free form response: Presented using text input 2. Multiple choice: Presented using drop down list 3. Multiple response: Presented using checkbox We can have also included a special field named Location data. If that field is included, android will send the location of the user when submitting the form. The fields can be added to the form by selecting a new field type from the Add New Field from the drop down menu. Different fields have different types of attributes. User can rearrange the fields, remove a field, or hide a field. User can mark a field as required or not (Figure 14). User can see the Preview on Android in the preview panel. When user is satisfied with the form he can save the form and the form will be created.

Figure 14: Using the Create new form section to create new survey form The system show popup dialog to indicate that a form has been created successfully (Figure. 15).

Figure 15: Confirmation of new form creation

d) Form management
The form management section allows the admin user to Edit / Delete and share form among mobile users. This section can be accessed using Form management link from the top menu (Figure 16). If the user clicks Edit the edit form is shown. The edit form is similar to the Create new form. If the user clicks Delete link, the system asks confirmation from the user.

Figure 16: The form management section

When the user clicks the Share form link, the system shows the form to share with existing mobile users. The right side shows the list of current users that has access to this form. There is Revoke access link on the right side of each user. Clicking this link will revoke access of this form from that user. In the User name textbox admin can type the name of a user to provide access. The system will show an autocomplete list to allow easy selection of a user that do not have access to the form (Figure 17). Admin selects a user from that autocomplete list and clicks on the Allow acccess button. The mobile user will be added to the access list of that form. When that user now log in from an android device he will see this form in the list of his forms.

Figure 17: Sharing / Revoking the survey form from mobile users

2. Android application
A login page will be shown to the user after lunching the application on an android phone (Figure. 18 left). An email id is required as login id. This login was created in the user management section of the web frontend. First default password is set by administrator and informed via email to user. If the user provides incorrect login id or password a pop up will be shown to the user informing an invalid login try (Figure. 18- right).

Figure 18: User login and validation on android

Application will communicate will the server for user authentication. After successful authentication menu screen will be shown (Figure. 19).

Figure 19: Main menu of the android application

After clicking on Show forms icon a user will be directed to show forms page. Application will send a REST message to server with user authentication token and server runs a query to database for form list and send the list to application. A list view of forms will be showed up where first bold line shows name of the forms and italic lines shows description of the forms. Those forms were created and access granted to current user in form management section of the web-frontend.

Figure 20: List of forms accessible to the user

After selecting a form from the list, application will again send a REST message to server and server will reply with list consists of field name, type. Application will build a dynamic form based on that field list (Figure 21).

Figure 21: A sample dynamic form

After touching submit button, application will collect data from all fields, perform required validation and if all the validation is passed then submit those data to database server. A confirmation message will be shown to the user (Figure 22). The system will return to the main menu.

Figure 22: Confirmation of data submission to server

The android application allows user to change the password. Change password icon will navigate to change password window (Figure 23).

Figure 23: Change password form on android

3. REST service
The rest service acts as bridge between the database and android application. Our implementation of the service is stateless. On first authentication, the service sends an authentication token to the client. For using other services a client needs to send that authentication token with their request. The methods created by our service are described below. Authentication URL: http://SYSTEM_URL/index.php/webservice/login Parameter: username, password Return: Authentication token on success, on failure returns 401 HTTP status Getting form list URL: http://SYSTEM_URL/index.php/webservice/getformlist Parameter: authentication token Return: List of forms that user is allowed to access Getting form field URL: http://SYSTEM_URL/index.php/webservice/getformfields Parameter: authentication token, form_id Return: Details of the form fields Saving a form data URL: http://SYSTEM_URL/index.php/webservice/saveform Parameter: authentication token, form_id, data of the form Return: HTTP 200 on success, HTTP 500 on error Changing password URL: http://SYSTEM_URL/index.php/webservice/changepassword Parameter: authentication token, current password, new password Return: HTTP 200 on success, HTTP 500 on error

Possible Future Enhancement

The system functionalities can be enhanced further with some possible enhancements. Firstly, we can add one more field type in form creation to handle uploading file from android device. This will allow the users to upload images, audios and videos. Moreover, we can add video, audio or image data type fields directly to form. When user activates one of this type of form the application will access corresponding device to collect that type of data. For example, for image fields the application will access camera of the device to take capture an image.

On the web frontend, we can provide a data management section. From this section, user will be able to browse and download the data in various formats. The data management section can also show some simple descriptive statistics of uploaded data.

Conclusion
Portable devices are very popular now. Using these portable devices in data collection in survey is a very convenient option. But problem of using portable devices is presentation of the data collection form of different forms in different devices. Our developed system is a flexible and convenient solution for these scenarios. Designing the for is very flexible and user friendly. User can view preview of the form on android device from the web-frontend. This enhances the usability of the system greatly. The attributes of the forms are downloaded from server and build dynamically in portable devices. User interfaces are enhanced than consisting approach.