Chapter 6 Results

This chapter presents the results obtained by applying ve metaheuristic algorithms to the optimization problem of the peer-to-peer transfer of information in VANET networks using VDTP (OFTC problem). In such networks, the information transfers varie critically depending on the scenario where vehicles are moving, so we have dened two dierent scenarios, an urban and a highway, over which it has solved the proposed optimization problem. The chapter is organized as follows: In Section 6.1 we analyze the results in terms of objective function for each of the algorithms. The performance study of each of the algorithms to solve the problem is presented in Section 6.2. Finally, in Section 6.3 we evaluate the obtained VDTP parameter congurations by simulating them.

6.1

Global Results

This section discusses the results obtained by the ve algorithms studied when solving the Optimal File Transfer Conguration (OFTC) problem using VDTP. As it is a minimization problem, the lower the tness value, the better the solution. Table 6.1 shows the resulting tness values regarding the urban and highway VANET scenarios. The third column contains the mean and standard deviation of the values returned by the objective function in the 30 independent executions. In turn, it shows minimum (best tness), maximum (worst tness), and median values for each of the algorithms and scenarios.

85

CHAPTER 6. RESULTS

Scenario

Urban

Highway

Algorithm PSO DE GA ES SA PSO DE GA ES SA

Mean Std.dev. Minimum Median Maximum 1.6346 0.2899 0.9077 1.7809 1.8918 1.7423 0.3717 0.7389 1.8658 2.0228 1.9086 0.2260 0.8799 1.9731 2.1614 2.1517 0.1266 1.8862 2.1222 2.4246 2.7850 0.8718 0.8730 2.1663 3.8025 4.1761 0.2556 3.3301 4.2513 4.4554 4.6631 0.9328 2.7145 4.2272 7.0531 4.3805 0.8695 2.5345 4.1918 5.8608 5.7833 0.9705 3.8836 6.1347 6.9421 4.4246 0.7401 3.1498 4.0855 5.7922

Table 6.1: Final tness values for both VANET scenarios and the ve optimization algorithms.

For the urban instance, we can observe in Table 6.1 that PSO obtained the best result in terms of the mean tness (1.6346 0.2899). This smallest mean value leads us to believe that using PSO the resulting VDTP conguration ends in an ecient communication which is fast and accurate between vehicles. In addition, the best median and maximum values were also obtained by PSO, although the best minimum, i.e., the best VDTP conguration found for urban scenario, was reached by DE (0.7389). This is an expected value, since DE generally shows a pronounced exploitative behavior (using a parametrization close to the standard one) [113], while PSO tends to have an explorative performance using a high inertia (as in this study w = 0.5) [1]. In the highway scenario the results are similar, PSO obtained also the best mean tness value (4.1761 0.2556). Besides, PSO presents the lowest value of standard deviation. This implies a considerable advantage, since it provides our model with a high robustness, which is a crucial issue when designing VANETs. In terms of the minimum tness, GA (2.5345) obtained the best VDTP conguration for this scenario. As we assumed previously, the dierence in results between the two scenarios is quite signicant, tness values obtained in the interurban scenario are more than twice those obtained in the urban area, that is, the QoS of VDTP protocol is worst in highways. This conrms the assumption that in highways the transfer of information is more dicult.

86

6.1. GLOBAL RESULTS

In order to provide a comparison with a statistical meaning we have applied a statistical test. Such tests may be parametric or nonparametric [124]. A parametric test is one that uses real-valued data belonging to an interval. However, it may be the case that one or more initial assumptions for using parametric tests fail, causing the loss reliability statistical analysis. In order to use a parametric test it is necessary that the distribution satises the conditions of independence, normality, and heteroscedasticity [45]. The condition of normality checks whether a data set is well-modeled by a normal distribution or not, or to compute how likely an underlying random variable is to be normally distributed [45]. In this work we have used Kolmogorov-Smirnov test (KS-test) [90]. This test compares the cumulative distribution of observed data with the expected cumulative distribution of a Gaussian distribution. As in any hypothesis test, the null hypothesis is rejected when the p value exceeds a critical value obtained from a probability table. In order to perform these statistical tests we have used SPSS1 . Table 6.2 shows the results of applying Kolmogorov-Smirnov test to the results of the ve metaheuristic algorithms for boths scenarios. We have performed this test with condence level of 95% (p value = 0.05). As we can see in the second column of Table 6.2 (the asymptote) there are samples that do not satises the test (DE in urban scenario and SA and PSO in highway scenario). Finally, we have applied a non-parametric test because the distributions violate the condition of normality. Scenario Algorithm PSO DE GA ES SA PSO DE GA ES SA Z of K-S 1.169 1.617 1.182 0.774 1.322 1.403 1.265 1.113 1.186 1.525 Asymp. Sig.(2-sided) 0.130 0.011 0.122 0.587 0.061 0.039 0.082 0.168 0.120 0.019

Urban

Highway

Table 6.2: KS-test results for the ve algorithms and two scenarios.
1

SPSS (Statistical Package for the Social Sciences) - http://www.spss.com/

87

CHAPTER 6. RESULTS

A general comparison can be made using the Friedman [124] statistical test by means of which the algorithms are sorted in a ranked list. Table 6.3 shows the Friedman ranking of the compared algorithms in urban and highway instances. In this table, the best ranked algorithm is located in the top. For urban instance, PSO and DE are the best ranked algorithms, showing SA the last position. Nevertheless, for highway scenario, SA obtains the best rank, whereas PSO is located in the third position. Urban Algorithm Rank PSO 1.27 DE 1.83 GA 3.07 ES 4.33 SA 4.50 Highway Algorithm Rank SA 1.83 GA 1.97 PSO 2.17 DE 3.67 ES 4.97

Table 6.3: Friedman Rank test with condence level 95% Theses statistical results lead us to think that, in spite of the global best behavior of PSO, the dierent requirements implicit to both instances implies that each algorithm can show quite dierent results depending on the VANET scenario on which it operates. For example, DE shows a competitive performance in urban scenario whereas it is the second worst in highway. The opposite example can be observed in GA and SA which show weak results in urban but highly competitive ones in highway. Therefore, the VANET designer can select the optimization model more suited to his/her requirements, and choose the best option for each studied VANET scenario.

6.2

Algorithms Performance Study

This section studies the performance of the ve used algorithms solving OFTC problem in both scenarios . In order to do so, we analyze the evolution of tness value for the best solution during the whole evolution process. In addition, we discuss the required execution times. Figures 6.1 and 6.2 illustrate the graphics of the best tness values obtained through the median execution in urban and highway scenarios, respectively.

88

6.2. ALGORITHMS PERFORMANCE STUDY

Figure 6.1: Median tness performance in urban scenario.

Figure 6.2: Median tness performance in highway scenario. Observing both graphs we can draw the following conclusions: PSO and DE tend to converge in the same range of solution evaluations. However they could improved their tness even during the nal generations of the evolution process. 89

CHAPTER 6. RESULTS

For both scenarios. GA starts from good solutions but it is subjected to an early stagnation. The lack of robustness of ES and SA in this specic application is remarkable, because they present dierent behavior depending on the VANET scenario (urban or highway). Concerning to the execution times that each algorithm spent in the experiments carried out in this work, Table 6.4 presents both, the mean time in which the best solution was found (Tbest ) and the global mean run time (Trun ). The detailed execution time for each independent run of each algorithm are included in Appendices (see Section A.2). Scenario Algorithm Tbest (seconds) Trun (seconds) PSO 4.68E+03 7.95E+03 DE 4.37E+03 7.12E+03 Urban GA 3.48E+03 6.68E+03 ES 5.46E+03 9.00E+03 SA 2.18E+03 4.76E+03 PSO 1.39E+03 2.19E+03 DE 9.82E+02 2.10E+03 Highway GA 8.83E+02 1.56E+03 ES 9.84E+02 1.47E+03 SA 5.85E+02 8.45E+02 Table 6.4: Mean execution time (seconds) per independent run of each algorithm for both, urban and highway, scenarios. SA has spent shortest times to nd its best solution and to nish the execution. PSO and DE spent close execution times for the two VANET scenarios because they perform similar internal operations. Finally, the two evolutionary algorithms (GA and ES) consume also closed times. In short, the algorithms require a time between 4.76E+03 and 9.00E+03 seconds (80 and 150 minutes, respectively) to solve the problem in the urban scenario and a time between 8.45E+02 and 2.19E+03 seconds (23 and 60 minutes, respectively) to solve the same on highway instance. This is a relative high computational eort. However it is justifcable since the benets of nding an optimal conguration for the VDTP protocol are considerable and during the life of the network. 90

6.3. VANET QOS ANALYSIS

6.3

VANET QoS Analysis

After obtaining the parameter settings for VDTP protocol returned by the ve metaheuristic algorithms, we have simulated them to test their performance on transferring information in VANETs. In addition, we have compared the results with the conguration settings used by the experts of CARLINK [22] (chunk size: 25,600 bytes, retransmission time: 8 seconds, and max. retransmissions per packet: 8 attempts). We have simulated le transfers in the VANET scenarios dened in Section 5.1.1. In these two scenarios (urban and highway) eleven 1 Mbyte (1,024 Kbytes) le transfers are performed among the vehicles. Tables 6.5 and 6.6 show the optimal parameter settings for the VDTP protocol automatically obtained by the proposed algorithms and the average values of the metrics used to analyze the QoS for both VANET scenarios. They also include the parameter setting that has been used by experts of CARLINK followed by the results of simulations (last row). The rst three columns contain the dierent parameter settings of the VDTP protocol, in this order: the chunk size, the timeout, and the maximum number of retransmissions of a request packet. The following three columns are the metrics obtained from the simulations in this order: the transmission time, the total number of packets lost during transmission, and the amount of data downloaded correctly. The results are available in more detail in Section A.1 of the Appendicces.

Algorithm

VDTP conguration Chunk size Timeout Attempts

(Bytes) (seconds)

PSO ES GA DE SA CARLINK

41,358 23,433 31,196 28,278 19,756 25,600

10.00 10.00 3.83 6.00 6.43 8.00

3 8 9 9 3 8

Simulation results Transmission time Lost packs. Transfer data (seconds) (Kbytes) 3.41 0.27 1,024 3.50 0.27 1,024 3.61 0.27 1,024 3.59 0.63 1,024 4.22 0.36 1,024 4.24 0.36 1,024

Table 6.5: Optimal congurations achieved in the median execution and the CARLINK experts one for VDTP protocol and simulation values in urban scenario.

91

CHAPTER 6. RESULTS

Algorithm

VDTP conguration Chunk size Timeout Attempts

(Bytes) (seconds)

Simulation results Transmission time Lost packs. Transfer data

(seconds) (Kbytes)

PSO DE GA ES SA CARLINK

29,257 19,810 34,542 38,490 32,002 25600

6.42 6.91 9.54 8.15 8.21 8.00

9 8 10 12 4 8

24.67 27.66 26.96 33.99 25.43 33.08

3.18 3.45 2.72 3.36 2.54 3.27

1,024 1,024 1,024 1,024 1,024 1,024

Table 6.6: Optimal congurations achieved in the median execution and the CARLINK experts one for VDTP protocol and simulation values in highway scenario. For the urban VANET scenario, the VDTP conguration obtained by PSO (chunk size=41,358 bytes, retransmission time=10 secs, and max. attempts=3 ) achieves the best performance in terms of transmission time (3.41 secs) and mean number of lost packets (0.27). Specically, in comparison with the human experts parameter conguration of CARLINK, PSO reduces the transmission time in 0.83 secs (19.5%) registering also a lower number of lost packets. Something similar happens with the results obtained in the highway scenario, the conguration obtained by PSO (chunk size=29,257 bytes, retransmission time=6.42 secs, and max. attempts=9 ) transfers the les faster than the others, regarding the human experts conguration from 33.08 to 24.67 secs (25%). We must notice that, in spite of achieving the PSO a higher reduction in the transmission time than SA and GA, the fact of losing more packets (3.18 in PSO, 2.71 in GA, and 2.54 in SA) in the global transference leads SA and GA to calculate a better tness value.

a) Urban scenario

b) Highway scenario

Figure 6.3: Eective transmission data rates (throughput) in Kbps achieved during the simulations of the nal VDTP congurations in comparison with values given by human expert congurations of CARLINK consortium. 92

6.3. VANET QOS ANALYSIS

According to gures 6.3.a and 6.3.b, the eective transmission data rate obtained by practically all algorithms in the two VANET scenarios are higher than the CARLINK experts conguration, except in the case of the parameter conguration returned by ES in highway scenario. Specically, PSO achieves the highest eective data rate (2402.32 Kbps in urban and 332 Kbps in highway scenarios). We again remind that the actual reduction of eective data rates among vehicles are in the order of hundreds of Kbps, so our savings (470.32 Kbps in urban and 84.4 Kbps in highway) are truly meaningful in current real applications such as safety and trac control. This clearly claims for the utilization of these automatic algorithms to help human designers. Notice the importance of the VANET scenario where there are the transfers of information, because in highway the eective transmission data rates are 70% lower than in the urban areas. This is due mainly to the high speed at which vehicles move out of towns. The following highlights the most important results presented in this chapter: Urban scenario: Best algorithm according Friedman test: PSO. Algorithm which obtains the best tness value: DE. Algorithm which obtains the best mean tness value: PSO. The most robust algorithm: ES. Fastest algorithm: SA. Percentage improvement of PSO against CARLINK experts: 19.6%. Mean percentage improvement of automatic computed congurations against the humans experts ones: 13.5%. Highway scenario: Best algorithm according Friedman test: SA. Algorithm which obtains the best tness value: GA. Algorithm which obtains the best mean tness value: PSO. The most robust algorithm: PSO. 93

CHAPTER 6. RESULTS

Fastest algorithm: SA. Percentage improvement of PSO against CARLINK experts: 25.4%. Mean percentage improvement of automatic computed congurations against the humans experts ones: 6.1%.

94