Forecasting Electricity Prices

Derek W. Bunn1 and Nektaria Karakatsani

London Business School 2003 v1

Abstract This is a review paper documenting the main issues and recent research on modeling and forecasting electricity prices. The special market microstructure of electricity is described, as an explanation of the extraordinary stochastic properties of electricity price time series. The research literature deriving from the application of models adapted from financial assets, for both spot and forward prices, is reviewed and criticised. Final emphasis is placed upon the virtues of computationally intensive structural modeling. Keywords: Electricity, Volatility, Regime-switching, Structural Models

Introduction Developing predictive models for electricity prices is a relatively new area of application for the forecasting profession. Until recently, electricity was a monopoly in most countries, often government owned, and if not, highly regulated. As such, electricity prices reflected the governments social and industrial policy, and any price forecasting which was undertaken was really focussed on thinking about underlying costs. In this respect, it tended to be over the longer term, taking a view on fuel prices, technological innovation and generation efficiency. This changed dramatically, however, during the 1990s. Following the examples of structural reforms and market liberalizations in Chile,

Corresponding author: Professor Derek W. Bunn, London Business School, Sussex Place, Regents Park, London NW1 4SA, UK. Email dbunn@london.edu

Britain, Norway, Argentina, and Australasia in the early 1990s, other European countries such as Spain and Germany followed suit a few years later, as well as various regions in North and South America, and this trend has continued so that power sector reform has now become a major issue worldwide. Ownership in the sector has generally become private rather than public, competitive markets (pools and power exchanges) have been introduced for wholesale trading and retail markets gradually liberalised to erode local franchises. Typically the industry has been split up into separate companies for generation, transmission, local distribution and retail supply. Transmission and distribution are network services and, as natural monopolies, are regulated. Generation is progressively deregulated as competition develops between a sufficient number of companies to promote an efficient wholesale market. Retail suppliers buy from the wholesale market and sell to customers. Industrial and commercial customers have generally been the first to receive full market liberalisation. The residential sector has been opened in many countries, but often quite slowly, and in some cases not at all. All of this structural change has been motivated by a faith in the ability of competitive forces to create a more efficient and enterprising industry, than either the public sector or regulated monopolies could deliver. Of course, it is the prospect of business risks that should drive the efficiency gains, and the major new element of risk is wholesale price uncertainty. In that the majority of the new wholesale spot markets are imperfect and inefficient, and the emergent power exchanges incomplete, in the financial sense, and insufficiently liquid, the need for careful and detailed modeling of prices becomes an essential aspect of risk management in the industry. If competition were so efficient that prices reflected marginal costs, even in peak periods of demand, then there would be a complete price convergence of electricity with the underlying fuel costs (eg gas), and spot price modeling would be relatively straightforward to specify. If the futures markets were liquid and complete, as well, then forward prices would have a simple dynamic structure similar to other financial assets, and conventional risk management techniques could be applied using financial products from the forward markets. Neither of these situations

prevails in most electricity markets, however, with the result that price models are considerably richer in structure than is seen in most other commodities, and the forecasting techniques which are being applied are still in their early stages of maturity. This paper summarises the special characteristics of price formation in electricity markets in the next section, followed in subsequent sections by a review of the various empirical models which the research literature is now manifesting. We are focusing upon the empirical time-series class of models, as these are now the basis of short and medium term forecasts in practice. This focus is distinct from the large research literature on ex ante economic modeling of electricity markets, using stylized game theoretic or simulation methods, to understand the price implications of various market designs, or equilibrium conditions, but where it is generally recognised that accurate price forecasts will not generally be obtained (Green and Newbery, 1992; Joskow and Frame, 1998; Green, 1999; Batstone, 2000, Skantze et al., 2000; Bunn and Oliveira, 2001; Routledge et al, 2001; Baldick, 2002, Day et al, 2002, Andersen and Xu, 2002; Bessembinder and Lemmon, 2002).

Price Formation in Electricity Markets The crucial feature of price formation in wholesale electricity spot markets, is the instantaneous nature of the product. The physical laws that determine the delivery of power across a transmission grid require a synchonised energy balance between the injection of power at generating points and the offtake at demand points (plus some allowance for transmission losses). Across the grid, production and consumption are perfectly synchonised, without any capability for storage. If the two get out of balance, even for a moment, the frequency and voltage of the power fluctuates. Furthermore, endusers treat this product as a service at their convenience, and there is very little short term elasticity of demand to price. The task of the grid operator, therefore, is to be continuously monitoring the demand process and to call on those generators who have the technical capability and the capacity to respond quickly to the fluctuations in demand.

Most spot markets for electricity are defined on hourly intervals (although the British market is half-hourly), and therefore it is clear that throughout the day and throughout the year, a wide variety of plant will be in action and therefore setting the prices at different times. However, when we look a comparison of demand and prices, we see that the price series exhibit much greater complexity than might initially be expected simply from the activity of scheduling different plant to meet fluctuations in demand. In Figure 1, the demand evolution for three price-setting periods in the day is shown for almost a year, and this should be compared with the corresponding market prices in Figure 2. It is clear that spot electricity prices display a rich structure much more complicated than a simple functional rescaling of demand to reflect the marginal costs of generation. A number of salient characteristics of the typical electricity spot price series have been noted in the literature, and apparently correspond to the profiles of Figure 2. These include mean-reversion to a long-run level (e.g. Johnson and Barz, 1999), multi-scale seasonality (intra-day, weekly, seasonal), calendar effects, erratic extreme behaviour with fast-reverting spikes as opposed to smooth regime-switching (e.g. Kaminski, 1997) and non-normality manifested as positive skewness and leptokurtosis. Spot prices also display excessive volatility, orders of magnitude higher than other commodities and finacial assets, with annualised values of 200%, o rmore. This volatility is time-varying (e.g. Escribano et al. 2001) with evidence of heteroscedasticity both in unconditional and conditional variance. The former reflects the influences of demand, capacity margin and trading volume on volatility levels. The latter describes the observed clustering of tranquil or unstable periods (GARCH effects), specifying volatility as a function of its lagged values and previous disturbances. There is also evidence that conditional variance reacts asymmetrically to positive and negative past shocks and in addition, displays an inverse to financial assets leverage effect (Knittel and Roberts, 2001). There are a number of market microstructure elements, which help to explain these unusual time series characteristics. The simplest observation is that with a diversity of

plant, of different technologies and fuel efficiencies on the system, at different levels of demand, different plant will be setting the market-clearing price. Furthermore, we would expect such a diversity of plant on the system for at least two reasons. The first and obvious one, is obsolescence. With power plant lasting for some 40 years, new technologies will come in and be more efficient. So prices will be fluctuating because of the varying efficiencies of the set of plant being used for generation at any particular moment in time. The more subtle, and more significant, reason for diversity is, however, again due to the instantaneous nature of the product. The most efficient plant, with the lowest marginal costs (the baseload plant), will operate most of the time (eg period 16), but during some of the peaks in demand (eg period 36 in winter), some of the power plants (the peaking plant) may only be operating for a few hours. The recovery of capital costs on peaking plant, through market prices, may have to be achieved over a relatively few hours of operation compared to the 8760 hours in a normal year for which a baseload plant, without maintenance breaks, could, in principle, serve. This will favour both the construction of low capital/high operating cost plant for peaking purposes and the overrecovery of marginal costs in operation, with the consequence that prices are much higher in the peaks. So, whilst the fundamental nature of fuel price convergence has a mean-reverting implication, the instantaneous production process of following a highly variable demand profile, with a diversity of plant costs, creates the high spot price volatility. Other factors also come into play in the short term. There may be technical failures with plant, causing even more expensive standby generators to come online. The transmission system may become congested so that rather expensive, but locally necessary plant gets called upon. And, of course, there may be unexpected fluctuations in demand. All of these events show up in spot prices, and reflect the fundamental economic and technical nature of pricing electricity as a real-time, non-storable, commodity.

There is a further important characteristic of electricity markets, with major implications for price behaviour, and that is their oligopolistic nature. Most power markets are characterised by a few dominant players, and even in those less common situations where there may appear to be sufficient competitors to achieve efficient prices, at particular times and in special locations, individual companies may have the ability to influence prices. Of the academic research on liberalised electricity markets, by far the bulk of work that has been published has been done on the analysis of, and strategies for the mitigation of, the abuse of market power by the generating companies. As a result of the presence of this market power, prices are generally much higher, and even more volatile, than the fundamentals suggest.

Parsimonious Stochastic Modelling of Spot Electricity Prices One class of methodological work on empirical price modeling is inspired by a desire to adapt some of the familiar models from financial assets, to the characteristics of electricity, in order to evaluate electricity derivatives and support trading. Table 1 summarises the progression of models within this theme of research. One of the earliest examples is Kaminski (1997), where the spiky characteristic is addressed through a random walk jump-diffusion model, adopted from Merton (1976). However, the model ignores another fundamental feature of electricity prices, the mean-reversion in the baseline regime. This property is established in Johnson and Barz (1998). The comparison of alternative models (Geometric Brownian motion and mean-reversion with/without jumps) across several deregulated markets suggests a mean-reverting model with a jump component as more adequate specification. Although this type of financial asset modelling addresses crucial characteristics of price dynamics, namely meanreversion and spikes, it still assumes deterministic price volatility, which clearly contradicts empirical evidence.

Table 1. Stochastic Models for Spot Electricity Prices. Random-walk Mean-reversion

dPt = Pt dt + dWt dPt = a( Pt )dt + dWt Constant parameters dPt = a( Pt )dt + dWt + kdqt ( ) Time-varying parameters (long-run level, jump intensity and volatility (GARCH)) Pt = f t + X t dX t = a( t X t )dt + t dWt + kdqt (t ) dvt = k v ( v vt )dt + vt1 / 2dZ t , t = vt1 / 2

Mean-reversion with jumps

t = 1W int er + 2 Spring + 3 Summer + 4 Autumn

Pt denotes the spot price, Wt a Weiner process, f t a deterministic component, qt a Poisson process with intensity
that

describes

the

jump

occurrence

and k = k ( J , J ) a random variable that describes the jump magnitude.

This class of modelling was, however, further extended in Deng (2000) with additional non-linearities in the price dynamics, such as regime-switching and stochastic volatility. These aspects allow richer dynamics to emerge, although they are not captured simultaneously in a single specification. In the same article, a multivariate framework is constructed for the joint-dynamics of electricity price and a correlated state variable. Defining the correlated process as demand, weather or fuel price, the multivariate model allows for cross-commodity hedging. Analytical results are also derived for several electricity derivatives under the three proposed models, which differentiates the paper from the simulation approaches previously implemented (eg Kaminski, 1997). Generalising previous modelling, Escribano et al. (2001) suggests a price formulation that captures two additional price features; volatility clustering in the form of GARCH effects and seasonality (emphasised by Lucia and Schwartz, 2001), both in the deterministic component of prices and the jump intensity.

It should be emphasised that the continuous-time models reduce to familiar formulations in discrete time. For instance, mean reversion is equivalent to an AR(1) process: P t =a o + o Pt 1 + nt , where a o = (1 e ), 1 = e , nt = e ( s t )dW ( s ) .
t 1 t

Similarly, jump diffusion implies:

o + Pt -1 + 1t with prob 1 - Pt = , where 1t , 2t ~ N(0,1) o + Pt -1 + J + J 2t with prob Despite their intuitive interpretation and non-linear behaviour, jump-diffusion models present some limitations. Firstly, it is assumed that all shocks affecting the price series die out at the same rate. In reality however, two types of shocks exist implying different reversion rates; large disturbances, which diminish rapidly due to economic forces, and moderate ones, which might persist for a while. Estimation bias is inevitable, as Maximum Likelihood methods tend to capture the smallest and most frequent jumps in the data. As emphasised by Huisman and Mahieu (2001), stochastic jump-models do not disentangle mean-reversion from the reversal of spikes to normal levels. Secondly, the model assumptions for jump intensity (constant or seasonal) are convenient for simulating the distribution of prices over several periods of time, but restrictive for actual short-term predictions for a particular time. The Poisson assumption for jumps may be valid empirically, but only provides the average probability of jumps for particular transitions. Tight demand-supply conditions and hence spikes, could emerge irrespectively of season, for instance due to anomalous fuel prices or a technical failure in supply. For this reason, a causa, structural representation, instead of probabilistic seasonal formulations could be more appealing for the parameters of the jump distribution for specific day ahead forecasting. An alternative modelling framework to jump-diffusion is regime-switching, and this may be more suitable for actual price forecasting. This can replicate the price discontinuities, observed in practice, and could detach the effects of mean-reversion and spike reversal, aliased in jump-diffusion. Ethier and Mount (1999) assume two latent market states and

an AR(1) price process under both the regular and the abnormal regime and constant transition probabilities, i.e. Pt St = ( Pt 1 St 1 ) + t , where t ~ N (0, s2t ) . This retains however the misspecification and also imposes stationarity in the irregular spike process. The model suggested by Huisman and Mahieu (2001), allows an isolation of the two effects assuming three market regimes; a regular state with mean-reverting price, a jump regime that creates the spike and finally, a jump reversal regime that ensures with certainty reversion of prices to their previous normal level. This regime-transition structure is however restrictive, as it does not allow for consecutive irregular prices. This constraint is relaxed in de Jong and Huisman (2002). The two-state model proposed assumes a stable mean-reverting regime and an independent spike regime of log-normal prices. Regime independence allows for multiple consecutive regimes, closed-form solutions and translates to a Kalman Filter algorithm in the implementation stage. Formally:
ln PM ,t = ln PM ,t 1 + a ( ln PM ,t 1 ) + M ,t , ln PS ,t = s + S ,t

where PM and PS denote the mean-reverting and spike regimes respectively,

2 M ,t ~ N (0, M ) and S ,t ~ N (0, s2 ) .

Assuming a regime-switching price model, day-ahead price forecasts can be derived as: i) The expected price across regimes i.e. a linear combination of the predicted prices across regimes with weights the predicted regime probabilities.
s

Pt +1 = P i t +1 Pr( S t +1 = i I t ) , where It denotes the information set up to day t.

i =1

ii) The prediction of the regime with the higher predicted probability i.e.

Pt +1 = P i t +1 , where Pr( S t +1 = i I t ) > 0.5 .

The former scheme typically over-estimates regular prices, whereas the latter generally selects the normal regime. Both findings reflect the fact that the irregular states in the semi-deregulated electricity markets are recurrent but not persistent. Thus, the predicted probabilities of the extreme state very rarely exceed a conventional threshold level. Whereas this problem might be averaged out when simulating several periods ahead with the intention to price a financial instrument, it is critical for the precision required in a day-ahead prediction. Imposing a lower threshold for the selection of the irregular regimes would seem arbitrary. An intuitive alternative would be to specify the regimeprobability as a logistic function of a strategic effect, e.g. Margin. Convergence complications could still emerge due to the non-linear nature of the problem, the nondisjoint probability densities across regimes and possibly the inability of an S-shaped function to capture kinked effects. A potentially more accurate description of electricity prices is proposed in Bystrom (2001). After assuming an AR-GARCH price process with a seasonal component in volatility, the residuals are modelled with distributions from extreme value theory. This approach avoids the estimation complexities and forecasting limitations present in the previous stochastic models due to sudden and fast-reverting spikes. Finally, it should be noted that the stationarity of electricity prices is not present in the same way in different power markets. If the modelling involves daily average or byperiod spot prices, a mean-reverting process with a seasonal trend, proposed for instance in Lucia and Schwartz (2002), seems appealing for some markets. However, discrepancies do exist. In Atkins and Chen (2002), time and frequency-domain tests reject the null hypotheses of I (1) and I (0) processes for the electricity prices in Alberta. Long memory features are subsequently identified in the price evolution and described with autoregressive fractional difference (ARFIMA) models. In Stevenson (2002), a unit root is identified in the Victorian market, possibly because all hourly prices are retained in the same data set and not divided by load period.

A common feature of the finance-inspired stochastic models reviewed in this section is their main intention to replicate the statistical properties of spot prices with the ultimate objective of derivatives evaluation. In order to retain simplicity and/or analytical tractability, the models include only a few factors and typically focus on daily average prices, which are sensitive however to outliers. Albeit convenient for options pricing, the previous properties, i.e. aggregation of intra-day information and modelling of nonrobust measures, are restrictive from a forecasting prospective. Estimation complexities and forecasting limitations are further enhanced due to the abrupt and fast-reverting nature of price spikes. Also, a subtle issue systematically neglected is the assessment of model adequacy with market data. In this framework, although stochastic models have been substantially adapted to the peculiarities of electricity, they have still got a long way to go in revealing the main components of price structure. Recent papers (e.g. Knittel and Roberts, 2001) have emphasised the need to explore this structure and include it in price specifications. A realted challenge is to explore how the sensitivities of prices to influential factors vary throughout the day as a response to the fundamentals of intra-day variation in demand, plant-operating constraints as well as the generators strategic potentials.

Forward Spot Price Dynamics in Electricity

In financial markets, spot and forward prices are usually linked with an analytical formula derived from non-arbitrage conditions: F (t , T ) = e
r (T t )

S (t ) , where F (t , T ) is

the forward price of the underlying asset at time t with maturity at T, S (t ) is the spot price of the asset and r the risk-free interest rate. For storable commodities, the above relationship is adapted to include the convenience yield y: F (t , T ) = e
( r y ) (T t )

S (t ) , where

T refers to delivery date. In electricity markets, however, the convenience link between forward and spot price clearly does not exist in the storable sense. The dominant perception, as expressed in Skantze and Illic (2000), is that the spot price S (t ) reflects only the current state of demand and supply, and is independent, due to non-storability, of

forward prices F (t , T ) with maturities in the medium and long future. The limited impact of long-term price movements on short-term dynamics is further illustrated with the low correlation between short and long-term forward prices, identified in Koekebakker and Ollmar (2001), which would be an unconventional feature for storable commodities. Spot price volatility however, strongly encourages contract coverage and hence raises forward electricity prices. Due to this causality, forward price is often specified as a function of expected spot price, its variance and a random disturbance. Harris (2003) emphasises the link between spot and forward in terms of risk aversion and risk management. The impact of forward prices with maturity T on the spot price S(T) is also idiosyncratic. The efficient markets hypothesis (EMH), stating that forward prices are unbiased estimators of future spot prices, is often invalid for storable commodities, where a positive risk premium tends to exist for holding a futures contract. In electricity however, the sign of the forward premium seems indefinite or irregular. To assess efficiency in US electricity futures markets, Avsar and Goss (2001) adopt the forecast error approach. Under the EMH, no systematic relationship should exist between current prediction error of the spot price, S (t + k ) F (t , t + k ) , and prior errors, S (t ) F (t k , t ) , for the same and related commodities. A solid econometric analysis indicates time-varying risk premia, which reflect volatility in the form of an M-GARCH term. The significant negative impact of trading volume on forecast errors suggests a violation of the rational expectations hypothesis, possibly due to agents learning. In a simplistic analysis that does not account for fundamentals, Botterud et al. (2002) report a negative risk premium for the Scandinavian futures markets. This peculiarity is attributed to the difference in flexibility between demand and generation side, which creates a higher incentive to the former to hedge their positions. Beyond stating the irregularity of the premium sign, Longstaff and Wang (2002) explore the economic properties of percentage forward premia regressing them on ex-ante risk factors. These include a measure of price risk, an indicator of demand uncertainty and the expected load, perceived as a proxy for the probability of spike occurrence. Significant structure is

detected in forward premia with strong intra-day variation, but many aspects of pricing remain vague, as indicated by the very low R2. At this point, it should be emphasised that even for short time differences, electricity prices could display persistent deviations. In Borenstein et al. (2001), absence of convergence is identified between the California day ahead and hourly real-time markets. This discrepancy is interpreted as trading inefficiency. Proposed explanations include transaction costs, information barriers or dominance of risk aversion. The selected justification is that a shock occurred in the price formation process following the revision of market rules and the strategic behaviour of one company. The above reasoning still underestimates the fact that the different timing of the two markets implies different information uncertainties and plant flexibility requirements, which are converted to costs. This remark is also consistent with Walls (1999), where maturity effects appear much stronger for electricity than other energy futures.
Forward Curve Modelling

One class of approaches inspired by the financial assets literature abandons the modelling of spot price dynamics and focuses instead on the term structure of forward/futures commodity prices across different maturities. This modelling has been more appealing to mature markets, such as the Nordic and Victoria, in Australia, which introduced hedging tools early in their liberalisation process. Clewlow and Strickland (1999) adapt the multifactor model of Cortazar and Schwartz (1994). process for futures prices is:
n dF (t , T ) = i (t , T )dwi (t ) F (t , T ) i =1

Formally, the general risk-adjusted

d ln F (t , T ) =

1 n 2 i (t , T )dt + 2 i =1

i =1

(t , T )dwi (t )

where F(t,T) denotes the futures price at time t for delivery at T, wi are independent Brownian motions under the equivalent-martingale measure and i are volatility functions of spot prices. Each factor i is associated with a volatility function that determines the magnitude and direction of the shift of each point in the forward curve due to the arrival of information related to a particular source of uncertainty. The forward curve movements are such that preclude arbitrage opportunities when trading among future contracts. The number n of volatility components is suggested from eigenvalue decomposition of the covariance matrix of forward returns. Usually, there are volatility factors that shift, tilt and bend the forward curve describing the relative movements of short, medium and long-term contracts. The mathematical formulation of this procedure is summarised below: Consider M forward contracts with relative maturities 1 ,..., j , ... M and values recorded on N time points (e.g. days). This equation can be discretised for small time changes t as follows: ln F (t , t + j ) =
n 1 n 2 i (t , t + j )t + i (t , t + j )W j 2 i =1 i =1

Let x jk denote the return of the forward contract j on day tk , x j the sample mean, jl the covariance between the returns of contracts j and l and = ( jl ) the sample covariance matrix of forward returns. Then, x jk = ln F (t k , t k + j ) = ln( F (t k , t k + j )) ln( F (t k t , t k + j t )) , k =1,2,..,N.

jl =

1 N

(x
k =1

jk

x j )( xlk xl )'

The symmetric matrix can be factorised as = , where = diag{i } is a diagonal matrix of eigenvalues and an orthogonal matrix of corresponding eigenvectors vi = (v ji ) . The Principal Component Analysis of forward returns, described above, reveals uncorrelated (orthogonal) dimensions of the variability in forward returns.

From the M calculated eigenvalues, the n larger are selected, which account for a fraction

of the total variation equal to

i =1 i =1 M

. The forward curve movement at point t due to factor

i

i is calculated as: i (t , t + j ) = v ji i .

General volatility functions are proposed from which existing models are derived as special cases and analytical formulae are obtained for various hedging products. A different approach for modelling the electricity forward curve is proposed in Audet et al. (2002). Forward prices are specified as log-normally distributed, strongly correlated for adjacent maturities and with lower than spot volatility, which is assumed deterministic. Despite this constraint, which contradicts the empirical evidence of stochastic volatility, the model captures attributes of the term structure not addressed in previous specifications. Whereas forward curve modelling allows a parsimonious description of the term structure, its extension to forecasting remains an open question. Complexities also arise at the estimation level. Firstly, as in convenience yield modelling, the parameters are estimated by equating observed market prices (or derivative prices and implied volatilities) to the ones derived by the theoretical model. In the forward curve approach, the model validation involves forward prices and their sample covariance matrix. This implies exclusive focus on the risk-neutral forward price measure and absence of link to actual spot prices. Although this is a desirable feature for commodities where spot data are unobservable or scarce, such an obstacle is rarely present in electricity markets, which are usually immature and in some cases possess limited forward data. To avoid loss of critical information, the volatility function could be linked with spot volatility S , e.g.
n dF (t , T ) = S (t ) i (T t )dz i (t ) F (t , T ) i =1

Secondly, even in deep forward markets, reliance of estimation only on linear payout assets may lead to poor estimates of volatility parameters, as their impact on futures prices could be delayed or not strong enough. Exploiting options prices in addition to futures could correct some of these problems. Thirdly, the forward price dynamics of electricity are influenced by several institutional and market structure factors, which are not addressed in the theoretical models. Finally, the discreteness of prices and market immaturity impose the need for estimating forward prices for dates with no transactions. The substitution of missing values is often implemented with a maximum smoothness criterion. An attempt to resolve some of the above issues is presented in two papers which exploit a richer set of market data than simply forward prices. The first utilises both spot and forward prices, whereas the second links a bottom-up model with forward data. More specifically, in Karesen and Husby (2001), observed futures prices are assumed to equate the theoretical ones, implied from a spot price specification, plus some noise resulting from market imperfections such as bid-ask spread and risk preferences. Assuming this divergence is consistent with market incompleteness and allows model validation with richer information sets. Further research could be conducted on the proper specification and estimation of the parameters that cause the discrepancy. In Fleten and Lemming (2003), a smooth forward curve is obtained by constraining the theoretical prices, derived from an optimisation bottom-up model, with the bid/ask prices observed in the market. Although it may restrict extreme deviations, the heuristic of imposing bounds in predicted prices does not ensure the consistency of structure between observed and model prices. Depending on the model structure, the spot and forward approaches are often equivalent for storable commodities. A reduced specification of the forward curve dynamics implies a process for the spot price and vice versa. In incomplete electricity markets however, this link is not as direct. If a stochastic differential equation is assumed for the dynamics of F (t , T ) , then an explicit specification is derived with Itos lemma and the spot price is simply calculated as S (t ) = F (t , t ) . This ignores however that the long-run

dynamics implicit in the forward curve are inadequate to explain the spot variations. The reverse task entails deriving the forward price as expected future spot price under the risk-neutral measure. Although the presence of a risk premium can be addressed, the fundamental issue of non-unique risk-neutral measure in incomplete markets is rather neglected in the literature. Defining an optimal risk measure for the unhedgable part of the risk, according to a distance criterion that also exploits actual market data, is a challenging research issue.

Structural Modelling

Moving away from the financial perspective of seeking to develop parsimonious stochastic models that can feed into derivative pricing formula, structural models intend to uncover a richer structure for electricity prices in order to understand market performance and enable to most accurate forecasting, per se. They synthesise two sources of information; historic market prices and fundamentals such as load, weather and plant data. For instance, a simple regression model that relates spot price to lagged price and demand values is suggested in Nogales et al. (2002). Their model wasrefined by adjusting the number of lags until the assumption of uncorrelated errors was satisfied. It is expressed as: Pt = c + d ( B) Dt + p( B) Pt + t where Dt is the demand at time t,

d (B) and p (B) polynomial functions of the

backshift operator B : Bxt = xt 1 with order d and p respectively. The predictive ability of this model seems limited, however, in the case of markets with strategic market power and complex trading environments. Other structural formulations address non-linear aspects of electricity price dynamics, such as multiple price regimes and jumps. Vucetin et al. (2001) implement a discovery algorithm of regression regimes, which reveals multiple price-load relationships in spot

trading. The assumption of a moderate switching rate between regimes, necessary for convergence, is unappealing for the sudden spikes in electricity, but could describe smooth regime transitions in the medium term. As the regime-switching process is not modelled, the algorithm is constrained to the analysis of past data, rather than forecasting. In Davison et al. (2002), prices are assumed to follow a mixture of two normal distributions. A nice property of the model is that, in contrast with Markov-regime switching, the regime probabilities are related empirically to a variable with economic and strategic attributes, the ratio demand/supply. To derive a more general formulation that allows medium-term forecasting, load is specified as a sinusoidal function and capacity across the year as a two-level categorical variable. This approximation ignores the interaction between prices and capacity availability decisions and could be replaced by a richer stochastic equation. In Skantze et al. (2000), hourly price is specified as an exponential function of demand and supply. Both are assumed stochastic with a deterministic monthly component plus a random term. Due to the pronounced intra-day correlation, the random terms are derived from a Principal Component Analysis, similarly to Wolak (1997). To capture stochastic effects, the loadings are specified as mean-reverting to a stochastic mean. A distinct feature of the model, compared to standard jump-diffusion, is the utilization of technical knowledge in the supply equation. More specifically, a Markovian process is assumed for plant outages with parameters related to technologies. This does not however include the possibility of strategic capacity withholding, leading to price manipulation, which, of course, has been a major concern to regulators in electricity markets, and has strong implications for accurate price forecasting.

The existence of multiple, different, components in electricity pricing is reflected in Stevenson (2002). The price and demand series are decomposed into multiple levels of resolution with wavelet analysis and signal is differentiated from noise with a robust smoother-cleaner transformation. For the reconstructed data, a threshold autoregressive (TAR) model is suggested with demand as a critical variable, i.e.
ao + D + a1 Pt 1 + ... + q Pt q , if Dt < 0 Pt = o + D + 1 Pt 1 + ... + q Pt q , if Dt 0 Price changes are modelled due to the presence of a unit root and assigned to one of two regimes depending on whether the change in demand is positive or negative. As this restriction excludes the impact on prices of other fundamentals, such as supply, the model could be more flexible by defining the threshold variable as a function of the ratio demand/ supply. The smoothing procedure eliminates the leakage of rapidly reverting price spikes to more fundamental resolution levels, where information takes progressively longer to be impounded into price. This allows a more reliable estimation of the baseline regime. Price spikes however, are perceived as noise and despite their information content, are essentially removed from the data.

Non-parametric Modelling

As the forecasting, rather than modeling, emphasis gradually becomes more pragmatic, several non-parametric techniques, such as genetic algorithms and neural networks, have inevitably been adopted for price prediction. An indicative list includes neural networks applications for the England -Wales pool by Ramsay and Wang (1997), for the California market by Gao et al. (2000), Spain by Centano Hernandez et al (2003) and Victoria by Szkuta et al. (1999); fuzzy regression models linking demand and price by Nakashima et al (2000); Fourier and Hartley transformations by Nicolaisen et al. (2000). Although nonparametric models tend to be flexible, can handle complexity and thereby promising for

short-term predictions, they do not provide structural insights and forecasts of the price distribution, which limits their application to risk management.

Electricity Price Forecasting: Research Challenges and Practice

Although much of the statistical characteristics of electricity prices are replicated with existing stochastic models, their idiosyncratic price structure has not been fully represented adequately in the empirical research literature. The research inspired by the needs of risk management is more concerned with capturing the distribution of prices over a period of time, than in the actual levels of prices at particular times. For the latter case, the structural approach is the most appropriate but unresolved modelling issues include: i) The factors reflected upon spot prices, for instance economic fundamentals, plant constraints, strategic behaviour, perceived risks, forward contracting, trading inefficiencies and market design effects. ii) iii) iv) The magnitude, relative importance and intra-day variation of the above impacts on prices. The non-linear nature and dynamics of structural effects as markets evolve. The response of volatility to fundamentals and shocks, and particularly the sources of residual uncertainty when structural components are removed from prices. v) vi) vii) viii) The structural characteristics of market cycles and the pricing scheme under temporal market irregularities, such as plant failure How to fully endogenise the market drivers of regime switching, such as reserve margin. The feedback of electricity prices on input variables, eg gas prices, when there is increasing evidence of cointegration between the two commodities. Electricity markets tend to be inter-connected through networks that occasionally become congested; again regime switching between spatial

models of price and volatility interaction become an added dimension for modelling.

In order to address the structural questions raised above and model more fully the idiosyncrasies of electricity price dynamics, an rich econometric modelling framework for variance decomposition would appear to be required. Firstly, a vector Xt of aggregated factors likely to influence spot prices should be defined. A categorisation of structural effects would include: Market mechanism, Market structure, Non-strategic uncertainties, Efficiency variables, Behavioural parameters and Time effects, as for example, in Table 2.

Table 2. Classification of Factors Influencing Spot Prices Market Mechanism

Demand Polynomial, Capacity Margin, Fuel Prices, Demand Slope, Demand Curvature, Demand Volatility, Forward Prices

Market Structure Non-Strategic Uncertainties Efficiency Variables Behavioural Variables

Margin, Concentration Indices Ratio = Margin / Demand Forecast Demand Forecast Error Volume, Availability Indices Lagged Price, Lagged Daily Average Price, Price Volatility, Demand Volatility, Lagged Spread, Time

Time Effects (Temporal / Systematic)

Daily, Weekly, Seasonal

Tables 3 and 4 indicates a standard approach to rich structural modelling, although it is rather remarkable that a full implementation of this comprehensive model-building framework does not yet appear to have been applied to electricity spot prices.

Table 3. Structural Price Modelling. OLS Regression Model (I)

Pjt = X jt j + jt , jt ~ i.i.d .N (0, 2j ) Pjt = X jt jt + jt

Which factors influence spot prices?

Time-Varying Regression Model (II)

Are the structural impacts stable over time?

jt = j (t 1) + v jt - Random walks jt ~ i.i.d .N (0, 2 ) , v jt ~ N k (0, j ) ,

j

E ( jt v jt ) = 0 , j = diag{ v2 } , v jt = (v j1t , v j 2t ,..., v jkt )

jk

Regression

Model

with

Markov

P jt = X t j + jt , jt ~ N (0, j S )
2

Regime-Switching (III)

St

How do prices react to temporal market irregularities?

Pr( S t = 1 S t 1 = 1) = p j , Pr( S t = 2 S t 1 = 2) = q j

Table 4. Structural Volatility Modelling. GLS Structural Model

Pjt = X jt j + jt unconditional

How

does

residual

j ~ N n (0, j ),
Var ( jt ) = f (Y jt ), j = diag{ f (Y jt )} I n ,

variance respond to fundamentals?

e.g. f (Y) = exptY or f (Y) = ( + |Y| b) 2

Regression GARCH Model

Pjt = X jt j + jt

How does residual conditional volatility react to past volatility and shocks?

jt = h jt u jt , u jt ~ t-distribution
Var ( jt I t 1 ) = h jt = g ( j ( t 1) , h j ( t 1) )

GARCH (1,1)
h jt = a o + a1 2( t 1) + a 2 h j ( t 1) j

Do positive and negative shocks have the Asymmetric GARCH (1,1) same impact on future volatility?
h jt = a 0 + 1 2( t 1) + At 1 2( t 1) + a 2 h j ( t 1) j j

[In the above tables, Pjt denotes the spot price in period j and day t, j a kx1 vector of explanatory variables, S t an index variable of market state, Y the variable driving the volatility and At an indicator variable taking the value 1 if t < 0 and 0 otherwise].

Finally, although the imperfections of electricity market create a richness of structure for modellers, and most of these economic, technical and behavioural influences can be captured by a mixture of econometric and stochastic specifications, the political sensitivity of electricity should not be underestimated. Even though the markets have been liberalised, regulatory interference is never far away (see Bower, 2003), and high prices only have to persist for a few months before price caps emerge, as indeed they have in the Britain, Spain and California. Similarly social, industrial and environmental polices are always on the horizon, as we have seen recently, with carbon taxes and renewable credits. It would appear prudent, therefore, that an analysis of institutional intent should provide the very basic set of background assumptions, followed by a strategic analysis of the major players, before a dynamic structural and stochastic specification of the pricing model is attempted. From a forecasting perspective, this will require the time series and econometric specifications to be consistent with higher level political and strategic models, which will inevitably be of quite a different, more subjective character.

References

Anderson E.J. and H. Xu, 2002. Nash Equilibria in Electricity Markets with Discrete Prices. Working paper, University of New South Wales. Atkins F. and J. Chen, 2002. Some Statistical Properties of Deregulated Electricity Prices in Alberta. Working Paper, University of Calgary. Audet N, P. Heiskanen, and I. Vehvilinen, 2003. Modeling Electricity Forward Curve Dynamics in the Nordic Market. In Bunn, D.W. (ed) Modeling Prices in Competitive Electricity Markets. Wiley, 2003 Avsar S.G. and Goss, B.A., 2001. Forecast Errors and Efficiency in the Electricity Futures Markets. Australian Economic Papers, 40, 479-499. Baldick R., 2002. Electricity Market Equilibrium Models: The Effect of Parametrisation. IEEE Transactions on Power Systems, 18, 4. Baldick R., R. Grant, and E. Kahn, 2000. Linear Supply Function Equilibrium: Generalizations, Application, and Limitations, Working paper, PWP-078, University of California Energy Institute. Batstone S., 2000. An Equilibrium Model of an Imperfect Electricity Market. Working paper, University of Canterbury. Bjerksund P., H. Rasmussen and G. Stensland. 2000. Valuation and Risk Management in the Norwegian Electricity Market, Discussion paper 20/2000, Norwegian School of Economics and Business Administration. Borenstein S., J.B. Bushnell, C. Knittel and C. Wolfram, 2001. Trading Inefficiencies in Californias Electricity Markets. Journal of Political Economy, submitted.

Botterud A., A.Bhattacharyya and M. Illic.,2002. Futures and Spot Prices - An Analysis of the Scandinavian Electricity Market. In: Proc 34th Annual North Americal Power Symposium, Tempe, October 2002. Bower, J. 2003. Price Impact of Horizontal Mergers in the British Generation Market. In Bunn, D.W (ed) 2003 Bunn, D.W, (ed), 2003. Modelling Prices in Competitive Electricity Markets. Wiley Bunn D.W. and F. Oliveira, 2001. Agent-Based Simulation - An Application to the New Electricity Trading Arrangements of England and Wales. IEEE Transactions on Evolutionary Computation, 5. Bystrom H., 2001. Extreme value theory and extremely large electricity price changes. Working paper, Lund University. Centano Hernandez, E., et al., 2003. Competitors Response Representation for Market Simulation in the Spanish Daily Market. In Bunn, D.W. (2003). Clewlow L. and C. Strickland, 2000. Valuing Energy Options In A One-Factor Model Fitted To Forward Prices. Working paper, University of Sidney. Cortazar G. and E.S. Schwartz, 1994. The Valuation of Commodity Contingent Claims. Journal of Derivatives, 1, 4, 27-39. Davison M., L. Anderson, B. Marcus and K. Anderson. 2002. Development of a Hybrid Model for Electrical Power Spot Prices, IEEE Transactions on Power Systems, 17, 2. Day C., B. Hobbs B. and J.-S. Pang. 2002. Oligopolistic Competition in Power Networks: A Conjectured Supply Function Approach. IEEE Transactions on Power Systems, in press.

de Jong C. and R.Huisman, 2002. Option Formulas for Mean-Reverting Power Prices With Spikes. Working Paper, Erasmus University. Deng S., 2000. Stochastic Models of Energy Commodity Prices and their applications: Mean Reversion with Jumps and Spikes. Working paper, PWP-073, University of California Energy Institute. Deng, S., B. Johnson and A. Sogomonian, 2001. Exotic Electricity Options and the Valuation of Electricity Genertaion and Transmission Assets. Decision Support Systems,
30, 383-392.

Duffie, D., S. Gray and P.Hoang, 1998. Volatility in Energy Prices. Managing Energy Price Risk. Risk Publications. Escribano A., J. Peaea J. and P. Villaplana, 2002. Modelling Electricity Prices: International Evidence. Working paper, Universidad Carlos III de Madrid. Ethier R. and T. Mount, 1998. Estimating The Volatility of Spot Prices In Restructured Electricity Markets And The Implications For Option Values. Working paper, Cornell University. Gao F., X.Guan, X-R Cao and A. Papalexopoulos. 2000. Forecasting Power Market Clearing Price and Quantity Using a Neural Network Method. In: Proc. Power Engineering Summer Meeting, Seattle, July 2000. Green R.J. and D.M. Newbery, 1992. Competition in the British Electricity Spot Market. Journal of Political Economy, 100, 5, 929-53. Green R.J. 1999. The Electricity Contract Market in England and Wales. Journal of Industrial Economics, XLVII, 107-124.

Harris, C. (2003). Forecasting Higher Moments of the Medium term Power Price Curve using Equilibrium Economics and the OptionValue of the Security of Supply. In Bunn, D.W, (ed). Modelling Prices in Competitive Electricity Markets. Wiley Huisman R. and R. Mahieu, 2001. Regime Jumps In Power Prices. Working paper, Energy & Power Risk Management. Johnson and Barz, 1999. Selecting Stochastic Processes For Modelling Electricity Prices. Energy Modelling and the Management of Uncertainty, Risk Publications. Kaminski V., 1997. The Challenge of Pricing And Risk Managing Electricity Derivatives. The U.S. Power Market, 149-71. Karesen K.F. and E. Husby. 2000.A joint State-Space Model for Electricity Spot and Futures Prices. Working paper, Norwegian Computing Center. Kellerhals B., 2001. Pricing Electricity Forwards Under Stochastic Volatility. Working paper, Eberhand-Karl University, Tubingen. Kholodnyi V., 2001. A Non-Markovian Process for Power Prices With Spikes for Valuation of European Contingent Claims On Power. Working paper, TXU Energy Trading. Knittel C.R. and M. Roberts, 2001. An Empirical Examination of Deregulated Electricity Prices. Working paper, PWP-087, University of California Energy Institute. Koekebakker S. and F. Ollmar, 2001. Forward Curve Dynamics in The Nordic Electricity Market. Working paper, Norwegian School of Economics and Business Administration.

Lemming J. and S.E. Fleten. 2003. Constructing Forward Price Curves in Electricity Markets. Energy Economics, In Press. Longstaff F. and A. Wang, 2002. Electricity forward prices: A high-frequency empirical analysis. Working paper, University of California Los Angeles. Lucia and Schwartz, 2002. Electricity Prices And Power Derivatives: Evidence From The Nordic Power Exchange. Review of Derivatives Research, 5, 5-50. Merton R.C., 1976. Option Pricing When Underlying Stock Returns are Discontinuous. Journal of Financial Economics, 3, 125-144. Nakasima T, M. Dhalival and T.Nimura. 2000. Electricity Market Data Representation by fuzzy regression models. In Proc: Power Engineering Summer Meeting, Seattle, July 2000. Nicolaisen J.D., C.W. Richter and G.B. Sheble. 2000. Signal Analysis for Competitive Electric Generation Companies. In Proc: Conference on Electric Utility Regulation and Restructuring and Power Technologies, London, April 2000, 4-7. Nogales F., J. Contreras, A. Conejo and R. Espinola, 2002. Forecasting Next-Day Electricity Prices by Time-Series Models. IEEE Transactions on Power Systems, 17, 2. Peroni E. and R. McNown, 1998. Noninformative and Informative Test of Efficiency in Three Energy Futures Markets. The Journal of Futures Markets, 18, 939-964. Pindyck. R. 2001.Dynamics of Commodity Spot and Futures Markets: A Primer. The Energy Journal, 22, 3.

Skantze, P., A. Gubina, and M. Ilic. 2000. Bid-based Stochastic Model for Electricity Prices: The Impact of Fundamental Drivers on Market Dynamics. Energy Laboratory Report, No. MIT-EL 00-004. Skantze, P., and M. Illic. 2000 The Joint Dynamics of Electricity Spot and Forward Markets: Implications of Formulating Dynamic Hedging Strategies. Energy Laboratory Report No. MIT-EL 00-005. Stevenson M., 2002. Filtering and Forecasting Spot Electricity Prices in the Increasingly Deregulated Australian Electricity Market, Working paper, University of Technology, Sydney. Szkuta B., L.A. Sanabria and T.S. Dillon, 1999. Electricity Price Short-term Forecasting Using Artificial Neural Networks. IEEE Transactions on Power Systems, 4, 851-857. Vucetin S., K. Tomsovic and Z. Obradovic, 2001. Discovering Price-Load Relationships in California Electricity Market. IEEE Transactions on Power Systems, 16, 2. Walls W.D., 1999. Volatility, Volume And Maturity In Electricity Futures.''

Applied Financial Economics, 9, 3, 283-287. Weron R., 2000. Pricing Electricity Options in the Scandinavian Market. Working paper, Wroslaw University of Technology. Wolak F., 1997. Market Design And Price Behaviour In Restructured Electricity Markets: An International Comparison. Working paper, Stanford University.

Period 16
35000 25000

MWh

50

100

150

200

250

300

Day

Period 26

MWh

30000

40000

50

100

150

200

250

300

Day

Period 36
45000 30000

MWh

50

100

150

200

250

300

Day

Figure 1. Demand Evolution for load periods 16 (6.30-7am), 26 (11.30-12pm) and 36 (4.30-5pm) in 6thJune, 2001 1st April, 2002.

Period 16
10 15 20 25 5

/MWh

50

100

150

200

250

300

Day

Period 26

/MWh

10

30

50

50

100

150

200

250

300

Day

Period 36
100 20 60

/MWh

50

100

150

200

250

300

Day

Figure 2. Price Evolution for load periods 16 (6.30-7 am), 26 (11.30-12 pm) and 36 (4.30-5 pm) in 6thJune, 2001 1st April, 2002.