Competition Instructions

1. Register your email
   Please enter your email to receive the login information to download the dataset, access the datasets and descriptions of previous competitions (NN3) and to receive future announcements. (Please note, that your login information for the NN3 competition will still allow you to access the NN3 datasets and presentations, but not the new datasets of NN5!):

2. Select a Dataset
   The competition will offer 2 datasets. All time series of the sets is drawn from a homogeneous population of cash withdrawals at different cash machines located at different, unrelated locations in England. Choose only one of the two datasets! Those forecasting the complete set will automatically be evaluated (and eligible to win awards and prices) on both data sets. Consequently we particularly encourage submissions for the dataset A with 111 time series.
   • Dataset A is a complete dataset of 111 daily time series from 111 different cash machines .
   • Dataset B is a sub sample of 11 time series from the 111 time series, and is therefore contained in the larger dataset.
      
3. Download the data
   • Click on the download link below and enter your login & password in the dialog-box (case sensitive entry!) to download the datasets. The login is provided in step 1 when you register your email-address and personal details. 
4. Forecast
   • Develop a single methodology to use on all time series - ideally in software code or though exactly repeated steps & tests conducted by a human expert (see the FAQ if this is unclear).
   • Forecast all of the in-sample training data. For the in-sample data please provide the 1-step ahead forecasts, as there is no room to provide 56-step ahead forecasts for each time origin of the training data. This data will be used to validate goodness of fit, but will not be used to evaluate and rank the performance of your submission. If you cannot provide this, please leave it empty
   • Forecast the last 56 observations as a trace forecast for a forecasting horizon of 1, 2, ..., 56 for each of the 11 or 111 time series .
      
5. Write Description of your Method
   • Documented the methodology you have used in a brief summary of 2 to 6 pages IEEE format (you may use the Microsoft Word or LaTeX template found on the DMIN submission website).
   • Only papers prepared in PDF format will be accepted. Violations of any of the above paper specifications may result in rejection of your paper.
     • Paper Size: US Letter format (8.5" x 11").
     • Paper Length: Maximum 6 pages, including figures, tables & references.
     • Paper Formatting: double column, single spaced, 10pt font.
     • Margins: Left, Right, and Bottom: 0.75" (19mm). The top margin must be 0.75" in (19 mm), except for the title page where it must be 1" (25 mm).
     • File Size Limitation: 5.0MB.
     • Do not number your manuscript pages.

     If you have already submitted a full paper to the DMIN'08 session of the competition please resubmit this paper with your results! If you have only submitted an abstract to ISF'08 or presented at WCCI'08 or DMIN'07 without submitting a formal paper submitting this document is mandatory to receive your error results and ranking. All submissions not fully documented will not be evaluated for forecasting accuracy!

      

6. Submit your forecasts
   • Record your forecasts in the original Microsoft Excel data files you downloaded.
   • Rename the forecasting file to include your last name (of the main author / contestant from a group).
   • Create the PDF of your method descriptions
   • Create an email which MUST:
     • be addressed to submission@neural-forecasting-competition.com
     • have the text "NN5 submission" in the subject line
     • Include your name & email contact information in the main text
     • inlcude the names & contact emails of ALL co-workers
   • Attach the following two files to the email:
     • attach the Excel file of your preditions
     • attach the PDF-description of the methodology used
     • Send the email by the submission deadline 18 MAY 2008, 0:00 CET

If you encounter any problems in submitting please contact sven.crone@neural-forecasting.com immediately!

General Instructions

• Submissions are restricted to one entrance per competitor.
• The competitors must certify upon submission that they didn’t try to retrieve the original data.
• As this is predominantly an academic competition, all advertising based upon or referencing the results or participation in this competition requires prior written consent from the organisers.

  Submitting your predictions to us will not automatically allow you to present your method at a conference. In addition to submitting, we therefore encourage you to submit to one of the conferences where we will host special sessions. This will allow you to

  • submit only an abstract & present at the International Symposium on Forecasting, ISF'08 without publication of a paper
  • submit a full draft-paper, publish your paper and present your results at the International Conference on Data Mining DMIN'08
  • submit abstract & paper and attend & present at two or more conferences!
  • submit to the post-competition publications (currently under negotiation)

  Please check back here regularly for information on submission deadlines & dates for theses conferences.

Experimental Design

The competition design and dataset adhere to previously identified requirements to derive valid and reliable results.

• Evaluation on multiple time series, using 11 and 111 daily time series
• Representative time series structure for cash machine demand 
• No domain knowledge, no user intervention in the forecasting methodology
• Ex ante (out-of-sample) evaluation
• Single time series origin (1-fold cross validation) in order to limit effort in computation & comparisons
• Fixed time horizon of 56 days into the future t+1, t+2, ..., t+56
• Evaluation using multiple, unbiased error measures
• Evaluation of "novel" methods against established statistical methods & software benchmarks
• Evaluation of "novel" methods against standard Neural Networks software packages
• Testing of conditions under which NN & statistical methods perform well (using multiple hypothesis)

Datasets

Two datasets are provided, which may be found [here].

Methods

The competition is open to all methods from Computational Intelligence, listed below. The objective requires a single methodology, that is implemented across all time series. This does not require you to build a single neural network with a pre-specified input-, hidden and output-node structure but allows you to develop a process in which to run tests and determine a best setup for each time series. Hence you can come up with 111 different network architectures, fuzzy membership functions, mix of ensemble members etc. for your submission. However, the process should always lead to selecting the same final model structure as a rigorous process.

• Feed forward Neural Networks (MLP etc.)
• Recurrent Neural Networks (TLRNN, ENN, ec.)
• Fuzzy Predictors
• Decision & Regression Trees
• Particle Swarm Optimisation
• Support Vector Regression (SVR)
• Evolutionary &  Genetic Algorithms
• Composite & Hybrid approaches
• Others

These will be evaluated against established statistical forecasting methods

• Naïve
• Single, Linear, Seasonal & Dampened Trend Exponential Smoothing
• ARIMA-Methods

Statistical benchmarks will be calculated using the software AUTOBOX and ForecastPro, two of the leading expert system software packages for automatic forecasting (provided by courtesy of Dave Reilly and Eric Stellwagen -THANKS!). We hope to also evaluate a number of additional packages: SAS, NeuralWorks (pending), Alyuda Forecatser (peding), NeuroDimensions (pending). In addition, the competition is open for submissions from statistical benchmark methods. Although these can be submitted and evaluated as benchmarks, only methods from computational intelligence are eligible to "win".

Evaluation

We assume no particular decision problem of the underlying forecasting competition and hence assume symmetric cost of errors. To account for a different number of observations in the individual data sub-samples of training and test set, and the different scale between individual series we propose to use a mean percentage error metric, which is also established best-practice in industry and in previous competitions. All submissions will be evaluated using the mean Symmteric Mean Absolute Percent Error (SMAPE) across al time series. The SMAPE  calculates the symmetric absolute error in percent between the actuals X and the forecast F  across all observations t of the test set of size n for each time series s with

The SMAPE of each series will then be averaged over all time series in the dataset for a mean SMAPE. To determine a winner, all submissions will be ranked by mean SMAPE across all series. However, biases may be introduced in selecting a “best” method based upon a single metric, particularly in the lack of a true objective or loss function. Therefore, while our primary means of ranking forecasting approaches is mean SMAPE, alternative metrics will be used so as to guarantee the integrity of the presented results. All submitted forecasts will also be evaluated on a number of additional statistical error measures in order to analyse sensitivity to different error metrics. Additional Metrics for reporting purposes include:

• Average SMAPE (main metric to determine winner)
• Median SMAPE
• Median absolute percentage error (MdAPE)
• Median relative absolute error (MdRAE)
• Average Ranking based upon the error measures
• …

Publication & Non-Disclosure of Results

We respect the decision of individuals to withhold their name should they feel unsatisfied with their results. Therefore each contestant reserves the right to withdraw their name and software package used after they have learned their relative rank on the datasets. However, we reserve the right to publish an anonymised version of the descriptions of themethod and methodology used, i.e. MLP, SVR etc without the name of the contributor.