One Step Ahead: A Framework for Detecting Unexpected Incidents and Predicting the Stock Markets


    Unexpected incidents can be destructive or even disastrous, affecting the financial markets. Incidents such as 9/11 attacks (2001), Fukushima nuclear disaster (2011) and Boston Marathon bombing (2013) severely shocked both local and global markets while less extreme ones that happen more often may also be disruptive. Various factors determine their impacts. For investors, it is crucial to quantify the key facts in a timely and accurate manner and estimate the reactions of the markets precisely and instantly. Though the abundance and timeliness of Web data allows such possibility, there are still three major challenges: 1. detecting such incidents and extracting comprehensive facts from the noisy and biased unstructured data; 2. establishing robust market models at scale across time and regions; 3. generating reliable real world predictions with the constraint that the incidents are detected in real time while the models are built on well-curated historical data. As one of the first such attempts, we build a framework that extracts incident facts globally based on a deep neural network, feeds them into models built with a global event database and improved by satellite data, and predicts the stock markets in a simulated real world setting with interpretable results that outperform various baselines.


    We build a framework that instantly detects unexpected incidents, extracts key facts, and feeds them into a market model to predict the market movements with transparent and explainable decision logic.

Fig shows its architecture. First, we cascade two classifiers to decide whether a piece of text reports an incident and which category of incidents it belongs to. According to the category, an information extraction model extracts facts such as time, location, casualty, victims, weapon (for terrorist attacks), magnitude (for earthquakes) and level (for typhoons). For predicting the market, we build a model with market data and historical event data cross-referenced with novel socioeconomic datasets such as satellite data. In a real world setting, the system receives pieces of text and outputs facts about incidents, market direction predictions and demonstrations of logic for the predictions.

Framework architecture