The data for analytics in different fields are abundantly available in social media like Twitter, Facebook and many such micro blogging forums. The topics discussed by the people in the media range from cooking, reviewing films to strategic discussions on important business decisions. There are lots of technologies available today to harness the power of the facts posted in social media. Many interested stakeholders collect data relevant to their businesses and apply analytics techniques like regression, and machine learning algorithms to reap the insights hidden in the facts.
Few years earlier, collection and storage of data posed a big challenge. Thanks to the technologists, we live in the days where we need not worry about the aforementioned difficulties. But the big challenge lies in making sense of those data.
When we tried to find the factors impacting food price rise crisis, we are left in the midst of the haystack as the related tweets had lots of shorthand words and emoticons. Many researchers studied and applied various methods to get sense out of them. Parts of speech tags are also used to extract feature sets for applying necessary machine learning algorithms. The features thus extracted are high dimensional. Clustering or classification or applying any evolutionary computing techniques on such a high dimensional data presents poor accuracy and recall and further increases the time complexity.
To solve the problem of accuracy and time complexity, we have pruned association rules and eliminated the redundant ones. To find the facts impacting a particular crisis, we need to cluster and identify the common causes. The feature set is added with hyponyms, hypernymns, meronyms and many other related terms from Wordnet. Even though the addition of those terms increases the dimensionality, we clearly saw the improvement in accuracy of clustering. The expectations from a data analytics framework lie in accuracy rather than time complexity. Therefore a tradeoff exists between accuracy and time. Still many works are going on to improve the accuracy of the framework reducing manual interventions as far as possible.
By Dr.J. Akilandeswari- HOD/IT
