Big Data Overview

Big Data is, nowadays, on the radar of many non-IT people, as it has the possibility to be considerably disruptive from a Business model perspective; a few enlightened organisations have even made Big Data (or at least the outcomes) a core element of their Business Strategy.

The challenges faced in Big Data are many and varied, but perhaps the most significant is the understanding of what it means to the individual organisation. There is some need to have an unconstrained vision of what a future could look like without the classic shackles that technologies have imposed on the thinking of the organisation. That means not constraining thought by what was previously considered the art of the possible. In doing this there is an opportunity to focus on what it means to them as opposed to what the industry is saying and therefore what pre-conceived ideas individuals might have.

In considering the implementation of a Big Data initiative, the number one consideration has to be how that initiative fits in with the existing business and IT estate. From a business perspective, there’s no point in generating new insights that cannot be analysed side-by-side with the core information that has and still does make the business tick; from an IT perspective there needs to be coherence in implementation such that a Big Data solution can sit comfortably with the existing IT estate.

Architectural discipline plays a significant part to the key aspects of Big Data. Techniques like Business Capability Planning provides a structure to allow the business to express the capabilities the business requires, and in this context will include the outcomes that are implicitly or explicitly envisaged from Big Data. Information Architecture is important too – the information dimension is where the new opportunity really can materialise, and one of the areas where consistency adds long-term value. Finally, technology architecture is important for an implementation that realises the vision. In implementation of a Big Data solution it is imperative to initially plan a proof-of-concept; whilst many so-called ‘proof-of-concept’ phases prove very little that isn’t already known, the problem the Big Data solution is addressing might never have been solved in the way the organisation is attempting to solve it. A proof-of-concept should also consider in tandem the needs for Big Data analytics to sit alongside other related emerging areas – both informational and general. In the informational space, there are other trends around concepts such as genuine real-time operational intelligence, data virtualisation and in-memory technologies; in the more general space Big Data can relate quite significantly to Cloud Computing, Internet of Things and Social Media.

Big Data should be seen as a journey to go on not an end in itself, because the technology that enables this paradigm is still advancing substantially, as is the opportunity to combine it with technologies such as Complex Event Processing and Business Rules Engines to create even more useful outcomes sometimes never thought possible and other times never even considered.