Why do Big Data projects fail? Part I. The Business Perspective.

In a recent post on out Big Data blog, "Big Data for E-commerce", I wrote about how Big Data solutions are becoming indispensable in modern business because digital data has become an crucial part of it. Undoubtedly, data analysis and management is one of the main challenges faced by companies on the threshold of a new, more digital reality.

Still, many people are skeptical about introducing such solutions, due to the often mentioned uncertainty of their effectiveness. Many reports on Big Data solutions (such as Gartner’s) indicate that over 80% of projects fail in some way, with only 20% reaching their goals. 

While considering such information, it is not difficult to remain skeptical about the value promised by Big Data tools. However, it is enough to look at the reasons behind the incorrect implementation of projects, and it turns out that the construction of even complex Big Data solutions doesn’t have to end in a fiasco. You just need to be aware of the glitches (both on the business and technological side) that may appear during the preparation for the implementation of the project and its actual execution. Taking care of them, we can be sure that the chance of a problem arising during the construction and subsequent use of a Big Data solution will be much smaller.

Business reasons of Big Data projects fail.

Here we will try to identify the most serious business problems that can be struggled with when building Big Data solutions. Additionally, we will try to indicate the possibility of avoiding them. Of course, these will not be all the problems that can be encountered in this matter, but we will mention the most important ones from our point of view. So, let’s begin!

1. Badly recognized problem - At the very beginning, we can make a critical mistake that will make even the best-designed solution fail to fulfill its task. Why? Because we misspelled its purpose! While it seems obvious to thoroughly prepare for the start of a project, it is not uncommon that both the goals and the problem have been misdefined. If the Big Data solution we designed does not provide data to help solve the problem, even the best infrastructure will not make the project successful. How to avoid this problem? The stakeholders should analyze their needs very meticulously, and then discuss it with the Project Team, Big Data tools, together defining the goal of the project. Thanks to an in-depth analysis, the project will be created with the right approach from the very beginning and will achieve required goals.
2. Lack of engagement -  Building a Big Data infrastructure is a complicated process that requires the involvement of not one, but two parties. If, for some reason, stakeholders do not take an active part in the process of building a solution, it often turns out that it does not meet their expectations. The reasons for this lack of interest can vary, from the wrong setting of the cooperation process, through changing the company's vision, to personnel changes. In the case of the latter, a new person who has not been introduced to the project may have problems with finding himself and thus sideline him. For projects of this scale, it may be a factor in the success of the undertaking.
3. Time - Time factor itself is not a problem, of course, but building comprehensive Big Data solutions is laborious. Expecting too quick results is the first problem, and the second is changing stakeholders' priorities. In an ever-changing market, even large enterprises are able to change their profile and this may result in the abandonment of the project, as it will be useless for the company. There is also the issue of changing people responsible for the project on the stakeholder side, mentioned in point 2. Personnel changes can be dangerous in this case. How can these problems be dealt with? There is no easy answer here, but you should think carefully about the architecture of Big Data infrastructure and be aware of the time it will take to implement and run it
4. Cooperation and communication -  Lack of cooperation between stakeholders and the team building the Big Data infrastructure. The importance of such cooperation should be clearly emphasized - it can be a problem, due to which many projects that had a chance to function successfully, failed. This catastrophe can only be prevented by a clear and transparent process describing project collaboration and implementation, as well as optimization. Only then we will be sure that both parties work together to maximize the benefits that the Big Data project is about to bring.
5. People - it's not the people who are the problem, but the efficient management of the team. Many companies with excellent specialists often have a problem with project management, which results in problems and delays. That is why it is so important for the people responsible for the project to understand the business issues. Thanks to experienced project managers, sensitive to the needs of the business, you can ensure that the project will be built properly and without delays.  The organization's approach to crucially important management.  For example, GetInData has implemented the Swedish model of teamwork, which you can read about in the recent post of our CEO, Adam Kawa.

These five points we have listed above are the real Achilles' heel of Big Data projects when it comes to business issues. Identifying and eliminating them will help introduce functional Big Data solutions that will bring significant benefits. While that doesn't sound too hard, the truth is quite different. Without a very serious approach to the subject and preparation of both stakeholders and the solution provider, it is difficult to create and maintain the correct process of creating, implementing, and then optimizing the solution.

How to ensure Big Data project success?

Building a Big Data project is undoubtedly a complicated process, and requires considerable commitment from both parties. Nevertheless, if approached in the right way, it will bring the expected benefits. The key to success in the Big Data project is  communication at every level. Thanks to this communication and constant exchange of information, stakeholders will be able to follow the creation process and keep the creators informed about their goals and problems. This, in turn, will help infrastructure providers to stay focused on the effects desired by stakeholders and without having to make too many changes in the process of building and implementing the solution. It should also be remembered that the success of the project is not only about the design and its implementation. It is also an effort put into its optimization and maintenance. And using the delivered insights correctly. The implementation is actually the beginning of the road.

If you are curious about the technological reasons for the failure of the Big Data project, subscribe to the newsletter so as not to miss the next part of the post!