7 min read

Why do Big Data projects fail? Part I. The Business Perspective.

In a recent post on out Big Data blog, "Big Data for E-commerce", I wrote about how Big Data solutions are becoming indispensable in modern business because digital data has become an crucial part of it. Undoubtedly, data analysis and management is one of the main challenges faced by companies on the threshold of a new, more digital reality.

Still, many people are skeptical about introducing such solutions, due to the often mentioned uncertainty of their effectiveness. Many reports on Big Data solutions (such as Gartner’s) indicate that over 80% of projects fail in some way, with only 20% reaching their goals. 

While considering such information, it is not difficult to remain skeptical about the value promised by Big Data tools. However, it is enough to look at the reasons behind the incorrect implementation of projects, and it turns out that the construction of even complex Big Data solutions doesn’t have to end in a fiasco. You just need to be aware of the glitches (both on the business and technological side) that may appear during the preparation for the implementation of the project and its actual execution. Taking care of them, we can be sure that the chance of a problem arising during the construction and subsequent use of a Big Data solution will be much smaller.

Business reasons of Big Data projects fail.

Here we will try to identify the most serious business problems that can be struggled with when building Big Data solutions. Additionally, we will try to indicate the possibility of avoiding them. Of course, these will not be all the problems that can be encountered in this matter, but we will mention the most important ones from our point of view. So, let’s begin!

  1. Badly recognized problem - At the very beginning, we can make a critical mistake that will make even the best-designed solution fail to fulfill its task. Why? Because we misspelled its purpose! While it seems obvious to thoroughly prepare for the start of a project, it is not uncommon that both the goals and the problem have been misdefined. If the Big Data solution we designed does not provide data to help solve the problem, even the best infrastructure will not make the project successful. How to avoid this problem? The stakeholders should analyze their needs very meticulously, and then discuss it with the Project Team, Big Data tools, together defining the goal of the project. Thanks to an in-depth analysis, the project will be created with the right approach from the very beginning and will achieve required goals.
  2. Lack of engagement -  Building a Big Data infrastructure is a complicated process that requires the involvement of not one, but two parties. If, for some reason, stakeholders do not take an active part in the process of building a solution, it often turns out that it does not meet their expectations. The reasons for this lack of interest can vary, from the wrong setting of the cooperation process, through changing the company's vision, to personnel changes. In the case of the latter, a new person who has not been introduced to the project may have problems with finding himself and thus sideline him. For projects of this scale, it may be a factor in the success of the undertaking.
  3. Time - Time factor itself is not a problem, of course, but building comprehensive Big Data solutions is laborious. Expecting too quick results is the first problem, and the second is changing stakeholders' priorities. In an ever-changing market, even large enterprises are able to change their profile and this may result in the abandonment of the project, as it will be useless for the company. There is also the issue of changing people responsible for the project on the stakeholder side, mentioned in point 2. Personnel changes can be dangerous in this case. How can these problems be dealt with? There is no easy answer here, but you should think carefully about the architecture of Big Data infrastructure and be aware of the time it will take to implement and run it
  4. Cooperation and communication -  Lack of cooperation between stakeholders and the team building the Big Data infrastructure. The importance of such cooperation should be clearly emphasized - it can be a problem, due to which many projects that had a chance to function successfully, failed. This catastrophe can only be prevented by a clear and transparent process describing project collaboration and implementation, as well as optimization. Only then we will be sure that both parties work together to maximize the benefits that the Big Data project is about to bring.
  5. People - it's not the people who are the problem, but the efficient management of the team. Many companies with excellent specialists often have a problem with project management, which results in problems and delays. That is why it is so important for the people responsible for the project to understand the business issues. Thanks to experienced project managers, sensitive to the needs of the business, you can ensure that the project will be built properly and without delays.  The organization's approach to crucially important management.  For example, GetInData has implemented the Swedish model of teamwork, which you can read about in the recent post of our CEO, Adam Kawa.

Adam Kawa, CEO of GetInData speaks about team managment in Big Data Projects.
Team Managment in Big Data Projects

These five points we have listed above are the real Achilles' heel of Big Data projects when it comes to business issues. Identifying and eliminating them will help introduce functional Big Data solutions that will bring significant benefits. While that doesn't sound too hard, the truth is quite different. Without a very serious approach to the subject and preparation of both stakeholders and the solution provider, it is difficult to create and maintain the correct process of creating, implementing, and then optimizing the solution.

How to ensure Big Data project success?

Building a Big Data project is undoubtedly a complicated process, and requires considerable commitment from both parties. Nevertheless, if approached in the right way, it will bring the expected benefits. The key to success in the Big Data project is  communication at every level. Thanks to this communication and constant exchange of information, stakeholders will be able to follow the creation process and keep the creators informed about their goals and problems. This, in turn, will help infrastructure providers to stay focused on the effects desired by stakeholders and without having to make too many changes in the process of building and implementing the solution. It should also be remembered that the success of the project is not only about the design and its implementation. It is also an effort put into its optimization and maintenance. And using the delivered insights correctly. The implementation is actually the beginning of the road.

If you are curious about the technological reasons for the failure of the Big Data project, subscribe to the newsletter so as not to miss the next part of the post!

big data
Software Development
big data project
30 March 2021

Want more? Check our articles

getindata nifi ingestion universe made out flow files nifi architecture big data

NiFi Ingestion Blog Series. PART IV - Universe made out of flow files - NiFi architecture

Apache NiFi, a big data processing engine with graphical WebUI, was created to give non-programmers the ability to swiftly and codelessly create data…

Read more

5 main data-related trends to be covered at Big Data Tech Warsaw 2021. Part I.

A year is definitely a long enough time to see new trends or technologies that get more traction. The Big Data landscape changes increasingly fast…

Read more

ETL 2.0 Why you should switch into stream processing

If you are looking at Nifi to help you in your data ingestions pipeline, there might be an interesting alternative. Let’s assume we want to simply…

Read more
lean big data 1

Lean Big Data - How to avoid wasting money with Big Data technologies and get some ROI

During my 6-year Hadoop adventure, I had an opportunity to work with Big Data technologies at several companies ranging from fast-growing startups (e…

Read more
mamava getindata cloud google bigquery prostooleh
Success Stories

Success story: Breastfeeding supported with modern IoT and app features

Outstanding customer experience is usually backed by robust data analytics. Same applies to Mamava, a business that celebrates and supports…

Read more
1 06fVzfDygMpOGKTvnlXAJQ
Tech News

Panem et circenses — how does the Netflix’s recommendation system work.

Panem et circenses can be literally translated to “bread and circuses”. This phrase, first said by Juvenal, a once well-known Roman poet is simple but…

Read more

Contact us

Interested in our solutions?
Contact us!

Together, we will select the best Big Data solutions for your organization and build a project that will have a real impact on your organization.

By submitting this form, you agree to our  Terms & Conditions