7 min read

Why do Big Data projects fail? Part I. The Business Perspective.

In a recent post on out Big Data blog, "Big Data for E-commerce", I wrote about how Big Data solutions are becoming indispensable in modern business because digital data has become an crucial part of it. Undoubtedly, data analysis and management is one of the main challenges faced by companies on the threshold of a new, more digital reality.

Still, many people are skeptical about introducing such solutions, due to the often mentioned uncertainty of their effectiveness. Many reports on Big Data solutions (such as Gartner’s) indicate that over 80% of projects fail in some way, with only 20% reaching their goals. 

While considering such information, it is not difficult to remain skeptical about the value promised by Big Data tools. However, it is enough to look at the reasons behind the incorrect implementation of projects, and it turns out that the construction of even complex Big Data solutions doesn’t have to end in a fiasco. You just need to be aware of the glitches (both on the business and technological side) that may appear during the preparation for the implementation of the project and its actual execution. Taking care of them, we can be sure that the chance of a problem arising during the construction and subsequent use of a Big Data solution will be much smaller.

Business reasons of Big Data projects fail.

Here we will try to identify the most serious business problems that can be struggled with when building Big Data solutions. Additionally, we will try to indicate the possibility of avoiding them. Of course, these will not be all the problems that can be encountered in this matter, but we will mention the most important ones from our point of view. So, let’s begin!

  1. Badly recognized problem - At the very beginning, we can make a critical mistake that will make even the best-designed solution fail to fulfill its task. Why? Because we misspelled its purpose! While it seems obvious to thoroughly prepare for the start of a project, it is not uncommon that both the goals and the problem have been misdefined. If the Big Data solution we designed does not provide data to help solve the problem, even the best infrastructure will not make the project successful. How to avoid this problem? The stakeholders should analyze their needs very meticulously, and then discuss it with the Project Team, Big Data tools, together defining the goal of the project. Thanks to an in-depth analysis, the project will be created with the right approach from the very beginning and will achieve required goals.
  2. Lack of engagement -  Building a Big Data infrastructure is a complicated process that requires the involvement of not one, but two parties. If, for some reason, stakeholders do not take an active part in the process of building a solution, it often turns out that it does not meet their expectations. The reasons for this lack of interest can vary, from the wrong setting of the cooperation process, through changing the company's vision, to personnel changes. In the case of the latter, a new person who has not been introduced to the project may have problems with finding himself and thus sideline him. For projects of this scale, it may be a factor in the success of the undertaking.
  3. Time - Time factor itself is not a problem, of course, but building comprehensive Big Data solutions is laborious. Expecting too quick results is the first problem, and the second is changing stakeholders' priorities. In an ever-changing market, even large enterprises are able to change their profile and this may result in the abandonment of the project, as it will be useless for the company. There is also the issue of changing people responsible for the project on the stakeholder side, mentioned in point 2. Personnel changes can be dangerous in this case. How can these problems be dealt with? There is no easy answer here, but you should think carefully about the architecture of Big Data infrastructure and be aware of the time it will take to implement and run it
  4. Cooperation and communication -  Lack of cooperation between stakeholders and the team building the Big Data infrastructure. The importance of such cooperation should be clearly emphasized - it can be a problem, due to which many projects that had a chance to function successfully, failed. This catastrophe can only be prevented by a clear and transparent process describing project collaboration and implementation, as well as optimization. Only then we will be sure that both parties work together to maximize the benefits that the Big Data project is about to bring.
  5. People - it's not the people who are the problem, but the efficient management of the team. Many companies with excellent specialists often have a problem with project management, which results in problems and delays. That is why it is so important for the people responsible for the project to understand the business issues. Thanks to experienced project managers, sensitive to the needs of the business, you can ensure that the project will be built properly and without delays.  The organization's approach to crucially important management.  For example, GetInData has implemented the Swedish model of teamwork, which you can read about in the recent post of our CEO, Adam Kawa.

Adam Kawa, CEO of GetInData speaks about team managment in Big Data Projects.
Team Managment in Big Data Projects

These five points we have listed above are the real Achilles' heel of Big Data projects when it comes to business issues. Identifying and eliminating them will help introduce functional Big Data solutions that will bring significant benefits. While that doesn't sound too hard, the truth is quite different. Without a very serious approach to the subject and preparation of both stakeholders and the solution provider, it is difficult to create and maintain the correct process of creating, implementing, and then optimizing the solution.

How to ensure Big Data project success?

Building a Big Data project is undoubtedly a complicated process, and requires considerable commitment from both parties. Nevertheless, if approached in the right way, it will bring the expected benefits. The key to success in the Big Data project is  communication at every level. Thanks to this communication and constant exchange of information, stakeholders will be able to follow the creation process and keep the creators informed about their goals and problems. This, in turn, will help infrastructure providers to stay focused on the effects desired by stakeholders and without having to make too many changes in the process of building and implementing the solution. It should also be remembered that the success of the project is not only about the design and its implementation. It is also an effort put into its optimization and maintenance. And using the delivered insights correctly. The implementation is actually the beginning of the road.

If you are curious about the technological reasons for the failure of the Big Data project, subscribe to the newsletter so as not to miss the next part of the post!

big data
Software Development
big data project
30 March 2021

Want more? Check our articles

getindata amundsen feast machine learining notext

Machine Learning Features discovery with Feast and Amundsen

One of the main challenges of today's Machine Learning initiatives is the need for a centralized store of high-quality data that can be reused by Data…

Read more
blogobszar roboczy 1 4

Power of Big Data: MLOps for business.

Welcome to the next instalment of the “Power of Big Data” series. The entire series aims to make readers aware of how much Big Data is needed and how…

Read more
podcast swedbank mlops cloud getindata
Radio DaTa Podcast

MLOps in the Cloud at Swedbank - Enterprise Analytics Platform

In this episode of the RadioData Podcast, Adama Kawa talks with Varun Bhatnagar from Swedbank. Mentioned topics include: Enterprise Analytics Platform…

Read more
power of big data ii obszar roboczy 1 3x 100

Power of Big Data: Healthcare

Welcome to another Power of Big Data series post. In the series, we present the possibilities offered by solutions related to the management, analysis…

Read more
logs analytics in cloud loki albert lewandowski getindata big data blog notext

Logs analytics at scale in the cloud with Loki

Logs can provide a lot of useful information about the environment and status of the application and should be part of our monitoring stack. We'll…

Read more
getindator green santa watching a dashboard on laptop with real 9bc272ff 58b5 400a a10d 1b1639be8b3e

Nailing e-commerce: Black Friday's data in near real-time analytics with Snowflake Dynamic Tables & Snowflake Alerts

Black Friday or Black Week and the pre-Christmas period is the prime time of year for the e-commerce and retail industry. This means masses of…

Read more

Contact us

Interested in our solutions?
Contact us!

Together, we will select the best Big Data solutions for your organization and build a project that will have a real impact on your organization.

What did you find most impressive about GetInData?

They did a very good job in finding people that fitted in Acast both technically as well as culturally.
Type the form or send a e-mail: hello@getindata.com
The administrator of your personal data is GetInData Poland Sp. z o.o. with its registered seat in Warsaw (02-508), 39/20 Pulawska St. Your data is processed for the purpose of provision of electronic services in accordance with the Terms & Conditions. For more information on personal data processing and your rights please see Privacy Policy.

By submitting this form, you agree to our Terms & Conditions and Privacy Policy