Tutorial
6 min read

Why do Big Data projects fail: Part. 2. The Technological Issues.

In the previous post on our Big Data Blog, we discussed the business reasons behind the failures of Big Data projects. We've listed five major mistakes that you should avoid in order to make sure that the complicated project of implementing Big Data tools and solutions turns out to be a success story. If you want to read about business issues that may arise during the implementation of the project, go to the post "Why do Big Data Project fail: Business Issues".   In today's post, we will look at the technological challenges that await the Teams involved in Big Data projects.

Recall that over 80% of Big Data projects are not implemented and end in failure. What's worse, even those that are implemented rarely bring the expected benefits (only 20% of them meet the expectations of stakeholders). Admittedly, it can discourage attempts to implement Big Data solutions. However, this is not a good decision - in the new reality in which we operate, the amount of generated data will only increase. 

Fear of making mistakes cannot stop the development that often depends on the existence on the market because of the inability to properly analyze and manage them is a critical error that may affect the functioning of the enterprise. 

You can read about why it is worth making bold decisions related to Big data projects in the post of our CEO, Adam Kawa:** The benefits of the Swedish approach to Big Data.**

Many of the problems encountered during implementation are of a business nature, as we wrote in the previous post, but the challenges faced by the initiators and beneficiaries of projects may be of a different nature. It should be remembered that Big Data solutions belong to complex IT projects, often requiring not only excellent specialists but also a refined technological process that will ensure success. The technological side of Big Data products can also cause significant problems and they can sink a project before it can be put into practice. 

Technical reasons for Big Data projects fail.

Below, we will try to present frequent mistakes that the technology team can make and which may affect the project. These are guidelines on what to pay attention to when discussing, planning, and then implementing each Big Data project that is to be successful.

  1. Labyrinth of experiments - Yes - although you should try to choose the best solution for a specific problem. However, it may turn out that the technology team will spend too much time experimenting, approaching the problem in a less business and more academic way. This will result in a too long time spent looking for solutions and getting stuck at the level of preparing the solution. Each experiment should have defined clear goals, measures and success criteria, so we can assess results and learn from the outcomes.
  2. Lack of technological knowledge and skills - Big Data solutions and tools, although more and more often used, require specialist knowledge and skills. Thanks to them, you can be sure that the project we are implementing won’t be wasted. This problem affects both the team from the supplier's side and internal specialists who may not have the skills to properly handle the solution. That is why cooperation between teams and training is so important, allowing the developed Big Data solution to bring the expected results.
  3. Inexperienced company - The problem of the experience of a company that builds Big Data solutions. We have already talked about the experience of managing a team in an earlier post, but the technological experience is just as important. Organizations with experience in managing Big Data projects are certainly able to share Case Studies and Success Stories of their previous projects, they are also present at technological conferences. This can make us sure that the company has the experience needed to implement our project successfully.
  4. Too rigid approach - The lack of a flexible approach to the project may result in its failure. Many technologies have their limitations and you should remember not to get stuck on one solution when other tools will be able to deal with the problem much faster. The lack of this flexibility may, at best, extend the implementation of the project, at worst… it may lead to its failure. Remember there is no best tool for everything, but you can always find the best tool for your particular scenario at a certain moment in time. The whole technological landscape changes very rapidly so you should be prepared for constantly upgrading and replacing technologies.
  5. Losing focus on the business case - building a data platform is a chance to work with exciting new technologies and can be really thrilling. However, the platform should be built on purpose, to support some business cases, not for the sake of playing with new tools. 

How to ensure Big Data project success by overcoming technological issues.

So how to avoid technological problems related to Big Data projects? Let's start with the obvious - you should find a good supplier to work with! It is good to check whether the providers of Big Data solutions have previously worked on projects similar to the one we want to implement, it is also worth reviewing the Case Studies and White Papers they present. Thanks to this, we will be able to make sure that they are specialists in the technologies in which they work. After selecting the partner specialised in  Big Data solutions we need, make sure that both parties know what the project aims to achieve. Thanks to this, it will be possible to determine the technological stack that will be used during the construction of the project.

Secondly, make sure that our own team has the technological competence to cooperate and then handle the project. If there are none, it should either be extended or trained prior to implementation so that technology-related problems do not get in the way of the goal. It is a good idea to arrange a relationship between our team and the supplier's team to make sure they are compatible. It is also crucial to have constant support from business stakeholders, it might be even worth to get them involved in the project Team.

At the same time, you should take care of a holistic view of the project and flexibility of solutions. Identify problems in real time and implement new solutions as needed. Thanks to this, the work will run more efficiently.

To the Big Data project success.

Is there one foolproof way to be successful in implementing a Big Data project? I do not think so. Certainly, however, by avoiding the mistakes that are described in this post and those mentioned in "Why do Big Data Project fail: Business Issues", it will give us more possibilities and will allow us to anticipate impending problems. Without a doubt, however, the most important issue is communication. It is thanks to the mutual understanding of the possibilities and expectations that we will achieve success.

big data
analytics
technology
bigdatatech
getindata
open source
big data project
big data experts
19 May 2021

Want more? Check our articles

5apacheobszar roboczy 1 4
Tutorial

Real-time ingestion to Iceberg with Kafka Connect - Apache Iceberg Sink

What is Apache Iceberg? Apache Iceberg is an open table format for huge analytics datasets which can be used with commonly-used big data processing…

Read more
getindata blog big data knowledge sharing it jobs

How do we apply knowledge sharing in our teams? GetInData’s internal initiatives

Knowledge sharing is one of our main missions. We regularly speak at international conferences, we contribute to open-source technologies, organize…

Read more
getindata adam goscicki terraform cloud infrastructure notext
Tutorial

Terraform your Cloud Infrastructure

So, you have an existing infrastructure in the cloud and want to wrap it up as code in a new, shiny IaC style? Splendid! Oh… it’s spanning through two…

Read more
why big data projects fail blog

Why do Big Data projects fail? Part I. The Business Perspective.

In a recent post on out Big Data blog, "Big Data for E-commerce", I wrote about how Big Data solutions are becoming indispensable in modern business…

Read more
getindata grafana loki monitoring
Use-cases/Project

Why are log analytics so important in a monitoring system?

A monitoring system is a necessary component of any data platform. We can find a lot of different services that use different approaches to the same…

Read more
getindata cover nifi ingestion kafka poc notext
Tutorial

NiFi Ingestion Blog Series. PART V - It’s fast and easy, what could possibly go wrong - one year history of certain nifi flow

Apache NiFi, a big data processing engine with graphical WebUI, was created to give non-programmers the ability to swiftly and codelessly create data…

Read more

Contact us

Interested in our solutions?
Contact us!

Together, we will select the best Big Data solutions for your organization and build a project that will have a real impact on your organization.

The administrator of your personal data is GetInData Sp. z o.o. Sp.k with its registered seat in Warsaw (02-508), 39/20 Pulawska St. Your data is processed for the purpose of provision of electronic services in accordance with the  Terms & Conditions. For more information on personal data processing and your rights please see  Privacy Policy.

By submitting this form, you agree to our Terms & Conditions and Privacy Policy