Tutorial
Highly available Airflow cluster in Amazon AWS
These days, companies getting into Big Data are granted to compose their set of technologies from a huge variety of available solutions. Even though…
Read moreUse-cases/Project
2 min read
Running Spark on Amazon Web Services (AWS)
When you search thought the net looking for methods of running Apache Spark on AWS infrastructure you are most likely to be redirected to the documentation of AWS EMR (Elastic Map Reduce) service, which is Amazon's Hadoop distribution suited to run in AWS cloud environment. It's quite an easy way to deploy your data pipelines, but sometimes bootstrapping a huge cluster to perform simple ad-hoc analysis it's a cumbersome task. They say:
"to a man with a hammer everything looks like a nail" :)
and we felt into this trap with EMR once.
The article below describes two other ways of running Apache Spark jobs on AWS-managed infrastructure - AWS Glue and AWS Fargate - that we use on our clients' data warehousing projects. You will find there the key differences between these methods when it comes to flexibility and pricing, showing why there is no place for "one service fits all" approach in AWS world.
Check out!
18 December 2019
Want more? Check our articles
Big Data Event
A Review of the Presentations at the Big Data Technology Warsaw Summit 2022!
The 8th edition of the Big Data Tech Summit is already over, and we would like to thank all of the attendees for joining us this year. It was a real…
Read moreWhy do Big Data projects fail? Part I. The Business Perspective.
In a recent post on out Big Data blog, "Big Data for E-commerce", I wrote about how Big Data solutions are becoming indispensable in modern business…
Read moreTutorial
Apache NiFi - why do data engineers love it and hate it at the same time? Blog Series Introduction
Learning new technologies is like falling in love. At the beginning, you enjoy it totally and it is like wearing pink glasses that prevent you from…
Read moreTutorial
Deploying secure MLflow on AWS
One of the core features of an MLOps platform is the capability of tracking and recording experiments, which can then be shared and compared. It also…
Read moreTutorial
Data online generation for event stream processing
In a lot of business cases that we solve at Getindata when working with our clients, we need to analyze sessions: a series of related events of actors…
Read moreContact us
Interested in our solutions?
Contact us!
Together, we will select the best Big Data solutions for your organization and build a project that will have a real impact on your organization.