GetInData in 2021 - let’s celebrate our achievements in the Big Data world!

The year 2021 passed in the blink of an eye and the time has come to summarize our goals at GetinData and define our challenges for the next year. Today, we would like to share our achievements with you in the fields of knowledge sharing and the Big Data community. Here's a quick summary:

• Throughout 2021, we published a lot of blogs about our solutions, Big Data technologies and Big Data events
• We shared interesting facts and news from the world of Big Data on our social media channels - follow our profiles and stay up to date!
• We took part in one of the biggest events in Poland’s IT industry - the Google Cloud Region launch
• Our Big Data Experts had the pleasure of performing in many Big Data events around the world
• We took part in GITEX Global - one of the biggest tech events in the field of #bigdata #ai #cloud #cybersecurity and more
• We started a new project dedicated to our Polish fans - Big Data Club. This  group focuses on Big Data solutions for business. It is primarily intended for people operating in the interface between business and technology.
• We were recognised for our expertise in Cloud Analytics, Stream Data Analytics, Enterprise BI solutions and remote monitoring using Google Cloud
•  This year we have exceeded 100+ people being on board and we are not slowing down! (check our job offers)
• Organizing internal Lunch & Learn sessions, which is already a tradition of ours 
• Our experts passed a countless number of assessments in the area of ​​Google Cloud, and the number is constantly growing
• We launched GetInData Labs, whose mission is to research and produce innovative solutions that develop our business and people (soon we will publish more information about this)
• We contributed to many open-source projects such as Marquez, Flink HTTP Connector, Amundsen, Kedro-kubeflow plugin

Below you can find a list of our publications, webinars and conference talks so…  sit down comfortably and dive into the topics that interest you the most! 

Our Top 5 blog posts

In 2021 we published 39 blog posts in the field of Big Data. You can check out the top 5 most viewed blogs (based on data from the 5th of January) below:

• Apache NiFi and Apache NiFi Registry on Kubernetes by Albert Lewandowski 
• Google Data Studio on BigQuery - usage and cost control by Emil Ruchała
• OAuth2-based authentication on Istio-powered Kubernetes clusters by Mariusz Strzelecki 
• Flink on Kubernetes - how and why? by Albert Lewandowski 
• Running Machine Learning Pipelines with Kedro, Kubeflow and Airflow by Mariusz Strzelecki

That's not all, of course!  We also published lots  of blog posts about some interesting technology:  

• Introduction to Apache Sedona (incubating) by Paweł Kociński
• Different generations of CICD tools by Albert Lewandowski 
• Apache Spark with Apache Iceberg - a way to boost your data pipeline performance and safety by Paweł Kociński 
• Logs analytics at scale in the cloud with Loki byAlbert Lewandowski
• Up & Running: data pipeline with BigQuery and dbt by Jakub Jurczak
• Terraform your Cloud Infrastructure by Adam Gościcki 
• Amazon DynamoDB - single table design by Marek Jędraszewski 
• NiFi Scripted Components - the missing link between scripts and fully custom stuff by Tomasz Nazarewicz 

More blog posts correlated to MLOps and our Machine Learning Platform such us:

• Kubeflow Pipelines up and running in 5 minutes by Marcin Zabłocki 
• Online ML Model serving using MLeap by Bartosz Chodnicki 

A lot of blog posts correlated to stream-processing and our Complex Event Processing Platform: 

• ETL 2.0 Why you should switch into stream processing by Maciej Maciejko
• Alert backoff with Flink CEP byPaweł Leszczyński
• Data Enrichment in Flink SQL using HTTP Connector For Flink - Part One by Krzysztof Chmielewski
• Change Data Capture by JDBC with FlinkSQL by Jakub Jurczak
• Success Story: Fintech data platform gets a boost from stream processing by Mateusz Pytel 

We also published a series of blog posts dedicated to businesses such as: 

• 5 questions you need to answer before starting a big data project by Grzegorz Kołpuć 
• Why do Big Data Projects Fail?** Part I Part II, Big Data in e-commerce, Power of Big Data: Marketing, Power of Big Data Sales, Power of Big Data: Industry, Power of Big Data: Healthcare** by Michał Talaśka
• Cloud computing standards for the insurance industry by Włodzimierz Marat 
• Artificial Intelligence regulatory initiatives of EU countries by Włodzimierz Marat

and last but not least some blog posts about our people and company:

• Data Analyst - how to start your career with a non-technical background by Piotr Menclewicz
• How do we apply knowledge sharing in our teams? GetInData’s internal initiatives by Klaudia Wachnio

Whitepapers and our first E-book

Together with our Big Data expert Paweł Leszczyński, we prepared the whitepaper “Stream Processing Explained”. There you can read about the characteristics of streaming, challenges and the open-source streaming playground such as Apache Flink, Kafka Connect and Spark Streaming. We encourage you to download it here! 

Did you have the chance to check out “Apache NiFi: A Complete Guide E-book” created by our specialists: Albert Lewandowski, Paweł Leszczyński and Tomasz Nazarewicz? If not, you definitely should.
What will you find there? You'll learn about NiFi architecture, NiFi flow, the management & operations of NiFi and NiFi Registry and some recommendations for using Apache NiFi. If you are interested in NiFi, check it out!

Our Big Data Experts at the conference

Last year we organized the 7th edition of the Big Data Technology Warsaw Summit, the first time this was done in a fully online version! In our blog, you can read a review of presentations and of course, we encourage you to join us during this year's edition on the 26-28th April 2022. 

During the first half of the year, we had the pleasure of taking part and performing  in many interesting Big Data Events:

• Albert Lewandowski talked about Kubernetes and real-time analytics at Warszawskie Dni Informatyki 

• At the launch of Google Cloud in Warsaw, our CTOs Krzysztof Zarzycki and Rafał Małanij talked about our MLOps Platform in the cloud 

• In May our team members Marek Wiewiórka and Krzysztof Zarzycki took part in the NDSML Summit. They had the pleasure of showing how we combine the cloud and open-source to boost data scientist work

• At the Big Data Conference Bigger Opportunity, organized by SGH, Piotr Menclewicz discussed how he started his career as a Data Analyst without any technical background 

• At Buzzwords Berlin, Albert Lewandowski talked about Kubernetes and real-time analytics and how we can connect these two worlds with Apache Flink

• In June we took part in the Data Science Summit ML Edition 2021:

  • Mariusz Strzelecki gave a presentation about Model Serving Made Easy Using Kedro pipelines 
  • Albert Lewandowski talked about predicting startup market trends based on the news
  • Piotr Menclewicz presented How We Built an Analytics Platform From Scratch While Developing Production Solutions as well.  

During the second half of the year at the ClubCloud Conference, Data Science Summit and during Flink Forward Global 2021, our team members Maciej Bryński and Rafał Małanij had the pleasure of presenting   Networks! Project - Real-Time Analytics That Control 50% of Mobile Networks in Poland (slides). 

Our Big Data DevOps Expert Albert Lewandowski also performed at:

• The Big Data Conference Europe, where he talked about best practices for ETL with Apache NiFi and Kubernetes
• The Open Source Summit + Embedded Linux Conference 2021 where he presented an Open-source Data Platform in the Cloud
• DataArt IT Nonstop, where he gave a presentation on creating Real-Time Data Streaming Powered by SQL on Kubernetes

Big Data Webinars and Meetups

In 2021 we organized a few webinars:

• Q&A Big Data Google Cloud Platform - ask experts webinar in which our Cloud experts: Mateusz Pytel, Tomasz Żukowski, Arkadiusz Gąsiorand Piotr Pilis answered some non-standard questions that we accumulated during the implementation of cloud projects.
• Why do you need to fix your Data Science processes? MLOps platform in details where Rafał Małanijexplained how an MLOps platform should be designed.
• Understanding Big Data Analytics - solutions for growing hosted by Rafał Małanij.
• MLOps Implemented - How we combine the cloud and open-source to boost data scientist work. During this webinar, Krzysztof Zarzycki and Marek Wiewiórka shared our experience in building MLOps platforms that we gained during recent implementation projects. You can watch this video at any time, just sign up here.

Our CTO Krzysztof Zarzycki also had the pleasure of hosting a webinar organized by Polish Executives Switzerland and Polish Professionals in Switzerland. During the meeting, we talked about the latest big data trends, how data processing works nowadays and how companies can benefit from data analytics (check slides). 

That’s not all! We also organized a Lighting Talks in a Warsaw Data Tech Talks meeting, where our team talked about: 

• Data Discovery with Apache Atlas and Amundsen - Improving the Productivity of Users Interacting with Data, hosted by Dominik Choma and Mariusz Górski (ING)
• Open-source vs Cloud-Managed - Data Engineers Dilemmas in the Cloud, hosted by Marcin Kaceperek
• Real-time Monitoring that Controls 50% of Mobile Networks in Poland, hosted by Maciej Bryński and Rafał Małanij
• Kedro, Data Scientist’s Swiss-Army Knife, hosted by Mariusz Strzelecki 

Our experts Dominik Choma and Mariusz Górski (ING) performed at the Amundsen Community meeting about Pandas profiling with Amundsen and OpenLineage. You can watch the clip here. At another meeting of the Amundsen Community, Mariusz Strzelecki talked about Feast + Amundsen Integration. Feel free to watch the video.

Knowledge Sharing and Big Data for Business

In 2021 we made some changes to our website and published two new subpages.

• Knowledge Base - this is our library of Big Data Knowledge. This page gives you full access to our whitepapers, recordings of conferences and webinars and the most interesting blog posts or contributions that we have made to open-source software.
• Big Data for Business - if you would like to know more about the process of cooperation with our company, the model that allows us to successfully deliver even complex Big Data projects for our clients or how Big Data can benefit your business, this is the place dedicated for you!

Plans for 2022?

We have a lot of new ideas that we want to bring to life in 2021. We also have plans for lots more knowledge sharing in our profile, so if you are interested in MLOps, Stream-processing, DevOps, DataOps, Cloud, Data Science or Big Data at all, please sign up to the newsletter and follow us on Linkedin, Facebook and Twitter, and subscribe to our channel on Youtube.