In this post we will see what a data architect is and what he does , but first, we must know what big data architecture is , its main characteristics and types.
What is Big Data Architecture?
Big data architecture is a process that involves analyzing unconventional analysis methods for high volumes of data.
In order to be able to evaluate this amount of information, work bc data india schemes and information structures are designed in a personalized way. In this way, we manage to understand the processes related to data storage, management and processing.
Characteristics of Big Data architecture
The characteristics that form it are the 5 that we present below:
Scalability: Large and constantly growing processing and storage capacity.
Distribution and processing of data on different machines (not to be confused with data science, you can do data science but not big data).
Data locality: proximity to the data to achieve speed.
Fault tolerance.
Division into 3 layers: one for analysis, another for management and the last one for storing and processing data.
What is a data architect and what do they do?
Types of Big Data architecture
Now that the characteristics have been explained, let's look at the two most well-known types of architecture in the data sector.
Lambda Architecture
It is a generic, scalable and fault-tolerant data processing architecture. It was created in 2012 and is divided into three layers:
Batch Layer: raw data is managed. If you have worked with Google Analytics, you will be familiar with the “Raw Data View” where there is no segmentation or filtering of the data.
Serving layer: indexes data from the previous layer so that it can be processed for a specific purpose. This is a slow process.
Speed layer: works only with new data.
Although it is true that the Lambda architecture keeps the input data unchanged and is very suitable for processing it, it is not fast enough and maintaining the layers can have high costs.
Kappa Architecture
It was created two years later to speed up the process of Lambda architecture. It does so by eliminating the Batch layer and performing all processing in a new layer called Real-time layer, which offers support for real-time processing. It is a more versatile simplification for a faster pace of data processing.
The main problem is that data systems must support large volumes of data and storage space must be larger .
It is most suitable when the analysis and processing is going to be the same in the batch layer and in the speed layer.
What type of data architecture is best?
So, which of the two is better? It depends on the type of business and the amount of data to be processed.
If the objective is to develop and operate systems on a single information processing flow and still obtain the best results from it, it is better to use the Kappa architecture, since it does not have the batch layer.
On the other hand, if there is a strong dependence on latency, that is, on the response time of information processing , it is better to use the Lambda architecture.
Now that we know a little more about data architecture, let's define the professional who designs it, explaining what their functions and skills are.
