22-03-2023
How to build an Azure Analytics solution from scratch?
by Rodrigo Silva, Data Analytics Engineer @ Xpand IT
The importance of data in business, services and industries is no longer the subject of the future — it is already in the present. The technology market has been developing and investing in the area of Analytics so that its customers can make more informed and effective decisions. It is on the basis of the data, especially those with the highest quality of treatment, that sustained steps are taken towards the objectives set.
This work is carried out by data consultants and engineers, who have various solutions for customer needs. One of these is Azure Data Factory, created by Microsoft with the promise of revolutionising the future of analytics. Azure Data Factory allows integration, ingestion, preparation, and transformation of data at scale and without a server, it allows for easy creation of “code-free” ETL and ELT processes in a visual, intuitive, and especially cloud environment.
What is the purpose of Azure Data Factory?
Azure Data Factory is a cloud-based data integration service developed by Microsoft. This allows you to create and manage pipelines moving and transforming data from multiple sources (up to 90+ different data sources) to multiple destinations, either cloud or local storage.
The question among the data & analytics communities is whether Azure Data Factory is going to be a replacement for SSIS (Server Integration Services), which operates on a local scope. The answer is not entirely definitive. At this stage of development, one of the main advantages of Azure Data Factory is that there is no concern with the creation and development of data integration infrastructures — that is, there is a greater focus on configuring the cloud solution. Even for projects that still run on SSSIS packages, Microsoft already allows you to rehost them with Azure Data Factory so everything we need can be migrated to the cloud.
Is Azure Data Factory a service that works on Big Data? Yes, because it has support for the vast majority of data types and integrates them, regardless of their origin. Data can be contextualised across the various applications in the Azure environment — and that’s where the power of this service comes in in the context of Analytics.
Azure Data Factory is a platform…
- Low code? More complex solutions will require knowledge of coding and/or programming
- For all skill levels? As mentioned in the previous point, it always depends on the level of knowledge that each one has. You should have coding knowledge because challenging projects can appear at any moment
- For all types of integration? Yes! From our team experience, we have not yet found a project/solution where something cannot be integrated
Conclusion
Microsoft’s big focus on working with data from various sources is with the aim of making integration impossible and, in turn, creating smarter businesses. The existence of these solutions is designed for industries to make better decisions, supported by data.
Developing and integrating data with Azure Data Factory allows you to work:
- Pipelines: logical sets/groupings that aim to work the data in the way needed for analysis.
- Connection to Datasets (reference to the data structure – which can be a file, folder or table) and Linked Services (technology layer where the data is located)
When compared to SSIS, Azure Data Factory makes a developer more comfortable with the development environment – it is prepared for project/client changes as there is no need to reinstall drivers, which prevents malfunctions. e is a learning curve in learning to work in an Azure environment – Microsoft has a lot of material available to be studied, has several certifications to be obtained, encouraging research and obtaining skills.
From a business and commercial point of view, the development and deployment time to market is more interesting for customers because it optimises costs, there are possibilities of expansion and elasticity of service.
Leave a comment
Comments are closed.