Optimizing Android for Scale: Storage Strategies for Modern Mobile Ecosystems

Posted on by Editorial Team 

Contributed by Parth Menon, Software Engineer

Many of us in today’s age are familiar with the term Android. The latter has been among the most adopted mobile technologies in the world, powering billions of devices across the globe. As it scales, the need for mobile storage management efficiently has never been more important. Applications are becoming increasingly complex and store large media files, intricate data sets, and an increasing number of assets. Consequently, the performance and user experience of these apps have become vital challenges to address. What’s more, modern applications are no longer built by a single team. In fact, some of the world’s largest apps, like Facebook, Instagram, Deliveroo, and Google, are developed by multiple teams and organizations spread across different countries, time zones, and continents. This vast, global collaboration adds further layers of complexity to both app development and storage management. This article will delve into storage strategies that support scalability, enhance user experience, and optimize app performance while navigating the challenges of such widespread teamwork. 

The Increasingly Important World of Efficient Storage in Mobile Ecosystems

Starting with mobile storage, it is the backbone of performance in Android devices, from app load times to user interactions with content. Unlike desktops or laptops,where storage is scalable and users can easily upgrade their storage capacity, mobile devices are limited by the storage they come with. This means that once you buy a mobile device, you’re stuck with its storage capacity, making it even more important to optimize how an app manages its data. Additionally,  users interact with mobile devices at a faster pace, frequently switching between apps, which demands that apps load quickly and respond instantly. Basically a well-optimized storage system ensures that apps run efficiently while still offering rich user experiences.

Why It Matters:

User Expectations: First reason is that users on mobile expect the app to be quick and responsive. When applications consume a lot of storage or take longer to load due to poor data management, this results in user frustration. As a matter of fact, a recent report from UXCam indicates that 90% of users have stopped using an app due to poor performance, and 88% will abandon an app if it consistently experiences glitches or technical bugs. Additionally, 21% of mobile apps have been used only once, underscoring the necessity for apps to deliver immediate value and seamless functionality to engage users effectively.

Developer Challenges: Secondly, Android developers are tasked with the job of creating applications that scale well across the board, considering a wide field of devices that come with limited amounts of internal storage. Variations in hardware, screen size, and amount of storage have placed increasing demands on developers to find flexible and efficient means of storing data on Android, ensuring optimal performance regardless of the device type.

Key Strategies for Optimizing Android Storage

1. Using Scoped Storage for Security and Efficiency

Moving to scoped storage, it was an important behavior change that was introduced with time in Android 10, that fundamentally altered how apps share files and access external data. Apps used to have nearly free run of the device, for better or worse, due to the previous paradigm. In contrast, scoped storage provides a restricted environment whereby an app is only allowed to access specific directories. 

In addition, developers should migrate their applications to scoped storage to align with the privacy standards set by Google. This scoped storage not only restricts data access but also increases user control over which data can be shared, hence improving trust and security.

For instance, the MediaStore API can be used to manage user media files, such as photos and videos, without having direct access to other sensitive files. This API is quite handy in interacting with media files while abiding by scoped storage guidelines.

Real-World Example:

Applications such as Spotify and WhatsApp serve as examples for the successful usage of scoped storage to adapt with extended standards of privacy protection under the Android environment. It isolates apps from any interaction with external files or system data other than the ones they actually have created. For example, WhatsApp by default keeps all of its files in its scoped storage but does allow users to store media outside of it on the device, depending on their choice. This balances security and user control, enabling these apps to scale to millions of users while keeping both performance and privacy.

2. Effective Strategy for Caching Data

In order to optimize app performance and user experience in data-heavy applications, effective caching strategies play a vital role. Caching is a critical method for enhancing mobile app performance, especially in data-heavy apps. Cache storage temporarily holds frequently accessed data, reducing the need to repeatedly fetch it from remote servers or databases, thus improving speed and responsiveness. However, without proper management, caches can grow uncontrollably, leading to unnecessary storage consumption and slower app performance.

Best Practices for Caching:

Caching is best implemented by apps themselves, so by thoughtfully managing caching, apps can enhance performance and optimize user experience while conserving device resources.

A good example would be Youtube, which is an adaptive caching through its Smart Downloads feature. This functionality downloads and caches recommended videos, ensuring they are available for users even without internet connectivity. Additionally, YouTube’s approach optimizes cache size based on available storage, preventing bloat and performance regressions while maintaining a seamless user experience.

3. Using Cloud Integration to Expand Storage

Cloud storage solutions have revolutionized how apps manage data, giving a practical way in which the limitations brought about by local device storage can be overcome. By using the cloud, applications can offset large files and backups, thus helping the application run on devices with constrained storage smoothly. However, it’s important to note that cloud integration often benefits apps when there is a backend server for doing the processing.

For instance, there is Google Photos for seamless cloud integration. The app itself not only saves the local device from storage pressure by backing up the photos and videos on the cloud but also provides an opportunity for the backend servers to process the content by automatically adding tags, geolocation metadata, and other contextual information that enhance the power of search and retrieval. This processing, which would be inefficient or impossible on a local device, greatly improves the user experience by offering faster and more accurate search results.

Best Practices for Cloud Integration:

  • Selective Syncing: Allow users to decide which data gets uploaded to the cloud and which remains local, giving them greater control over their storage.
  • On-Demand Downloads: Only fetch data from the cloud when necessary to minimize storage usage on the device.
  • Real-Time Updates: Implement real-time synchronization with cloud storage to ensure that data remains up-to-date without manual intervention.
  • Enhanced User Privacy: Use encryption and secure transfer protocols to protect user data both in transit and at rest.

So by utilizing cloud storage effectively, developers can optimize app performance, conserve local device resources, and unlock advanced functionalities through server side processing. This strategy is particularly valuable for apps managing large media files or requiring computationally intensive features that extend beyond the capabilities of a mobile device.

Advanced Solutions: Beyond Traditional Storage Management

Custom Scoped Storage Management 

While the above solutions use already existing methods to improve Storage Management on device, as the application scales, it becomes harder to manage storage at an app level with multiple sub products and services competing for the same storage space.

As applications are sandboxed since Android 9, developers have 2 main directories to store files.

Context.getFilesDir() returns a directory within the app’s sandbox where devs can store files related to the app. These files are generally only deleted when the app is uninstalled or all data of the app is cleared.
Context.getCacheDir() returns a similar directory but where cached files are stored. Cached files should be cleaned up by the app, but they can also be cleaned up by the OS or other third party storage cleaner apps.

As the app scales, a way to provide better storage management would be to provide a single entry point or service that acts as a Storage Layer above Android’s APIs.
The Storage Layer can then provide managed subdirectories to products and services, under the Cache or Files app sandbox directories based on configuration.

This API layer has many advantages:

  1. Ownership: The subdirectory requested by the product or service has clear ownership of it and all files under it. No other product or service should access or make changes within this directory
  2. Automatic cleanup: A great advantage of having a managed directory is that it can be automatically cleaned up after use. The configuration can have a parameter which states how long the data should be kept, which prevents stale data from taking up precious space on device
  3. Limits: Having managed partitioned directories means that it is possible to set limits to the data contained within it. Once the limit is exceeded, the directory can be cleaned up. Additionally, other cleanup algorithms can also be used to retain and re-use individual files in the directory which are frequently used, such as LRU based cleanup
  4. Versioning: App scaling and growing over time can mean changes to the data being stored, additional metadata or entire change to the storage itself. These can be versioned from the Storage Layer with migrators in place to move data between versions.
  5. User Scoping: An additional boon to having managed storage is User Scoped storage.
    Products and Services that have user data can be stored to UserScoped subdirectories, which can be auto cleaned up when the user logs out or switches. This significantly boosts the privacy of the app by ensuring no user data is kept once the user removes their account.

Conclusion: Towards Smart Storage Ecosystem

In conclusion, the Android mobile device storage landscape is evolving at a very fast pace. Optimizing storage in today’s world is no longer about just managing space; rather, it has to do with creating intelligent, scalable systems that balance user expectations with app performance. The more complex mobile apps are getting the greater the demand for strong storage solutions which can scale across millions of devices.

Further, developers are armed with a host of other features, from scoped storage to custom storage management  optimizations and embracing cloud-based solutions. These innovations ensure that the developers create applications that scale efficiently and offer seamless experiences that keep users coming back for more.

However, the big question into the future is, with further development in AI and cloud computing, how will these continue to redefine mobile app experiences and change the way we use our devices? The answer will likely depend on continued innovation and collaboration across the entire Android ecosystem.

What is LLMOps, MLOps for large language models, and their purpose

Posted on by Editorial Team

Why manage transfer learning of large language models and what is included in this management: getting acquainted with the MLOps extension for LLM called LLMOps.

(um) let’s make tech work for us…

How did LLMOps come to be? 

Large language models, embodied in generative neural networks (ChatGPT and other analogues), have become the main technology of the outgoing year, which is already actively used in practice by both individuals and large companies. However, the process of training LLM (Large Language Model) and their implementation in industrial use must be managed in the same way as any other ML system. A good practice for this has become the MLOps concept, aimed at eliminating organizational and technological gaps between all participants in the development, deployment and operation of machine learning systems.

As the popularity of GPT networks and their implementation in various application solutions grows, there is a need to adapt the principles and technologies of MLOps to transfer learning used in generative models. This is because language models are becoming increasingly large and complex to maintain and manage manually, which increases costs and reduces productivity. To avoid this, LLMOps, a type of MLOps that oversees the LLM lifecycle from training to maintenance using innovative tools and methodologies, can help.

LLMOps focuses on the operational capabilities and infrastructure required to fine-tune existing base models and deploy these improved models as part of a product. Because base language models are huge, such as GPT-3, which has 175 billion parameters, they require a huge amount of data to train, as well as time to map the computations. For example, it would take over 350 years to train GPT-3 on a single NVIDIA Tesla V100 GPU. Therefore, an infrastructure that can run GPU machines in parallel and process huge data sets is essential. LLM inference is also much more resource-intensive than more traditional machine learning, as it is not a single model, but a chain of models.

LLMOps provides developers with the necessary tools and best practices for managing the LLM development lifecycle. While the ideas behind LLMOps are largely the same as MLOps, large base language models require new methods, guidelines, and tools. For example, Apache Spark in Databricks works great for traditional machine learning, but it is not suitable for fine-tuning LLMs.

LLMOps focuses specifically on fine-tuning base models, since modern LLMs are rarely trained entirely from scratch. Modern LLMs are typically consumed as a service, where a provider such as OpenAI, Google AI, etc. offers an API of the LLM hosted on their infrastructure as a service. However, there is also a custom LLM stack, a broad category of tools for fine-tuning and deploying custom solutions built on top of open-source GPT models. The fine-tuning process starts with an already trained base model, which then needs to be trained on a more specific and smaller dataset to create a custom model. Once this custom model is deployed, queries are sent and the corresponding completion information is returned. Monitoring and retraining a model is essential to ensure its consistent performance, especially for LLM-driven AI systems.

Rapid engineering tools allow contextual training to be performed faster and cheaper than fine-tuning, without requiring sensitive data. In this case, vector databases extract contextually relevant information for specific queries, and prompt queries can optimize and improve model output based on patterns and chaining.

Similarities and differences with MLOps

In summary, LLMOps facilitates the practical application of LLM by incorporating operational management, LLM chaining, monitoring, and observation techniques that are not typically found in conventional MLOps. In particular, prompts are the primary means by which humans interact with LLMs. However, formulating a precise query is not a one-time process, but is typically performed iteratively, over several attempts, to achieve a satisfactory result. LLMOps tools offer features to track and version prompts and their results. This facilitates the evaluation of the overall performance of the model, including operational work with multiple LLMs.

LLM chaining links multiple LLM invocations in a sequential manner to provide a single application function. In this workflow, the output of one LLM invocation serves as the input to another to produce the final result. This design approach represents an innovative approach to developing AI applications by breaking down complex tasks into smaller steps. Chaining removes the inherent limitation on the maximum number of tokens that LLM can process simultaneously. LLMOps simplifies chaining management and combines it with other document retrieval methods, such as vector database access.

LLMOps’s LLM monitoring system collects real-time data points after a model is deployed to detect degradation in its performance. Continuous, real-time monitoring allows you to quickly identify, troubleshoot, and resolve performance issues before they affect end users. Specifically, prompts, tokens and their length, processing time, inference latency, and user metadata are monitored. This allows you to notice overfitting or changing the underlying model before performance actually degrades.

Monitoring models for drift and bias is also critical. While drift is a common problem in traditional machine learning models, as we’ve written about here, monitoring LLM solutions with LLMOps is even more important due to their reliance on underlying models. Bias can arise from the original datasets on which the base model was trained, custom datasets used for fine-tuning, or even from human evaluators judging fast completion. A thorough evaluation and monitoring system is needed to effectively remove bias.

LLM is difficult to evaluate using traditional machine learning metrics because there is often no single “right” answer, whereas traditional MLOps relies on human feedback, incorporating it into testing, monitoring, and collecting data for use in future fine-tuning.

Finally, there are differences in the way LLMOps and MLOps approach application design and development. LLMOps is designed to be fast, whereas traditional MLOps projects are typically iterative, starting with existing proprietary or open-source models and ending with custom fine-tuned or fully trained models on curated data.

Despite these differences, LLMOps is still a subset of MLOps. That’s why the authors of The Big Book of MLOps from Databricks have included the term in the second edition of this collection, which provides guiding principles, design considerations, and reference architectures for MLOps.

Data Fabric and Data Mesh: Complementary Forces or Competing Paradigms?

Posted on by Editorial Team

As the world continues to change, two frameworks have emerged to help businesses each manage their data ecosystems – Data Fabric and Data Mesh. While both these frameworks aim to simplify a business’s data governance, integration, and access, they differ quite a lot in their philosophy and how they operate. Data Fabric focuses more on technological orchestration over a distributed environment. Alternatively, Data Mesh focuses more on structural decentralization and domain-centric autonomy. This article looks at the powerful cloud-based architecture that integrates these two frameworks through its definitions, strengths, limitations, and the potential for synergy.

(um) let’s make tech work for us)

What is Data Fabric?

The Data Fabric concept originated in 2015 and came into focus after Gartner included it in the top analysis trends of 2020. In the DAMA DMBOK2 glossary, data architecture is defined as the plan for how to manage an organization’s data assets in a way that model of the organization’s data structures. Data Fabric implements this by offering a unified framework that automatically and logically integrates multiple disjointed data systems into one entity. 

Simply put, Data Fabric is a singular architectural layer that sits on top of multiple heterogeneous data ecosystems – on-premises systems, cloud infrastructures, edge servers –  and abstracts their individual complexities. It uses and combines several data integration approaches like the use of special data access interfaces (APIs), reusable data pipelines, automation through metadata, and AI orchestration to provide and facilitate non-restricted access and processing. Unlike older methods of data virtualization, which assisted in constructing a logical view, Data Fabric combines with the essence of containerization, which allows better management, control, and governance making masking it more powerful for modernizing applications than traditional methods.

Key Features of Data Fabric

  • Centralized Integration Layer: A virtualized access layer unifies data silos, governed by a central authority enforcing enterprise standards.
  • Hybrid Multi-Cloud Support: Consistent data management across diverse environments, ensuring visibility, security, and analytics readiness.
  • Low-Code/No-Code Enablement: Platforms like the Arenadata Enterprise Data Platform or Cloudera Data Platform simplify implementation with user-friendly tools and prebuilt services.

Practical Example: Fraud Detection with Data Fabric

Consider a financial institution building a fraud detection system:

  1. An ETL pipeline extracts customer claims data from multiple sources (e.g., CRM, transaction logs).
  2. Data is centralized in a governed repository (e.g., a data lake on Hadoop or AWS S3).
  3. An API layer, enriched with business rules (e.g., anomaly detection logic), connects tables and exposes the unified dataset to downstream applications.


While this approach excels at technical integration, it often sidesteps critical organizational aspects – such as data ownership, trust, and governance processes—leading to potential bottlenecks in scalability and adoption.

How Data Mesh Works

Data Mesh, introduced around 2019, is a new framework of data architecture that puts a greater emphasis on people rather than technology and processes. Like DDD, Data Mesh advocates for Domain-oriented decentralization, which promotes the fragmentation of data ownership among business units. Unlike Data Fabric, which controls everything from a single point, Data Mesh assigns domain teams with the responsibility of treating data as a product that can be owned, accessed, and interacted with in a self-service manner. 

Core Principles of Data Mesh

  • Domain-Oriented Decentralization: The closest teams to the data, whether it be its consumption or generation, have the ownership and management of the data. 
  • Data as a Product: More than just a simple dataset, each dataset can be marketed and comes with features such as access controls and metadata. 
  • Self-Service Infrastructure: Centralized domain teams are able to function autonomously because of a centralized platform. 
  • Federated Governance: Domains without a central data governance point are controlled centrally in terms of standards, data policies, and interfacing.

Practical Example: Fraud Detection with Data Mesh

Using the same fraud detection scenario:

  1. A domain team (e.g., the claims processing unit) defines and owns an ETL/ELT job to ingest claims data.
  2.  Datasets (e.g., claims, transactions, customer profiles) are stored separately, each with a designated owner.
  3.  A data product owner aggregates these datasets, writing logic to join them into a cohesive fraud detection model, delivered via an API or event stream.

This approach fosters accountability and trust by embedding governance into the process from the outset. However, its reliance on decentralized teams can strain organizations lacking mature data cultures or robust tooling.

Emerging Tools

Data Mesh is still maturing technologically. Google’s BigLake, launched in 2022, exemplifies an early attempt to support Data Mesh principles by enabling domain-specific data lakes with unified governance across structured and unstructured data.

Data Fabric works best with complex siloed infrastructures since it offers a top-down approach to data access. On the other hand, Data Mesh performs well in decentralized organizations that are willing to undergo a cultural shift and give more emphasis on trust and agility as compared to technical standardization.

Just like data fabric and data mesh, enterprise operational context and digital transformation journey determines the scope of its existence. The cloud provides a platform where both approaches can be integrated. Consider an architecture where there exists an event bus (for example Apache Kafka), which streams data to many different consumers. The consumers could include AWS S3, which acts as a data lake, and ETL pipelines (AirFlow for batch and NiFi for streaming), which serve to integrate operational and historical data. Add a robust Master Data Management (MDM) layer and analytics will be of good quality. 

This is the integration point where synergy shines: the centralized integration of data fabric sets up the infrastructure and data mesh domain autonomy makes it possible to innovate. A cloud native application platform which enables and controls innovation is the result. Business Intelligence (BI) dashboard is an example, which could be drawing the Mesh IoT dashboard clean data products, while Fabric governs seamless access to data.

A Call to Innovate

Marrying these paradigms isn’t without hurdles. Architects and engineers must grapple with:

  • Migration Complexity: How do you transition on-premises data to the cloud without disruption?
  •  Real-Time vs. Batch: Can the platform balance speed and depth to meet business demands?
  •  Data Quality: How do you embed quality checks into a decentralized model?
  •  Security and Access: What federated security model ensures ease without compromising safety?
  •  Lifecycle Management: How do you govern data from creation to destruction in a hybrid setup?


Moreover, the cloud isn’t a silver bullet. Relational databases often fall short for advanced analytics compared to NoSQL, and data lake security models can hinder experimentation. Siloed data and duplication further complicate scalability, while shifting from centralized to decentralized governance requires a cultural leap.

The Verdict: Together, Not Versus

So, is it Data Fabric versus Data Mesh? These methods have no real conflict; they work hand in hand. Data Fabric shows the threads of a technology metaphor for a superordinate access to information, while Data Mesh gives authority to the operational teams to manage their data. In a cloud-powered ecosystem, they have the potential to revolutionize data management by merging centralization’s productivity with decentralization’s creativity. The challenge that arises is not what to select, but how to combine the multifarious assets into a harmonious orchestra that nurtures trust, economic agility, and value to the enterprise. As the instruments undergo changes and institutions transform, these two concepts may as well be the paradigm shift that data architecture has long been waiting for, shaken, stirred and beautifully blended.

UnlockED Hackathon

Posted on by Editorial Team

Shaping the Future of Education with Technology – February 25-26, 2024

ExpertStack proudly hosted the UnlockED Hackathon, a high-energy innovation marathon focused on transforming education through technology. With over 550 participants from the Netherlands and a distinguished panel of 15 judges from BigTech, the event brought together some of the brightest minds to tackle pressing challenges in EdTech.

The Challenge: Reimagining Education through Tech
Participants were challenged to develop groundbreaking solutions that leverage technology to make education more accessible, engaging, and effective. The hackathon explored critical areas such as:

  • AI-powered personalized learning – Enhancing student experiences with adaptive, data-driven education.
  • Gamification & immersive tech – Using AR/VR and interactive platforms to improve engagement.
  • Bridging the digital divide – Creating tools that ensure equal learning opportunities for all.
  • EdTech for skill-building – Solutions focused on upskilling and reskilling for the digital economy.

For 48 hours, teams brainstormed, designed, and built innovative prototypes, pushing the boundaries of what’s possible in education technology.

And the Winner is… Team OXY!
After an intense round of final presentations, Team OXY took home the top prize with their AI-driven adaptive learning platform that personalizes study plans based on real-time student performance. Their solution impressed judges with its scalability, real-world impact, and seamless integration with existing education systems.

Driving Change in EdTech
The UnlockED Hackathon was more than just a competition—it was a movement toward revolutionizing education through technology. By fostering collaboration between developers, educators, and industry leaders, ExpertStack is committed to shaping a future where learning is smarter, more inclusive, and driven by innovation.

Want to be part of our next hackathon? Stay connected and join us in shaping the future of tech! 🚀

Mobile Development Trends to Follow in 2024

Posted on by Editorial Team

The mobile development industry witnessed tremendous innovations and shifts in the year 2023. The implementation of new programming languages and libraries, as well as the introduction of new tools and technologies, results in rapid growth in mobile development. Looking into 2024, the mobile development sector will be influenced by these several trends. Here’s what technologies to focus on, what will be in demand, and what developers need to monitor.

Artificial Intelligence: A Game-Changer in Mobile Development

Artificial Intelligence (AI) remains the most prominent and promising direction in mobile development for 2024. According to the McKinsey Global Institute, AI has the potential to increase company profits by $4.4 trillion annually. As AI continues to revolutionize industries across the board, mobile developers are increasingly incorporating generative AI into everyday applications.

Machine Learning (ML) technologies have already found a strong foothold in mobile development. Libraries for image recognition, text scanning, voice and video processing, and tools for improving app performance, such as those combating memory leaks, are just the beginning. FaceID, for instance, relies heavily on ML to enable secure and seamless authentication.

Since late 2022, there has been an explosion in the development of AI systems, and generative AI technologies like ChatGPT are now at the forefront of this revolution. In the Russian market, Yandex’s Yandex GPT is also making waves. Moving into 2024, companies are increasingly integrating AI-based solutions into their apps. This integration is not limited to improving user experiences through better content recommendations and personalized services but extends to tasks like automatic translation and smart user interaction.

For mobile developers, AI’s role is expanding beyond app functionality to the very development process itself. AI-driven tools are now capable of generating and optimizing code, which could significantly speed up the development cycle. However, questions still remain about the quality and safety of AI-generated code—particularly in terms of security and performance. Despite these concerns, the trend toward AI-enhanced development is undeniable and promises to evolve further in 2024.

Cross-Platform Development: Kotlin Multiplatform and Flutter Take Center Stage

Cross-platform mobile development has been a growing trend for several years, allowing developers to build apps for multiple platforms with a single codebase. In 2024, two solutions are leading the charge: Kotlin Multiplatform (KMP) by JetBrains and Flutter by Google. These technologies are not only popular but continue to evolve and improve, making them key players in the cross-platform development space.

Kotlin Multiplatform (KMP) is gaining traction as a versatile SDK for developing apps across Android, iOS, and other platforms. With its strong Kotlin ecosystem, KMP allows developers to share code between platforms while maintaining the ability to write platform-specific code where necessary. The result is a streamlined development process that reduces redundancy without sacrificing performance.

Flutter, Google’s open-source UI toolkit, is another heavyweight in the cross-platform development world. Known for its fast development cycle and rich set of pre-designed widgets, Flutter continues to evolve with regular updates that enhance its capabilities, performance, and integration with various platforms. Flutter’s flexibility makes it an appealing choice for developers seeking to create beautiful, natively compiled applications for mobile, web, and desktop from a single codebase.

Both KMP and Flutter will see further advancements in 2024, with JetBrains conducting an annual survey to gauge the growing popularity of Kotlin and KMP. Expect these tools to introduce new features, optimizations, and capabilities that make cross-platform mobile development even more efficient and powerful.

Source: https://www.jetbrains.com/lp/devecosystem-2023/kotlin/

In the fall of 2023, something that many developers and fans of KMP technology were waiting for happened. The technology moved to the Stable status and became completely ready for full use. This means that many issues that were relevant for the Alpha and Beta versions of the SDK were resolved. For example, they improved support for Kotlin/Native, work with multithreading, memory, etc. One of the goals of the roadmap for 2024 was to implement direct interaction between the Kotlin and Swift languages. Despite the fact that Google developers are the authors of the competing product Flutter, the company officially places a big bet on KMP. Cross-platform support is included in many solutions. So we advise you to take a closer look at this product.

Another trend related to cross-platform development is the use of Compose Multi Platform to implement a cross-platform UI. This declarative framework combines technologies such as Compose for Desktop, Compose iOS, Compose for Web, and Jetpack Compose for Android, and allows you to quickly and relatively easily create a common UI for different platforms. In 2023, the alpha version of Compose for iOS was released. In 2024, the Beta version is expected to be released, which will include improvements in working with the native iOS UI, as well as support for a cross-platform solution for navigation.

In global trends in mobile development, solutions such as React Native, hybrid development on Cordova and ionic, and Xamarin remain popular. The trend for PWA development also continues.

Native development

Technologies and frameworks change, native development remains. This is the foundation of foundations, the base that every developer should know. Using native languages ​​and tools, as well as the Native First approach, will always be relevant. In addition, for the implementation of projects with complex logic, complex UI, it is worth choosing native development.

Every year, developers of iOS/Android platforms, as well as Swift and Kotlin languages, release many new and interesting solutions, which they talk about at thematic conferences WWDC and Google I/O.

Particular attention should be paid to the growing trend towards declarative development in mobile applications. The SwiftUI and Jetpack Compose frameworks are actively developing, steadily improving, becoming more convenient and reliable in operation. They are increasingly used in the development of applications of varying complexity. Many libraries and ready-made solutions tailored for SwiftUI and Jetpack Compose. We can say that this is a new standard for mobile development for Android and iOS.

It is also worth paying attention to supplementing mobile applications with interactive widgets that will help draw attention to applications and provide instant access to a number of functions.

Virtual reality

In 2023, at the WWDC conference, Apple presented one of the most high-profile new products – Vision Pro virtual reality glasses running on the VisionOS platform. Of course, this is far from the first such device in the world, and similar devices have long been used in the gaming industry. The emphasis is on unique immersive technologies, improved sound and image quality. As part of the creation of the device, large-scale developments were carried out in the field of spatial computing. VisionOS support was included in such tools as: ARKit, RealityKit, Unity, RealityComposer, etc. Both the toolkit and documentation are currently available to all interested developers. The start of sales of the device itself is scheduled for 2024-2025.

Recently, the creators of Vision Pro announced the imminent launch of an app store for virtual reality glasses. This means that in 2024 we will see a boom in various applications for AR/VR devices: games, virtual fitting rooms, interior design applications, immersive movie viewing, listening to music, etc. In addition, according to Statista, by 2028 the number of AR and VR market users in the world will reach 3674.0 million users.

Both Google and Apple are also focusing on improving the immersive user experience on standard smartphones, watches, and tablets.

Not just smartphones. IoT

Every year, new various devices running Android and iOS are released. These are not only smartphones and tablets, but also watches, smart TVs, game consoles, fitness trackers, car computers, and smart home devices. OS development companies are interested not only in supporting new capabilities of devices and gadgets, but also in improving the tools for third-party developers.

The use of NFC, Bluetooth in applications is still relevant. Yahoo Finance predicts growth in the NFC field from 2023 to 2030 by 33.1 billion US dollars.

Security, improved network operation

Preservation of confidential information has been and remains one of the main tasks of developers. This is evidenced by the Gartner forecast. The introduction of enhanced security measures (biometric authentication, use of blockchain, etc.) is very relevant in 2024. Also, special attention should be paid to stable and secure operation of the network, including work with cloud services, with NFC, when connecting to other devices. Modern mobile OS offer a wide range of native tools for implementing and supporting secure operation of applications.

Let’s sum it up

In 2024, mobile platforms, languages ​​and development tools will continue to evolve. The main areas that we recommend paying attention to will be:

  • native development;
  • cross-platform;
  • development for various mobile devices, IoT;
  • support and use of Russian technologies;
  • security, network operations;
  • AR/VR.

Breathing New Life into Legacy Businesses with AI

Posted on by Editorial Team

Author: Jenn Cunningham, a Go-to-Market Lead, Strategic Alliances at PolyAI. Currently, she leads strategic alliances at PolyAI, where she manages key relationships with AWS and global consulting partners while collaborating closely with the PolyAI cofounders on product expansion and new market entry. Her unique journey from data science beginnings to implementation consulting gives her a front-row seat to how legacy businesses are leveraging AI to evolve and thrive.

***

Introduction: 

I was hired and trained as a data scientist for IBM fresh out of university. Data science and data analytics were the hot topic at the time, as businesses were ready to become more data-driven organizations. I was so excited to unlock new customer insights and inform business strategy, until my first project. After 8 weeks of cleansing data and crunching numbers for 12 hours a day, it was glaringly obvious that I was entirely too extroverted for a pure data science role. I quickly moved into implementation consulting, where I was fortunate to see firsthand how businesses evaluate, implement, and adopt different types of process automation technology as the technology itself continued to evolve.That evolution led me to realize the other data and AI’s capabilities, primarily focusing on what they could do to the operations of businesses labeled legacy – not only for efficiency, but also for improving user service. These companies tend to be branded or perceived as slower to adapt, but they’re full of indisputable value waiting for the right ‘nudge.’

AI is providing that nudge because today, AI does more than automating boring work; it is changing how businesses perceive value. A century-old bank, a global manufacturer, and a regional insurer, these are just a few examples of businesses that are evolving their core AI technologies, improving their internal systems while retaining their rich history. 

This didn’t suddenly happen though, but there were many steps involved, each more groundbreaking than the last. So to truly narrate the current state AI had to evolve in, we need to wind the clocks back to a time when data wasn’t an inevitability but a luxury.

The First Wave: Data as a Project

Back in the infancy of data science within companies, their treatment of data resembled a whiteboard and marker experiment. Businesses seemed lost on what to do with data, and therefore assumed it required a project-like treatment with a start and end, or a PowerPoint presentation in mid — something that showcases interim findings. Along the way, gathering “let’s get a data scientist to look at this” comments became a norm embracing a carefree approach of one-single-to-multi-business domain change. 

While working for IBM, I was fortunate to witness this phenomenon in real time. Shifts were just beginning where organizations were changing from relying on gut feelings to data informed strategies but everything still felt labored. One of my clients from IBM still sits fresh in my head because they took the tin can approach a little too literally, printing out .txt files like word documents containing customer interactions and using scissors to post these documents in a conference room where they would visually calculate key metrics, calculators and highlighters in hand. Data science in its untapped, unrefined glory was radical.

The purpose wasn’t to create sustainable systems. Instead, it was to respond to prompts such as “What’s our churn rate?” or “Was this campaign successful?” These questions, while important in their own right, were evasive at best. Each project felt like a fleeting victory without much future potential. There was no reusable framework, no collaboration across teams, and definitely no foresight for what data could evolve into.

However, this preliminary wave had significance as it allowed companies to recognize the boundaries of instinct-driven decision-making and the usefulness of evidence. Although the work was done in stages, it rarely resulted in foundational changes, and even when insights did materialize, they were not capable of driving widespread change.

The Second Wave: Building a Foundation for Ongoing Innovation

Gradually, a new understanding seemed to surface,  one that moved data from being a tactical resource to a strategic asset. In this second wave, companies sought answers to more advanced inquiries. How do we use data to enable proactive decision-making rather than only responsive actions? How can we incorporate insights into the operational fabric of our company? 

This phase experienced shifts from companies such as Publicis Groupe. Rather than bringing on freelance data scientists on a contractual basis, they transformed their approaches by building internal ecosystems of expertise composed of multidisciplinary teams and fostering a spirit of innovation. Thus, the focus shifted from immediate problem-solving to laying the foundational systems for comprehensive future infrastructure. 

Moreover, data started to shift from the back-office functions to the forefront. Marketing, sales, product, and customer service functions received access to real-time dashboards, AI tools, predictive analytics, and a host of other utilities. Therefore the democratization of data accelerated to bring the power of AI data insights to the decision makers who worked directly with customers and crafted user experiences.

What also became clear during this phase was that not all parts of the organization required the same level of AI maturity at the same time. Some teams were set for complete automation; others just required clean reporting which was perfectly fine. The goal was not standard adoption; rather, it was movement. The most advanced thinking companies understood that the pace of change didn’t have to happen everywhere all at once, it just needed a starting point and careful cultivation.

This was the turning point when data began evolving from a department to a capability; it could now single-handedly drive continuous enhancements instead of relying on project-based wins. That is when the flywheel of innovation had commenced spinning.

The Current Wave: Reimagining Processes with AI

Today, we are experiencing a third and possibly the most impactful wave of change. AI is no longer limited to enhancing analytics and operational efficiency; it now rethinks the very framework of how businesses are structured and run. What was previously regarded as an expenditure is now considered a divisive competitive advantage.  

Consider what PolyAI and Simplyhealth have done. Simplyhealth, a UK health insurer, partnered with PolyAI to implement voice AI within their customer service channels. However, this integration went beyond implementing basic chatbots. The AI was ‘empathetic AI’ since it could understand urgency, recognize vulnerable callers, and make judgment calls on whether patients should be passed to a human auxiliary.  

Everyone saw the difference. There was less waiting around, better call resolution, and most crucially, those that required care from a member of staff received it. Nonetheless, AI did not take the person out of the process; it elevated the person into the process, allowing them to experience empathy and enable humanity to work alongside effectiveness.

Such a focus on building technology around humans is rapidly becoming a signature of AI change in today’s world. You see it with retail AI, which customizes every touchpoint in the customer experience. It’s happening in manufacturing with costs associated with breakdowns being avoided through predictive maintenance. And in financial services, it’s experiencing massive shifts as AI technologies offer personalized financial consulting, fraud detection, and assistance to those missing traditional support.  

In all these examples, AI technologies support rather than replace people. Customer service representatives are equipped with richer context, which augments their responses, freelancers are liberated from doing repetitive work, and strategists get help concentrating on the right resources. Therefore, today’s best AI use cases focus on augmenting human experience instead of reducing the workforce.

Conclusion: 

Too often is the phrase “legacy business” misused to describe something as old-fashioned or boring. But in fact, these are businesses with long-standing customer relationships and histories, enabling them to evolve in meaningful ways.  

Modern AI solutions don’t simply replace manual labor as the advancement from spreadsheets and instinct-based decisions to fully integrated AI systems is more complex. Businesses progressively adopt modern practices all while having a vision and patience in terms of cultural branding. Plus, legacy businesses are contemporarily evolving and keeping up with the pace, and many are leading the race.  

AI today is changing everything and has now become a culture driving system. It impacts the very way we collaborate, deliver services, value customers, and so much more. Whether implementing new business strategies, redefining customer support, or optimizing computer science logistics, AI is proving to be a propellant for transformation focused on humans.  

Further, visionaries and team members who witnessed this automated evolution firsthand felt unity through action, fervently participating as data table-aligned pilots meshed with algorithms and numbers. Reminding us that change isn’t all technical; it’s human. It’s intricate, fulfilling, and simply put: essential.

To sum up, the future businesses are not the newest; rather, they are the oldest that choose to develop with a strong sense of intention behind it. In that development, legacy is not a hindrance, but rather, a powerful resource.

Testing Camera and Gallery in Android: Emulating Image Loading in UI Tests

Posted on by Editorial Team

Author: Dmitrii Nikitin, a Android Team Leader at Quadcode with over 7 years of experience in developing scalable mobile solutions and leading Android development teams.

A lot of Android apps request the user to upload images. Social media apps, document scanners, cloud storage providers, you name it. These scenarios are left without any automated tests because developers would rather not try to open the camera or the gallery.

But the fact is, such difficulties can be surpassed. In this article, I will discuss simulating the camera and gallery behavior in emulators, injecting specific images for testing purposes, intent mocking, and how to know when such methods are not enough for thorough testing.

Emulating Camera Images in Android Emulator

The Android emulator is able to display arbitrary images as camera sources, which is extremely convenient if you’re writing flows like “take a picture” or “scan a document” and you’d like the camera to display the same image under all circumstances.

Setting Up Custom Camera Images

The emulator uses a scene configuration file located at:

$ANDROID_HOME/emulator/resources/Toren1BD.posters

You can add a poster block to this file with these attributes:

poster custom
size 1.45 1.45
position 0.05 -0.15 -1.4
rotation -13 0 0
default custom-poster.jpg

This setting determines:

  • default: The path to the image used as the camera feed
  • size, position, rotation: Image size, position, and rotation angle parameters in the scene

Automating Image Setup

You can automatize this process through a shell command:

sed -i ’1s,^,poster custom\n size 1.45 1.45\n position 0.05 -0.15 -1.4\n rotation -13 0 0\n default custom-poster.jpg\n,’ $ANDROID_HOME/emulator/resources/Toren1BD.posters

Here is a Kotlin script that copies the required file into the correct position:

class SetupCameraImageScenario(private val imageFileName: String): BaseScenario<ScenarioData>() {
    override val steps: TestContext<ScenarioData>.() -> Unit = {
        val androidHome = System.getenv("ANDROID_HOME") ?: error("ANDROID_HOME is required")
        val posterPath = "$androidHome/emulator/resources/custom-poster.jpg"
        val localImagePath = "src/androidTest/resources/$imageFileName"
        val cmd = "cp $localImagePath $posterPath"
        Runtime.getRuntime().exec(cmd).waitFor()
    }
}

Injecting Images into Gallery

(Intent.ACTION_PICK) is less of a pain than camera testing, but with one crucial gotcha: copying to internal storage alone is not enough. If you simply copy an image file, it will not appear in the system picker.

An image must be written to the correct folder to be pickable, and must also be registered in MediaStore.

Proper Gallery Image Setup

The process involves:

  1. Declaring the name, type, and path of the image (e.g., Pictures/Test)
  2. Obtaining a URI from MediaStore and storing the image content into it

You can implement it as follows:

class SetupGalleryImageScenario(private val imageFileName: String) : BaseScenario<Unit>() {
    override val steps: TestContext<Unit>.() -> Unit = {
        step("Adding image to MediaStore") {
            val context = InstrumentationRegistry.getInstrumentation().targetContext
            val resolver = context.contentResolver
            
            val values = ContentValues().apply {
                put(MediaStore.Images.Media.DISPLAY_NAME, imageFileName)
                put(MediaStore.Images.Media.MIME_TYPE, "image/jpeg")
                put(MediaStore.Images.Media.RELATIVE_PATH, "Pictures/Test")
            }
            
            val uri = resolver.insert(MediaStore.Images.Media.EXTERNAL_CONTENT_URI, values)
            checkNotNull(uri) { "Failed to insert image into MediaStore" }
            
            resolver.openOutputStream(uri)?.use { output ->
                val assetStream = context.assets.open(imageFileName)
                assetStream.copyTo(output)
            }
        }
    }
}

Now that test is opening the gallery, the required image will be visible among the options.

Selecting Images from Gallery

After you’ve placed the image in the MediaStore, you need to call Intent.ACTION_PICK and select the appropriate file in the UI. That’s where UiAutomator is useful, as the picker UI varies across versions and Android:

  • Photo Picker (Android 13+)
  • System file picker or gallery on older Android versions

In order to support both, create a wrapper:

class ChooseImageScenario<ScenarioData>(
    onOpenFilePicker: () -> Unit,
) : BaseScenario<ScenarioData>() {
    override val steps: TestContext<ScenarioData>.() -> Unit = {
        if (PickVisualMedia.isPhotoPickerAvailable(appContext)) {
            scenario(ChooseImageInPhotoPickerScenario(onOpenFilePicker))
        } else {
            scenario(ChooseImageInFilesScenario(onOpenFilePicker))
        }
    }
}

Both approaches start in the same way by calling onOpenFilePicker() (typcally a button click in the UI), then:

  • ChooseImageInPhotoPickerScenario: Locates and taps the image within Photo Picker
  • ChooseImageInFilesScenario: Opens the system file manager (for example, locating the file name via UiSelector().text(“test_image.jpg”) and opening it)

This approach covers both kinds of scenarios, making picking an image general and robust.

Intent Mocking: When Real Camera or Gallery Isn’t Necessary

Most of the tests will not require opening actual camera or gallery apps. To test app response after an image is received, you can mock the response of the external app using Espresso Intents or Kaspresso.

For example, when you’re testing that a user “took a picture” and subsequently the UI displays the correct picture or triggers a button, you don’t need to open the camera. You can simulate the result to get this accomplished:

val resultIntent = Intent().apply {
    putExtra("some_result_key", "mocked_value")
}
Intents.intending(IntentMatchers.hasAction(MediaStore.ACTION_IMAGE_CAPTURE))
    .respondWith(Instrumentation.ActivityResult(Activity.RESULT_OK, resultIntent))

When startActivityForResult(.) is invoked by the app to launch the camera, the test gets an immediate precooked result, such as the image being captured and returned. The camera is not launched, so the test is fast and predictable.

This strategy proves useful when:

  • You are less concerned about selection or capture process but more concerned about processing outcomes
  • You have to make test execution faster
  • You need to avoid dependencies on different versions of camera/galleries on devices

When Mocking Isn’t Sufficient

Sometimes it’s also necessary to stress-test not just that the app returns results correctly, but that it acts correctly in actual usage, such as when the user isn’t using it and the system boots it out of RAM. An example of that is DNKA (Death Not Killed by Android).

Understanding DNKA

DNKA happens when Android quietly unloads your app because of memory pressure, loss of focus, or dev settings explicit unloading. onSaveInstanceState() may be invoked but onDestroy() may not. Users come back in and expect the app to “restore” itself into the same state. Ensure that you:

  • Check if ViewModel and State are properly rebuilt
  • Check that the screen crashes if no saved state exists
  • Check that SavedStateHandle is as expected
  • If user interaction (photo selection, form input, etc.) is preserved

Enabling DNKA

The simplest way of enabling behavior in which Android terminates activities forcefully is through developer system settings:

Developer Options → Always Finish Activities

You can achieve with ADB:

adb shell settings put global always_finish_activities 1
# 1 to enable, 0 to disable

That background aside, any external activity launch (camera or gallery) will result in your Activity destroyed. When you go back to the app, it’ll need to recreate state from scratch, precisely what we’d like to test.

Why Intent Mocks Don’t Help Here

When using mocked intents:

Intents.intending(IntentMatchers.hasAction(MediaStore.ACTION_IMAGE_CAPTURE))
    .respondWith(Instrumentation.ActivityResult(Activity.RESULT_OK, resultIntent))

The external application is never started, so Android won’t even unload your Activity. The mock is instant responsive, so it’s not possible to test DNKA scenarios.

When Real Intents Are Necessary

To verify DNKA behavior, Android actually needs to unload the Activity. This means actually performing an actual external Intents: take a picture, select from the gallery, or third-party apps. Only this is capable of simulating when users open another application and your application “dies” in the background.

Conclusion

Automated testing sometimes has added the requirement to “see” images, and this issue is not as sneaky as it may seem. Testing photo loading from camera or gallery choice actually doesn’t involve real devices or manual testing. Emulators let you pre-place required images and simulate them as though users just selected or took files.

While intent mocking can be sufficient in some cases, for others complete “real” experience is necessary in order to guarantee recovery from activity cancellation. The trick is choosing the right method for your specific test scenario. 

Understanding these methods enables you to gain complete testing of image-related functionality so that your app handles well in happy path and edge case scenarios like system-induced process death. With proper setup, you can create robust, stable tests for the full gamut of user activity across camera and gallery functionality.

Whether you are writing tests for profile picture uploads, document scanning, or something else that involves images, these practices provide the foundation for good automated testing without jeopardizing coverage or reliability.

Building serverless pipeline using AWS CDK and Lambda in Python

Posted on by Editorial Team

Creating a serverless pipeline using AWS CDK alongside AWS Lambda in Python allows for event-driven applications which can easily be scaled without worrying about the underlying infrastructure. This article describes the process of creating and setting up a serverless pipeline step by step in AWS CDK and Python Lambda with Visual Studio Code (VS Code) as the IDE.

Completing this guide enables the deployment of a fully working AWS Lambda function with AWS CDK.

Understanding Serverless Architecture and Its Benefits

A serverless architecture is a cloud computing paradigm where the developers need to write the code as functions and these functions get executed upon receiving an event or request. These functions will execute without any server provisioning or management. Execution and resource allocation are automatically managed by the cloud provider – in this instance, AWS.

Key Characteristics of Serverless Architecture:

  1. Event-Driven: Functions are triggered by events such as S3 uploads, API calls, or other AWS service actions.
  2. Automatic Scaling: The platform automatically scales based on workload, handling high traffic without requiring manual intervention.
  3. Cost Efficiency: Users pay only for the compute time used by the functions, making it cost-effective, especially for workloads with varying traffic.

Benefits:

Serverless architecture comes with numerous advantages that are beneficial for modern applications in the cloud. One of the most notable benefits of serverless architecture is improved operational efficiency due to the lack of server configuration and maintenance. Developers are free to focus on building and writing code instead of worrying about managing infrastructure. 

Serverless architecture has also enabled better workload management because automatic scaling allows serverless platforms to adjust to changing workloads without human interaction, making traffic spikes effortless. This kind of adaptability maintains high performance and efficiency while minimizing costs and resource waste.

In addition, serverless architecture has proven to be financially efficient, allowing users to pay solely for the computing resources they utilize, as opposed to pre-purchased server capacity. This flexibility is advantageous for workloads with unpredictable or fluctuating demand. Finally, the ease of use provided by serverless architecture leads to an accelerated market launch because developers can rapidly build, test, and deploy applications without the tedious task of configuring infrastructure, leading to faster development cycles.

Understanding ETL Pipelines and Their Benefits

ETL (Extract, Transform, Load) pipelines automate the movement and transformation of data between systems. In the context of serverless, AWS services like Lambda and S3 work together to build scalable, event-driven data pipelines.

Key Benefits of ETL Pipelines:

  1. Data Integration: Combines disparate data sources into a unified system.
  2. Scalability: Services like AWS Glue and S3 scale automatically to handle large datasets.
  3. Automation: Use AWS Step Functions or Python scripts to orchestrate tasks with minimal manual intervention.
  4. Cost Efficiency: Pay-as-you-go pricing models for services like Glue, Lambda, and S3 optimize costs.

Tech Stack Used in the Project

For this serverless ETL pipeline, Python is the programming language of choice while Visual Studio Code serves as the IDE. The architecture is built around AWS services such as AWS CDK for resource definition and deployment, Amazon S3 as the storage service, and AWS Lambda for running serverless functions. All these in combination build a strong robust and scalable serverless data pipeline.

The versatility and simplicity associated with Python, as well as its extensive library collection, make it an ideal language for Lambda functions and serverless applications. With AWS’s CDK (Cloud Development Kit), the deployment of cloud resources is made easier because infrastructure can be defined programmatically in Python and many other languages. AWS Lambda is a serverless compute service which scales automatically and charges only when functions are executed, making it very cost-effective for event-driven workloads. Amazon S3 is a highly scalable object storage service that features prominently in serverless pipelines as a staging area for raw data and the final store for the processed results. These components create the building blocks of a cost-effective and scalable serverless data pipeline.

  • Language: Python
  • IDE: Visual Studio Code
  • AWS Services:
    • AWS CDK: Infrastructure as Code (IaC) tool to define and deploy resources.
    • Amazon S3: Object storage for raw and processed data.
    • AWS Lambda: Serverless compute service to transform data.

Brief Description of Tools and Technologies:

  1. Python: A versatile programming language favored for its simplicity and vast ecosystem of libraries, making it ideal for Lambda functions and serverless applications.
  2. AWS CDK (Cloud Development Kit): An open-source framework that allows you to define AWS infrastructure in code using languages like Python. It simplifies the deployment of cloud resources.
  3. AWS Lambda: A serverless compute service that runs code in response to events. Lambda automatically scales and charges you only for the execution time of your function.
  4. Amazon S3: A scalable object storage service for storing and retrieving large amounts of data. In serverless pipelines, it acts as both a staging and final storage location for processed data.

Building the Serverless ETL Pipeline – Step by Step

In this tutorial, we’ll guide you through setting up a serverless pipeline using AWS CDK and AWS Lambda in Python. We’ll also use Amazon S3 to store data.

Step 1: Prerequisites

To get started, ensure you have the following installed on your local machine:

  • Node.js (v18 or later) → Download Here
  • AWS CLI (Latest version) → Install Guide
  • Python 3.x (v3.9 or later) → Install Here
  • AWS CDK (Latest version) → Install via npm.
  • Visual Studio Code → Download Here
  • AWS Toolkit for VS Code (Optional, but recommended for easy interaction with AWS)
Configure AWS CLI

To configure AWS CLI, open a terminal and run:

A screenshot of a computer AI-generated content may be incorrect.

aws configure

A screenshot of a computer

Enter your AWS Access Key, Secret Access Key, default region, and output format when prompted.

Install AWS CDK
A screenshot of a computer AI-generated content may be incorrect.

To install AWS CDK globally, run:

npm install -g aws-cdk

Verify the installation by checking the version:

cdk --version

Login to AWS from Visual Studio Code

Click on the AWS logo on the left side, it will ask for credentials for the first time

A screenshot of a computer AI-generated content may be incorrect.

For the profile name use the Iam user name

A screenshot of a computer

After signing in the IDE will appear as below.

A screenshot of a computer

Step 2: Create a New AWS CDK Project

Open Visual Studio Code and create a new project directory:

mkdir serverless_pipeline_project

cd serverless_pipeline_project

A screenshot of a computer
A computer screen shot of a computer screen
A screenshot of a computer

Initialize the AWS CDK project with Python:

cdk init app --language python
This sets up a Python-based AWS CDK project with the necessary files.

Step 3: Set Up a Virtual Environment

Create and activate a virtual environment to manage your project’s dependencies:

python3 -m venv .venv

source .venv/bin/activate  # For macOS/Linux

# OR

.venv\Scripts\activate  # For Windows

python3 -m venv .venv

source .venv/bin/activate  # For macOS/Linux

# OR

.venv\Scripts\activate  # For Windows

Install the project dependencies:

pip install -r requirements.txt

Step 4: Define the Lambda Function

Create a directory for the Lambda function:

mkdir lambda

Write your Lambda function in lambda/handler.py:

import boto3

import os

s3 = boto3.client('s3')

bucket_name = os.environ['BUCKET_NAME']

def handler(event, context):

    # Example: Upload processed data to S3

    s3.put_object(Bucket=bucket_name, Key='output/data.json', Body='{"result": "ETL complete"}')

    return {"statusCode": 200, "body": "Data successfully written to S3"}

Step 5: Define AWS Resources in AWS CDK

In the serverless_pipeline/serverless_pipeline_stack.py, define the Lambda function and the S3 bucket for data storage:

from aws_cdk import (

    Stack,

    aws_lambda as _lambda,

    aws_s3 as s3

)

from constructs import Construct

class ServerlessPipelineProjectStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:

        super().__init__(scope, construct_id, **kwargs)

        # Create an S3 bucket

        bucket = s3.Bucket(self, "ServerlessPipelineProjectS3Bucket")

        # Create a Lambda function

        lambda_function = _lambda.Function(

            self, 

            "ServerlessPipelineProjectLambdaFunction",

            runtime=_lambda.Runtime.PYTHON_3_9,

            handler="handler.handler",

            code=_lambda.Code.from_asset("lambda"),

            environment={

                "BUCKET_NAME": bucket.bucket_name

            }

        )

        # Grant Lambda permissions to read/write to the S3 bucket

        bucket.grant_read_write(lambda_function)

Step 6: Bootstrap and Deploy the AWS CDK Stack

Before deploying the stack, bootstrap your AWS environment:

cdk bootstrap

Then, synthesize and deploy the CDK stack:

cdk synth

cdk deploy

A screen shot of a computer code

You’ll see a message confirming the deployment.

Step 7: Test the Lambda Function

Once deployed, test the Lambda function using the AWS CLI:

aws lambda invoke --function-name ServerlessPipelineProjectLambdaFunction output.txt

You should see a response like:

{

    "StatusCode": 200,

    "ExecutedVersion": "$LATEST"

}

Check the output.txt file; it will contain:

{"statusCode": 200, "body": "Data successfully written to S3"}

A folder called output will be created in S3 with a file data.json inside it, containing:

{"result": "ETL complete"}

Step 8: Clean Up Resources (Optional)

To delete all deployed resources and avoid AWS charges, run:

cdk destroy

Summary of What We Built

For this project, we configured AWS CDK within a Python environment. This was done to create and manage the infrastructure that is needed for a serverless ETL pipeline. The processing unit of the pipeline is an AWS Lambda serverless function which we developed for data processing. We also added Amazon S3 to use as a scalable and durable storage solution for raw and processed data. We deployed the required AWS resources using AWS CDK which automated the deployment processes. Finally, we confirmed that the entire setup was as expected by invoking the Lambda function and assured the data flowed properly through the pipeline.

Next Steps

In the future, I see multiple opportunities to improve and extend this serverless pipeline. An improvement that could be added is the use of AWS Glue for data transformation since it can automate and scale complicated ETL processes. Also, integrating Amazon Athena will enable serverless querying of the processed data which will allow for efficient analytics and reporting. Furthermore, we could use Amazon QuickSight for data visualization that can enhance the insights obtained from the data, allowing users to interact with the data presented on dashboards. These steps will build upon fundamentally what we have already built and will create a more comprehensive and sophisticated data pipeline.

By following this tutorial, you’ve laid the foundation for building a scalable, event-driven serverless pipeline in AWS using Python. Now, you can further expand the architecture based on your needs and integrate more services to automate and scale your workflows.

Author: Ashis Chowdhury, a Lead Software Engineer at Mastercard with over 22 years of experience designing and deploying data-driven IT solutions for top-tier firms including Tata, Accenture, Deloitte, Barclays Capital, Bupa, Cognizant, and Mastercard.