Scientific innovation and drug discovery rely on quickly analyzing massive amounts of experimental data. When a team of dedicated scientists runs intricate experiments generating thousands of high-res microscope images, manually managing, transferring and archiving all of that information from physical hard drives creates a severe bottleneck that impedes digital image processing and the rapid collaboration necessary for groundbreaking scientific discoveries.
Imagine this: As researchers conduct more experiments leveraging high-speed, cloud-connected lab instruments in laboratories across the globe, that vast amount of experiment and microscopic image data quickly fills several multiterabyte hard drives, complicating local storage. When local data storage becomes unwieldy, the cloud provides a crucial lifeline — delivering limitless capacity, data accessibility for distributed teams and, ultimately, faster insights at lower costs.
Keep reading to learn how we built an Amazon Web Services-based proof-of-concept solution to address our client’s challenges managing the huge amounts of data generated by their customers’ experiments.
Data Retention + On-Site Storage — A Technical Challenge
Our client, a leading innovator in laboratory technology and software, has thousands of advanced automated microscopes installed in labs worldwide. However, the storage and processing of the image data these instruments produce is expensive.
Microscopy datasets can rapidly proliferate, with a single lab microscope producing data at a rate of 10 megabytes/second for extended time periods. Consequently, research teams quickly fill terabyte drives with high-resolution microscope images.
And processing all of that information demands significant computing power. While image analysis tasks are computationally intensive and extremely parallelizable, our client’s customers do not perform them frequently enough to justify purchasing and maintaining large local parallel processing systems.
The inefficiencies extend beyond the sheer volume of data. With data siloed on local physical drives, scientists have to manually transfer drives between desks to share recent results — a logistic and productivity nightmare for distributed teams, especially when multiple scientists need simultaneous access to experiment images. Unfortunately, this challenge is all too common for research teams generating huge volumes of visual data.
Consequently, our client needed a solution to centralize storage, accelerate processing and enable seamless collaboration for customers worldwide.
The cloud-based lab data management and digital image processing solution consolidates storage in the cloud so scientists can easily access experiment results anywhere.
Building a Cloud Data Storage and Digital Image Processing System
Recognizing the need for a scalable and efficient solution, our client engaged us to build a proof-of-concept multitenant cloud data storage and processing system. As an AWS IoT partner with skills in hardware development, embedded software, cloud services and mobile applications, we sought to design and build the system in AWS, which offers valuable services such as IoT Core, thus shortening the development cycle and reducing the risk of deploying new technology.
We took our client’s digital image processing software and wrapped it, using Python, to access the app in the cloud. The uploader app installs on the personal computers our client’s customers use, allowing researchers to easily upload lab microscope images to the server, which are then stored in Amazon Elastic File System (EFS) for quick access. To users, EFS looks like a local file system. Together with RDS, it serves as the source of truth, bypassing the message synchronization requirement.
For processing, we leverage containerized applications running on Amazon Elastic Container Service (ECS) clusters to rapidly process, stitch and analyze large volumes of cellular image data, for example. The application can run a plate with 96 wells and each well with 12×12 tile montages.
Digital Image Processing System: Simplified Cloud Architecture
Amazon Elastic File System (EFS) for Message Synchronization
Consolidated Cloud-Based Storage
The cloud-based lab data management and digital image processing solution consolidates storage in the cloud so scientists can easily access experiment results anywhere. Shipping the images off to the cloud as they’re produced eliminates the requirement for long-term local storage and storage backup.
Leveraging managed AWS services like EFS and ECS helps ensure no storage capacity or growth limits exist. The proof-of-concept solution also enables significantly faster processing by auto-scaling compute resources to handle image workloads in parallel. With a cloud environment performing the work, users benefit from parallelism — gaining speed — without the exorbitant cost of buying the system. With the AWS-based platform, our client only pays for the resources used.
AWS’s core storage and computation services shorten the development cycle and reduce the risk associated with migrating the critical aspects of our client’s digital image processing software to the cloud. AWS provides for the secure transfer and processing of data demanded by the lab managers and secure access to the data in the cloud.
Additionally, our Agile product development process appealed to our client, which previously employed a waterfall-style monthslong release cycle. Minimizing the development, verification and validation time by rapidly implementing feedback in short, two-week cycles empowered our client with valuable insight early. This level of knowledge enabled decision-makers to refine requirements and shorten the testing loop.
The Advantages of Moving a Laboratory Digital Image Processing System to the Cloud
Providing the right combination of speed, scalability and collaboration features that scientific research demands, our implementation of the AWS cloud-based proof-of-concept system yielded remarkable results.
Accelerated image processing
Parallel processing on ECS cut analysis time nearly in half — compared to the traditional cellular analysis algorithm — for large workloads.
Easy collaboration and multitenancy
Our unique cloud system allows multiple scientists to share and access data from anywhere, collaborating seamlessly in near real time.
Improved scalability and storage
The AWS-powered solution provides an infinitely scalable environment, eliminating storage space and processing power concerns. No more filling local drives because the cloud offers unlimited capacity.
Greater cost-efficiency
Cloud-based storage and processing significantly reduces costs associated with traditional on-site storage and maintenance, as well as the corresponding productivity losses.
AI and Machine Learning for Lab Analysis
Our collaborative effort exemplifies the transformative power of cloud-based lab data management solutions in overcoming complex data storage and microscope image processing challenges. Even better, the success of the proof-of-concept product prompted our client to engage Cardinal Peak in transforming it into an official cloud-based data storage and machine learning image processing product featuring artificial intelligence-driven cellular analysis.
Our client’s scientists provided us with AI models trained to recognize different morphologies within various cellular samples:
|
|
We’ve run those models on GPU processors since those are faster for the AI, unlocking even more substantial performance benefits.
Modernize Medical Technology for Speed, Storage and AI Insights
The seamless integration of AWS services addressed the immediate concerns while paving the way for a new era of efficiency and collaboration in high-speed lab instrument image processing. This success story illustrates the healthcare product design and cloud transformation and migration expertise that Cardinal Peak brings to its clients, providing innovative solutions that can even lead to breakthrough discoveries that save lives.
If you are looking to modernize your outdated software solution to unlock speed, storage and efficiency, our team of AWS cloud experts is ready to partner with you on the next stage of your cloud adoption journey. Tell us how we can help!
