CASE STUDY
M1 Finance
Intelligent Financial Management Simplified with Automation on AWS
Accelerating Fraud Detection and Enabling Real-time Dynamic Data Querying through an Efficient Data Pipeline
M1 Finance is a groundbreaking online financial services company headquartered in Chicago. The company offers an innovative money management application for intelligent investors who want to access features such as automation, leveraged investing and banking services, all wrapped into one integrated solution. Their mission is to empower people with tools and automation to improve their financial well-being.
M1 Finance has been growing rapidly, scaling its employee count from the mid-30s in 2019 to more than 65 employees in 2020. The company’s platform has also experienced significant growth, with over 200,000 accounts totaling approximately $2 billion in assets and counting. To ensure a secure user experience, M1 Finance aims to provide a seamless interface with robust security and fraud detection.
Industry
Finance, SaaS
Challenge
Develop an efficient and scalable data pipeline on AWS to capture and process login data, allowing rapid querying and significantly accelerating fraud detection to improve application security.
Services & Tech
AWS Cloud, Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift Spectrum, and AWS Lambda
The Problem
M1 Finance’s platform was built on AWS prior to their engagement with Onica. The company had a data warehouse comprising an Amazon Redshift cluster managing just under 400GB of data. The cluster facilitates their internal analytics needs and serves as a clearinghouse for data from backend services as well as third party vendors and other sources.
In addition to the data warehouse, M1 Finance has a data lake using Amazon S3 that serves as a staging ground for loading data into Amazon Redshift and as an archive store. However, the data lake wasn’t equipped with capabilities for ad hoc queries – a functionality M1 Finance was looking for in order to improve their fraud detection capabilities.
With rapid user growth, the company found they were collecting significantly higher amounts of data from internal systems than what could fit in their existing data warehouse. Handling such high data volume could get very expensive with traditional data warehousing. Additionally, the company also had Java applications which had accumulated data over many years that could be highly useful for fraud analytics. This data was trapped in silos of legacy databases and they weren’t able to find a solution to query a high volume of rapidly moving data with their existing system.
Being in the financial industry, M1 Finance takes user data security extremely seriously, and it’s imperative their team has the ability to track system access and usage in order to detect and prevent fraud. The company’s existing process for reviewing system logs was manual and not scalable. As the volume of data and logs continues to increase, the company requires a more efficient process to execute a task that is highly time-sensitive.
The M1 Finance team wanted real-time visibility into suspicious activity so they could respond as soon as possible. They learned about Onica from the AWS Jumpstart program and were impressed with Onica’s expertise and capabilities and found the team had insights and knowledge that could be valuable to M1 Finance’s needs.
The Solution
Onica worked closely with the M1 Finance team to architect and build a solution leveraging AWS services for enhanced automation, visualization, analysis, and reporting of data. The Onica and M1 Finance team worked collaboratively to achieve clarity on the scope of the project.
The Onica team designed an efficient and scalable data pipeline to capture small AWS Database Migration Service files dropped into M1 Finance’s Amazon S3 bucket from their authentication database. Login event data are captured, organized, partitioned and data partitions are registered within an AWS Glue data catalog. The files are compacted into fewer, larger files for optimal I/O performance which can then be queried through Amazon Redshift Spectrum or Amazon Athena.
Onica extended the capabilities of their database by creating a separate data path. This process facilitates a variety of cost controls, access controls and governance that can be applied at the Amazon Redshift Spectrum or Amazon Athena query interfaces. Together, this allows M1’s engineering team and frontline fraud analysts with SQL knowledge to query the data dynamically and in real-time without requiring an engineer’s expertise.
The solution implements Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift Spectrum, and AWS Lambda. Amazon Redshift Spectrum was used because the data was also going to be combined with other data already within an existing Amazon Redshift cluster. Amazon Redshift Spectrum uniquely provides an additional layer where data can live outside of the data warehouse on Amazon S3 but still be queried like it is within the cluster. This allows M1 Finance to avoid paying large costs for querying a large scale of data and still maintain optimal functionality.
“Onica’s team really went above and beyond in ideating and implementing the solution,” said Richard Whaling, Lead Data Engineer at M1 Finance. “A great degree of passion and craftsmanship was visible in the code style and design of the implementation, which is rare to find in these kinds of engagements and really awesome to work with.”
"Onica’s team demonstrated a strong ability to be thought leaders and also be implementation experts. It is rare to find a partner who is so honest about bill rates and targets to come under budget for customers. The Onica team builds trust and confidence that they care about our problems and about producing custom solutions that are highly effective and cost-efficient."
Steve Gall
VP, Engineering at M1 Finance
The Outcome
The new solution streamlined the process significantly, automating what was previously done manually over multiple hours to a practically real-time solution. The design of the solution provides scalability, low cost, and integration with existing data sets. Furthermore, the solution offers great flexibility, replacing a process of having to queue tasks for an engineer with one where data analysts can perform ad hoc dynamic queries. M1 Finance business users now have the ability to build robust queries at scale and are able to address their multi-petabyte size data sources and return meaningful timely results. This provides a significant advantage in fraud detection where every minute reduced in processing time can mean proportionately greater success and improved security.
To help mitigate cost risk for M1 Finance, Onica built alerts for Amazon Redshift Spectrum and Amazon Athena usage. M1 Finance’s team was also pleased with the delivery of the solution that was carefully tailored to their existing practice and unique use case. They felt confident about the way Onica’s team handed-off the solution, educating the M1 Finance team in effective implementation and management.
WHY US
Why Onica
Onica is one of the largest and fastest-growing Amazon Web Services (AWS) Premier Consulting Partners in the world, helping companies enable, operate, and innovate in the cloud. From migration strategy to operational excellence and immersive transformation, Onica is a full spectrum AWS integrator. Learn more at www.onica.com.
