INSIGHT / Big Data

Log retentions management

Big Data

About Client

A global full-service, therapeutically focused, clinical research organization (CRO) headquartered in Raleigh (NC) running operations across six continents in more than 110 countries was helped in building a security analytics solution called “Secure Logs” using the latest Apache Metron – that ties big-data into security analytics.

Problem:

Client was facing a host of issues ranging from monitoring the access & activity logs in real-time (more than two dozen servers with about 24000 users accessing them) and scaling the infrastructure to meet the requirement, to business unit heads (who are invariably non-technical / non-networking guys) wanting to see what was happening and how secure the infrastructure was.

Solution Delivered:

MSRCOSMOS designed a solution that was developed in four weeks flat (excluding the requirements gathering & solution discovery phase).Employing some of the modern technology platforms that suited the requirements was key.

Log-retentions-Management-casestudies-img

Solution Architecture

Technologies employed:

1. Apache Metron (Beta 2) – Integrate Kafka, Storm, Elastic Search, etc., with in-built features to separate the data
2. Apache Kafka – Stream processing / handling real-time data feeds
3. Apache Storm – Filtering / Distributed Processing
4. Elastic Search – Indexing & Retrieving
5. Kibana – Data Visualization (Dashboards)

Technical Challenges:

The project could be completed in record time primarily due to adopting the still in beta platform Apache Metron (Beta 2 was out in August 2016) for integrating the several parts of the solution to work in unison and as desired. While Metron drastically reduced the development time from a possible 7-12 months to just one (1) month, since it still being in beta, there is currently no support available. However, as the benefits of Metron outweighed the constraints, there had to be many work-arounds, twists, and tweaks induced to get the desired functionality.

Kafka-Case-Study

Impact of the solution

  • Customer and its various business heads now have real-time visibility into access and activity logs of the more than 25,000 users across the globe
  • Data security increased many-fold owing to the Server Logs – which now even visualize any unusual activity or unauthorized access and helped in arresting breaches and loss of IP.
  • Savings of up to 85% on the development cost (what would have otherwise taken a year was accomplished in just 1 month)
  • The solution is scalable as well as built to easily absorb any dynamic requirements, hence very marginal incremental costs, if any, in the future, meaning further savings in cap-ex & op-ex

We have proven expertise in ensuring security of business IT enviroments for our clients. Get in touch with us to discuss as to how we can help you or send in your queries to info@msrcosmos.com, or T +1 925 399 4218

SHARE: