LogScale

(formerly known as Humio)

BEST PRACTICES

Based on Vijilan’s proven approach to log analytics and security operations

Abstract:

LogScale (former Humio) provides a solution for effective log management. It enables real time, large-scale log analytics, a necessity for Security Operations (SecOps), and DevOps. Vijilan, a longtime LogScale (former Humio) partner, offers a unique LogScale (former Humio) implementation process based on best practices and its own proprietary technology developed to make LogScale (former Humio) work optimally. This paper explores these best practices and highlights what they mean for a SecOps and DevOps team who want to improve their logs collection, log storage and log analytics. While this paper focuses on SecOps, the methodology for logging and storing log in LogScale (former Humio) for DevOps is performed in a similar fashion.

Introduction

Effective log management is a critical, though often elusive, element of Security Operations (SecOps). Successful incident response is impossible without it. However, getting on top of logs from multiple devices, users, applications, and other log sources can be a significant challenge for a SecOps team. Humio offers a solution, one that delivers real time, large-scale log analytics. Making Humio work requires a focused, meticulous implementation. Vijilan, a longtime Humio partner, has developed a proven set of best practices for Humio implementation based on numerous client engagements over the last several years. Vijilan augments these practices with its own proprietary technology. This paper explores Vijilan’s Humio implementation best practices and highlights what they mean for a SecOps team that wants to improve its log analytics and incident response capabilities

Overview: Log management, analytics and SecOps

Log management challenges in SecOps

Log management presents several challenges to SecOps teams. In terms of requirements, log management demand fast ingestion of large amounts of log data from a variety of sources at great velocity. This is easier said than done, and some solutions require the SecOps team to set up server clusters to manage growth in log data sources. As log data volume grows—and it usually grows exponentially—it may also be necessary to deploy indexing farms. Storage requirements ballooning the process, too. Performance can lag, which may negatively affect the security analysis that is the point of the whole endeavor. SecOps may need to assign engineers just to the work of managing and updating the infrastructure that supports log management.

Humio: Solving log management challenges

Humio is a modern log management platform designed to process the scale and complexity of today’s log management workloads. Humio is designed with two key differentiators which make real-time analytics at scale possible: Data-streaming in an index-free architecture and high compression storage. These two factors combine to enable Humio users to ask anything and get instant responses from log data.

Best practices for Humio implementation

Vijilan has assessed and validated Humio as a log management solution for applications and security for over four hundred organizations in finance, healthcare, government, manufacturing, legal and education. A growing set of best practices has emerged from these experiences. When Humio is deployed in accordance with best practices, the client enjoys optimal usage of Humio, in terms of SecOps results, utilization of IT assets and team productivity.

Vijilan uses Humio for long-term storage, analysis, and reporting of log data, leveraging Humio as a Security Information and Event Management (SIEM) solution with advanced analytical capabilities. Through this partnership, Vijilan can provide a complete, end-to-end SIEM with log collection, parsing and normalization, real-time detection, analysis, and reporting

LogScale can run ultra-fast searches and queries against raw log data in seconds and presenting the information visually to users. The query window generates results from inputs ranging from a simple text-based search to a multi-variable structured query of billions of records within seconds.

Core processes

Best practices for Humio integration start with a series of core processes:

Determine log integration requirements: current and future

Determine alerts and align with SecOps staffing and workflows

Document Humio log analytics solution for specific SecOps parameters

Operationalize log collection

Determine alerts and align with SecOps staffing and workflows

Some of these are not directly related to log management and analysis. The broader SecOps context matters for getting log management and analysis right. Humio implementers should understand where log management fits into the SecOps team’s mandate and obligations, e.g., if real time ransomware detection is a priority, then decisions about which logs to integrate into Humio ought to align with that priority. In this way, SecOps’ strategy and goals form a baseline for the Humio implementation project.

Determine log integration requirements: current and future

The Vijilan approach to Humio implementation includes the development of an architecture definition that includes project requirements. As part of this architecture definition, the Humio implementation team must determine log integration requirements: which logs will be integrated? What connectors and other technologies will be needed? What are the parameters of the log data parsing and normalization? This thought process and development of requirements should apply to current and future log integrations alike. It may be difficult to know future requirements for sure but thinking through likely scenarios for future integration is a best practice, one that will pay off as the Humio instance inevitably evolves over time

Operationalize log collection

The implementation process continues with the setup of log collection. This will involve the deployment of threat sensors and cloud connectors. At the functional level, what is happening is the creation of a data lake into which a huge, diverse amount of log data will flow. Making this work means ingesting and normalizing all the various log data streams and then “shipping” them into Humio. Vijilan has a pre-built solution for on-premises and cloud log collection. A best practice at this stage is to collect historical log data in addition to real time and recent data flows. Forensic analysis of old logs can reveal critical information about past attacks, which may have not been detected. Having access to historical log data in LogScale can also facilitate predicttive threat detection going forward

Determine alerts and align with SecOps staffing and workflows

Vijilan collaborates closely with clients on determining alerts for Humio users. It is essential that alerts align with SecOps staffing and workflows. For any alert, there needs to be a SecOps team member to deal with it. or the alert needs to feed into an automated incident response system, such as ITSMs olution. These matters because alert fatigue is a major issue in SecOps. Flooding the team with alerts is never optimal. Indeed, the practice can cause important threats to be missed. The best practice is to map the staffing and workflows carefully and design alerts to match those parameters.

Design a log analytics dashboard in alignment with SecOps staffing and workflows

Visualization of Humio log management is a key to SecOps effectiveness and team productivity. Vijilan works with Humio implementation clients to design and deploy customized dashboards. The best practice is to follow a dashboard development process that includes widget design, mapping of log sources, validation, and documentation.

Document Humio log analytics solution for specific SecOps parameters

Documentation tends to be neglected in IT projects, but that does not make it any less important. A Humio implementation must be documented to some extent. Because Humio already offers documentation of its solution, there is no need to draft a big book to go along with the implementation. A succinct piece of documentation, which highlights the parameters of the implementation, will prove itself to be extremely valuable as the Humio instance expands and evolves.

Run test cases and provide feedback to SecOps team

Setting up Humio is not a push button process. The solution needs tuning. The best practice, therefore, i is a continuous and iterative process with well-defined milestones and goals. As Humio processes the test cases, Vijilan provides feedback to the SecOps team. In this way, Vijilan can get Humio properly tuned while simultaneously mentoring the SecOps team on how to improve their log analytics and incident response processes.

Identify and train key personnel

People will need to make Humio work once it’s deployed. If the client does not already have staff trained for Humio, it’s a wise idea to use the implementation project period to identify and train key personnel. Management and admin for Humio does not have to be a full-time job. It can be part of someone’s role in SecOps. But someone will need to take responsibility for monitoring Humio and its supporting infrastructure.

Assess the suitability of outsourcing log monitoring

Outsourcing log monitoring can be a best practice for some organizations. The decision to engage with an external service provider for this workload may arise from a lack of personnel on staff or a desire to focus human resources elsewhere. Given the constraints on SecOps staff and the general difficulty in finding competent employees in this specialized work, letting an outsourced provider handle log monitoring may be a good move.

Vijlian’s toolset offers a range of security analytics capabilities integrated with LogScale functionality, including

About
LogScale

LogScale is a modern log management platform that is purpose-built for today’s complex systems and scale. Humio is built with two key differentiators which make real-time analytics at scale possible: Data-streaming and index-free architecture and high compression storage. This enables customers to ask anything and get instant responses. The Humio platform also has a robust ecosystem that integrates with technologies (such as Vijilan) that will allow Humio to be used as a security solution among large-scale enterprise and education institutions.

About
Vijilan

Vijilan Security has over fifteen years of experience specializing in monitoring, detecting, and responding to information security incidents. Vijilan is a US-based Limited Liability Company found in Aventura, Florida. Vijilan’s primary Security Operations Centers (SOCs) run 24/7, collecting events from private and public networks globally. Vijilan operates and stores all US-based customers’ information in the USA. This company is led by a team of engineers, developers, and highly skilled information security professionals, and services more than 500 small and medium businesses (SMB), focused on Banking, Health Care, Government and Education, in the US, Australia, South Africa, Brazil, and the UK.

Conclusion

Getting Humio to work effectively means following best practices. With Vijilan as an implementation partner, an organization can realize its goals for real time, large-scale log analytics. SecOps will improve as a result. Best practices include determining log integration requirements for the present and the future, operationalizing log collection and solving the problem of “log shipping” and determining alerts and aligning them with SecOps staffing and workflows. A well-designed dashboard is essential, as is the process of documentation. Vijilan offers its own unique, proprietary toolset to facilitate the implementation in accordance with these best practices, and others, such as running test cases and training key personnel. As these factors come together, a successful Humio implementation will be the result.

LogScale

(formerly known as Humio)

BEST PRACTICES

Based on Vijilan’s proven approach to log analytics and security operations

Abstract:

Introduction