2024 Trend Preview: Cloud Visibility and Automation

(Editor's note: This is a multipart series in which Futuriom profiles the trends we expect to drive cloud technology in 2024, as well as a preview of some of the companies that are being considered for our 2024 Futuriom list. The new report will be published at the end of January.)

Visibility is essential to making the most of cloud applications and services. You can’t manage what you can’t see, as they say. And in today’s cloud-native environments, there are multiple methods of obtaining information on infrastructure health – including network telemetry, application monitoring, API data, log data, and security diagnostics – to name a few. Combined with data streaming for real-time retrieval, along with artificial intelligence and machine learning (AI/ML), visibility becomes observability, or performance and security predictability emerging from comprehensive data collection across multiple sources.

The shift to hybrid cloud and multicloud infrastructure (another one of our top trends) has made it imperative that IT operators and developers collect a composite picture of their systems to manage and orchestrate services across multiple domains. They also need to collect the data to fuel automation with AI/ML. But until recently, that meant wrangling lots of disparate software to get the necessary views and information – all while risking downtime from assorted failures and inefficiencies. Legacy systems based on open-source monitoring via Prometheus were troubled by the scale of data rushing into enterprise IT coffers.

Many innovative startups leverage visibility to create automated responses to alerted conditions and trends. This too is key to the successful operation of cloud services. The ability to identify what’s happening on a cloud network is needed to optimize its performance and effectiveness. But the observability spade is also coming under some criticism because in some cases it can become expensive – and generate lots of data, with sometimes vague results. The most successful observability companies will tie data collection and analysis to actions, in some cases automation.

There are many approaches to cloud visibility and automation. One product may be network oriented, while another may specialize in the application environment. Still others are focused on cloud infrastructure, including microservices and Kubernetes. Some incorporate multiple iterations of these. Following are some key terms used in this area:

• Network observability
• Full stack observability
• Network orchestration
• AIOps
• Cost optimization
• Performance management

Network Observability and Automation

For cloud networking companies, visibility is crucial to managing connectivity. For instance, Aryaka’s SD-WAN and SASE service automatically orchestrates network traffic for optimal performance through its global network of PoPs. Meanwhile, it gathers data from network and security devices, identity systems, and applications to generate User and Entity Behavioral Analytics (UEBA). This approach uses AI/ML to set baselines that help identify abnormal user and device behavior.

Through its CoPilot interface, Aviatrix provides an overall view of the multicloud networks it generates. CoPilot maps all connections enabled by the Aviatrix Controller through multicloud and hybrid cloud networks. Security and diagnostics are incorporated to enable troubleshooting.

IP Fabric is a network analytics platform that discovers and creates visualizations and models of network infrastructure, including cloud environments. The goal is to provide network visibility and assurance that performance and security are up to expected levels. IP Fabric automatically highlights points of network weakness that could potentially cause problems.

Kentik offers a network observability platform that gathers large amounts of data about network traffic on premises, in hybrid and multicloud environments, and over the Internet and correlates it and enriches it with information from multiple sources (e.g., telemetry, routing, traffic monitoring, metrics, performance testing). The SaaS also applies machine learning to reveal potential problems in real time.

In terms of network automation, Itential has been a leader in the field of NetDevOps by building low-code and no-code automation platforms for the NetDevOps markets. The vendor’s Automation Platform uses APIs to integrate and automate the orchestration of physical and virtual networks. It ties together IT service management, inventory, analytics, and orchestration tools and supports end-to-end and closed loop network automation. (Closed loop refers to the ability for a system to self-correct based on continuous monitoring of network status.) Itential is a partner with some of the network observability companies such as Kentik.

Cloud Observability and Automation

As noted, network visibility is one aspect of cloud observability. As you move deeper into the cloud-native stack, there are many other ares to monitor and analyze. Following the success of cloud visibility and applications monitoring giants such as DataDog and Dynatrace, there are emerging specialists in the realm of cloud visiblity.

Chronosphere was founded by former Uber lead engineers Martin Mao and Rob Skillington in 2019. The company helps end users deploy Prometheus and other open source software tools, such as OpenTelemetry (OTel) and Grafana, to gather information on events in cloud infrastructure, including Kubernetes and microservices. The goal is a full-stack observability tool that enables developers to identify where workloads can be adjusted for greater efficiency and cost optimization. Chronosphere is also part of the cloud-cost optimization trend. Early last year, Chronosphere claimed to have tripled its annual recurring revenue (ARR) and achieve 145% net revenue retention. Customers include Robinhood, Snap, Obsidian Security, DoorDash, Zillow, and Visa.

Selector correlates metrics, logs, configuration, events, and alerts from multicloud and hybrid cloud environments, using network, application, and security data sources to achieve “full stack” visibility. AI/ML is applied to gathered data to detect anomalies and issue alerts – a function called AI for Operations (AIOps), which refers to using AI/ML in IT operations. The user interface features a Copilot tool that responds to natural language queries. Selector’s platform is integrated Slack, Microsoft Teams, ServiceNow, and similar programs.

Some companies offer specific types of visibility. Engflow’s DevOps solution streamlines the building and testing of source code for cloud or on-premises applications by caching results across multiple machines and regulating resources as needed. In the process, Engflow also produces observability metrics related to source code builds. The goal is to create a faster, more efficient method of producing cloud applications.

Cloud visibility and automation remains a highly technical but high growth area of cloud technology. As enterprises continue to adopt cloud infrastructure, including both public cloud infrastructure and private cloud infra, this segment of the market will increase in importance, and the number of firms involved will expand. And in particular, the visibility and automation of cloud networking will play a crucial role in the spread of hybrid and multicloud environments.