At OVHcloud, we see many different use cases around log management. In addition, we have recently had the opportunity to discuss with several client companies with different approaches and maturity on these topics. Based on these insights, we have made major changes to our LDP solution to address these new ways of consuming logs.
In this blog post we will see the historical features that made the success of this platform and the improvements resulting from your feedback.
Let’s start from the beginning
Why are logs important?
It’s critical you know at anytime, what’s going on into your Information System and applications.. You must be able to understand, analyze and monitor the health of your IT and applications to fix any issue. Log files are created each time an event occurs on your computer systems. They therefore provide useful information about the state of your services and infrastructure. These files come from different sources (physical servers, virtual servers, mobile applications, websites, etc.) and are counted in the millions every day, making them difficult to analyze. From your logs, you can extract valuable information about the behavior of your customers or your systems, and then act accordingly.
How do I analyze my logs?
With so many logs to collect, store and analyse, we know most IT administrators need turnkey integrated solution. That what we had in mind when we designed our platform. Indeed, Logs Data Platform is a turnkey solution, which allows you to collect, store and analyze logs. It supports the different log files, whether they are related to applications, servers or security. You have the possibility to use the log collector of your choice (Syslog-ng, Fluentd, NXLog) or to use our dedicated collectors (Flowgger and Logstash). These collectors work regardless of the source, format or structure of your data. For analysis, you can rely on visualization via Graylog, Kibana or Grafana. For example, in the context of infrastructure supervision, you can monitor logs at the application or server level. By making you benefit from the ELK (Elasticsearch Logstash Kibana) ecosystem, Logs Data Platform is a powerful log analysis solution.
3 ways to use Logs Data Platform
We saw that we could diffrenciate 3 major way to store and analyze logs, depending on the business usecase. So we decided that for each log stream you collect, you can activate any of the 3 following approach (or all 3 at the same time) :
- WebSocket broadcasting allows you to see what’s going on in your application or server in real time. Indeed with this feature our Logs Data Platform allows you to connect different applications or servers to one unique endpoint and make all of them appear in one stream if needed. ldp-tail is able to follow one your stream in real-time with sub-second latency.
- Logs Data Platform allows you to index your logs with a flexible retention period ranging from 14 days to 1 year, which allows you to analyze the data over a given period of time.
- You also have the possibility to archive your logs for a long period of time (from 1 year to 10 years) thanks to the Cold Storage feature of Logs Data Platform. This can be very useful for example within the framework of the GDPR requirements or simply to keep the log history of your infrastructure.
How does it work?
Now, I think you see broadly how you can leverage the platform. Let’s dig deeper on the technologies that power it and how you can leverage them for ingest, query and analysis.
Ingestion
Logs Data Platform is compatible with most of the standard protocols on the market: GELF, SYSLOG, Cap’n’Proto, LTSV, RFC5425, Beats…
Moreover you also have the possibility to subscribe to dedicated collectors such as Logstash or Flowgger for more flexibility.
Query
If you have chosen to index your logs, then you have different ways to analyze the results: you can choose to use one of the visualization tools provided by OVHcloud (Graylog, Grafana or Kibana) or use the Elasticsearch, Graylog or OVHcloud APIs in order to use your own analysis tools.
And the little bonus
Logs Data Platform also allows you to index data other than logs thanks to its Index as a Service feature based on Elasticsearch, you can for example index documents.
Thanks to this feature you can for example create powerful search engines thanks to the performance of Elasticsearch and all this without worrying about the integration of Elasticsearch because the Index as a Service of Logs Data Platform is a turnkey solution fully managed by OVHcloud.
So what’s new in this new version?
- You expressed us the wish to have a more flexible invoicing, so we changed our pricing model to pay-as-you-go. Indeed, pay-as-you-go makes invoicing simpler, more readable and predictable. Moreover, you can now take advantage of thresholds and alerts to improve your consumption efficiency.
- Until recently our logs analysis platform was only available for French customers, now Logs Data Platform is available in all countries and in all languages!
- Security and confidentiality are becoming more and more important in company policies, so in order to comply with your stringent security requirements, we have created the Enterprise Logs Account on Logs Data Platform. Thanks to this dedicated cluster you are totally isolated. It will allow us to offer you brand new features such as the Network Access Control List or customizable retention period.
Moreover
Moreover, Logs Data Platform will soon be ISO/IEC certified.
Indeed, we are only a few weeks away from obtaining the ISO/IEC 27001, 27017, 27018 and 27701 certificates.
What do these norms correspond to?
We won’t go into boring legal details here, but to put it simply:
- ISO/IEC 27001:2013 Certification and ISMS relating to Information security management systems for cloud services
- ISO/IEC 27017:2015 Certification relating to information security controls for cloud services
- ISO/IEC 27018:2014 Code of practice for protection of personally identifiable information for cloud services
- ISO/IEC 27701:2019 Certification and PIMS relating to personal data processing security management
To summarize, these ISO/IEC certifications ensure the presence of an Information Security Management System (ISMS) for the management of risks, vulnerabilities and business continuity, as well as a Privacy Information Management System (PIMS).
That said, if the legal is your passion, you’ll find more details on iso.org.
And that’s not all, at the same time our Enterprise Logs Account offer will be HDS compatible to host health data. You will find more information here.
NB:
A few weeks after we released these improvements on our Logs Data Platform Product, Elastic announced changes in licensing for the future versions of Elasticsearch and Kibana offered as a service.
Other members of the Elastic open-source community announced that open versions of those components will continue to exist. Be reassured that the platform as it exists now is not impacted by the change and we will in the mid term future explore the best options to keep offering your the latest feature of the ecosystem, sticking to our S.M.A.R.T. values.
Contact us
https://community.ovh.com/c/platform/data-platforms
Some useful links
Data Product Manager