Interview with Technology Evangelist | Podcaster | Speaker | Blogger at Logz.io
Dotan started his career as a developer, moved to become a System Architect and Director of Product Management. This gives a broad understanding of various aspects of app development, which is extremely relevant for this blog where we focus on app quality in a broad sense. Today, Dotan is a Technology Evangelist, Podcaster, Speaker and Blogger
On LinkedIn you present yourself as Principal Developer Advocate and Technology Evangelist. Can you describe what you mean by these titles?
Essentially what this means is that I help the developer community use the products at my company Logz.io, as well as with the broader open source projects in the domain, in my case DevOps and observability domains by sharing knowledge about tools, best practices, design patterns, etc. I also keep watch on the community pulse to make sure our own products meet the needs and desires of the community. I make sure it's a bidirectional channel where I can bring in the actual feedback and voice of our users into our company practice.
How do you see the place of logs in monitoring and solving issues?
Logs are an essential part of monitoring obviously, maybe the oldest and most established method of understanding why things happen in our system, especially why things go wrong. This is due to the fact that the developer who wrote the specific piece of code, essentially creates the output for what is going on in that piece of code. Therefore logs are just a very rich and verbose piece of information, and are the basic foundation for monitoring. It is important to say that logs today are not enough. You need to obtain fuller observability. Logs are one essential pillar, but they are usually accompanied by metrics and traces as well. These are the classic pillars of Observability upon which, together with potentially additional signals, one can get the broader picture of what goes on in the system.
Where do you see logging platforms in 5 years?
Logging has traditionally been very focused on the human aspect of the users. As I said, the developer outputs what goes on in the business logic that he wrote. Usually, it was outputted as plain text for humans to read. Plain text logs that are created for humans to read, they were unstructured and simply not scalable. These days when you have such a massive amount of logs in today’s distributed systems and microservices, we must use automated systems, and I foresee that these automated solutions will continue to grow and develop and we will see that people will be increasingly using log management systems and automation to aggregate, analyze and visualize log data. For that purpose we will see a growing shift going from unstructured to structured formats, plain text to machine-readable formats such as JSON. We will see more work toward log enrichment with metadata in order to attain better system insights, we will see more advanced centralized log collection, ingestion, indexing, and management. Instead of developers having to traverse hundreds of servers, there will be one centralized place where one can visualize, analyze, register, and create automatic alerts based on the logs. This shift is already underway and will accelerate. Going beyond that, as I said, logging will become much more integrated with the other signals in the observability domain, everything from the correlation between logs and metrics, logs and traces, as well as traces back to logs, and by traces I am referring to Distributed Tracing. This is another way in which I foresee logs will be integrated with other signals and events. The final direction perhaps in which I see for the future development of logging is through AI and machine learning. I believe that they will take an increasingly larger role with respect to analyzing and processing logs efficiently as well as proactively because as opposed to the constraints people face when attempting similar processes, AI and machine learning systems have the power to develop and advance incomparably faster.
I know that you are a big champion of Open Source. On your LinkedIn profile you write that your passion is: Tech, Innovation, and the power of communities & open source. I would like to hear your reasoning behind this, but maybe you can start by talking about what limitations you experience with Open Source.
I am a great advocate of Open Source. It’s hugely popular. Many developers already have the skill sets, so you have lots of users who are very familiar with the platforms, and a massive community pushing it forward. There are however several issues with it. The first thing that comes to mind is a common misconception. Often people think Open Source is equivalent to free. It is not zero cost. Yes you may not need to pay for a software or service license however you still need to invest in developers, devops, to install, operate, manage and scale. It is important to understand this. Many times new startups will begin with Open Source, as it is relatively manageable and cheap, however as they scale out they realize that the do-it-yourself model of Open Source is not as trivial as the organization scales out.
Another thing that I often encounter with individuals who are just starting to use Open Source, is as they adapt it to use in production, they realize that Open Source projects are really focused on one core capability. They put less focus on peripheral or administrative capabilities such as user and role management, team collaboration, single sign on (SSO), easy deployment, and so on. These features, I call them commercial grade or enterprise grade, and are not usually an inherent part of Open Source Software, but are needed to run things in production environments. Some vendors develop these additional capabilities on top of the open source project: so Grafana Labs or Elastic, or my company Logz.io develop an additional wrap-around to Open Source to provide another layer of enterprise grade capabilities that is applicable for businesses in production as well.
Recently there has been a security breach with log4j, which is also based on Open Source. Is Open Source less safe than Closed Source?
The Log4j CVE was definitely painful for many of us, including for us in my company. Actually many Java products are based on log4j, but we are happy to say that Logz.io is based on Logback, so we are not actually users of log4j, so specifically for us it was pretty smooth all things considered. Then again, the claim that Open Source is less safe would be one that I would have to disagree with, in principle. I believe the contrary is true. Open Source is safer than closed source. At least the popular Open Source projects are less prone to bugs and security vulnerabilities than closed source. That is thanks to the transparency and involvement of the vast user base and developer community that are constantly detecting, advancing, and fixing issues together. I understand that log4j had this bug for quite some time and no one detected it. However in commercial products the bugs are no less severe, perhaps they are much lower profile only because they are being detected and managed behind closed doors, whereas log4j was Open Source. For known CVE’s out there, I think over 90%, you can see patches and releases out there already, so the turnaround of the community is often much faster than commercial products. I definitely feel very comfortable with the security of open source, and as a developer community and community of software owners, we must remember that at the end of the day it is our responsibility to manage the dependencies and control them just as we need to do for licencing because it is ultimately our architecture and thus our responsibility to mitigate all this.
Elasticsearch has changed their licensing model. What does it mean for the user community?
Elasticsearch has become widely popular for log management and analytics, open source under Apache 2.0 license. Less than a year ago, Elastic B.V. the company that owns and is backing the elasticsearch project has decided to relicense elasticsearch and Kibana to a non-open-source license, it is actually a dual license, SSPL and Elastic License, bottom line, it is non-OSI license (OSI being the organization that essentially defines and validates the open source licensing). For the users, this essentially means that those who were counting on the open source being something they would be free to modify the code and adapt it to their needs, this is no longer the case. So, some users are fine using it as is, closed source. For others, it is important to have the flexibility and freedom to make these sorts of changes. Those who care about continuing to develop and better the software through the community, this will no longer be the case for elasticsearch and kibana. The other pieces of the ELK Stack, if you use the client libraries and the Beats, for example, filebeat, metricbeat, logstash and so on, are still Apache 2.0, however it is important to note that even there they inserted all sorts of version checks to verify the version of the elasticsearch cluster to which they ship logs. This is a breaking change, so if you don’t work with an approved distro, or even an older elasticsearch open source clusters (Elasticsearch 7.10 or earlier), it may break so you won’t be able to actually ship your logs to your elasticsearch distro anymore. So, it is a problem and the community has had adverse reactions. For example the community, after seeing this, then gathered around a fork of elasticsearch and kibana, it’s called OpenSearch, which is backed by significant vendors, the most prominent one being AWS, or Amazon Web Services. My company Logz.io is part of this endeavor, whose goal is to keep elasticsearch and kibana open source. So OpenSearch is really an independent project that started off as a fork, and it is also a very interesting path forward for those who like elasticsearch but who wish to continue using open source.
You work at Logz.io, can you tell us what Logz.io offers?
Logz.io offers a cloud native observability platform that is based on popular open source tools. These include elasticsearch, as we mentioned, now OpenSearch, Jaeger, Prometheus, OpenTelemetry, and others. Essentially, we provide you with the ability to send your logs, metrics, and traces into one managed central location. There you can search, visualize, and correlate across your different telemetry data. So you can correlate your logs with your metrics, your traces with your logs, and so on in one central place. As I said it’s all based on open source so if you are already familiar with the Kibana UI or with Jaeger UI or with Prometheus and Grafana, you won't need to learn some proprietary UI or APIs or formats or query languages, it’s all based on open source formats that you already know and are familiar working with.
Where do you see product quality going in the future?
I think the most interesting paradigm I see is the shift of responsibilities towards engineers. Engineers are owning more of their respective features all the way to production deployment and beyond. Which means, we need to make it easy and accessible for engineers to understand the piece of software that they wrote and how it will behave in production and then once it’s in production to be able to monitor and make sure the quality of the product is maintained. This is why I am very passionate about observability. I believe that observability will play the role of the hero in enabling engineers to have at their fingertips the power to understand the telemetry and state of the system at any given time. They will be able to understand trends over time, even historically. By the way, with Logz.io we understand that you cannot separate the issue of security from devs and DevOps practices, these are currently being called DevSecOps, and they are becoming very integrative. So even down to understanding how my system is vulnerable, and including all of this under one single pane of glass. This is how I see enabling the product quality starting from engineering and going all the way to production and beyond.
Do you find that there is a difference in user sensitivity towards poor product quality between different age groups or across different countries? Has it developed over time?
I think we as consumers have grown used to better user experience and new services. If you look at Saas services, on-demand, PLG (product-led growth), such as Netflix, etc. This user experience as consumers has educated us to expect a certain level of quality from the products that we use, even in our business interactions. This is why I believe that younger generations who are more exposed to newer SaaS offerings, are more informed, educated, and accustomed to these high levels of services. However, in general we, as a population are becoming much more keen in our expectation to receive a high level of quality in our user experience and this is something that we will see become very prominent. This is a nice segway to mention my podcast, OpenObservability Talks, because I am actually going to have an episode this week devoted to SaaS observability which will discuss how SaaS observability is key to enabling this level of quality of experience with SaaS products.
Do you have a dream tool that hasn’t been developed yet?
Well, I have many but I guess the one that is most relevant for these cold winter days that we are having here is a tool that will be able to turn on the boiler on time so that I can have a hot shower when I get home. Certainly for my children, this would be the one that would have the most impact these days. Perhaps an app that would be able to help you detect and avoid exposure to the areas with the highest Covid levels. These are the two tools that would be most useful these days.
It was great speaking to you, you gave us an eye-opening perspective on logs as well as on the subject of open source software. As a developer I have always appreciated open source, and as such Shipbook is open source on all levels, including our SDK, so I found your input very interesting. Thank you for your time.
Shipbook gives you the power to remotely gather, search and analyze your user logs and exceptions in the cloud, on a per-user & session basis.