No laptop at the control cabinet: scaling device & application management in OT

Welcome to the IoT Use Case Podcast, the channel featuring the latest IoT projects from our implementation partners in the market. Today, once again, we are at the Hannover Messe. We are here in a small Mercedes bus, specially set up for podcasts. With me are Tobias Mühlnikel from Portainer and Jürgen Pfeifer from WAGO. I’m very pleased that you’re both here with us today at the Hannover Messe. Could you briefly describe your highlights from the trade fair?

Jürgen

My highlight is actually this bus we’re sitting in right now. It’s quite an interesting setting: four microphones, a camera. And you’re sitting in the middle of a large hall, right next to the Center Stage, talking about the topics ahead. I find that really inspiring.

Very nice. Tobias, what was your highlight here?

Tobias

In general, of course, the previously scheduled meetings with our strategic partners. It’s always good to see people in person again. Apart from that, from a messaging perspective, it’s quite interesting to see that much more attention is being paid to sovereignty and data sovereignty. There are a few fairly large players in the market now saying: the location of the data, or where it is stored, is becoming meaningful and relevant again. But other than that, the most important thing is seeing partners in person again.

I also found this distinction exciting, between what was discussed on the Center Stage, for example by Roland Busch, also together with Friedrich Merz: the distinction between industrial data and public personal data, and that the same regulation and so on should not apply there. I think this distinction, what is particularly relevant in industry, is coming across very clearly.

Jürgen

I would underline that. We certainly need a special dynamic in order to use machine learning models, AI and so on, and no regulations at that point. But I think that is also a good path, because personal data is not present in most shopfloor applications we think of, where we might want to do condition monitoring or other things. So we do not fall under those regulations. That is a good path.

Tobias

And if it does come to the point where you are looking for computer vision solutions and maybe also capturing a few people in the factory hall, then there are very, very good solutions for anonymizing that, even directly at the edge, so that you do not run into conflicts.

That is exactly what today is about: How can you learn from IT “on the edge”, meaning locally in the plant, for example hosting AI models and similar things, largely based on containers? That is our main topic today. Tobias, maybe you can tell us a bit about this: How did you get into container-based device management, and also application management, in industry? How did you personally get involved, and how did your company Portainer get involved?

Tobias

Software container management originally comes from IT, and today it is nothing new there. It is common practice. In OT, it is still fairly new and currently gaining momentum.

Portainer started in IT with simple software container management, in classic industries such as insurance or banking. Then we found out that this approach, making this technology consumable, meaning easy for the end user to handle, is also relevant for OT, where there are not as many IT experts. That is why there has been very organic growth in this sector.
So four or five years ago, we started moving from IT toward OT, and now we generate a large share of our revenue there.

What role do you personally play in that? What experiences have you had?

Tobias

I originally came in as a former customer of Portainer. I was at Volkswagen in digitalization planning and used Portainer there. Then I worked with the team for more than a year, found out that it was very exciting, and then switched from the customer side to the supplier side.

Those are, of course, the most credible stories. When you have used something yourself and say: The solution is so cool, I want to be part of it, that is a good step.

Tobias

Exactly. And there was not just one supplier involved in digitalization projects, but several. Then there was also a lively exchange with WAGO, and that is how we originally got to know each other a few years ago, Jürgen and I.

Jürgen, over to you: How long have you been with WAGO? What is your role, and how did you come into contact with Portainer?

Jürgen

It has been 25 years. Put differently: a quarter of a century. It is pretty crazy how long I have been with WAGO. Of course, we have not been doing containerized software for that long. As Tobias said, it started slowly and is now perhaps moving slightly into the exponential range, with usage increasing enormously.
At the time when Tobias and I were in contact at his former employer, we could of course also work with containers, but more on a “black-window-and-code-line” level. Back then, it was more something for experts.
We at WAGO come from the automation and PLC environment. There, the work is more engineering-oriented, not development-oriented. And that is why Portainer’s graphical interface appealed to us pretty quickly. We said: This is a level that our typical customer base can also work with well.

Tobias

Jürgen sometimes presents WAGO differently, too. At joint conferences, when people ask, “What is WAGO?”, an object is often pulled out of a jacket pocket.

Jürgen

Today you have it in your trouser pocket.

Now I’m curious to see what you pull out of your trouser pocket.

Jürgen

The WAGO terminal block, of course, always has to be there, and the ones with a spring have this very distinctive click. I think by now everyone who has anything to do with electrical engineering knows them. He is probably similar to me in that regard and has a WAGO terminal block in his pocket.

Very good. But you have also developed this quite a bit further. Especially the topic of containers: I would actually be interested in hearing both of you define it. What is it, how does it work for someone who has not dealt with it yet, who does not know it? Jürgen, maybe we’ll start with you, more from the OT perspective, and then I’m curious to hear what Tobias adds. How do you explain containers to a layperson?

Jürgen

I think the word “container” is already very good, and I am happy that this word exists, however it came about. Because you can explain it nicely using the image of a shipping container.
When we think of a huge freighter with lots of containers on it, the same analogy can be made as with a truck carrying a shipping container. And just as flexible. Meaning: It is encapsulated software that is fully executable in itself, but then needs a base, in other words a host, like a truck or a ship in the image, which provides the capability for the container to be used there. Very briefly and abstractly. Tobias can certainly add to that well.

Tobias

The shipping and container analogy is very fitting. It has sometimes also put us in a tricky position: If you do not specify and say you do software container management, but only say “container management”, people quickly think of physical containers, as in managing 40-foot containers.
But the analogy fits very well because it is also this modular structure in software. In shipping, the port operator does not need to know what is inside a container. It is the same with the operating system in IT: It does not have to determine what is actually running inside the container; the standardized format is what matters in order to roll it out at scale, regardless of the hardware.
This offers customers advantages such as hardware independence, but also portability. That means it can be scaled quickly. Once you test it, you know it works that way. The state can be frozen, but at the same time it remains very flexible to update or roll back.

Could you describe that very practically using an example, maybe also a joint example where you work together?

Jürgen

We have had a Linux operating system at WAGO for 20 years. Then it was always a challenge: People who really knew their way around a Linux operating system could expand the functionalities beyond the basic functionality, meaning the PLC function. For others, this was only possible after a longer familiarization period, but not “out of the box”.
With container technology, you can look at platforms such as GitHub, Docker Hub or similar and ask: What might already be available there? Take the example of Node-RED, which has become quite popular for non-real-time-critical or non-deterministic applications. You only need to pull this so-called container.
The only thing you need to pay attention to is the standardization Tobias mentioned: What hardware do I have in front of me? Is it a 32-bit or 64-bit architecture? Accordingly, you select the image for the Docker container, deploy it, install it on the hardware, and it runs.

Tobias

I think there are very, very many joint projects. One we can mention is on drilling ships in the States, and also in other regions of the world. That reflects very nicely where you can find all of this: where you can find WAGO hardware, but then also our software.
For the end customer, it was very important to be able to use this flexibility. That means: They are not dependent on a hardware or software manufacturer when they want to make changes, but can look at what is available, what they can run on it, and do not have to pay a very large amount of money afterward for change requests in a very closed ecosystem.
In this oil and gas industry, it was about millions for small change requests. The customer is very happy that they have broken open this closed ecosystem with it.

If you imagine it like this: We have the container, and for that we need the ship, so to speak, meaning the hardware. What limits or limitations are there? What are the requirements for this kind of hardware? At some point it will probably become too small. Of course, that depends on the use case. Can you describe the range you work in and the hardware topics involved?

Tobias

It depends very strongly on the end use case. If I want to deploy a very large LLM, it is clear that I need memory for that. If it is a very lightweight solution, then it really goes all the way down to very small embedded devices.
We are allowed to name the customer: in this case, Cummins. Containers also run there on very small Bosch control units. That is exciting because you can then also run on embedded devices. There is also a separate podcast episode with you about that.
For us as a control plane, it is not so decisive how large the hardware is. We are talking here about 10 MB of memory, which should usually be available.

Jürgen

That is the basis for the initial footprint, and then it depends on the application and what it actually needs.

One application: You just mentioned Large Language Models, LLMs. That is currently the topic of generative AI. Many people still confuse it with classical AI, where you train your own model for a very specific use case. Now suddenly you have these transformer models, these foundation models, generative AI. They do require computing power. Do you already have examples or ideas of how generative AI is being used on the shopfloor, perhaps in a container to manage it well? Which use cases are you seeing?

Jürgen

I would preface this by saying that we have many applications where machine learning models or anomaly detection, or whatever else ultimately falls under the umbrella term AI, are already running. They run on the normal CPU of the devices. In this case, they are very often edge computers from WAGO.
The topic of LLMs sometimes requires a bit more computing power, but it can also run on edge computers in a control cabinet. Then they are often Tiny LLMs. For this, additional hardware, so-called AI accelerators, is installed in our edge computers if needed, and for the target market, they offer a good price-performance ratio compared with the edge computer. This does not double the effort and investment, but is a percentage share of it.

Tobias

This special hardware can usually also be routed through to containers and consumed in that way, without leaving IT best practices, and it remains scalable. That is exciting for the end user because they can continue using the familiar environment in which they have managed their software containers so far, and can then simply upgrade the hardware and add it.
Regarding LLMs in industry, we often see inline inspection. Logically, you should not deploy these world models, because that does not scale, simply because of current RAM prices and the hardware required.

Jürgen

I would also advise against using these world models. The more specifically you want to work, and in an industrial context you often want to work very specifically because you want to solve limited tasks, the larger and more extensive the model is, the greater the risk of hallucination.
If it is accordingly “gated content” or predefined LLMs, the more specific it is, the better I can get a specific use case and results.
The use cases are currently not that complex, at least: evaluating the user manual for specific questions or translating error codes. That is limited to the user manual, the error code and the queries. You do not need the really large model for that.

Tobias

It is also important: If the customer is interested in this and wants to implement it, they will often also use an LLM to define the use case or ask a chatbot. Then it is good if you are represented there and the LLM outputs how it should be implemented. It is interesting that such an LLM then says you should build the software stack as a container-based software stack. A control plane was also mentioned repeatedly, for example. It was quite funny to hear a chatbot giving instructions on what it should look like.

An invitation to all listeners: Try that out. Ask exactly how your setup is and how AI would implement it, potentially with containers. Independently of generative AI, back to the topic of containers on the shopfloor: We are in the OT area. What can OT learn from IT? What is different? Which functionalities should be adopted?

Jürgen

Very often, you actually had to go to the control cabinet with your engineering tool, meaning your notebook: notebook on your knees, a 1:1 cable connection to the PLC, and then carry out the program download. Step by step, this improved with network technology and TCP/IP, but it was often the case that even with similar applications distributed across a property or factory hall, you still had to perform the download one after another.
If we think in the IoT context, that we want to connect machines, whether for MES applications or energy monitoring: both during the initial rollout and especially when making changes, this 1:1 relationship becomes extremely cumbersome.
The great advantage that containerization can offer, or what we can learn from IT, is that distribution mechanisms or deployment mechanisms are much more flexible. You can roll out at greater scale and also at scheduled times, so that production is not affected.

Can you tie this flexibility to a specific customer project?

Jürgen

I would come back to what you just said: What can we learn from IT? There is an abbreviation called CI/CD, meaning Continuous Integration, Continuous Deployment or Development.
A customer once asked a colleague of mine in sales: Do you at WAGO support this concept? Fortunately, this question was routed through to Business Development, and we were able to find an answer to it. As a result, this customer now actually develops their PLC program for many similar machines, machine connectivity as outlined earlier, via a CI/CD pipeline and then distributes the applications to several hundred devices, in this case also with Portainer. That is really something where you can absolutely learn from it: tools from IT that can be used well in OT.

Tobias

And that without being ignorant and saying: Now everyone in OT uses different tools and has to be an IT expert. You provide it implicitly as best practices, without every employee needing a two-month training course.
Especially with regard to rollout, but also version control: Previously you had the legendary golden USB stick. The current state is on it, but nobody knows which state is currently running on the machine. This brings additional advantages such as version control, where you not only know what is currently running, but also what you can roll back to if something does not work. These are crucial topics. In IT, this is common practice; in OT, it is sometimes still a small miracle when something like this appears on site, just like a dedicated test environment. A lot is still tested very strongly in production.

Now we have a whole range of technical terms that are not all that common for people in the OT field. Can we go into that in a little more detail? CI/CD: Can you define that? You just mentioned rollback. Can you describe from your perspective what it does?

Tobias

I have different version states in the software. For example, version state A is currently running. Then I do a rollout, want to deploy a new software version, and I am at version state B. But I realize that I am not satisfied with it. Then I reverse course, do the rollback, and I am back at software state A, which worked, and I can do that very easily with technical tools. Something that was, or in some cases still is, very difficult in an older or still common world.
The other topic is that you design the development path and the rollout path in an automated way. That is what IT understands by CI/CD: from creating the software through to delivery, meaning how it is rolled out, automated and accelerated as much as possible.
The topic of test environments is totally relevant in IT. I honestly do not know how OT does this without a test environment, because you constantly have to apply and test new versions.

Tobias

There are definitely customers who have their own laboratory environments. To be sure, you usually need test hardware, because often you cannot simply test it in the cloud, also with actuators and sensors. That is cost-intensive. Surely not everyone can afford that, but they should, because in critical environments you should ideally test once beforehand with real hardware.

When you are in conversations with potential customers: What are the main topics? Are there objections? This IT/OT conflict is sometimes also a personal conflict, where people feel attacked. Do you notice resistance, or is it more a matter of information and explanation, where once people understand it, they use it?

Jürgen

When we think about the use cases, it is very gladly used once the advantages are recognizable. But what still exists, and will probably continue to exist for good reasons, is the OT-IT gap. You can bridge it very well technically, but organizationally it is separated, also in terms of areas of responsibility and cost centers.
Things that annoy us as users of company computers, when an update suddenly runs over them, are annoying in IT. In OT, to put it drastically, that would be fatal. I cannot simply run an update at any time without knowing what the physics behind it is doing. The physical impact in OT is completely different from that in IT.
But there are mechanisms that allow those responsible for operating machines and systems to “gate” these technologies: They determine at what time they allow it. It is not simply pushed through, as we as PC users have sometimes painfully experienced.

Tobias

What we have said is not meant to give IT a position of dominance, but rather to enable OT to use tools and best practices. IT also depends on understanding these relationships. That should not be ignored by saying: We are now taking over the entire shopfloor, and we are not interested in what is happening on site because we are 20 kilometers away.

When you talk to customers: They are not only convinced by architecture; it is often about facts, figures and data. Do you have KPIs that you address? Examples where you say: updates are X times faster, error rates decrease? What figures are on the table?

Jürgen

It very quickly comes down to arguments like: laptop on your knees in front of the control cabinet versus one mouse click. And not only updating my production cell, where I update the IoT gateways or controllers, but also the other, similar production site in a completely different country at the same time. There are enough scenarios where technicians still fly around the world today. You do not have to discuss for long when you present the functions. That is well received.
It is a nice business case when you can calculate travel costs and travel time. It usually pays off relatively quickly.

Tobias

Other topics are the strategic advantage. I mentioned the gas customer earlier: flexible architecture to retrofit later, so as not to be dependent on a solution provider when it comes to taking an additional data point into account in energy monitoring, which suddenly costs two million US dollars. That is transparent to everyone: Implementation and support do not cost two million US dollars.
You can also see this in the cloud: If you make yourself dependent on certain cloud providers, there can quickly be unpleasant surprises in terms of cost increases, automatic cost increases, even though no additional value is being created.
Customers appreciate this independence: Either they build it themselves, if they are able and want to, or a system integrator takes it on, but the flexibility remains.

Jürgen

I would add to that: In the past, it was often the case that exactly one application ran on very OT-close devices. Then you needed the integrator who could open it, who had the version, the software version. Even if you as the owner had stored and documented it yourself, that does not mean you understand the software and can make a change.
With containerized and flexibly distributable software, you can, for example, deploy another container, or a second or third one, which additionally accesses existing signals in read-only mode, performs its own calculations, and delivers other evaluations “on the edge” that the previous software did not provide. Because both are containerized, they do not affect each other. You have modularity in the software, and it can grow regardless of who initially implemented it and how, as long as it is containerized and access to the desired data is available.

Tobias

Exactly, I can then really keep a kind of reserve in the control cabinet, because buying new hardware is effort for the end customer. They do not want to do that every six months for a new application. Then they have to touch the circuit diagrams, and the whole thing has to be installed. Ideally, you do not do that every six months, but maybe only again in a few years.

Suppose someone is now convinced and says: Yes, I’m doing this now. I’m getting started. What are typical mistakes or problems that occur?

Jürgen

When it comes to an OT-close device, perhaps also a PLC, so that you have the possibilities described earlier to retrofit applications, you need to keep access to all signals, to all I/Os, available so that other applications can access them. That is the basic architecture. If I encapsulate it too much and only provide one application, I cannot add a second one. That is something you should consider at the beginning at that level.

Tobias

Beyond that, you should not reduce it to just one component, but ideally address it holistically: keep the whole project in mind, talk to your own security department, IT security, bring everyone on board, because it is often a joint project between OT and IT. It is pragmatic if one team starts, but at some point everyone needs to be brought along. If that happens too late, there are internal conflicts that can ideally be avoided.

Thank you. That was very exciting. Especially this combination of IT, OT and learning from one another is an important topic. Do you have any final additions? Otherwise, my question would be what else you are doing at the Hannover Messe.

Tobias

I have no further additions. Next up is a meetup, our partner dinner, where we will sit down with strategic partners and come together again this evening. Then tomorrow we continue with a hardware manufacturer from Nuremberg.

Jürgen

I am here today to gain inspiration, and I will take another close look at the topic of AI and how IoT and AI are now developing.

Then I am also looking forward to the exchange we are having now, in the IoT Use Case Podcast, also within the team, in the network. See you next time.