• The birth of agile telemetry at OVHcloud – Part I
• Agile telemetry at OVHCloud – Part II

In the final post relating to agile telemetry, we focus on the micro-vision. In any project, keeping micro and macro vision separate is vital for effective communication.

Macro and Micro vision

For development teams, micro-vision is crucial. While a macro perspective is important for meeting goals such as delivery deadlines, overseeing the micro details of a development ensures a product is good quality. Here is how we managed to achieve this important goal:

Micro-Vision

This part of our Dashboard displays three important pieces of information relating to the progress of our sprints: the expected, the unexpected and the reason for the unexpected.

(1) This is the progress of the current sprint and its percentage of completion. The percentage of completion is simply computed:

(2) This highlights the number of times the development team has gone out of its sprint to fix something and the completion times. The attached graph shows the peak days when the team was mobilized.

To report this type of information:

• Teams add in the “Label” field on JIRA the title: impediment.
• They log the time spent in the “log work” field on JIRA.

The accumulated information is then visible in our dashboard. This helps us to understand why a sprint does not progress. When teams go through weeks of troubleshooting, we can explain why.

NB: It is essential to separate the sprint from your impediments. These two pieces of information allow you to compare the “expected” and the “unexpected”. This allows us to propose 2 types of graphs.

(3) The last part highlights why the teams intervened when they came out of their sprint. This precious data gives us a clear vision of the situation. If a team spends 90% of its time dealing with a bug, or troubleshooting in support, it informs decisions making processes for future projects.

Example : We provide 30 hours of bug correction time during the sprints; should we focus on creating something new, or reducing the technical debt?

Conclusion : Today, Agile telemetry is used by 19 teams at OVHCloud. It facilitates and allows us to measure the performance and progress of the teams remotely, and in real time. It also provides us with the up-to-date information we need to conduct our projects. It is this methodology that has won Innovations Awards for OVHCloud employees.

If you want to dive into Agile Telemetry you can do it now!! Simply grab our open source dashboard and our bot Jerem.

• Establish an efficient, simple and universally-accepted methodology to support the development teams.
• Set out the different visions necessary for the good handling of a project.
• Define the micro vision, for development teams.
• Define the macro vision, for sponsors.
• Industrialise this innovative concept for other teams at OVHcloud.

As with every new challenge, there was an element of complexity. OVHcloud is an international company, and our teams are distributed all around the world. The tool therefore needed to be easily readable and simple to maintain, and the indicators had to update automatically, as our teams work in a changing context where there are often unplanned events

We needed to create a tool that would allow us to remotely measure the performance and progress of teams in real time, and update the information needed to manage a project. This is our telemetry.

“Before engaging a development team, create your construction plans… draw!” With this idea in mind, I started to sketch a few drafts on a notebook, then on software.

This first sketch highlighted what I wanted to show my teams. We could track our progress on every subject we were working on (the Epics we talked about in part I). We were able to see if the roadmap was achievable, based on the rhythm of the team, and measure the team’s performance. Finally, we had a tool that reconciled the micro and macro visions.

But how do you make this board come alive?

At OVHcloud we use a ticketing tool: JIRA. This tool helps us in our everyday project organisation. However, it doesn’t show the information we need. Nevertheless, we can make it talk…

Based on this concept, we and the teams have built a metrics table with the GRAFANA tool.

This is our Agile telemetry

In this second blog post, we will be focusing on the macro vision. The next post will look at the micro vision part. 😉

Macro vision

This part of the table gives us a MACRO view of the progression throughout the quarter, and the associated charts.

1^st data block: Quarter KPIs

• (A) Predictability. Are you able to deliver all projects/requests? With this indicator, top management and development teams can monitor their capacity regarding deadlines. We are able to monitor this data with the following formula:

• (B) Quarter progression. Shows the completed parts of the quarter.

• (C) Team velocity. The number of story point completed per day. With this information, we are able to built our future sprints, and maintain the team’s rhythm.

• (D) Average impediment. This information is critical to building the next sprint and estimating the security coefficient.

Example

Let’s say we have a seven-person team, with a capacity of 350 hours per sprint (7 people x 5 hours/day x 10 days). If our team logged 58 hours as impediments during a sprint, we could use this information to tailor the security coefficient for the next sprint.

On average, the team use 17% of their time to resolve impediments during a sprint. With this information, we are able to make a security coefficient.

2^nd data block: Sum of Roadmap Epics

This table collects the expected epics. It shows the number of story points evaluated by the teams, the number of points they have already completed, the number of tasks “not estimated”, and the rate of progress. With this table, we can give figures concerning a progression. 

3^rd data block: Quarter Completion

1. Epic completion (1): During each week, we can watch the roadmap’s progression. Sometimes, we will see the completion roadmap decrease. It’s normal, because we have added a story point in our plan. (4)
2. Impediment count (2): This is the number of times the teams came out of the sprint due to impediments.
3. Impediment duration (3): The total time spent on impediments.
4. Delta story point planned (4): The number of points entering or leaving the roadmap, allowing us to see any variations on the charts.

4^th data block: Draw me a graph!

• A.  This figure represents the progress curve of the roadmap. It is created using the sum of the completed epics.
• B.  Our burnup roadmap.

This graph represents two curves:

• a – The curve of the elements quantified and estimated by the teams.
• b – This is the limit, calculated using the velocity of the team on the number of days in the quarter. We then add the team’s safety coefficient.

• c – The curve of the tasks completed by the teams.

Conclusion

This dashboard brings together all the information that we have managed to pull together, thanks to the amazing contributions of the teams. By believing in the values and trusting the method, the group already has a maturity that they have achieved themselves.

For our top management, this board shows all the key information they need. When the predictability is green, all is well. When something is wrong, they are able to take the right decision, whether this involves changing priorities on the roadmap, or hiring new people to help teams successfully complete it (for example).

In our next post, I will explain the micro vision in detail. Until then, keep Agile !

Agility concepts from a developer’s point of view

To help you understand our goals for Jerem, we need to explain some Agility concepts we will be using. First, we will establish a technical quarterly roadmap for a product, which sets out all features we plan to release every three months. This is what we call an epic. 

For each epic, we identify the tasks that will need to be completed. For all of those tasks, we then evaluate the complexity of tasks using story points, during a team preparation session. A story point reflects the effort required to complete the specific JIRA task.

Then, to advance our roadmap, we will conduct regular sprints that correspond to a period of ten days, during which the team will onboard several tasks. The amount of story points taken in a sprint should match, or be close to, the team velocity. In other words, the average number of story points that the team is able to complete each day.

However, other urgent tasks may arise unexpectedly during sprints. That’s what we call an impediment. We might, for example, need to factor in helping customers, bug fixes, or urgent infrastructure tasks.   

How Jerem works

At OVH we use JIRA to track our activity. Our Jerem bot scraps our projects from JIRA and exports all necessary data to our OVHCloud Metrics Data Platform. Jerem can also push data to any Warp 10 compatible database. In Grafana, you simply query the Metrics platform (using Warp10 datasource) with for example our program management dashboard. All your KPI are now available in a nice dashboard!

Discover Jerem metrics

Now that we have an overview of the main Agility concepts involved, let’s dive into Jerem! How do we convert those Agility concepts into metrics? First of all, we’ll retrieve all metrics related to epics (i.e. new features). Then, we will have a deep look at the sprint metrics.

Epic data

To explain Jerem epic metrics, we’ll start by creating a new one. In this example, we called it Agile Telemetry. We add a Q2-20 label, which means that we plan to release it for Q2. To record an epic with Jerem, you need to set a quarter for the final delivery! Next, we’ll simply add four tasks, as shown below:

To get the metrics, we need to evaluate each individual task. We we’ll do this together during preparation sessions. In this example, we have custom story points for each task. For example, we estimated the write a BlogPost about Jerem task as being a 3.

As a result, Jerem now has everything it needs to start collecting epic metrics. This example provides five metrics:

• jerem.jira.epic.storypoint: the total number of story points needed to complete this epic. The value here is 14 (the sum of all the epic story points). This metric will evolve whenever as the epic is updated by adding or removing tasks. 
• jerem.jira.epic.storypoint.done: the number of completed tasks. In our example, we have already completed the Write Jerem bot and Deploy Jerem Bot, so we have already completed eight story points.
• jerem.jira.epic.storypoint.inprogress: the number of ‘in progress’ tasks, such as Write a BlogPost about Jerem.
• jerem.jira.epic.unestimated: the number of unestimated tasks, shown as Unestimated Task in our example.
• jerem.jira.epic.dependency: the number of tasks that have dependency labels, indicating that they are mandatory for other epics or projects.

This way, for each epic in a project, Jerem collects five unique metrics.

Sprint data

To complete epic tasks, we work using a sprint process. When doing sprints, we want to provide a lot of insights into our achievements. That’s why Jerem collects sprint data too!

So let’s open a new sprint in JIRA and start working on our task. This gives us the following JIRA view:

Jerem collects the following metrics for each sprint: 

• jerem.jira.sprint.storypoint.total: the total number of story points onboarded into a sprint.
• jerem.jira.sprint.storypoint.inprogress: the story points currently in progress within a sprint.
• jerem.jira.sprint.storypoint.done: the number of story points currently completed within a sprint.
• jerem.jira.sprint.events: the ‘start’ and of the ‘end’ dates of sprint events, recorded as Warp10 string values.

As you can see in the Metrics view above, we will record every sprint metric twice. We do this to provide a quick view of the active sprint, which is why we use the ‘current’ label’. This also enables us to query past sprints, using the real sprint name. Of course, an active sprint can also be queried using its name.

Impediment data

Starting a sprint means you need to know all the tasks you will have to work on over the next few days. But how can we track and measure unplanned tasks? For example, the very urgent one for your manager, or the teammate that needs a bit of help?

We can add special tickets on JIRA to keep track of those task. That’s what we call an ‘impediment’. They are labelled according their nature. If, for example, the production requires your attention, then it’s an ‘Infra’ impediment. You will also retrieve metrics for the ‘Total’ (all kinds of impediments), ‘Excess’ (the unplanned tasks), ‘Support’ (helping teammates), and ‘Bug fixes or other’ (for all other kinds of impediment).

Each impediment belongs to the active sprint it was closed in. To close an impediment, you only have to flag it as ‘Done’ or ‘Closed’.

We also retrieve metrics like:

• jerem.jira.impediment.TYPE.count: the number of impediments that occurred during a sprint.
• jerem.jira.impediment.TYPE.timespent: the amount of time spent on impediments during a sprint.

TYPE corresponds to the kind of recorded impediment. As we didn’t open any actual impediments, Jerem collects only the total metrics.

To start recording impediments, we simply create a new JIRA task, in which we add an ‘impediment’ label. We we also set its nature, and the actual time spent on it.

For the impediment, we’ll we also retrieve the global metrics that Jerem always records:

• jerem.jira.impediment.total.created: the time spent from the creation date to complete an impediment. This allows us to retrieve a total impediment count. We can also record all impediment actions, even outside sprints. 

For a single Jira project, like our example, you can expect around 300 metrics. This might increase depending on the epic you create and flag on Jira, and the one you close.

Grafana dashboard

We love building Grafana dashboards! They provide both the team and the manager a lot of insights into KPIs. The best part of it for me, as a developer, is that I see why it’s nice to fill a JIRA task!

In our first Grafana dashboard, you will retrieve all the best program management KPIs. Let’s start with the global overview:

Quarter data overview

Here, you will find the current epic in progress. You will also find the global team KPIs, such as the predictability, the velocity, and the impediment stats. It’s here where the magic happens! When filled correctly, this dashboard will show you exactly what your team should deliver in the current quarter. This means you have quick access to all important current subjects. You will also be able to see if your team is expected to deliver on too many subjects, so you can quickly take action and delay some of the new features.

Active sprint data

The active sprint data panel is often a great support during our daily meetings. In this panel, we get a quick overview of the team’s achievements, and can establish the time spent on parallel tasks.

Detailed data

The last part provides more detailed data. Using the epic Grafana variable, we can check specific epics, along with the completion of more global projects. You have also a velocity chart, which plots the past sprint, and compares the expected story points to the ones actually completed.

The Grafana dashboard is directly availble in the Jerem project. You can import it directly in Grafana, provided you have a valid Warp 10 datasource configured.

To make the dashboard work as required, you have to configure the Grafana project variable in the form of a WarpScript list [ 'SAN' 'OTHER-PROJECT' ]. If our program manager can do it, I am sure you can! 😉

Setting up Jerem and automatically loading program management data give us a lot of insights. As a developer, I really enjoy it and I’ve quickly become used to tracking a lot more events in JIRA than I did before. You are able to directly see the impact of your tasks. For example, you see how quickly the roadmap is advancing, and you become able to identify any bottlenecks that are causing impediments. Those bottlenecks then become epics. In other words, once we start to use Jerem, we just auto-fill it! I hope you will enjoy it too! If you have any feedback, we would love to hear it.

My first action, as a newcomer, was to ask the team to express themselves. I wanted to understand how the group worked, as listening is one of the first steps in the agile construction process. 

When dealing with significant changes, it’s necessary to do some change management and accompany people throughout the process. All too often, the word “agile” is used in a way that sacrifices common sense, becoming the scapegoat when unforeseen events or delays in delivery occur. Reassuring the team and explaining them that agility wasn’t going to make their lives harder, or add an overhead to the daily tasks, was a key objective. We achieved this in several ways:

• Organising several presentations and methodology training sessions
• Holding physical meetings with the teams at Paris and Lyon 
• Explaining how the method would work during a real sprint
• Collectively drafting a guide to our implementation of the SCRUM method 

Once we had validated the fundamentals together, we moved on to the methodology. At this stage, we could begin to add value to the project by using agility.

Value in our agility

After a few sprints using the agile method, our group found some stability, i.e. we had reached a good “velocity”, as we call it in agile practitioner slang. Velocity is an important value, and a determining factor in our ongoing journey towards agile telemetry. 

To quickly (re)define it, velocity is the number of “effort points” the team is able to accomplish during a sprint. This key indicator allowed us to assess how much work our team could deliver over a given period of time.  

Velocity and effort points in our Managed Kubernetes project

Let’s look at a simplified schema of how this method worked in our Managed Kubernetes project: 

Here, we have three main steps. For each of them, we collected its quantitative values, to get a consolidated view of each step’s velocity, in addition to a global view of the whole project. With these velocities, we got a good view of the project’s advancement, and some visibility on the deadlines.

Example :

This agile model allowed us to achieve our goal and deliver the Managed Kubernetes project to our customers on time, and this success subsequently led us to share these good practices with all teams in the Product Unit.

But a second challenge presented itself, as maintaining this process required a lot of discipline, with numerous opportunities for mistakes to be made. And so our next step was to create a tool capable of reducing the risk of error, while maintaining full visibility of projects at both the macro and micro levels. This is what we will present in detail in a future blog post: Agile telemetry by OVHcloud. 

An FPGA is a hardware accelerator, a reconfigurable chip that can behave as custom silicon, designed for a specific application. We use FPGAs as custom network devices for our attacks mitigation system. But FPGA development is quite different from software development: it requires specialized proprietary software and long development cycles.

In this article, I would like to focus on the way we integrate FPGA development to the agile software development workflow used by all other developers at OVHcloud.

Why use FPGAs?

FPGAs are extremely generic chips, they can be used to build circuits for a very wide range of applications:

• signal processing
• finance
• machine learning (classification)
• networking

Their main interest is that you are not constrained by the architecture of a CPU, you can build your custom hardware architecture tailored to an application. This usually brings better performance, lower power consumption and lower latency.

For networking applications, the advantages are:

• a direct connection to 100GbE links: no network interface card, no PCIe link, packets are received directly on the chip
• an access to extremely low latency memory with very fast random accesses (QDR SRAM: each bank allows about 250 millions reads and writes per second)
• the ability to build custom packet processing pipelines to use the resources of the chip at their maximum.

This allows us to handle 300 millions packets per second and 400 Gb/s on a single FPGA board with a power consumption under 70W.

To know more about FPGAs, the ebook FPGAs For Dummies is a good resource.

A traditional FPGA development workflow

Languages used to develop on FPGA have a strong specificity: contrary to standard sequential languages, everything happens in parallel, to model the behavior of millions of transistors working in parallel on a chip. Two main languages are used: VHDL and SystemVerilog. We use SystemVerilog. Here is an example SystemVerilog module:

// Simple example module: a counter
// Will clear if clear is 1 during one clock cycle.
// Will increment at each clock cycle when enable is 1.
 
`timescale 1ns / 1ps
 
module counter
    #(
        // Number of bits of counter result
        parameter WIDTH = 5
    )
    (
        input                    clk,
 
        // Control
        input                    enable,
        input                    clear,
         
        // Result
        output reg [WIDTH-1:0]   count = '0
    );
 
    always_ff @(posedge clk) begin
        if (clear) begin
            count <= '0;
        end else if (enable) begin
            count <= count + 1;
        end
    end
 
endmodule

Modules can be combined together by connecting their inputs and outputs to create complex systems.

Testing on the simulator

A very important tool when developing on FPGA is the simulator: it is complex and slow to test the code directly on an actual FPGA. To speed things up, simulators can run code without specific hardware. They are used both for unit tests, to test each module separately, and for functional tests, simulating the whole design, controlling its inputs and checking its outputs. Here is the result on the counter module:

This is the wave showing the value of each signal at each clock cycle. Of course, the simulator can also run headless, and the testbench can be modified to return a pass/fail result.

A basic simulator is provided by Xilinx, an FPGA manufacturer. More advanced simulators are provided by Mentor or Synopsys. These simulators are commercial and require expensive licenses.

Building the binary

Once all tests pass, it is time to get a binary that can be used to configure the FPGA. The biggest FPGA providers, Intel and Xilinx, both provide proprietary tools for this process. A first phase, the synthesis, transforms the source code into a circuit. The second phase, “place and route”, is a very complex optimization problem to fit the circuit onto the resources provided by the FPGA, while respecting the timing constraints, so that the circuit can run at the wanted frequency. It can last multiple hours, even up to one day on very complex designs. This process might fail if the design is over-constrained, so it is usual to have to start multiple processes with different seeds to have more chances to get a working binary at the end.

Our current FPGA development workflow

Our current development process is very close to a traditional one. But usually, FPGA development is much slower than software development. At OVHcloud, we are able to develop and ship a small feature in one day. We achieve that by leveraging the workflow used by software developers, and by using our cloud infrastructure. Here is the global workflow:

The whole workflow is controlled by CDS, our open-source continuous delivery system. All tests, as well as the compilation jobs, run on Public Cloud, except the tests on board, which are run in our lab.

Using our Public Cloud

The setup of all machines is done by Ansible. There are a few important roles, to install different important components:

• the simulator
• the Xilinx compiler, Vivado
• the Intel compiler, Quartus
• the license server for the simulator and the compilers

The license server as well as development boxes are long-running Public Cloud instances. The license server is the smallest instance possible, the development box is an instance with a fast CPU and a lot of RAM. The simulator and the compilers are installed on the development box. Authorizations to access the license server are managed with OpenStack security groups.

The instances used for simulated tests and for compilation are started using the OpenStack API when needed. It is very important because it allows to run multiple test sets in parallel, for different developers. It is also very important for compilation. We compile our designs for multiple targets (Stratix V FPGAs for 10G and Ultrascale+ FPGAs for 100G), so we need to run multiple compilation jobs in parallel. In addition, we run jobs in parallel with multiple seeds to improve our chances to get a correct binary. As build jobs can last 12 hours with our designs, it is very important to start enough in parallel to be sure that we get at least one working binary.

Running the tests

Functional tests are very important because they validate each feature our designs provide. The tests are developed in Python, using scapy to send traffic and analyze the results. They can be run with a simulated design or with the real design on the actual FPGA boards. CDS is able to run tests automatically on the real boards by booking lab servers and connecting to them through SSH. The same process is used for performance tests.

The result of this infrastructure is that developers can push a new feature on a branch of our git repository, and they will have full unit and functional tests results in 30 minutes. If all is ok, they can trigger the compilation, and have the result tested on board the next day. Then they just have to tag the new package version to have a new release available. After that, the team managing the production can deploy the new version using ansible.

Going further

We have automated as much as possible our process and we use the public cloud infrastructure to accelerate the workflow. But currently we are still using a quite traditional FPGA development flow. Many different approches exist to go further, and as we want to push the FPGA development process as close to software development as we can, we have looked into many of them.

HLS

A very common approach is to use High Level Synthesis (HLS). It consists in using a high-level language to develop modules instead of SystemVerilog. With Vivado HLS, it is possible to develop in C++. It is also possible to use OpenCL, which we have tested on Intel boards. The principle of HLS is to extract the algorithm from the high level code, and then to build automatically the best pipelined architecture on FPGA. But we are doing packet processing, our algorithms are extremely basic. The complexity of our code lies in the architecture itself, to be able to support very high data rates. So we were not able to use HLS efficiently, the code we got was actually more complex than the same function in SystemVerilog.

Chisel

SystemVerilog is extremely low-level and does not allow to use high level of abstractions (at least not if you want the code to be usable by Intel and Xilinx compilers). What we really need to simplify development is to be able to use higher levels of abstraction. We do not need a complex compiler to try and guess the best architecture itself. For that, we have a PhD student currently collaborating on an Open Source project: Chisel.

Chisel is a hardware design language based on Scala. Its main interest is that it allows to use the whole level of abstraction offered by Scala to describe hardware. It is also fully Open Source, which is highly unusual in the hardware development world. It relies on Verilator, an Open Source simulator, for testing. This means that we could get rid of proprietary simulators and have a fully open source toolchain, up to the compilation.

There are currently no open source tools for the place and route phase, at least on the most recent Xilinx and Intel FPGAs. But Chisel can generate Verilog that can be used by proprietary compilers.

We plan to have our first modules designed in Chisel used in production soon. This should help us to have more reusable and easier to write code, and to get slowly rid of proprietary tools.

A paradigm change

The open source community is extremely important to keep making FPGA development closer and closer to software development. A sign of improvement is the progressive arrival of low-end FPGAs in FabLabs and hobby electronics projects. We hope Xilinx and Intel FPGA will follow and that they will one-day embrace open source for their compilers, which could allow to make them more efficient and interoperable. FPGAs are accelerators that offer an incredible flexibility and can be powerful alternatives to CPUs and GPUs, but to democratize their use in cloud environments, the open source community has to get much stronger.