Mark Hampton, CTO and Co-founder of Certess is joining us today here on Think Verification to give us a glimpse of what Functional Qualification, a breakthrough in the concept of a complete verification environment, is all about. So without further ado - here’s the full interview you can’t afford to miss:
Think Verification: Hi Mark, thank you so much for taking the time to be with us today, I’m sure your schedule is pretty busy. So where did we catch you today?
Mark Hampton: Shinagawa in Tokyo. I’ve been spending quite a lot of my time in Japan during the last year.
Think Verification: Before we start our technical discussion, could you tell us a little about your background and what made you join Certess’ executive team?
Mark Hampton: Certess was founded by three engineers Joerg Grosse, Mel Gilmore and myself, all from the user side of EDA, having designed and verified ASICs before starting Certess. We had a common desire to build a successful company. I’d had an ongoing interest in some basic market analysis techniques and we used some of those to explore different ideas during about 6 months, before focusing on the current technology which turned out to be in EDA. This process was challenging but I believe it was more realistic than waiting for the right idea. Certess is relatively unique in EDA for at least a couple of reasons. Firstly we are solving a problem that has existed for a long time, rather than an emerging problem in the EDA flow. Secondly we haven’t overhyped the technology and have a solid solution.
Think Verification: So, Certess seems to be a pioneer in the Functional Qualification area. I’m sure most of our readers have heard the buzzword in the past but don’t really know what’s behind it How would you explain the essence of Functional Qualification?
Mark Hampton: Verification quality has never been well measured by EDA tools. The metrics provided in the past do not guarantee that the verification would be effective in finding potential design bugs. Functional qualification really does measure the ability of the verification to find bugs. It does this by automatically inserting faults or artificial bugs into the HDL design and then analysing the verification environment to see if they can be detected. Our tool, named Certitude, can provide a global score, the percentage of faults that would be detected. Or Certitude can present details of specific faults that can’t be detected so the user can then improve the verification. The concept is very simple, I’m the CTO so it had to be !
Think Verification: Where in the current verification flow would you fit the Functional Qualification process? And more importantly, could any part of the so-called traditional verification flow become redundant if Functional Qualification were to be applied
Mark Hampton: Before verification starts we suggest running Certitude in the metric mode on any 3rd party IPs. The resulting scores can be used to judge if the IPs are well verified before being integrated into your system. Once your verification environment has been built and the first testcase is passing then Certitude can be used. Initially Certitude can be used to identify problems with checkers in the verification environment such as coding errors or missing checkers. Later in the project, during verification closure, Certitude can provide more detailed information. Based on this testcases can be augmented by the user to improve the propagation of potential design bugs to
checkers. At the end of the project Certitude should be run in metric mode to get a final qualification score that reflects the quality of the verification.
It is perhaps important to highlight that functional qualification is not trying to do verification. It is measuring how effective an existing verification environment is. Verification is a very expensive activity and without a good measurement of verification quality it will remain that way. The first step in improving something efficiently is measuring it, until now the industry had forgotten this principle in the verification space.
Certainly after adopting Certitude some clients are no longer using code coverage and have reduced the amount of functional coverage code being developed. But this is only part of the information Certitude provides. Information about things like missing checkers or poor propagation is completely unique to functional qualification.
We can imagine the undetected faults found by Certitude as virtual silicon bugs. If the designer makes a mistake in related functionality it could really get to silicon with the current verification. With such an effective feedback on verification quality engineers can gain skills much faster. This leads to a more effective verification methodology in future projects.
Think Verification: So if I understand correctly, Certitude can give you an objective and unbiased feedback on the quality of your verification environment. That is a real breakthrough. I remember this one time a couple of years ago, when a critical bug was discovered at the lab long after unit level verification had been completed. Management was curious (hm.. I mean, angrily eager) to find out how on earth the verification team had missed it. Quick investigation showed that the checker had been modified. Obviously a tool like Certitude would have saved us My concern, however, as with all automatic tools, is the amount of extra work that might be required - integrating the tool, analyzing the raw reports, getting rid of false alarms, etc. How do you cope with this challenge?
Mark Hampton: Like all EDA technologies functional qualification is not a silver bullet. The key is building a methodology. Certess has been helping clients do this for nearly four years now. Today’s profitable business units, which will be in business tomorrow, know how to do verification. But they also know that verification keeps getting tougher. Functional correctness is critical to enable plugging together massively complex chips. Functional qualification provides an objective measurement that helps teams improve their verification skills faster. When deployed this accelerates the entire organization’s ability to reliably deliver working chips on time. You only need to leverage from a fraction of the qualification information potentially available to ensure you are achieving higher quality than teams who don’t have access to this technology. Certitude presents the most valuable information early and this filters most of the potential noise. One of the key points is to focus on minimizing project risk - not blindly aiming for 100% of a metric.
Functional qualification is typically integrated into the verification environment within 1 day - it really is that simple. But leveraging efficiently from this new information source requires a fresh perspective on verification improvement and this is where our field application engineers help clients adopt more effectively.
Think Verification: Mark, you mentioned before that Certitude is addressing a problem that has actually existed for a long time, at least as far as Functional Verification is concerned. In your initial market analysis you must have talked about that as well. What do you think took the industry so long?
Mark Hampton: Functional qualification is derived from the field of mutation analysis in software testing and I think a lot of the researchers in that field are asking the same question. The hardware world followed in the footsteps of the software world regarding the adoption of code coverage. In the early 1990’s the HDL flow didn’t have the maturity to question that choice. Mainstream software testing typically isn’t trying to achieve the same levels of functional correctness as hardware systems. I had felt the pain when working on hardware verification projects. The idea of injecting faults in the design to check the verification is relatively obvious but there are performance challenges that Certess addresses in unique ways. These innovations allow the technique to scale to industry size projects.
Think Verification: Can you tell us a little bit about your clients and perhaps share with us some of the feedback you’ve been getting from them about their experience with Certitude?
Mark Hampton: Today there are 14 clients in 4 continents with over 50 different design sites. This is not mainstream technology yet but it is widely deployed in several organizations. The leading user is STMicroelectronics, they were the first adopter and have now deployed company wide. Other clients I can name include Juniper Networks in the United States and Toshiba in Japan. The most rewarding feedback for me is to hear clients explaining how this is impacting the culture of verification in their company. We see the direct link between improved quality and improved productivity. Customers are claiming to see an improvement in the quality of their chips and in overall productivity. For example they are able to allocate verification resources more effectively.
Think Verification: And lastly, as the CTO of Certess, how do you see the future of Functional Qualification and Certess in the industry?
Mark Hampton: I believe functional qualification will become common sense in the verification community. I hope there will be a strategic advantage for some of our early clients, once you have an edge in the quality improvement process it is very difficult for competitors to catch up. Toyata and Ford in the automobile industry are stricking examples of this.
I also hope that functional qualification will touch industries beyond EDA. The broader software community is becoming aware of the cost of poor verification - it touches diverse issues from security to maintainability. I would love the world’s most popular operating systems to be even more robust !
Globally I would say the EDA industry today is not a great environment for startups. I’m from the EDA user camp and clearly EDA technology is commoditized. Increasingly users are not looking to EDA for business differentiation - EDA is outsourced as a cost reduction exercise. Today Certess is in the fortunate position of being the only company offering functional qualification.
Certess is focused on point tool solutions in the EDA space. We have a couple of active research projects exploring new applications of functional qualification. We are the first to have the opportunity to imagine industrial applications for this technology so I think we can get by on simple ideas for quite some time yet.
Think Verification: Thanks a lot for sharing all this with us today Mark, that was really insightful. In fact, I’m now curious to try out Certitude myself. What I find really exciting is the ability to measure my testbench effectiveness and possibly find ways to refine my existing methodology with the help of an objective tool. So once again, it was a pleasure to host you here on Think Verification and I wish you and Certess much success in the future!