( DVcon 07 Item 6 ) --------------------------------------------- [ 04/24/07 ]
Subject: Nusym "DeNibulator"
THE OTHER OTHER COVERAGE TOOL -- These are the comments on the other "other"
coverage tool. (In the preceeding section of this report, I wrote about
finding the Certess Certitude other coverage tool.) On a close read the
users here are reporting about a Nusym demo; not actual use. And since I
can't find any mention of the name of the mystery Nusym tool anywhere, I've
decided to christen it the Nusym "DeNibulator" -- a safe made-up name as
anything else. What does it do? It denibulates your RTL code! What? You
haven't denibulized your source code yet? Oh, man, your project is in such
big trouble with all those dangerous undiscovered nibs still in it...
I recently saw a demo of Nusym's beta verification product. Here is
my first impression of their stuff.
1) Nusym automates both coverage and test generation. To run Nusym in
this mode, you'll need a testbench that already includes BFMs and
checkers. When you run the test generation part of Nusym, you can
rerun the transactions created with much lower overhead in
regression mode. The tool will still collect supplemental coverage,
so you can look at the regression run and make sure you are covering
everything that you hit with your original analysis runs, plus make
sure you haven't created new coverage holes in your design. Nusym
works in conjunction with testbenches written in Verilog or Vera.
2) If you run Nusym in analysis mode, it will provide some architectural
analysis of your design, giving insight as to how some signals in the
design are being used. E.g. it distinguishes between "persistent"
signals where the values are stored from one transaction to the next,
and "temporary" signals where values are only kept while a transaction
is pending. For this mode, Nusym requires only that the testbench is
able to drive transactions. No checkers are required. It then
outputs a report showing which signals are of which type. Then you
can easily see whether your signals are being used as expected.
3) Nusym's debug lets you runs single, specific transactions to recreate
a bug, rather having to running long sequences or transactions to get
to a bug.
4) Their "path tracing" feature also has some useful debug. If the tool
isn't covering something that it should, it will tell you why and
show you the path. For example, if you have a branch in your code
that is not covered, Nusym will indicate that it can't be covered
because the controlling variable is a constant. Since the constant
may be tied to signals passing through multiple layers of logic and
it may not be obvious, the tool shows you what the original constant
was and the path to get to the branch.
I didn't see any major functionality missing. As it is still in beta,
it does still need some work on usability and polish. For example,
the user must still sift through extra data in the reports, but I
expect that this will be refined as it moves closer to production.
- Steve Pearlmutter of TwoTents Systems
I recently saw a demo of Nusym's verification product. I haven't had
any hands on experience yet, but I can tell you my impressions based
on the demo.
First off, our projects take about a year, and a good 6 months of it
is verification - verification can be bigger than the design effort!
Our current verification methodology is:
1) Write the testbench, borrowing from the previous project but
adding support for many new features with each chip.
2) Write a series of directed tests at the block and chip level
that sift out many of the easy bugs.
3) Do random testing for different configurations and packet types
using Synopsys VERA and VCS. These random tests always run a really
long time before finding any faults. Plus they are harder to debug
than directed tests, because the tests have larger traces to look at,
and a lot of the errors that pop up are just illegal configurations.
So we have to sort through the errors to see if they are real.
Based on the demo I saw, there are two ways to run Nusym:
1. You can use it to build a test bench from scratch. Building the
verification environment can be like building the chip (!) and the
Nusym environment lets you do it very quickly. Nusym looks at
your design up front and automates the test generation, trying for
100% coverage. For example, it will look specifically at a coverage
point, e.g. part of an IF clause, and automatically generate the
test to exercise it.
Nusym's algorithms look at the design and automatically generate
tests, e.g. covering every IF statement in your design. For the
small test design (simple state machine) that they demoed, I was
impressed with how quickly it created test stimulus that would
exercise the design completely. It will be interesting to see
how it scales. Most current formal verification tools have a hard
time scaling to larger designs, but since Nusym is more of a
simulator, it should scale much better.
Nusym has a command line interface to drive the tool. They don't
have a GUI yet, but that doesn't really matter because with
verification tools, we tend to use the GUI the first time to see
the features, then we write scripts to actually run the tool.
2. The second method is to use Nusym with your existing environment
(test bench and checker), targeting 100% coverage. Most testbenches
are flawed and can't reach all coverage points. Nusym very quickly
tells you what kind of coverage you can get. Nusym said they will
support Vera and Verilog for the first release. It will read in
testbenches in Vera and analyze the design to determine the coverage
points. If it can't reach a certain state, it will give you a trace
to the original problem in your testbench.
I've found with most formal tools you need to write a description of what
your design is supposed to do in their language, and it is not always
easy to understand how to do this. Plus it is hard to write assertions
that completely describe the design you are trying to test. Unlike formal
tools, Nusym doesn't require you to learn a new language or change our
methodology, which I liked. We can leverage our verification environments
by reusing them from project to project. Nusym has the potential to save
us a lot of time there and get a better quality result.
- [ An Anon Engineer ]
I saw a demo of Nusym's verification tool recently, and wanted to share
what they are doing.
Functionality: From the demo, Nusym does automated coverage closure. It
automatically directs the random constraints so as to target coverage
points. It can also direct you to bugs. It turns an error statement
into a coverage point, and then targets it as if it were a coverage
point -- i.e. automatic bug finding.
Nusym could save us from having to modify our constraints to improve
coverage. It could also improve our verification environment by
simplifying the generation component that you must code, because we
wouldn't need to put in hooks for modification later. It automates the
process of going between coverage goals and determining what constraints
to modify, then modifying the constraint.
Automated coverage closure versus Formal Verification:
There are 2 types of formal algorithms
1) those looking for exhaustive proofs
and
2) those looking for counter examples, e.g. traces showing what's wrong.
Nusym is like a bug hunting formal tool - except that unlike a formal tool
it can work with testbench constructs, plus it is not just limited to
synthesizable RTL.
If the Nusym tool does what it claims, it could potentially replace the
bug hunting algorithms of formal tools, but not the proof algorithms.
Features from the demo:
- Debug. Nusym's has a pretty neat debug feature that gives short and
specific tests for functionality, and lets you replay the bug. With
traditional simulation, you might find a bug after 4 hours of run
time. Then when you fix it, you must run the 4 hours of simulation
again to make sure it works.
Nusym says it not only helps you find the bug in minutes, but it also
only takes minutes to rerun it again.
- Replaying non-bugs. Nusym also lets you replay non-bugs (which has
its place but is not as useful.) You can use it to document what
the design does if you don't believe the checkers, or if you want
to show the design to someone else.
- Replaying regression. It makes sure everything keeps working as you
get the design more stable. You must run millions of test vectors to
run a regression today, which takes hours and hours. Nusym claims it
is precise in terms of which sequence of test vectors hit the coverage
point, which saves you from running the other 95% of test vectors.
Nusym works with Open Vera testbench constructs and Verilog 2001. They
also claim to be working on testbench constructs for System Verilog. It
should be easy to come up to speed on, rather than having to learn a new
methodology as you do with formal tools. I am looking for them to add
support for System Verilog Assertions (SVA) next.
- [ An Anon Engineer ]
Index
Next->Item
|
|
