( ESNUG 475 Item 2 ) -------------------------------------------- [09/18/08]
Subject: NuSym, Certess, Mentor Infact, Jasper, Avery, OneSpin, Dafca
NUSYM KICKS ASS: Hands down, the one tool/company that ruled DAC this year
in terms of user mindshare was the NuSym DeNibulator. Certess Certitude
seems to have a little "me, too" effect going for it, but it's NuSym that
rules. The other independent bug hunters (Jasper, Avery, OneSpin) got some
mention; while nobody noticed Mentor's 0-in nor Synopsys Magellan at all.
"What were the 1 or 2 or 3 INTERESTING specific tools that you
saw at DAC this year? WHY where they interesting to you?
(If any were under NDA say it and I'll keep you anon on them.)"
---- ---- ---- ---- ---- ---- ----
NuSym DeNibulator is the most interesting new tool I have seen this year
at DAC. It solves a major problem of constrained random verification
flows by providing an automated way to reach coverage closure. Avery
Insight in the same area looks less advanced. Technology however should
be applied with a lot of care to the checking system of the environment.
With respect to this, it is complementary (they are not at all "rivals")
to Certess Certitude. I would consider Certitude newly interesting this
year, since we are using it for more than 3 years. DeNibulator without
Certitude is useless.
Breker Trek & Mentor Infact were also interesting, even if already known.
This is another possible alternative as well to solve CRV limitations by
going more directed, but "intelligently".
- Olivier Haller of STMicroelectronics
---- ---- ---- ---- ---- ---- ----
I found NuSym's tool the most interesting at DAC this year because they
demonstrated actual accelerated coverage closure within several
otherwise conventional verification environments. Verification coverage
-- functional, code and assertion -- typically approaches 100% in an
asymptotic fashion. NuSym is able to morph that convergence to
essentially linear. In a constrained random verification environment
their technology appears to infer additional generation constraints from
the coverage goals, accelerating coverage closure. For an autonomous
verification environment -- one in which all verification constraints
required for achieving coverage goals are embedded in the environment
rather than distributed among a set of tests -- this is an ideal
enhancement.
- Andrew Piziali, author of "ESL Design and Verification" (2007)
---- ---- ---- ---- ---- ---- ----
NuSym DeNibulator:
Essentially a RTL simulator where instead of the $random function returns
not pseudo random number but instead returns numbers that gives you the
highest coverage of the RTL DUT. Their algorithm trys it's hardest to
cover the most lines of code with the smallest number of input vectors.
Seems to work extremely well and in cases where it can't find inputs to
cover a line in will give feedback about why it can't. Downside to this
tool is that the entire test bench has to be written in System Verilog,
and many things are extremely dependent on Perl to generate input.
- Norm Zhou of Ambarella
---- ---- ---- ---- ---- ---- ----
My #3 was NuSym, which claims to be able to automagically generate
tests to reach uncovered parts of a design. This would obviously be
an amazing breakthrough if it works, but I am skeptical. What I saw
did not appear to be ready for prime time, and I am classing this as
'probably too good to be true'. We will likely do at least a limited
evaluation in case it is real. I would also class this as a possible
candidate for 'biggest lie' if it actually is vaporware.
- [ An Anon Engineer ]
---- ---- ---- ---- ---- ---- ----
1. NuSym's tool
2. Infact (Mentor)
3. none
- Satoshi Namematsu of Mitsubishi Electric
---- ---- ---- ---- ---- ---- ----
The NuSym touted "intelligent verification" capabilities piqued my
interest and the hype seemed warranted.
- [ An Anon Engineer ]
---- ---- ---- ---- ---- ---- ----
Most verification uses a pseudo-random coverage-driven methodology. The
"coverage-driven" portion still requires a lot of manual work, even after
coverage points have been identified. Typically several fairly directed
tests need to be written to hit coverage points that "luck" did not reach
despite billions of simulation cycles have been thrown at the design.
NuSym should change this flow by providing intelligent automation that
links stimulus generation with coverage results. It should enable users
to get better coverage results in far less simulation cycles and help
them detect and debug potentially unreachable coverage goals.
I have not evaluated the tool on a real project, so cannot really say if
the technology is mature enough to deliver results on real projects
rather then just a hand-picked DAC demo, but if the tool keeps its
promises then this is true innovation that makes a lot of business sense.
- [ An Anon Engineer ]
---- ---- ---- ---- ---- ---- ----
NuSym looks very interesting, it promises to help close the gap in CDV
where the user has to figure out how to write test cases to hit specific
coverage items.
An evaluation of the tool is our next step.
- [ An Anon Engineer ]
---- ---- ---- ---- ---- ---- ----
NuSym DeNibulator:
Upside:
The demo was interesting but not enough technical detail for me to
recommend using it. The concept as I understand it is to automate
hitting and 'generating?' coverage points
This was done by using knowledge of the HDL model to do this.
Cool idea, impressive performance
Downside:
I did not understand what coverage points looked like. In my
experience, coverage points take on many forms. From simple assertions
to complex sequences built into score boards and cross products of
various 'coverage' points.
It was not clear how the feed back was supposed to be done.
The killer in my opinion and in the opinion of the few people I talked to
about it was the requirement that the NuSym simulator had to be used.
Given the state of the System Verilog simulators (not mature) supplied by
the big name companies it would be considered high risk to introduce
another unknown into the verification mix.
If I misunderstood the role of coverage points then the NuSym tool becomes
little more than another way to do model code coverage which is not a
great use of resources.
- [ An Anon Engineer ]
---- ---- ---- ---- ---- ---- ----
NuSym's approach looks helpful to reach a higher function coverage by
hitting the corner case faster.
However, SVA-based verification (which NuSym says they will support in
Q4) is used to verify complicated protocols. I don't know if NuSym's
approach could help designers effectively and efficiently do SVA-based
verification.
- Joseph Ku, consultant
---- ---- ---- ---- ---- ---- ----
I sat in on a demo of NuSym's verification tool at DAC. It looks like
the type of product that can improve verification time with respect to
coverage analysis. The impressive part of the tool is that it generates
coverage points by analyzing the design. We rely on our designers to
write coverage events, which could be coded incorrectly. By removing the
middle man, we should be able to receive always accurate, and more,
coverage events. This is because their tool considers everything a
coverage point. What I am unsure of is if that philosophy also applies
to the stimulus generators, covering all possible combinations of input
commands, or packets, or controls.
The tool is compatable with various verification languages, Vera, System
Verilog, Verilog, and C++ (work just beginning), which is attractive to
us, since we are currently working with a C++ based environment. Also,
the ability to use regular-expressions to limit the coverage zone
targeted by NuSym is great for handling changes to large designs.
Overall, it looks like a really useful product that will help drive home
the effectiveness of coverage based verification.
- Shaun Uldrikis of IBM
---- ---- ---- ---- ---- ---- ----
Most interesting tool: NuSym. NuSym was interesting because it offered
reduction in the amount of time required to close out coverage. This
is one of the issues which drags on before tapeout and a tool which can
shrink this time could reduce my schedule. Other tools made similar
claims, but the other claims contained a certain amount of mysticism
that makes me skeptical of the claims. The NuSym approach seemed to
make sense without the mystic claims and without training a team on a
whole new verification paradigm. I'm not sure how, or if, the NuSym
handles temporal dependencies, but the tool is compelling enough to
require another look.
- [ An Anon Engineer ]
---- ---- ---- ---- ---- ---- ----
Nusym biggest product/marketing error is that it is built in into their
own simulator, and not a coupled product. This means that instead of
offering a verification solution/capability - Nusym actually offers an
RTL simulator and a verification capability coupled together. This on
top of legacy test benches and other validation collateral will be
significant headaches to any verification team.
Needless to say all other capabilities that hook to the simulator through
PLI will pose a question mark on integration with Nusym's.
I can't wait to see how this will pan out.
- [ An Anon Engineer ]
---- ---- ---- ---- ---- ---- ----
NuSym leverages simulation data to either increase functional coverage
or to tell the user why the coverage is low. It tells user whether the
issue is in the design or in the test-bench itself. It appears to be a
good step towards reducing the verification effort.
- Bhanu Kapoor of Mimasic
---- ---- ---- ---- ---- ---- ----
NuSym - intelligent testbench: Basically, NuSym aims to generate tests
based on the random variables of the testbench and on the design
specification and structure, and automatically adapts tests to target
specific functional and code coverage points. It complements, rather
than replace, existing verification tools and methodologies, and don't
require a new language. They support SVTB to TestBenches, and Verilog
for design, they say they will have support of SV design later and
depends on customers pull.
The (-) is that you need to compile the model using their compiler, they
add some annotations to the simulator, and they need to run their own
simulator, which may be a problem. Their methodology is to use the
existing methodology to get the 80% coverage and then to use NuSym to
get higher coverage.
The tool and technology look promising, interesting to watch progress.
Avery Insight - semi-formal: Insight is aimed for bug hunting and
coverage closure by combining logic and symbolic simulation with formal
solvers.
Insight uses assertions, constraints, scoreboard checkers and coverage
points from SVTB and SVA to drive formal analysis to interesting areas
for bug hunting. Insight has its own simulator, which may be a (-).
They still do not support SV designs; they plan to have it by end of
the year. In a typical use model, a validator will use a conventional
RTL simulator to run constrained-random simulation, get to a given
coverage level, and then turn to Insight for a deeper look into the
design.
They are in some kind of Beta, will be interesting to watch progress.
Certess Certitude - functional qualification: The Certitude tool aims
to measure the quality of validation environment by inserting
faults/mutations to the design and see if they are caught.
They insert one fault at a time in the RTL model and then check by
simulation if the fault is detected by existing checkers or not.
They did not show any major progress over past year but they claim to
support more types of faults, and bigger customer's base.
- [ An Anon Engineer ]
---- ---- ---- ---- ---- ---- ----
My #1 most interesting was Certess, which makes a tool that measures
quality of a DV test set by inserting faults and seeing whether they
are detected. This is far better than any previous coverage metric I
have seen, and the tool actually appears to be pretty 'industrial
strength' in terms of completeness and usability.
- [ An Anon Engineer ]
---- ---- ---- ---- ---- ---- ----
1. Certess - Certess sells a verification coverage tool named Certitude
which uses a method of mutation analysis. The theory behind the coverage
tool is that they can grade coverage of the verification suite by
injecting faults or changing operators in the design, and then checking
if the verification suite can detect them.
Certess sponsored a presentation from Prof. David Dill at their suite
with only minimal shilling. Dill spoke about his research group's work
in "Perspective-Based Verification".
- [ An Anon Engineer ]
---- ---- ---- ---- ---- ---- ----
Jasper - I get the sense that they've turned another corner in terms of
end-to-end capability, and completeness coverage.
- [ An Anon Engineer ]
---- ---- ---- ---- ---- ---- ----
3 - OneSpin Solutions 360 module verifier: guides in verification
planning, generates SVA properties, identifies gaps and guides
verification. Also has high capacity.
- [ An Anon Engineer ]
---- ---- ---- ---- ---- ---- ----
2. DAFCA - for building a methodology & tools around an old and accepted
concept and productizing it. Many design teams custom-build this kind of
post-silicon debug instrumentation into their chips; having a structured
methodology around it could be a winning play.
- [ An Anon Engineer ]
Index
Next->Item
|
|
