Article: 65371 of comp.arch
From: mash@mash.engr.sgi.com (John R. Mashey)
Newsgroups: comp.arch
Subject: Re: DEC/Intel deal and Alpha future...
Date: 12 Nov 1997 04:24:23 GMT
Organization: Silicon Graphics, Inc.
Message-ID: <64bb1n$caa$1@murrow.corp.sgi.com>
References: <63njvi$913$1@elektron.et.tudelft.nl> <63pdfn$95o@usenet.pa.dec.com> <64035e$8si$1@elektron.et.tudelft.nl> <3463E2E4.61FA@EasyInternet.net> <6424p7$n2f$1@murrow.corp.sgi.com> <3464DD62.778B@EasyInternet.net> <642uln$3n5$1@mark.ucdavis.edu> <34655CDF.AFA@EasyInternet.net>
Lines: 75

In article <34655CDF.AFA@EasyInternet.net>, Paul DeMone <PaulDeMone@EasyInternet.net> writes:

|> > :      2) integer performance compared to x86 and other RISCs
|> > 
|> > :        - FALSE: with a few noticably exceptions (0.35 um P6 intro)
|> > :          Alpha has held a consistent integer lead, typically as high as
|> > :          50%, occasionaly a lot more, over other RISC and x86 flagship
|> > :          CPUs
|> > 
|> > Well today it's like 4.29 % (hmmm less then 5%).  (vs c240)
|> 
|>   Well if you want to compare with the spanking new C240 why not try
|>   the latest Alpha results of 18.8 SPECint95 (AS 4100 IIRC);  this


Sigh. "Integer performance" != SPECint.
	"floating point performance" != SPECfp

SPECint is a set of benchmarks that tell you something about integer performance
for certain sizes of code, often with ultra-heroic compiler tuning that
sometimes doesn't happen on real applications, some fo which includes
pattern recognition of the specific source code that breaks real applications,
etc.

SPECfp is a set of benchmarks that tell you something about floating poinbt
code of certain sizes, often with... (etc)

In general, statements of the form:
	X has better integer perofrmance than Y ....
	For example, SPECint....
are wrong.

Reality would be much better served if people would say:
	"Integer performance, as measured by SPECint, of X is better than
	that of Y" to remind people of what is, or isn't measured.

For instance, in an Origin system, the following:
	- Improving the L2 cache speed by 1.5X.
	- Quadrupling the L2 cache.
	- Increasing the main memory bandwidth.
	- Decreasing the main memory latency.

Don't affect SPECint95 very much, since the on-chip caches get fairly
good hit rates.  Nevertheless, in larger codes, in various combinations,
such changes can have noticable effects [of course, all of these tests are
on unnanounced machines, or one we may never actually build].
In fact, a whole lot of hardware features found to be useful i nvarious
real codes don't really help SPECint very much.

I've said this before, and I guess I'll have to keep saying it:
1) SPEC benchmarks are useful; they sure improved the state of the world over
	Whetstone & Dhrystone & vacuous mips-ratings; newer ones will
	continue to be useful, I hope.

2) People have *got* to use them for what they're good for, and not
propagate over-generalizations.

3) The best thing about SPEC is the provision of large amounts of consistent
data: search among the benchamrks for ones that are good predictors for
your own codes, and ignore most of the rest.  The problem is that many codes
are not very well predicted by them, despite being real codes of the
sort that McCalpin cited.

4) Finally, remember that SPECint & SPECfp  (done as 1-CPU benchmarks on
multiprocessors) tell you very little themselves about scaling, i.e.,
a processor A might have SPECfp 1.5X higher than processor B,
but 8 of them might not have higher SPECthruput 1.5X higher than B.



-- 
-john mashey    DISCLAIMER: <generic disclaimer: I speak for me only...>
EMAIL:  mash@sgi.com  DDD: 650-933-3090 FAX: 650-932-3090 
USPS:   Silicon Graphics/Cray Research 6L-005,
2011 N. Shoreline Blvd, Mountain View, CA 94043-1389