GATT Benchmarking Development forum

[W1nTry]

This thread was formally 3DMark Vantage, however i've taken this opportunity to turn it into a development discussion lookinf forward for a weighted benchmarking scheme to determine overall ranking of rigs as opposed to just 3DMark.

Verification Apps
CPU-Z (1.50) (CPU, cache, memory, mainboard, spd) must be shown
GPU-Z (0.3.3)

Gaming/Overall Benchmark Apps
3DMark06 - Run @ 1280x1024 Default settings
3DMark Vantage Run @ 1280x1024, Default settings
PCMark Vantage Run @ 1024x768, Default settings

Memory Benchmark
SiSoftware Sandra - Within this suite are MANY benchmarking, stress testing and information tools. We would be interested primarily with the Memory Bandwidth Benchmark

HD
HDTach - Run with Long Bench (32 mb zones) N.B. Must be run as Administrator in Vista and in Windows XP SP2 compatibility mode.

Stability Apps (not part of the results system, just to verify validity of results as REPEATABLE)
OCCT - Linpack Medium, 16-20 minute run
ATI Tool - Used to detect Artifacts
Linkpack/Linx
Prime95 32-bit - Stress Test with as many threads as there are physical cores
Prime95 64-bit - Stress Test with as many threads as there are physical cores

Then we establish a weighted system to calculate a final score.

[Spazosaurus]

I dont agree with the system must be stable bit. If a man wanna do a max oc to post his ultimate score, he should be able to do so.

[W1nTry]

OK we'll discuss for now but as soon as it's finalized i'll remove all these 'conversational posts'. So why shouldn't it be stable? Doesn't that already apply to the 3DMark06 rankings that determines the top rig in gatt? I mean if I had the time i'd develop the weighted system... in fact let's discuss a weighted system now. I will expect feedback from you, arc, n00b and all others on gatt that benchmark as we will develop a suite of apps that are easy to acquire, tried and tested and the process for such. So first we decide the apps to use for verification:

CPU-Z
GPU-Z

Wanna add aything to that?

[Spazosaurus]

CoreTemp or Everest preferably coretemp as its free. im not sure if there's a non paid version of Everest? (Arc can clear up)

Also, multiple tabs of CPU-Z. E.g one instance showing the cpu, one the memory and even another one showing the mainboard.

(More suggestions for verification programs are welcome)

As far as system stability is concerned, I can speak for myself that my 4ghz oc is not entirely stable. The rule of thumb around the forums is to run 10 LinX tests for 'guaranteed' stability. My thread will show that my 4ghz overclock does indeed pass that test, but when put under extreme test to setermine MAXIMUM stability (read 100tests plus), it fails. So can THAT be considered "Stable"? What is stable? 100%? 10%?

Most of us are aware that some of us (you know, those of us who are not 'p*$$ies) overclock systems to the MAX so that it is just stable enough to successfully run benchies. For a lot of us, benchie speed is NOWHERE near what we would run our systems for everyday use,If we WERE to go down the stability road, i'd say a minimum of 100 LinX tests utilizing at least HALF of your installed memory should be the standard, but I wont recomment going down that road.

[Nephilim]

i think stable is a relative term here.

what might fail a 20 pass at linpak might pass a 06/vantage. my system runs 15 passes and fails. but everything else runs without fail, and im hard pressed to decide...should i tweak more to get more passes stable? or should i be content that the system has been running fine, gaming, multitasking etc for the last few days?

So perhaps before we say 'it should be stable' we should define what 'stable' is for our purposes

^what he said

[W1nTry]

Ok, now we don't want to have TOO much variety in the verification programs since standardization I think is a must. What about Prime95 for stress testing stability? I for one also know that my system fails prime95 when OCed, hence just as with Capn and Linpack 10 vs 100 there has to be tolerences for what 'stability' means. We need a vote on if marginally stable system score are acceptable or if some semblence of stability should be shown. My OC fails prime95, but there s a built in stress test in AOD that I can runn 100% utilization for over an hour and it's ROCK solid. So we will have to devise what we consider 'stable'

E.g. run Prime95 or Linkpack or AOD for x minutes without reboot/failures. Run a PI calculation program (whilst not an app to 'stress' a CPU, it DOES reveal instability when the calculations fail or reveal errors). We'll address 'STABILITY' shortly. Let's continue to develop the verification app list first. I will add to the FIRST post the results of our discussions.

[Netizen1]

How about we post the latest version along with the URL to acquire the said version?

e.g.

CPU-Z (1.50) http://www.cpuid.com/download/cpuz_150.zip
GPU-Z (0.3.3) http://www.techpowerup.com/downloads/1363/TechPowerUp_GPU-Z_v0.3.3.html

[TriniXaeno]

The KISS approach to development might apply here.

[Kraeoss]

I dont agree with the system must be stable bit. If a man wanna do a max oc to post his ultimate score, he should be able to do so.

world records are made @ suicide lol so if it passes the bench then it's good even if it bsod's right after pics are taken lol

[Spazosaurus]

Ive been out of the Prime game for at least a couple years now but I do remember enough to know that there is no specific test you can run that stops after a predetermined period of time. Linpack of LinX on the other hand is capable of that. Now im not pushing it down yall throats eh.

That said, I think 10 tests in Linx should be enough to prove at least low to medium stability.

That coupled with the gpuz and cpuz shots should be sufficient.

Fullstop.

[W1nTry]

You've raised another issue... GPU stability... you can complete 3DMark Benchmarks and suffer artifacting! that should not be counted... now here's where honesty comes into play as you'll have to either fess up or lower the clocks. I don't think insane conditions should be considered, meaning if you rig is border line stable (aka passes benchmarks and crashes or artifacts) those results should NOT be acceptable. If we do decide to go that route of accepting borderline stable, then I will insist that a run of no less than 5 passes should be made (aka a batch run with 5 repeats for EACH test). That would increase both the time the benchmark takes as well as stress on the system to at least show the results are SOMEWHAT REPEATABLE!

As was Originally intended we should also have performance based benchmarks, I like SiSoftware Sandra for Memory benchmarks (we'll ignore latency and cache) and perhaps HDtach for HDs. Audio is DIFFICULT to benchmark especially with the issues with Soundblasters and Vista so we'll ignore sound unless someone can recommend a 'turn-key' application for testing audio.

Last I would like 1 game benchmark (this will be more controversial) e.g. Farcry 2 has a built in benchmark app.

[Kraeoss]

occt 3.0 you have linx pack from IBT and a P95 derivative type test you can also stress your gfx card and psu lol

[Kraeoss]

crysis too but wrt the hdd benching not everyone has a raptor or a super fast un-fragged drive so... we could leave that out for now ?

[Arcmanov]

Everest is not free.  You will have to acquire that one.  There are some very nice memory read/write benchies
built into it.  Current version is 5.0.

As for the standard benching apps, I think its all been covered.

As for game benchmarks, we have to standardise the version of the game being used.
I think Far Cry 2 already has about 2 patches out.

[Spazosaurus]

I think we going a wee bit too far. W1n, I understand you point on artifacts, but there's no easy way for someone to prove that they dont have artifacts on a level of overclocked that they achieved. We just have to have faith our members that they wont post scores in which artifacts occur. We cant police every single last detail.

As for cpu stability, seeing as it can work with both AMD and Intel cpu's now, can everyone agree that 10 passes of LinX is sufficient in proving at least moderate stability and go with that for 'stability proof'?

[Nephilim]

yeah we going lil too far.

the futurmark tests
cpuz, gpuz with validation, and 10 pass linX should be good.

a game bench and prob Everest after and we shall call him..'george'

[W1nTry]

I don't have a raptor myself nor do I intend to get on or a SSD drive for that matter but as was originally the benchmark idea it was supposed to be the entire rig as the 'gaming' experience is best when one combines not just the best CPU and GPU but also sound and supporting hardware i.e. HD, Sound Card, speakers, keyboard & Mouse, Screen and even layout. I have scaled back tremendously from that rather rigorous undertaking and HDTack s a 'turn-key' benchmark for HD so I don't see a reason to leave it out.

Since Everest is NOT free, I suggest Sandra for the memory benchmark and stability that LinX that capn talking bout, once I can run it on the various AMD rigs that I have sure I don't see a reason to not use it as the stability verification engine.

Lastly PCMark Vantage test the PC overall to see how it performs in apps, gaming, everything. It's free to download and I don't see a particular reason why it can't be included. As Arc said, we'll have to decide on what game though, but I think synthetic benchmarks whilst great cannot replace actual IN GAME experience. So I vote FC2.

[TriniXaeno]

FC2 as in the freely available FC2 DEMO that allows benchmarking???

I think these apps should be free and readily available.

What about having your cake and eating it too?

Why not have both a Amateur Benchmark Top Rig and a Hard Core Benchmark Top Rig list.

The Amateurs can get in the door by simply running 3dmark 2006. Simple and sweet. If the machine can survive the benchmark and screenshot taken with accompanying cpuz info, then voila, score accepted.

The Hard Core benchmark will be a more elite list, with rigorous testing as is being decided here.

We could look to doing something as an incentive for those who jump through all the hoops to earn a certified spot on the GATT Hard Core Benchmark Top 20 list. (An OC'er T-Shirt, Medocino Core amulet, Brazilian chick, etc...)

[W1nTry]

I was always an advocate for incentive for jumping through hoops, just never found the time to create a long, detailed guide to the benchmark. Even the suggestion for various catergories instead of just best of the best, at least giving credit to the older rigs, etc. We can still bring all this up here and then create appropriate threads for each.

[Nephilim]

all the vantages are run once and then pay for.

so you gonna have to use your acquire-no-jutsus