long-term ccache statistics for a portage-dedicated instance

[robbat2]

Migrating data and cleaning up my old desktop display head machine, I decided to check out my ccache statistics. This is a very old cache, having first started 2006-01-13. The oldest item in the present cache is 2008-01-12, but the statistics are valid for the entire period. hits 229k and 834k misses = approximately 21% hit rate. This wasn't any crazy repeated compiling of my own code, just a dedicated ccache directory for Portage to use.

cache hit                         228637
cache miss                        834113
called for link                   100293
multiple source files                526
compile failed                     20645
ccache internal error                 14
preprocessor error                 12425
cache file missing                     9
bad compiler arguments                 1
not a C/C++ file                   39097
autoconf compile/link             183802
unsupported compiler option        34481
no input file                      96690
files in cache                    204344
cache size                           1.8 Gbytes
max cache size                       2.0 Gbytes

Comments

High hit ratio due to FEATURES='test'

[(Anonymous)]

On my system I have a hit ration of about 50% because I have FEATURES='test', and on some packages, the tests take forever, on other (stable) packages, the tests fail. For the first case, the usual scenario is Control + C, FEATURES='-test' and reemerge, and this is the case where ccache comes in really handy.

Re: High hit ratio due to FEATURES='test'

[robbat2.livejournal.com]

The box was purely a desktop display head. It basically ran X, ssh, and a single xterm instance (fluxbox and urxvtd were run remotely).

My copy of ccache for the kernel, or my own development gets much higher rates, >=70%.

Similar

[(Anonymous)]

Hmmm, my ccache statistics is remarkably similar — is this a good or a bad thing? I suppose the bigger the hit rate the better, right?

--hook (stupid certifi.ca OpenID server seems down)

Re: Similar

[(Anonymous)]

Well, here's mine. Hit rate: 0.25.

cache directory /root/.ccache
cache hit 265906
cache miss 795511
called for link 82714
multiple source files 345
compile failed 18453
preprocessor error 15916
bad compiler arguments 9
not a C/C++ file 42239
autoconf compile/link 208573
unsupported compiler option 33121
no input file 67459
files in cache 105217
cache size 1.8 Gbytes
max cache size 2.0 Gbytes

--hook

Re: Similar

[(Anonymous)]

SPSS 17 new Improvements:
- Complete datasets for more reliable analysis using multiple imputation procedure in SPSS Missing Values
- Improved administrative tools to facilitate software configuration
- Syntax editor update makes it simpler to generate, test and correct syntax
- Better integration with Office
- SPSS EZ RFM (Recency, Frequency, Monetary value) helps analysts pick out top customers
- Capable of integrating third party applications, procedures, and graphics packages that are created in R
- Several new algorithms, faster speed and performance

when you need help - http://www.spssstatistics.com

~2.5M chances, 29.35% hit rate, here (from 2004-06)

[(Anonymous)]

cache hit 739238
cache miss 1779705
called for link 241305
multiple source files 718
compile failed 52094
ccache internal error 3
preprocessor error 41296
bad compiler arguments 38
not a C/C++ file 120620
autoconf compile/link 439280
unsupported compiler option 98390
output to a non-regular file 1
no input file 171030
files in cache 172557
cache size 1.8 Gbytes
max cache size 2.0 Gbytes


Figuring your ccache at 975 days (365+365+245, give or take a couple), with 1,062,750 "chances" (hits + misses), you're running 1090 chances/day, average.

My ccache is even older... 2004-06-25 (pretty close to as soon as possible while installing Gentoo, IIRC I installed ccache before I did the first system remerge on an install from stage-one), 4 years plus, total 1541 days, with 2,518,943 "chances", 1634.6 chances/day, average.

The higher chances and higher it rate correspond, and are likely due to three reasons.

First, back then I was running "cheap" memory that could barely keep up with its rated bandwidth. It'd occasionally drop a bit, more often with an active system... like when compiling... so I relied heavily on ccache to even be able to complete merges, on big packages often crashing, rebooting and restarting the merge, getting a bit farther with the help of ccache, then crashing again... two, three, more times before I'd actually complete a big package. Of course that upped my ccache hitcount and chances/day rate too.

Second, I've a much better system now, with 2x dual-core and 8 gig ram (stable now!), and run PORTAGE_TMPDIR pointed at tmpfs. With PORTAGE_NICENESS=19, I can run a huge amount of makeops and with portage 2.2 now automated parallel merges in the background, without significantly killing system responsiveness. It's no longer a big deal to do an emerge --emptytree @system @world every so often... so I do. (Last time was to test the new automated parallel merge functionality in portage 2.2.) Again, that ups both the hitcount and chances/day.

Third, while I'm only running a couple "live" packages ATM, I do run ~arch, and run testing snapshots and live packages fairly routinely. Again, that's going to increase both the hitcount and the per-day average.

So... comparing your figures and mine, it looks like an ordinary Gentoo user can expect perhaps a 20% hitrate on maybe a thousand chances per day, while an active (still non-dev) user may up that to 30-ish percent and 1.5-1.6 Kchances/day. That 50% mentioned in the first comment... that's a LOT of repeat building, rather atypical for the ordinary user, I'd say (tho a dev using ccache on his active projects could well hit that).

I'm glad you posted this! I'd not had a chance to compare with anyone else's figures. I hope a few others post their figures as well! =:^)

Duncan

Re: ~2.5M chances, 29.35% hit rate, here (from 2004-06)

[robbat2.livejournal.com]

I specifically gave the age of the stats and the age of the oldest item. The chances statistic is way out, because of my high uptimes on that box. It basically regularly went 1-2 months without ANY updates. Average peak uptime was 14 days, but only because of a glitch in the disk controller where the box locked up about every 2 weeks depending on use.

It didn't have much on it either, basically just X, ssh and an xterm. xterm over to the G5, and start fluxbox and everything else over X forwarding.

Re: ~2.5M chances, 29.35% hit rate, here (from 2004-06)

[(Anonymous)]

So is the moral of the story that ccache makes sense for normal portage users? Should this be added to the Gentoo docs?

37%, one year old desktop installation

[(Anonymous)]

cache hit                          52146
cache miss                        140702
called for link                    15041
multiple source files                 30
compile failed                      3356
preprocessor error                  2430
not a C/C++ file                    9673
autoconf compile/link              38242
unsupported compiler option         3108
no input file                      14797
files in cache                    259839
cache size                           1.8 Gbytes
max cache size                       2.0 Gbytes

29.9% x86 server since March 2005

[(Anonymous)]

My stable server gets world emerged every week or so.
It runs mail, web, proxy, filtering, and X for monitoring.
It initially had limited drive space hence the small size (I should probably increase that now that it has MUCH more space).

cache hit 25853
cache miss 60535
called for link 5562
multiple source files 54
compile failed 1183
preprocessor error 610
not a C/C++ file 3096
autoconf compile/link 11517
unsupported compiler option 4537
no input file 13636
files in cache 55194
cache size 462.9 Mbytes
max cache size 500.0 Mbytes