robbat2: (Default)

As a recent random time-waster, I went and read all of the bugs in the "Recruitment" product of the Gentoo Bugzilla. In doing so, I found twelve developers (ebuild or other) that weren't listed in our LDAP or historical tracking at all. I added them back now, I have gentoo-core announcements from when several of them joined as well that I double-checked.

The "lost" developers
  • pihta - bug 20756
  • ct - bug 22211
  • srcerer - bug 23184 (retire date approximate)
  • fede2 - bug 25464
  • vlaci - bug 31795
  • teval - bug 36753
  • mccabemt - bug 43029
  • rip7 - bug 46353
  • twk-b - bug 53723
  • dj-submerge - bug 57051
  • little_bob - bug 69742
  • ruth - bug 70469
Other LDAP changes from my review:
  • svyatogor - bug 20756 - updated join date for original docs work, he had commit rights two years before his previously stated join date
  • archaelus - bug 30835 - data fixup
  • apokorny - bug 70188 - add join date
Further plans:

There are 92 developers without join dates. We need to find join dates for them via BugZilla and CVS/SVN. Also audit all join dates for every other developer. Lastly, discover and capture retirement dates for every past developer.

Present statistics: 673 developers total. 247 active, 426 retired.

robbat2: (Default)

Following up on my earlier posting on the AD2000BX/AD1989B SPDIF support being broken, I figured out the required fixes, and they are waiting in the sound-2.6 kernel tree for the next merge window

robbat2: (Default)

Migrating data and cleaning up my old desktop display head machine, I decided to check out my ccache statistics. This is a very old cache, having first started 2006-01-13. The oldest item in the present cache is 2008-01-12, but the statistics are valid for the entire period. hits 229k and 834k misses = approximately 21% hit rate. This wasn't any crazy repeated compiling of my own code, just a dedicated ccache directory for Portage to use.

Raw numbers )
robbat2: (Default)

Setting up the storage on my new machine, I just ran into something really interesting, what seems to be deliberate usable and useful, but completely undocumented functionality in the MD RAID layer.

It's possible to create RAID devices with the initial array having 'missing' slots, and then add the devices for those missing slots later. RAID1 lets you have one or more, RAID5 only one, RAID6 one or two, RAID10 up to half of the total. That functionality is documented in both the Documentation/md.txt of the kernel, as well as the manpage for mdadm.

What isn't documented is when you later add devices, how to get them to take up the 'missing' slots, rather than remain as spares. Nothing in md(7), mdadm(8), or Documentation/md.txt. Nothing I tried with mdadm could do it either, leaving only the sysfs interface for the RAID device.

Documentation/md.txt does describe the sysfs interface in detail, but seems to have some omissions and outdated material - the code has moved on, but the documentation hasn't caught up yet.

So, below the jump, I present my small HOWTO on creating a RAID10 with missing devices and how to later add them properly.

MD with missing devices HOWTO )
robbat2: (Default)
Edit 2008/09/16:

Code fixed now, no specs available yet See my patches here.

Edit 2008/09/05:

A private source that I inquired of indicates that the AD2000B part was only a special run of the AD1989B part. There shouldn't be any functional differences. On the side of a spec sheet, the AD1989B specs should be available "shortly" from Analog Devices.

Original posting:

So in more details to follow, I picked up hardware for a new workstation to replace my G5. The only part of the hardware that isn't working yet, is the digital audio (SPDIF/Toslink) output. My motherboard is an Asus P5Q-Premium, and the specifications claim to have "ADIĀ® AD2000B 8-Channel High Definition Audio CODEC" as the audio chip. This chip is apparently the successor to the AD1988B chip. The analog audio part works fine, just that I use optical to overcome an interference issue on the run between my computers and my actual working area of my desk (with a small digital decoder and stereo speakers).

Digging around in the ALSA drivers, it just seems I need to find a different set of controls to toggle the digital lines to be outputs or enabled - and that this data would be in the public datasheet, just like previous versions of the chip. I submitted a technical request to Asus a few days ago, with no response yet. I also contacted Analog Devices directly. Their customer support referred me to their application engineers, whom I phoned, and they then proceeded to deny the existence of the chip, and I quote: "It's not in my system, we don't manufacture it." That's really interesting, because I've got it on my motherboard!

Either the divisions of Analog Devices aren't talking, or Asus is using chips from a 3rd party that's ripping off Analog Device's trademark amongst other things.

Here's the text off the chip:

AD2000BX
14??793.1
#0816 0.3
SINGAPORE

I tried to take a photo, but it's really annoying and hard to read, without dis-assembling my machine, which I'd prefer not to do at this point.

However, I did find another photo on the web, of the same area from a review of the motherboard. The Analog Devices logo is also clearly visible after the 'BX' portion of the text. From the photo I could make out:

AD2000BX
1383055.1
#0808 0.2
SINGAPORE

If I had to make a guess about it, the chip is AD2000BX, the second line is the serial number, the third is the year and week of manufacturer, plus the revision of the chip, and the last line is the manufacture location.

If you're from Asus or Analog Devices, and you're reading this, where's the datasheet for the chip? Is it a real ADI part? I simply want the public datasheet like the rest of models so that I can fix digital audio output in Linux myself, and contribute it back to the ALSA project.

P.S. The upstream ALSA bug is here. There's no downstream Gentoo bug.

robbat2: (Default)

This is a copy+paste from my email to the gentoo-dev mailing list, simply because some developers and users follow the RSS feeds rather than read email. If you want the bot in your channel and you are a channel founder/lead op, please respond on the thread in the mailing list

Hi folks,

Sorry that it's taken this long to get completed, but the Jeeves
replacement, Willikins, is finally 99% done, and ready to join lots of
channels.

Getting the bot out there
-------------------------
If you would like to have the new bot in your #gentoo-* channel, would
each channel founder/leader please respond to this thread, stating the
channel name, and that they are the contact for any problems/troubles.

Bug reports
-----------
Please open a bug in the Gentoo Infrastructure product, using the
'Other' component, and assign it directly to me.

Custom bot functionality:
-------------------------
Here's all the functionality that we have assembled, beyond the standard
rbot stuff.
Bugzilla
========
!bug [ZILLA] ID
Looks up bug #ID in the per-channel default or specified bugzilla.

!bugstats [ZILLA]
Totals of bugs per the bugzilla 'status' field.

!archstats [ZILLA] [STATUS] [RESO]
Totals of bugs per architecture, optionally with some specific set of
status or resolution values, comma delimited.

status = OPEN, DONE, UNCONFIRMED,NEW,ASSIGNED,REOPENED, RESOLVED, VERIFIED, CLOSED
Reso = FIXED, INVALID, WONTFIX, LATER, REMIND, DUPLICATE, WORKSFORME,
       CANTFIX, NEEDINFO, TEST-REQUEST, UPSTREAM
zilla = gentoo xine sourcemage redhat mozilla kernel fdo abisource
        apache kde gnome
If you want another bugzilla, file a bug.

Gentoo-specific
===============
!meta [-v] [CAT/]PACKAGE
Print the metadata and optionally herd members for a given package.

!changelog [CAT/]PACKAGE
Changelog stats for a package

!devaway list
List all away developers.

!devaway DEVNAME
Display .away message for a single developer.

!herd HERD
Show herd members

!expn NAME
Show the expansion of any public Gentoo mail alias

!glsa GLSAID
Shows the title and external IDS for any given GLSA ID.

!earch [CAT/]PACKAGE
Earch output for a given package

!rdep [CAT/]PACKAGE
Reverse RDEPEND for a given package

!ddep
Reverse DEPEND for a given package

What isn't supported yet
------------------------
1. !glsa -s TEXT
This used to search for GLSAs that matched that string in their title or
external IDS.

2. New bug announcements
Jeeves used to announce brand new bugs to #gentoo-bugs as well as
targeted channels or users, depending on the product, component,
assignee, cc and a number of other factors (deeply nested if/else
trees). The old implementation had this in code entirely, and it would
be nice to avoid having to modify the code whatsoever, and instead have
some domain-specific language for doing this.

Source availability
-------------------
Gentoo specific:
http://git.overlays.gentoo.org/gitweb/?p=proj/rbot-gentoo.git
Bugzilla support:
http://git.overlays.gentoo.org/gitweb/?p=proj/rbot-bugzilla.git
(flameeyes has his own tree as well, but he's been sick lately, so it
was lagging behind my development)

Right now, if you want to run your own instance of the bot, you will
need the latest Git tree of the rBot itself, as upstream only fixed the
last remaining issue a couple of hours ago.

Thanks to
---------
solar:
Running the old Jeeves Eggdrop till now, and helping to document all of
the Eggdrop functionality we used.

flameeyes:
Bugzilla plugin development

halcy0n:
Gentoo-specific stuff

tango_, jsn-:
(rbot upstream developers) For fixing the bugs as I found them :-).
robbat2: (Default)

Cardoe was complaining that repeatedly hitting the Gentoo CVS server was too slow, and it turned out he wasn't using SSH ControlMaster at all. Other developers have blogged about it before, but here is a quick reminder how.

Without ControlMaster, running "time ssh robbat2@cvs.gentoo.org w" shows a turnaround of 1.9 seconds. With ControlMaster, It's more in the range of 0.07-0.09 seconds :-).

~/.ssh/config:
Host master-cvs.gentoo.org
    HostName cvs.gentoo.org
    User robbat2
    ControlMaster yes
    ControlPath ~/.ssh/master-%l-%h-%p-%r.sock
Host cvs.gentoo.org
    ControlMaster no 
    ControlPath ~/.ssh/master-%l-%h-%p-%r.sock
    BatchMode yes
Setup Usage:
ssh -f -n -N master-cvs.gentoo.org

Now just do anything like you would normally. For security, you should probably close the ControlMaster session if you're going away from your machine for a long time. It would be nice to detect the loss of the ControlMaster and re-initiate it always at the start of a sequence.

robbat2: (Default)

On Tuesday for OLS2008, I attended the wireless mini-summit. In past years, fellow Gentoo developer dsd has attended, and was remember by some of the attendees. I'm not so much involved with wireless stuff these days, but I have a good grasp of it from back when I worked at Net-Conex doing point-to-point links using Airaya WirelessGRID 802.11a gear doing 108Mbit w/ AES256, plus personal experimentation.

The vendors (Intel, Marvel, Broadcom, Atheros, Ralink, Nokia and others) and major distributions (Fedora/RH, Ubuntu, Debian, Suse) were present, but I was the only attendee from the smaller distros. Also present were some of the other core wireless developers, incl. Johannes Berg.

Most of the talk focused on 802.11 stack and driver issues, with a presentation about WiMax from Intel.

One of the really interesting things was the work from Luis R. Rodriguez, on the new Central Regulatory Domain Agent (CRDA). There were some large questions from the Intel crowd about API and interaction, but the general concept was very well received. The support for signing the domain file is probably going to not be used for the most part, as there are too many other places to subvert usage of the data even if the file is signed. 802.11d and 802.11h are mostly considered as useless as apparently no regulatory agencies have signed off on them.

Another interesting discussion came out of the discussion on power management. Stuff on the usage of the CARRIER interface flag. It's apparently quite inconsistent, and the UP/DOWN status on some wireless devices has large implications. Some devices go totally away on DOWN, and need firmware loaded on UP. In some, the power consumption in reset state prior to loading firmware is lower than any other powered-down state. Multiple power levels may be added later to try and allow devices to define what states are best/available for their power saving. Implications of firmware loss and DOWN state on associations to APs, esp. when some parts of WPA are in play. This all also sucks with some DHCP clients as they perform DOWN on release or failure, which loses the firmware - such behavior from userspace really needs to be stamped out.

For lunch, we went to a buffet resturant, Tuckers. I was a little dubious of this at first, as buffet is really not my thing, however I can say that it was quite decent, esp. their roast beef carvery, with some nice whole-grained mustard. The salads weren't so great, but overall I think I'd eat there again in a group if there was sufficient group demand.

On the way out of lunch, I ran into a cute girl (I'll call her A) with a rubber-spiked laptop bag, and started chatting to her. As she was an Ottawa resident, she was prepared with an umbrella for the torrential summer rain that started during lunch. Sharing her umbrella we returned to the hotel conference rooms, splitting up thereafter as she was in the virtualization mini-summit.

Post-lunch, resuming the wireless mini-summit, we discussed more issues about the CRDA, core mac80211 development, and then breakout sessions on power-management and ??? (I can't remember what the other side was, even though I was in it).

For dinner, I took a clear walk out via parliament, and a very long way, full route. Ended up at "Elgin Street Freehouse" for dinner, had an Indian-fusion twist on steak, and did manage to find virgin Mojitos successfully. Nice 5km walk for exploring.

robbat2: (Default)
As one talk I was really interested in, I went to John Hawley's talk entitled "Issues in Linux Mirroring: Or, BitTorrent Considered Harmful", as seen from the perspective of the kernel.org mirrors.

This paper was really interesting for me, both as the Gentoo releng infra liasion (I get the bits from releng onto the mirrors), as well as working for IsoHunt, since he was complaining about BitTorrent.

Before the actual material about BitTorrent, he had some harsh words about distributions and space usage, and the lack of co-ordination. Having multiple major distributions doing their releases in the same week really only hurt themselves, because the mirrors get saturated by users. Between two major distros, they use up fully half of the 5.5TiB at kernel.org, and having them doing new material at the same time just blows out the cache, even with stupid amounts of memory. (Comments were made about Mark Shuttleworth having the money to buy some boxes with TiB of RAM for kernel.org). Co-ordination between distributions is needed to resolve this issue, and the audience discussion suggested we should try the distributions@freedesktop list first, and if that's too much noise, start up a list at kernel.org instead.

Moving onto BitTorrent, he noted that in large Linux torrent swarms, the standard tracker balancing algorithms end up with a net effect that a few slow peers joining greatly slow down the swarm speed at present (based on analysis of the tracker used by Fedora for the F8 release). If mirror are performing seeding, in many cases, it will still be faster for the mirror to provide content for a given user than other client peers. If the objective is to move content as fast as possible, this is needed vs. the normal BT objective of balancing total bandwidth usage.

Issues for distributions in handling bittorrent to make life easy for mirrors, he had several complaints about the level of manual interaction needed, to which I responded with the Gentoo structure of symlink trees under experimental, which is used for mirrors to run torrents easily, as well as powering the HTTP seeding additions to the BitTorrent protocol.

In using rtorrent(libtorrent), he complained that it wasn't using sendfile at all, which had a large negative performance impact, should be tackled upstream.

The BitTorrent community also needs to look at tweaking the peer decision protocol in the announce protocols, to hand out a smarter selection of fast peers. Where fast is local (look at BGP looking-glass for clues) or is a designated fast mirror that should be used as a fast peer.

Lastly, he noted that the trackers seem to be badly run, as somebody from isoHunt, I offered to post up my own work on running effective trackers to the inter-distro discussion.
robbat2: (Default)
- In 2006, I went to MySQL UC, and OSCON.
- In 2007, I went to the Vancouver PHP conference and LWE-SF.
- For 2008, I went to MySQL UC, and I'm going to be at OLS2008 in Ottawa next week, July 21st thru 27th.1

I'll have the entire Sunday free in Ottawa (my flight home is in the evening, and the conference itself ends up Saturday). Anybody that wants to hang out, that would be cool, or sight-seeing.

Additionally, if you're interested in PGP keysigning, or CACert assurances, you should seek me out with some ID. This applies doubly to all Gentoo developers with the upcoming tree-signing work.

While I'm not going to OSCON since it conflicts with OLS, my friend Zak Greant (really I mean it, he lives just up the street from me!) is going to OSCON, and putting on a totally free mini-conference within it: FOSSCoach. If you're just trying to get a start in open source from a beginner's perspective, and would like to be more than just a user, it should be worth checking out. (I meant to hype it a while ago, but was too busy).

robbat2: (Default)

Ok, so this isn't a full one week period yet, but I'm going to be out tonight probably, so 8 hours ahead of time is close enough. These also don't account for anybody who went and picked a specific mirror manually. I could do a much better job, but this is just a quick scrape of the numbers. There are many pitfalls in them, so they are more for interest than serious statistics.

Downloads by bouncer product (no arch breakdown)
gentoo-2008.0-livecd (x86,amd64)72518
gentoo-2008.0-minimal26543
gentoo-2008.0-universal (hppa,ppc,sparc64)2925
gentoo-2008.0-packagecd (sparc64)385
'completed' from torrent tracker
livecd-i686-installer-2008.0-r1975
livecd-i686-installer-2008.0867
install-x86-minimal-2008.0681
livecd-amd64-installer-2008.0-r1451
livecd-amd64-installer-2008.0373
install-amd64-minimal-2008.0353
install-powerpc-universal-2008.069
install-powerpc-minimal-2008.061
install-alpha-minimal-2008.048
install-ia64-minimal-2008.046
install-hppa-universal-2008.042
install-sparc64-universal-2008.029
packages-sparc64-2008.028
install-sparc64-minimal-2008.028
install-hppa-minimal-2008.028
www node traffic

For the two machines that serve up exclusively the www.gentoo.org vhost, they normally do 6-9GiB/day in HTTP traffic, and on the day of the release they jumped to 21GiB (and 14GiB for the second day).

robbat2: (Default)

So on Slashdot today, there was a link to the latest research into Package manager security. Specifically, their focus was on defeating signed packages by use of malicious mirrors and replay attacks of signed content. Recording the source of client requests, and possibly denying specific security updates (having an older tree that doesn't contain the security updates).

This plays into some of my long-ongoing tree-signing research in Gentoo. The GLEPs with the exception of 02 and 03 have been mailed to the GLEP editors as well as the portage-dev mailing list, and will be going to the gentoo-dev mailing list after the GLEP editors have reviewed them.

For dealing with the new issues raised by Cappos et al, at Gentoo we are really lucky to have our own infra maintained hardened rotation of mirrors at rsync://rsync.gentoo.org/ in addition to the community mirrors at rsync://rsync$N.$CC.gentoo.org/. Nobody using just the infra-maintained mirrors (barring MITM attacks) would be vulnerable to the new attacks described by Cappos, however those using a community-maintained mirror could be.

Using the main mirrors for new signing purposes, this will enable us to deliver the new MetaManifests reliably via our own infrastructure, even when the user has a community mirror for their actual tree content. The actual changes to the GLEP for this weren't very big at all. Just a timestamp header inside the signed area, as well as distributing the MetaManifests via a trusted medium.

As a minor side note on the infra-maintained rsync.gentoo.org rotation, this would be a good time to consider sponsering a box to Gentoo for that purpose. Each of the 5 existing boxes in the rotation does 50-65GiB of traffic every day - averaging to 6.5Mbit/sec, over a 24-hour period. These boxes are bandwidth, memory and CPU intensive, however they don't hit disk very hard (we serve the trees directly from memory). 4GiB RAM, 2+ 64-bit processors (single core or dual core is fine), ~16GiB of disk (optional: software RAID1 is nice for avoiding downtime, and fancy fast disks aren't needed). We need a serial console or KVM to install it securely - you just boot the box to a livecd, get the access details to infra, we install it from there with our own stage4 tarball that links into cfengine. The machine continues to be owned by the sponsor, in your data centre.

robbat2: (Default)

So working on a cleanup of machines, I looked at the history of the infra pages in CVS, and I noticed that infra has had a lot of developers of the years.

I'm probably missing a few, that never made it to the list, or predated the list, but I think it's a good start. I've also listed what they did either from the webpage, or from memory, again apologies if I got it wrong.

I'd like to thank all of those that put work into infra in the past, but have retired from Gentoo

  • Alex Howells (astinus) - Mirrors, DNS.
  • Jeffrey Forman (jforman) - Mirrors, DNS, Bugzilla, sysadmin, lots
  • Andrea Barisani (lcars) - Lists, LDAP, mail, sysadmin, lots
  • Kyle England (kengland) - sysadmin, cfengine
  • Lars Weiler (pylon) - CVS, SVN, overlays
  • Robert Coie (rac) - Forums, DBA
  • Jon Portnoy (avenj) - Mirrors
  • Sascha Schwabbauer (cybersystem) - Mail, Jabber
  • Tim Haynes (piglet) - Mirrors
  • Corey Shields (cshields) - LOTS
  • Rob Holland (tigger) - sysadmin
  • Benjamin Coles (sj7trunks) - Bugzilla, sysadmin
  • Michael Cummings (mcummings) - sysadmin
  • David Olsen (lude) - Mirrors
  • Albert Hopkins (marduk) - packages.g.o
  • Luca Mercuri (siggy) - www
  • Andrew D. Fant (jfmuggs) - backups, www
  • Curtis Napier (curtis119) - www, torrents
  • ???? (little_bob) - nagios

I'm not forgetting our current infra team, I hope to do a followup about them sometime soon too.

robbat2: (Default)

Up until recently, I had thought most Gentoo users and developers to be adults, who made sensible choices in their actions (but not always their words). This may be generalized to acting professionally. I am saddened to report on the ongoing degradation of the community in this regard, and how infra will deal with their side of it.

I've been with the infrastructure team in general for a very long time, however, up until April 2007, I was only the CVS administrator, and had no roles nor access outside of that. Since then, I stepped in as an extra sysadmin, and I've ended up as one of the operational leads, which still means I do most of the work, I just get to make the choices about it too. While the 'old' infra were in some cases called tyrants, dictators, cabals, and other nasty things, we as the 'new infra' hoped to change this view.

We're charged with a lot for developers and users: procuring machines to run them on, maintaining them, developing new services, troubling some user and developer issues (eg: cvs/mirrors) and more.

For myself, in addition to the CVS/SVN/Git services that grew out of my CVS administration, I presently maintain LDAP, Lists and Bugzilla. I have also been the infra liaison to the releng team since 2007.0.

The various VCS and LDAP services are only of primary concern to developers, because extremely few users interact directly with them. However, Bugzilla and Lists are used by significantly more users than developers, and the interactions show.

All messages to mailing lists with 'unsubscribe' in the subject line get moderation and passed to me, and a great many of them are in the realm of blunt and abusive - usually on generic-sounding email accounts that have changed ownership to clueless people. There's also the fun of keeping the spam off (see my recent post to the mlmmj list, of which I should possibly blog about). That's the mundane side. There's also moderation of the actual moderated announcement lists, and tracing mis-delivered list bouncemail as it gets reported. Lastly, and perhaps most important to some, we are held accountable to userrel and devrel for enacting list bans.

Bugzilla gets less direct abuse, however when it happens, it's usually quite flagrant. jakub used to complain to me once or twice a month about users refusing to take no for an answer, and repeatedly filing duplicates, or deleting entire CC lists, or spamming a bug. Since his absence, I've caught less of these early on, simply because he basically read every bug that was filed, and I don't have the time for that (yes, I'd like him back, he did a good job). Bots that ignore robots.txt are a hassle, but are mostly manageable.

For developer issues, we haven't been offering executable homedirs for several years, since some former developers tried running BOINC, and various servers. It seems however that there has never been any codified warning, merely action on a case-by-case basis.

As of today, we're formalizing the handling of this. All infra-maintained machines either already, or will shortly have an AUP banner as follows:

 Any or all uses of this system and all files on this system may be
 intercepted, monitored, recorded, copied, audited, inspected, and
 disclosed to authorized site personnel, as well as authorized officials
 of federal law enforcement agencies, both domestic and foreign. By 
 using this system, the user consents to such interception, monitoring,
 recording, copying, auditing, inspection, and disclosure at the
 discretion of authorized site personnel. Use of this system constitutes
 consent to security monitoring and testing. All activity is logged with
 your host name and IP address. Unauthorized or improper use of this
 system may result in civil and criminal penalties. By continuing to use
 this system you indicate your awareness of and consent to these terms
 and conditions of use. -- Gentoo Linux Infrastructure Admins.

To make it more concise without the legalese: If you abuse a Gentoo infrastructure system, we have no compunctions about kicking your ass and handing you to the suitable authorities (userrel, devrel, $GOV_AUTHORITY).

What does this not mean? Aside from being proactive about patching security issues, we are not intended, nor do have no plans to target people that some of our group don't get along with - we're meant to be accountable and responsible to other authorities in Gentoo. We'll collect the evidence (logging) and execute you (retirement), but somebody else (devrel) gets to sentence you - the only exceptions to this are preemptive actions where we consider security to be at risk.

On the matter of logging, we aren't the Stasi either, we have far better things to do than babysit logs, and we've been logging a lot longer than I was ever even a Gentoo developer. Some former developers and infra folk automated the log analysis, so the only time we really need to look is when something has been brought to our direct attention and needs logs to back it up. The most common uses for the logs are finding abusive users and bots against rsync and bugzilla, plus doing audits after (in)security events.

robbat2: (Default)

Ok, so it seems that I'm blogging again a bit, but only about software bugs and treachery. Today's post is about how I've burnt about 6 hours of development time, working through what seemed to be a simple bug.

Some background first. I'd like to switch away from an older system that's presently only used as a display head, with dual 20" LCDs on an Nvidia card. It's too old and limited to run a lot of things on directly. The new system would be both a display head and runs most apps already is a quad G5, with 12GiB of RAM. The great holdup has been in graphics. While I have access to both the stock GeForce 6600 NV43 that shipped with the machine, and an ATI X1900 G5 edition that I purchased later, my luck with graphics drivers has been less than stellar. My choices are between Nouveau, of which this tale ensues, and the competing free ATI drivers (which, at my last testing, were both still stuck, unable to read the AtomBIOS from the G5 card).

Some months ago, I filed a bug for Nouveau, that the first display output worked perfectly, but the second did not. It sat for about a month, before I got a response to go and try again. I didn't get to it until yesterday, because I was away in South Africa for two weeks, and busy with a myriad other things.

Update to the latest Nouveau and x11-drm trees yesterday afternon, and I find that it no longer even starts up a single display now. The debugging thus begins.

  1. The Device Control Block datastructure seems to have a bad data signature.
  2. Trace it. This is because the pointer to it is byteswapped.
  3. Hack in a byteswap. The signature is also byteswapped, something more central is wrong.
  4. The functions for "le{16,32}_to_cpu" seem to be broken. Just force them to byteswap for now.
  5. Update the nouveau bug again.
  6. Now we get an -ENOMEM from a memory allocation seemingly, but digging deeper, it's actually from ioctl.
  7. Update the nouveau bug again. Give up for the night.
  8. Update the x11-drm Git tree in the morning, and see that there's a fix for the ioctl stuff. Rebuild with it, and find that X will now start, but...
  9. The colors are badly swapped too! Red and Green are exchanged, and everything white is in a horrible shade of yellow.
  10. Try to initialize the second screen. "xrandr --display DVI-D-1 --right-of DVI-D-0". The taskbar expands as if it was covering both screens, but the second display is still not actually enabled.
  11. Update the nouveau bug again. Go and do other stuff.
  12. Start prodding into the nouveau driver source for the third time, looking to see about color issues. Mention this in the IRC channel. pq mentions to check that actual X_BYTE_ORDER macro.
  13. Doing a quick C program gives the correct output, however during the Nouveau compile, it's defined to _X_BYTE_ORDER (with a leading underscore), and THAT isn't defined.
  14. Look at the FreeDesktop.org bugzilla again, locate a bug for the new issue.
  15. Leave a comment with some useful output on the bug, and then set out to trace it myself.
  16. Revert a Gentoo patch that removes _X_BYTE_ORDER (with the underscore) from xorg-server's configure.ac.
  17. Notice that _X_BYTE_ORDER is being defined to an EMPTY string now. That is NOT right.
  18. Start reading the rest of configure.ac. _X_BYTE_ORDER should have the value of "$ENDIAN".
  19. $ENDIAN is in turn defined by AC_C_BIGENDIAN([ENDIAN="X_BIG_ENDIAN"], [ENDIAN="X_LITTLE_ENDIAN"]).
  20. Absolutely great. So what the hell is the problem? Well, on powerpc, the macro returns "universal".
  21. Just what the hell is "universal"??? Well it seems that in autoconf-2.6.2, the autoconf maintainers made AC_C_BIGENDIAN take another input.
  22. AC_C_BIGENDIAN([ACTION-IF-TRUE], [ACTION-IF-FALSE], [ACTION-IF-UNKNOWN], [ACTION-IF-UNIVERSAL]).
  23. So where does leave us? Up the creek with a broken autoconf I think. I haven't figured out why autoconf is telling us that we are universal yet, beyond that it's designed for the OSX universal binaries.

Hopefully I'm not tromping off into the depths of Mordor (aka the hell of autoconf M4 as maintained by vapier and flameeyes.

robbat2: (Default)

I wanted to document how to deal with an annoying corner case with git-submodules here. This applies in two variations:

  • Changing the URL of a submodule to another tree with different ids (eg, two different git-svn conversions of the same SVN project).
  • Converting an entire existing directory to a submodule.

If you apply the changes to all of your branches right when you make them, they are easy to do, but if you have old rotting branches, and you haven't kept them up to date, you can run into fun errors like the following:

fatal: cannot read object c1a25b84627516da46b6c375f4dc874deedbb597 'vendor/plugins/rspec~a4c30e94d52232e958d4f53c6a633ed438c54bcc': It is a submodule!

"vendor/plugins/rspec" already existed as a directory in this case. So how to do we work with this problem?

  1. Look at the "git log ${PARENT_BRANCH} .gitmodules". Every time there was a change to the file, make a temporary tag. I suggest using tags with date+time in the name, as you will be making more tags in a bit. You should do this on the branch that is the parent of the branch you want to fix, to cut down on conflicts.
  2. Next, identify spots that directories were deleted immediately prior to conversion to submodules, or between submodule URLs. Also tag these.
  3. Edit your .git/config, and comment out ALL submodule references.
  4. Now one at a time, issue "git pull . tags/${TAGNAME}", for each of your temporary tags, in order.
  5. If you hit conflicts, fix them up as usual (edit files, "git-update-index ${FILES}", "git commit -F .git/MERGE_MSG")
  6. If you think you messed up, use "git reset --hard ${COMMITISH} && git clean -f" with a known good point to reset you.
  7. Remember to clean up your temporary tags afterwards.
  8. Do "git submodule init" again.

I don't see why Git couldn't be made to do this automatically, since the process is reasonable simple, if a bit long to do by hand.

robbat2: (Default)

A few days ago, nicoj posted about using Gitosis on Gentoo, and as the developer that put the package into the tree. However, I don't think he realized at the time, that the Gitosis I packaged up, does differ from the original upstream version.

Why is is different? It is different because I decided to use Gitosis to power the new Gentoo Overlays, and found some limitations with Gitosis, so ended up hacking the codebase heavily to make it do what I wanted. It seems my original Christmas email to the upstream author went AWOL, so I wrote him another in the meantime. I hope that he will be able to merge the changed sanely, and make life easier for evertbody.

So what's different? A lot. Here's a partial list of the big stuff that is actually visible to most folk

  • Relative git+ssh:// URLs! The original Gitosis required that you use : between the hostname and the repository for a relative URL, and assumed that you were using an absolute URL otherwise. This made URLs look a bit ugly, and also broke some classical URL parsers that expected a port number after the colon, then a path after the slash. So now Gitosis supports git+ssh://HOST/REPO style URLs, where the REPO is looked up directly in the Gitosis config to see if it is a valid relative URL.
  • gitosis-init, you can use the default STDIN input of your key, or you can use actual command-line arguments: gitosis-init --adminkey=FILE --adminname=STRING. The latter argument is for when then username portion of your SSH key does not contain anything useful to you, and you wish for gitosis to place it in a more suitably named file in the keydir/ set of SSH keys.
  • Handle SSH keys intelligently, validate the algorithm, supporting both SSH1 and SSH2 keys, extract the username (the field is actually a comment per the RFC) safely if possible, and handle the options correctly. Amongst the options, the from field of the key is now preserved, so that if you had a key (without a passphrase for example) that was limited to login from a certain location only, it does not become less secure.
  • Allow setting of the initial directory permissions, globally and per-repository. Gitosis used chmod 0750 on directories it created before, which caused problems if you were running git-daemon as nobody:nobody. For the Gentoo overlays, the default repos now use 0755. If you set this globally, you should ensure that your gitosis-admin repo gets dirmode=0750, so that it does not get shared out by gitweb or git-daemon.

There are also two pending TODO items that I have for the Gentoo Gitosis-powered Overlays

  • Gitweb has broken owner strings when UTF8 is involved
  • The permissions handling needs an overhaul, adding a repo presently requires adding two config sections, each with 3 lines long. I'd like to refactor and make the [repo ...] section get a single line extra for the common case, which would totally do away with the [group ...] section per repo. Groups would remain JUST for defining groups, and the [repo ...] sections would get the lists of members and groups directly.

On a total lark, something like Gitosis for managing SVN users would be great too.

2008/08/03: Comments have now been disabled due to the amount of spam comments on this post. Email me if you have something useful to say.

robbat2: (Default)

Not sure who out there can help, but I'm looking for a number of old Gentoo distfiles, that were located on the Gentoo mirrors directly, and not copied from some other location. I do have every other version of the mysql-extras, so I am only looking for those listed here.

mysql-extras-20050904.tar.bz2
mysql-extras-20050919.tar.bz2
mysql-extras-20051205.tar.bz2
mysql-extras-20060114.tar.bz2
mysql-extras-20070104.tar.bz2

I have every other version of that distfile, those are the only ones I'm missing, and I'm after making a nice Git repo to trace the history. The SVN tree that was used for a short while doesn't contain some of the details from these either, hence the need for the tarballs.

Beyond those tarballs, it would be interesting to try and build an archive of every distfile ever used in Gentoo. I've got the diskspace (and tape backup) to do it. I already have an LTO3 tape that is getting every bit of release media/stages from Gentoo, so distfiles would be the next logical step.

Edit: Thanks to Lisa for 20050904.

robbat2: (Default)

If you use 'PermitRootLogin no' in your sshd_config and a locked-down sudo (requiring a password to upgrade powers), logging in to a machine as root is not allowed. This can be a pain when you want to rsync files between two machines, as root on both sides to preserve permissions and ownership. There is a fun little hack that you can use to get around this, that I'll document here.

  1. Ensure your SSH agent is running and has a key present.
  2. Open two shells, we will call them A and B (instructions prefixed with either or 'Both' below)
  3. We will use A to connect to the source, and B to connect to the destination.
  4. Both: SSH to the relevant machines, forwarding your agent, using 'ssh -A hostname'
  5. Both: Run 'sudo su', authenticating to sudo. Do not use 'sudo su -', as we need to preserve our SSH agent information.
  6. B: Run your rsync command as normal, but include the following option: --rsync-path='sudo rsync'

You should not get a password prompt! If you do get one, your sudo authentication did not propagate on the source machine properly. You cannot enter a password at this prompt either, it will never reach sudo, as rsync does not pass your input to it.

Alternatively, if your rsync version does not have a usable 'rsync-path' option (non-existant or wants a full path to a single program), you can use the -e option as: -e 'ssh user@source sudo /usr/local/bin/ignorefirst'. /usr/local/bin/ignorefirst is the following tiny script:

#!/bin/sh
shift
exec "$@"

The '-e' method is a lot more flexible, you can chain SSHs in it for example. You only need the 'ignorefirst' script because rsync puts the the hostname as the immediate next argument to the contents of '-e' commands.

robbat2: (Default)

On one of my machines with a PCI-express Intel e1000 network controller, I get some weird behavior during startup. The driver loads, but the link lights are not lit until such time as I cycle the administrative link status. I did complain to the e1000 upstream folk about it months ago, but I still haven't seen a solution, as they seemed to bicker about it with Linus, in the name of power-saving.

The following is a snipped for /etc/conf.d/net that combines the solution with this problem with the more common check for link status check before trying to get a DHCP address.

check_link() {
  ethtool "${1}" | grep -q 'Link detected: yes'
}

preup() {
  # Try to force link up first, for e1000 special case
  i=0 
  while [ "${IFACE}" != "lo" ] && [ $i -lt 3 ] && ! check_link "${IFACE}"; do
    [ $i -gt 0 ] && sleep 1
    ip link set "${IFACE}" up  
    i=$(($i+1))
  done
  # Then check for actual link
  if ! check_link "${IFACE}"; then
    ewarn "No link on ${IFACE}, aborting configuration"
    ip link set "${IFACE}" down
    return 1
  fi  
}

May 2017

S M T W T F S
 123456
78910111213
141516171819 20
21222324252627
28293031   

Syndicate

RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags