robbat2: (Default)
2017-05-20 09:32 pm

ebtables for EC2-like metadata server on AND bridge

<>Playing with the concepts around Cloud metadata services, specifically those that are network-based, rather than the ConfigDrive or serial port variants.

EC2 ensures that is magically provide your instance with your data, and it won't be accessible to another instance. This is trivial to achieve if your instances are using routed or tap network; but is more complex if you are on a bridged network: the client will try to send the packets for to the MAC of the default gateway.

So far I can force bridged packets that would otherwise be headed for the gateway to be routed locally (and put locally on the host). I don't have a good way to associate the packets with a specific instance yet. Using kernel packet marks work, but isn't really scalable. Main requirement is that a simple web service should be able to uniquely identify the client, even if they try to spoof themselves (learn mac+IP of another instance on the same hypervisor & bridge, and ask for it's metadata from the wrong interface)

Variant 1

  ebtables -t nat -N metadata || ebtables -t nat -F metadata
for i in $(seq 0 20) ; do
   ebtables -t nat -A metadata -i vnet$i -j mark --mark-set $((256+$i)) --mark-target CONTINUE
ebtables -t broute -A metadata --limit 10/minute --limit-burst 2 --log --log-level debug --log-prefix "ebtables metadata" --log-ip
ebtables -t nat -A metadata -j redirect
ebtables -t nat -F PREROUTING
ebtables -t nat -A PREROUTING -p IPv4 --logical-in br0 --ip-src -j metadata
ebtables -t nat -A PREROUTING -p IPv4 --logical-in br0 --ip-dst -j metadata

Variant 2

ebtables -t broute -N metadata || ebtables -t broute -F metadata
ebtables -t broute -F BROUTING
ebtables -t broute -A BROUTING -p IPv4 --ip-src -j metadata
ebtables -t broute -A BROUTING -p IPv4 --ip-dst -j metadata
ebtables -t broute -A metadata --limit 10/minute --limit-burst 2 --log --log-level debug --log-prefix "ebtables metadata" --log-ip
# Repeat the marks if you want them.
ebtables -t broute -A metadata -j redirect
robbat2: (Default)
2015-06-05 10:21 am

gnupg-2.1 mutt

For the mutt users with GnuPG, depending on your configuration, you might notice that mutt's handling of GnuPG mail stopped working with GnuPG. There were a few specific cases that would have caused this, which I'll detail, but if you just want it to work again, put the below into your Muttrc, and make the tweak to gpg-agent.conf. The underlying cause for most if it is that secret key operations have moved to the agent, and many Mutt users used the agent-less mode, because Mutt handled the passphrase nicely on it's own.

  • -u must now come BEFORE --cleansign
  • Add allow-loopback-pinentry to gpg-agent.conf, and restart the agent
  • The below config adds --pinentry-mode loopback before --passphrase-fd 0, so that GnuPG (and the agent) will accept it from Mutt still.
  • --verbose is optional, depending what you're doing, you might find --no-verbose cleaner.
  • --trust-model always is a personal preference for my Mutt mail usage, because I do try and curate my keyring
set pgp_autosign = yes
set pgp_use_gpg_agent = no
set pgp_timeout = 600
set pgp_sign_as="(your key here)"
set pgp_ignore_subkeys = no

set pgp_decode_command="gpg %?p?--pinentry-mode loopback  --passphrase-fd 0? --verbose --no-auto-check-trustdb --batch --output - %f"
set pgp_verify_command="gpg --pinentry-mode loopback --verbose --batch --output - --no-auto-check-trustdb --verify %s %f"
set pgp_decrypt_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --output - %f"
set pgp_sign_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --output - --armor --textmode %?a?-u %a? --detach-sign %f"
set pgp_clearsign_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --output - --armor --textmode %?a?-u %a? --detach-sign %f"
set pgp_encrypt_sign_command="pgpewrap gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --textmode --trust-model always --output - %?a?-u %a? --armor --encrypt --sign --armor -- -r %r -- %f"
set pgp_encrypt_only_command="pgpewrap gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --trust-model always --output --output - --encrypt --textmode --armor -- -r %r -- %f"
set pgp_import_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --import -v %f"
set pgp_export_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --export --armor %r"
set pgp_verify_key_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --fingerprint --check-sigs %r"
set pgp_list_pubring_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --with-colons --list-keys %r"
set pgp_list_secring_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --with-colons --list-secret-keys %r"

robbat2: (ubercoder)
2014-03-29 03:35 pm

Mail Bounces & Gmail/GApps users: The ugly truth of DMARC in open-source mailing lists

This is a slightly edited copy of an email I send to the mailing lists for my local hackspace, VHS. I run their mailing lists presently for historical reasons, but we're working on migrating them slowly.

Hi all,

Speaking as your email list administrator here. I've tried to keep the logs below as intact as possible, I've censored only one user's domain as being identifying information explicitly, and then two other recipient addresses.

There have been a lot of reports lately of bounce notices from the list, and users have correctly contacted me, wondering what's going on. The bounce messages are seen primarily by users on Gmail and hosted Google Apps, but the problems do ultimately affect everybody.

67.6% of the vhs-general list uses either gmail or google apps (347 subs of 513). For the vhs-members list it's 68.3% (both of these stats created by checking if the MX record for the user's domain points to Google).

Google deciding that a certain list message is too much like spam, because of two things:

  • because of content
  • because of DMARC policy


We CAN do something about the content.

Please don't send email that has one or twos, containing a URL and a short line of text. It's really suspicious and spam-like.

Include a better description (two or three lines) with the URL.

This gets an entry in the mailserver logs like:

delivery 47198: failure:

That was triggered by this email earlier in the month:

> Subject: Kano OS for RasPi
> Apparently it's faster than Rasbian

DMARC policy:

TL;DR: If you work on an open-source mailing list app, please implement DMARC support ASAP!

Google and other big mail hosters have been working on an anti-spam measure called DMARC [1].

Unlike many prior attempts, it latches onto the From header as well as the SMTP envelope sender, and this unfortunately interferes with mailing lists [2], [3].

I do applaud the concept behind DMARC, but the rollout seems to be hurting lots of the small guys.

At least person (Eric Sachs) at Google is aware of this [4]. There is no useful workaround that I can enact as a list admin right now, other than asking the one present user to tweak his mailserver if possible.

There is also no completed open source support I can find for DMARC. Per the Google post above, the Mailman project is working on it [5], [6], but it's not yet available as of the last release. Our lists run on ezmlm-idx, and I run some other very large lists using mlmmj ( and sympa; none of them have DMARC support.

The problem is only triggering with a few conditions so far:

  • Recpient is on a mail service that implements DMARC (and DKIM and SPF)
  • Sender is on a domain that has a DMARC policy of reject

Of the 115 unique domains used by subscribers on this list, here are all the DMARC policies:       600  IN TXT "v=DMARC1\; p=none\;"   7200 IN TXT "v=DMARC1\; p=reject\;\;\; adkim=s\; aspf=s"      3600 IN TXT "v=DMARC1\; p=none\;,\;,\;rf=afrf\;pct=100"         3600 IN TXT "v=DMARC1\; p=none\;\;\;"          3600 IN TXT "v=DMARC1\; p=none\;\;\;"        7200 IN TXT "v=DMARC1\; p=none\; pct=100\;\;"       1800 IN TXT "v=DMARC1\; p=none\; pct=100\;\;"     1800 IN TXT "v=DMARC1\; p=none\; pct=100\;\;"

Only one of those includes a reject policy, but I suspect it's a matter of time until more of them will include it. I'm going to use here as the rest of the example, and that user is indirectly responsible for lots of the rejects we are seeing.

Step 1.

User sends this email.

From: A User <>

Delivered to list server via SMTP (these two addresses form the SMTP envelope)


Step 2.

If the MAIL-FROM envelope address is on the list of list subscribers, your message is accepted.

Step 3.0.

The list adjusts the mail to outgoing, and uses SMTP VERP [7] to get the mail server to send the new message. This means it hands off a single copy of the email, as well as a list of all recipients for the mail. Envelope from address in this case will encode the name of the list and the number of the mail in the archive.

If it was delivering to me (, the outgoing SMTP connection would look roughly like:


And the mail itself still looks like:

From: A User <>

Step 3.1.

I got this email, and if I open it I see this telling me about the SMTP details:

Return-Path: <>

I don't implement DMARC on my domain. If my system bounced the email, it would have gone to that address, and the list app would know that message 18094 on list vhs-general bounced to user

Step 3.2.

Google DOES implement DMARC, so lets run through that.

The key part of DMARC is that it takes the domain from the From header.   7200 IN TXT "v=DMARC1\; p=reject\;\;\; adkim=s\; aspf=s"

The relevant parts to us are:

p=reject, aspf=s

The ASPF section applies strict mode, and says the mail with a From header of, must have an exact match of the MAIL FROM transaction of

It doesn't match, as the list changed the MAIL FROM address. The p=reject says to reject the mail if this happens.

This runs counter to the design principles of mailing lists, so DMARC has a bunch of options, all of which require changing the mail in some way.

Here's the logs from the above failure:

> 2014-03-19 11:19:50.783996500 new msg 98907
> 2014-03-19 11:19:50.783998500 info msg 98907: bytes 8864 from <[]> qp 32511 uid 89
> 2014-03-19 11:19:50.785359500 starting delivery 211352: msg 98907 to remote
> 2014-03-19 11:19:50.785385500 status: local 1/10 remote 1/40
> 2014-03-19 11:19:50.785450500 starting delivery 211353: msg 98907 to remote
> ...
> 2014-03-19 11:19:58.713558500 delivery 211352: failure:
> 2014-03-19 11:19:59.053816500 delivery 211353: failure:


robbat2: (ubercoder)
2014-03-22 06:15 pm
Entry tags:

Terrible PHP hacks: making PHP/FI style file uploads work in PHP5.5 and newer

One of my past consulting customers, came to me with a problem. He'd been relatively diligent in upgrading his servers since last I spoke (it had been some years), and now the admin panel on one of his client's very old PHP websites was no longer working.

I knew the code had some roots back to at least PHP3, at the file headers I'd previously seen had copyright dates back to 1999. Little did I know, I was in for a treat today.

When last I visited this codebase, due to it's terrible nature with hundreds of globals, I had to put some hacks in for PHP 5.4, since register_globals were no longer an option. The hack for this is quite simple:

foreach($_POST as $__k => $__v) { $$__k = $__v; }
foreach($_GET as $__k => $__v) { $$__k = $__v; }

Well it seems since the last upgrade, they had also changed the register_long_arrays setting by demand of another project, and the login on the old site was broken. Quite simple this, just need to s/HTTP_SERVER_VARS/_SERVER/ (and similarly for POST/GET/COOKIE depending on your site).

Almost all was well now, except that the next complain was file uploads didn't work for several forms. I naively duplicated the _POST/_GET block above to $_FILES. No luck. Thus, my memory not remembering how file uploads used to work in early PHP, I set out to fix this.

I picked a good one to test with, and noticed that it used some of the very old PHP variables for file uploads (again globals). These files dated back to 1997 and PHP/FI!. The initial solution was to map $_FILES[x]['tmp_name'] to $x, and the rest of $_FILES[x][y] to $x_y. Great it seems to work now.

Except... one file upload form was still broken; it had multiple files allowed in a single form. Time for a more advanced hack:

# PHP/FI used this structure for files:
foreach($_FILES as $__k => $__v) { 
  if(!is_array($__v['tmp_name'])) {
    $s = $__k;
    $$s = $__v['tmp_name'];
    $keys = array('name','size','type');
    foreach($keys as $k) {
      $s = $__k.'_'.$k;
      $$s = $__v[$k];
  } else {
    for($i = 0; $i <= count($__v['tmp_name']); $i++) {
      if(defined($__v['tmp_name']) && defined($__v['tmp_name'][$i])) {
        $s = $__k.'['.$i.']';
        $$s = $__v['tmp_name'][$i];
        $keys = array('name','size','type');
        foreach($keys as $k) {
          $s = $__k.'_'.$k.'['.$i.']';
          $$s = $__v[$k][$i];

Thus I solved the problem, and had to relearn back how it used to be done with PHP/FI.

robbat2: (ubercoder)
2014-01-28 03:51 pm

Adding 95th Percentile in Munin, without any patches: undocumented setting graph_args_after

Munin is commonly used to graph lots of systems stuff, however it lacks a common piece of functionality: 95th percentile.

The Munin bug tracker has ticket #443 sitting open for 7 years now, asking for this, and proving a not-great patch for it.

I really wanted to add 95th percentile to one of my complicated graphs (4 base variables, and 3 derived variables deep), but I didn't like the above patch either. Reading the Munin source to consider implementing VDEF properly, I noticed an undocumented setting: graph_args_after. It was introduced by ticket #1032, as a way of passing things directly to rrdtool-graph.

Clever use of this variable can pass in ANYTHING else to rrdtool-graph, including VDEF! So without further ado, here's how to put 95th percentile into individual Munin graphs, relatively easily.

# GRAPHNAME is the name of the graph you want to render on.
# VARNAME is the name of the new variable to call the Percentile line.
# DEF_VAR is the name of the CDEF or DEF variable from earlier in your graph definition.
# LEGEND is whatever legend you want to display on the graph for the line.
#   FYI Normal rrdtool escaping rules apply for legend (spaces, pound, slash).
${GRAPHNAME}.graph_args_after \
  LINE1:${VARNAME}\#999999:${LEGEND}:dashes \
# Example of the above I'm using
bandwidth1.graph_args_after \
  VDEF:totalperc=gcdeftotal,95,PERCENT \
  LINE1:totalperc\#999999:95th\ Percentile\ (billable\):dashes \
robbat2: (ubercoder)
2014-01-22 09:32 pm
Entry tags:

APC PDU: resetting passwords with SNMP instead of a serial cable

So recently at one of the things I do for money, we got some used APC PDUs, AP7900. You can get them on eBay now for $100-$150USD, including shipping. They still sell the identical model, so there's nothing wrong with used gear. However, when they come, it's possible that the last owner didn't remove the passwords. There are some general guides on the Internet, but they almost exclusively revolve around using a custom serial cable.

While this guide is aimed at APC PDUs, APC actually uses a common embedded OS on many of their products, and the SNMP trick I have documented here was derived from their document: "Management Card Addendum", part number 990-6015A

Finding the device IP

If you're really lucky, the device will issue a BOOTP or DHCP request on boot. Then you can easily figure it out from there. If not, read on.

In the case of the PDUs, there is a large grey button. Hold it for 30 seconds, then release and press again, and it will cycle through displaying the IP. For other devices, you might been connect directly, and sniff for traffic to figure out the IP, or issue ARP requests for possible IPs (scanning,, an hour or two with nmap for example).

Default passwords

You might as well try all the default passwords first, it wouldn't hurt you. The protocols you want to try are Telnet, SSH, HTTP, HTTPS. Usernames of apc, device, readonly (web interface only); all with a password of apc. If the username of apc works, you don't need the rest of this document.

If your firmware is really old, you should also try any username with the password of TENmanUFactOryPOWER. This will drop you into factory test mode, and you can read the password from the EEPROM this way (option 13, then look at offset 0x1D0, but realize that the offset is different in various revisions). In later revisions, this password is only usable with a serial cable.


This is where we can get interesting. The PDUs come with a stock configuration of two SNMP communities: public and private. If the latter works, we'll use it to reset the device entirely. Test with: snmpget -v 1 -c private $IP SNMPv2-MIB::sysDescr.0, where $IP is the IP you found before.

Resetting the device with SNMP

If you've made it this far, you're stuck with an APC device, that you don't have administrator access with Telnet or SSH, but the SNMP private community does work

. You'll need to go and get the SNMP MIBS from APC next. Then you need a file from APC, it is a windows binary, but runs perfectly fine under WINE: i2c301.exe. Paste the file below to rpdu.ini:
SystemIP =
SubnetMask =
DefaultGateway =
Bootp = enabled
RemoteIP =
RemoteIP =
RemoteUserName = apc
RemotePassword = apc
Access = enabled
Port = 21
Access = enabled
Port = 23
Access = enabled
Port = 80
Access = enabled
AccessControl1Community = public
AccessControl1NMSIP =
AccessControl1AccessType = read
AccessControl2Community = private
AccessControl2NMSIP =
AccessControl2AccessType = write
DNSServerIP =
Name = Unknown
Contact = Unknown
Location = Unknown
Date = 01/01/2014
Time = 12:00:00
Authentication = Basic
AutoLogout = 10
AdminUserName = apc
AdminPassword = apc
AdminAuthPhrase = admin user phrase
DeviceUserName = device
DevicePassword = apc
DeviceAuthPhrase = device user phrase

Run i2c301.exe rpdu.ini. This will generate apc.cfg. Setup up a TFTP server on your local subset, so that the IP on the PDU will be able to reach it. Place that apc.cfg in a path where it can be reached, I used /apc/apc.cfg in my case. Now run the following commands, giving a second or so between them.

snmpset -v 1 -c private $DEVICEIP PowerNet-MIB::mfiletransferConfigTFTPServerAddress.0 s $SERVERIP
snmpset -v 1 -c private $DEVICEIP PowerNet-MIB::mfiletransferConfigSettingsFilename.0 s /apc/apc.cfg
snmpset -v 1 -c private $DEVICEIP PowerNet-MIB::mfiletransferControlInitiateFileTransfer.0 i initiateFileTransferDownloadViaTFTP
snmpget -v 1 -c private $DEVICEIP PowerNet-MIB::mfiletransferStatusLastTransferResult.0

The PDU will proceed to reset at this point, it can take up to two minutes. You should then be able to log in with the default of apc/apc. Beware that if you're running DHCP it will get a new IP.

You should probably upgrade the system at this point. If you grabbed the updated firmware from APC, it's a self-extracting zipfile (unpack with unzip in linux). FTP to the PDU, with the default login. Switch to binary mode (important!), and upload apc_hw02_aos_374.bin. Afterwards the device will reboot again. Reconnect afterwards and upload apc_hw02_rpdu_374.bin, again in binary mode.

Locking down your PDU

Now that you've reset the password and upgrade your device, it's time to lock it down PROPERLY. Switch to SSHv2 only, disable FTP, change all SNMP communities.

Giving up your PDU

If you're getting rid of old PDUs, please remember to remove the passwords on them! It makes it easier for the next sysadmin to deploy the PDU, but also prevents leaking any passwords to an attacker with a serial cable and the factory password.

robbat2: (ubercoder)
2013-11-15 01:34 pm

python-exec: solutions for package conflicts, and making it easier on users

Running into another system today with the fun python-exec block, I realise that while it has been discussed on the Gentoo mailing lists, and the forums slightly, there's been hardly any posts about it in the blog stream.

I'm not going to go into what caused it, but rather solutions for package conflicts in the short term, and also the long-term. The TL;DR general solution is running "emerge -1 dev-python/python-exec"

Here's the latest conflict I got on it; I wanted to install mirrorselect to compare some hosts

hostname / # emerge -pv mirrorselect

These are the packages that would be merged, in order:
[ebuild  N     ] net-analyzer/netselect-0.3-r3  22 kB
[ebuild     U  ] dev-lang/python-2.7.5-r3:2.7 [2.7.3-r2:2.7] USE="gdbm hardened%* ipv6 ncurses readline ssl threads (wide-unicode) xml -berkdb -build -doc -examples -sqlite -tk -wininst" 10,026 kB
[ebuild     U  ] dev-lang/python-3.2.5-r3:3.2 [3.2.3:3.2] USE="gdbm hardened%* ipv6 ncurses readline ssl threads (wide-unicode) xml -build -doc -examples -sqlite -tk -wininst" 9,020 kB
[ebuild  N     ] dev-lang/python-exec-2.0:2  PYTHON_TARGETS="(jython2_5) (jython2_7) (python2_6) (python2_7) (python3_2) (-pypy2_0) (-python3_3)" 79 kB
[ebuild  N     ] dev-util/dialog-1.2.20121230  USE="nls unicode -examples -minimal -static-libs" 422 kB
[ebuild  N     ] app-portage/mirrorselect-  PYTHON_TARGETS="python2_7 python3_2 -python2_6 (-python3_3)" 13 kB
[blocks B      ] <dev-python/python-exec-10000 ("<dev-python/python-exec-10000" is blocking dev-lang/python-exec-2.0)

Total: 6 packages (2 upgrades, 4 new), Size of downloads: 19,580 kB
Conflict: 1 block (1 unsatisfied)

 * Error: The above package list contains packages which cannot be
 * installed at the same time on the same system.

  (dev-python/python-exec-0.2::gentoo, installed) pulled in by
    dev-python/python-exec[python_targets_python2_7(-),-python_single_target_python2_5(-),-python_single_target_python2_6(-),-python_single_target_python2_7(-)] required by (dev-libs/libxml2-2.9.0-r2::gentoo, installed)

  (dev-lang/python-exec-2.0::gentoo, ebuild scheduled for merge) pulled in by
    dev-lang/python-exec:=[python_targets_python2_6(-)?,python_targets_python2_7(-)?,python_targets_python3_2(-)?,-python_single_target_python2_6(-),-python_single_target_python2_7(-),-python_single_target_python3_2(-)] (dev-lang/python-exec:=[python_targets_python2_7(-),python_targets_python3_2(-),-python_single_target_python2_6(-),-python_single_target_python2_7(-),-python_single_target_python3_2(-)]) required by (dev-python/setuptools-0.6.30-r1::gentoo, installed)
    dev-lang/python-exec:=[python_targets_python2_6(-)?,python_targets_python2_7(-)?,python_targets_python3_2(-)?,python_targets_python3_3(-)?,-python_single_target_python2_6(-),-python_single_target_python2_7(-),-python_single_target_python3_2(-),-python_single_target_python3_3(-)] (dev-lang/python-exec:=[python_targets_python2_7(-),python_targets_python3_2(-),-python_single_target_python2_6(-),-python_single_target_python2_7(-),-python_single_target_python3_2(-),-python_single_target_python3_3(-)]) required by (app-portage/mirrorselect-, ebuild scheduled for merge)
    dev-lang/python-exec:=[python_targets_python2_6(-)?,python_targets_python2_7(-)?,python_targets_python3_2(-)?,python_targets_python3_3(-)?,python_targets_pypy2_0(-)?,-python_single_target_python2_6(-),-python_single_target_python2_7(-),-python_single_target_python3_2(-),-python_single_target_python3_3(-),-python_single_target_pypy2_0(-)] (dev-lang/python-exec:=[python_targets_python2_7(-),python_targets_python3_2(-),-python_single_target_python2_6(-),-python_single_target_python2_7(-),-python_single_target_python3_2(-),-python_single_target_python3_3(-),-python_single_target_pypy2_0(-)]) required by (virtual/python-argparse-1::gentoo, installed)

For more information about Blocked Packages, please refer to the following
section of the Gentoo Linux x86 Handbook (architecture is irrelevant):

This system has just dev-lang/python-exec-2.0 presently. We can reduce the conflict down to a minimal version as follows:

HOST / # emerge -pv  dev-lang/python-exec

These are the packages that would be merged, in order:
[ebuild  N     ] dev-lang/python-exec-2.0:2  PYTHON_TARGETS="(jython2_5) (jython2_7) (python2_6) (python2_7) (python3_2) (-pypy2_0) (-python3_3)" 79 kB
[blocks B      ] <dev-python/python-exec-10000 ("<dev-python/python-exec-10000" is blocking dev-lang/python-exec-2.0)

Total: 1 package (1 new), Size of downloads: 79 kB
Conflict: 1 block (1 unsatisfied)

 * Error: The above package list contains packages which cannot be
 * installed at the same time on the same system.

  (dev-python/python-exec-0.2::gentoo, installed) pulled in by
    dev-python/python-exec[python_targets_python2_7(-),-python_single_target_python2_5(-),-python_single_target_python2_6(-),-python_single_target_python2_7(-)] required by (dev-libs/libxml2-2.9.0-r2::gentoo, installed)

  (dev-lang/python-exec-2.0::gentoo, ebuild scheduled for merge) pulled in by
    dev-lang/python-exec:=[python_targets_python2_6(-)?,python_targets_python2_7(-)?,python_targets_python3_2(-)?,-python_single_target_python2_6(-),-python_single_target_python2_7(-),-python_single_target_python3_2(-)] (dev-lang/python-exec:=[python_targets_python2_7(-),python_targets_python3_2(-),-python_single_target_python2_6(-),-python_single_target_python2_7(-),-python_single_target_python3_2(-)]) required by (dev-python/setuptools-0.6.30-r1::gentoo, installed)
    dev-lang/python-exec:=[python_targets_python2_6(-)?,python_targets_python2_7(-)?,python_targets_python3_2(-)?,python_targets_python3_3(-)?,python_targets_pypy2_0(-)?,-python_single_target_python2_6(-),-python_single_target_python2_7(-),-python_single_target_python3_2(-),-python_single_target_python3_3(-),-python_single_target_pypy2_0(-)] (dev-lang/python-exec:=[python_targets_python2_7(-),python_targets_python3_2(-),-python_single_target_python2_6(-),-python_single_target_python2_7(-),-python_single_target_python3_2(-),-python_single_target_python3_3(-),-python_single_target_pypy2_0(-)]) required by (virtual/python-argparse-1::gentoo, installed)

For more information about Blocked Packages, please refer to the following
section of the Gentoo Linux x86 Handbook (architecture is irrelevant):

So what do we know?

  1. We have dev-python/python-exec-0.2 installed, it has the default SLOT=0.
  2. Here's what the packages in the tree right now look like:

$ egrep '^R?DEPEND|^SLOT' dev-{python,lang}/python-exec/*ebuild
  • If we try to bring in dev-lang/python-exec directly, it will trigger the block, because our version of dev-python/python-exec is too old.
  • This entire problem happens because the python*r1 eclasses bring in dev-lang/python-exec.
  • This leads to a simple user-actionable solution of "emerge -1 dev-python/python-exec", which will work as follows (notice that portage uninstalls the old version for us):

    HOST / # emerge -pv  dev-python/python-exec
    These are the packages that would be merged, in order:
    [ebuild  N     ] dev-lang/python-exec-0.3.1  PYTHON_TARGETS="(jython2_5) (jython2_7) (python2_6) (python2_7) (python3_2) (-pypy2_0) (-python3_3)" 73 kB
    [ebuild  N     ] dev-lang/python-exec-2.0:2  PYTHON_TARGETS="(jython2_5) (jython2_7) (python2_6) (python2_7) (python3_2) (-pypy2_0) (-python3_3)" 79 kB
    [uninstall     ] dev-python/python-exec-0.2  PYTHON_TARGETS="(jython2_5) (jython2_7) python2_5 (python2_6) (python2_7) python3_1 (python3_2) -pypy1_9 (-pypy2_0) (-python3_3)" 
    [blocks b      ] <dev-python/python-exec-10000 ("<dev-python/python-exec-10000" is blocking dev-lang/python-exec-2.0, dev-lang/python-exec-0.3.1)
    [ebuild  NS    ] dev-python/python-exec-10000.2:2 [0.2:0] PYTHON_TARGETS="(jython2_5) (jython2_7) (python2_6) (python2_7) (python3_2) (-pypy2_0) (-python3_3)" 0 kB
    Total: 3 packages (2 new, 1 in new slot, 1 uninstall), Size of downloads: 152 kB
    Conflict: 1 block

    The above is not actually the minimal solution, but it is the best general solution. The minimal solution is to include the slot on the package, but in future if the slots change further and the default slot is removed, this won't work anymore.

    HOST / # emerge -pv dev-python/python-exec:0
    These are the packages that would be merged, in order:
    [ebuild  N     ] dev-lang/python-exec-0.3.1  PYTHON_TARGETS="(jython2_5) (jython2_7) (python2_6) (python2_7) (python3_2) (-pypy2_0) (-python3_3)" 73 kB
    [ebuild     U  ] dev-python/python-exec-10000.1 [0.2] PYTHON_TARGETS="(jython2_5) (jython2_7) (python2_6) (python2_7) (python3_2) (-pypy2_0) (-python3_3) (-pypy1_9%) (-python2_5%*) (-python3_1%*)" 0 kB
    [blocks b      ] <dev-python/python-exec-10000 ("<dev-python/python-exec-10000" is blocking dev-lang/python-exec-0.3.1)
    Total: 2 packages (1 upgrade, 1 new), Size of downloads: 73 kB
    Conflict: 1 block

    But now the better question, is as developers, can we help users prevent this, and at what cost? If we don't mind new users having an extra placeholder package, then yes, we CAN actually solve it for the users. In all of the dev-lang/python-exec ebuilds we need to make this simple change:


    This provides a nice solution as follows:

    # emerge -pv dev-lang/python-exec
    These are the packages that would be merged, in order:
    [ebuild  N     ] dev-lang/python-exec-0.3.1  PYTHON_TARGETS="(jython2_5) (jython2_7) (python2_6) (python2_7) (python3_2) (-pypy2_0) (-python3_3)" 73 kB
    [ebuild     U  ] dev-python/python-exec-10000.1 [0.2] PYTHON_TARGETS="(jython2_5) (jython2_7) (python2_6) (python2_7) (python3_2) (-pypy2_0) (-python3_3) (-pypy1_9%) (-python2_5%*) (-python3_1%*)" 0 kB
    [blocks b      ] <dev-python/python-exec-10000 ("<dev-python/python-exec-10000" is blocking dev-lang/python-exec-2.0, dev-lang/python-exec-0.3.1)
    [ebuild  N     ] dev-lang/python-exec-2.0:2  PYTHON_TARGETS="(jython2_5) (jython2_7) (python2_6) (python2_7) (python3_2) (-pypy2_0) (-python3_3)" 79 kB
    [ebuild  NS    ] dev-python/python-exec-10000.2:2 [0.2:0] PYTHON_TARGETS="(jython2_5) (jython2_7) (python2_6) (python2_7) (python3_2) (-pypy2_0) (-python3_3)" 0 kB
    Total: 4 packages (1 upgrade, 2 new, 1 in new slot), Size of downloads: 152 kB
    Conflict: 1 block

    All that remains is convincing the Python team to accept this solution for users...

    robbat2: (ubercoder)
    2013-10-25 11:32 am

    Tracking prior open-source contributions

    For the many other open-source contributors and developers out there, I'm wondering if anybody has a complete list of all works they have created. It came up recently that such a list would be useful in asserting my own prior copyrights in any future employment, and avoiding claims that I'd taken any code [1].

    For version control systems are still accessible, this isn't too much of a problem, but for past historical creations, this is a lot harder. Has anybody else done it? To what level of detail did your listing go?

    Here's my initial broad listing(I'm going to come back often to fill it in more)

    I realized that this does form a sort of portfolio of work that I've done, and it shows just how flexible I am, esp. if I went and wrote this up better including a blurb about some of the larger or more standalone projects I've done.

    Additional sources to look up this stuff are:

    (list last updated 2015/06/20, partially)

    And I'm sure that I'm missing many more.

    Flattr this
    robbat2: (ubercoder)
    2013-10-25 11:07 am
    Entry tags:

    Looking for a new graphics card

    So I'm hunting for a new graphics card, and my set of requirements make this a difficult quest. I welcome all suggestions, either as comments, or as email.

    I haven't seen any passive cards in the new R7/R9 lines from ATI, not sure if those will come out later only


    • MUST be supported by the open-source Radeon or Noveau drivers.
    • MUST support at least 3 digital displays (DVI/HDMI/DP etc, not VGA)
    • SHOULD be cost less than $300
    • SHOULD occupy only one PCI-e slot (trying to avoid double-height cards)
    • SHOULD be passively cooled, or after-market water-cooled
    • NICE TO HAVE: good 3D performance

    Options so far (for further review)

    Name & Link Outputs Passive Card height Price
    ATI FirePro 2460 4x MiniDP Yes 1 slot $256 NCIX (no PP)
    VISIONTEK Radeon HD 7750, 2GB GDDR5, PCIe x16, 6x Mini-DP, Retail 6x MiniDP No 1 slot $273 NCIX (no PP)
    Radeon HD 7750 Low Profile - Eyefinity 4 CGAX-7758LM4 - PCI Express 3.0 - 2048 - GDDR5 - 128 BIT 4x MiniDP No 1 slot 136GBP AmazonUK, no NCIX listing
    SAPPHIRE FleX 100322FLEX Radeon HD 6450 1GB 64-bit DDR3 PCI Express 2.1 x16 HDCP Ready Low Profile Ready Video Card 1x DVI-D, 1x DVI-S, 1x HDMI Yes 2 slots LP $60 Newegg, $75 NCIX no-PP
    PowerColor HD7750 2GB GDDR5 Eyefinity 4 LP Edition (UEFI) AX7750 2GBD5-4DL 4x MiniDP No 1 slot $190 Ebay, no NCIX listing
    NVIDIA Quadro NVS 450 by PNY 512MB GDDR3 PCI Express Gen 2 x16 Quad DisplayPort or DVI-D SL Profesional Business Graphics Board, VCQ450NVS-X16-DVI-PB 4x MiniDP Yes 1 slot $295 AmazonUS, $495 NCIX non-PP
    FirePro W600 6x MiniDp No 1 slot $570 NCIX non-PP, $480 AmazonUS
    HIS 7750 iSilence 5 1GB GDDR5 PCI-E DP/DVI/HDMI DVI, HDMI, DP Yes 2 slot internally, 1 slot physically $100 AmazonUS


    I have tried the following cards already without success:

    • Galaxy MDT X4 GeForce210
      This card implements 4 DVI in a design like a pair of inline Matrox DualHead2Go units, it mucked badly with the EDID and crashed Noveau.
    • Matrox ??? older 3 or 4-head card: only first output supported on open-source driver
    • Gigabyte Radeon HD 7970 OC Edition: radeon driver very buggy when DP used and other outputs

    For reference, my monitors are 2x Dell U2410, 1x Dell U2413. So I have DVI/HDMI/DP on all displays, and the U2413 has DP1.2 MST as well.

    robbat2: (Default)
    2012-08-03 03:15 pm
    Entry tags:

    Making transparent PDFs in Linux

    Just documenting this again, so I don't forget it, and it might help others too.

    Specifically, how to make a PDF with transparency, using ImageMagick, so that it can be used as a stamp for PDFTK. This requires a PNG with transparency as input.

    convert $INPUT.PNG -transparent white -background none $OUTPUT.PDF
    pdftk $FORM.PDF stamp $OUTPUT.PDF output $COMPLETED_FORM.PDF
    robbat2: (Default)
    2011-06-08 04:06 am
    Entry tags:

    Gentoo Linux participates in World IPv6 day

    In light of World IPv6 day, the Gentoo Linux Infrastructure team would like to announce new IPv6-availability of several services, and list the existing IPv6 services. Every service listed below is running a dual-stack native IPv4/IPv6 service, no tunnels.

    The new services available via IPv6 are:

    The existing services available via IPv6 are:

    • CVS/SVN/Git services for developers
    • rsync:// - our primary rsync rotation
    • rsync://${CC} - our regional community rsync rotations
    • A number of our mirrors

    All of our IPv6 services will remain online after today, unless serious IPv6 problems (esp. regarding routing) are encountered.

    Gentoo would like to extend thanks to all our sponsors & mirrors who have provided IPv6 service, and the servers to make use of it!

    robbat2: (Default)
    2011-01-16 02:34 am

    Robin's 2011 conferences plans, ideas & other travel

    Working on my conference travel plans and wishes for the year. I am downgrading OLS to a maybe, the cost is becoming more of a factor. Likewise, while I had incredible fun at FOSDEM last year, and OSCON in 2006, I cannot justify the airfare/hotel expenses for them. I would like to attend SCALE at some point as well, but uncertain for the same cost reason.

    • February 25-27, SCALE 9x @ Los Angeles, CA, USA. [SCALE9x].
    • April 11-14, MySQL UC @ Santa Clara, CA, USA [MySQLconf].
    • April 13, Embedded Linux Conference @ San Francisco, CA [ELC].
    • April 27-30, STS-134 launch @ Kennedy Space Centre, FL, USA [STS134].
    • May 26-29, Bowen Island, BC, Canada.
    • June 25-26, Mini Maker Faire Vancouver @ Vancouver, BC, Canada.
    • August 17-19, LinuxCon 2011 @ Vancouver, BC, Canada [LinuxCon].
    • August 25-28, PAX Prime 2011 @ Seattle, WA, USA.
    • July 25-29, OSCON @ Portland, OR, USA.
    • August 7-11, SIGGRAPH 2011 @ Vancouver, BC, Canada.
    • October 19-22, Access 2011, @ Vancouver, BC, Canada.
    • October 22-23, Google Summer of Code 2011 Mentor Summit @ Mountain View, CA, USA.
    Would like to go, but out of my financial reach:
    • February 5-6, FOSDEM @ Brussels, Belgium.
    • June 13-15, Linux Symposium @ Ottawa, ON, Canada.
    • September 7-9, Linux Plumbers @ Santa Rosa, CA, USA.
    Arriving on the 24th actually
    I will be manning the phpMyAdmin booth, like past 5 years.
    Dropped in for just one day for hallway track
    KSC grandstand seats to see the penultimate launch :-)
    Local this year, so no travel costs :-)
    Page History
    Added GSoC, Access 2011, Bowen Island, STS-134
    robbat2: (Default)
    2010-09-17 03:25 am
    Entry tags:

    NightBus isn't really all night long

    The province is saying that there is sufficient Nightbus service, so they aren't going to extend the SkyTrain hours:

    TransLink's Ken Hardie says they can't run SkyTrain later because the tracks need maintenance. "We have night bus routes that basically follow Canada Line, the Expo Line and the Millennium Line, so they duplicate those routes and they run all night."

    I don't disagree that the maintenance is needed, but my objection in your claim that the NightBus routes "run all night".

    Almost all of the NightBus routes have a final bus leaving the downtown core at 03h09. The first buses in the morning then start leaving the downtown area between 05h00 and 06h30. The exceptions: the N10, with downtown departures up to 04h39 (1 hour gap to the start of normal service). The N16, which stops at 03h28.

    This means that if you are downtown and want to leave AFTER that, perhaps because your job had you working downtown, or you were chatting with friends, then you're stuck.

    I would like to ask Translink to add the few more trips that it would take to continue to run 30-minute service intervals until the resumption of regular morning service. The N10 is almost there, it just needs one more set of Downtown departures. Make our transit system really 24-hours!

    robbat2: (Default)
    2010-06-22 10:33 pm
    Entry tags:

    Recent LJ spam

    I apologize for doing this, but recent onslaught of spammers (~35 in the last 5 days) have left me with no choice: I've changed comments from non-friends to be screened by default AND require a captcha.

    robbat2: (Default)
    2010-06-15 01:36 am
    Entry tags:

    Complaining at Journalists again: Gentoo Security and the UnrealIRCd backdoor

    Those that have followed me for a while might have seen me previously complain at journalism that's misleading, wrong, or outright fictitious. Now I've got another case...
    This article by Ed Bott at ZDNet:
    Linux infection proves Windows malware monopoly is over; Gentoo ships backdoor? [updated]

    The article was first published 2010/06/12 20:37 UTC.
    It claims to be "worse" when updated at 2010/06/14 19:30 UTC.

    Gentoo had a revision bump to a known good copy of the tarball at 2010/06/12 16:34 UTC (using a different filename, and verified against the GPG signature provided by upstream), so it was ALREADY fixed when the article was published. The old revision was explicitly removed at 2010/06/12 21:18 UTC.
    Commit data for fixes:
    Changes for unrealircd-
    Changes for unrealircd-

    The trojaned tarball was then removed from the Gentoo master mirror at 2010/06/13 08:00 UTC, about 11 hours after the article was published. It would have been sooner, but it was a matter of bad timing.

    Gentoo bug 323691.

    The article also claims: "There’s a great deal of comment in the Talkback section of this post about how official repositories can be trusted. It appears that system broke down thoroughly in this case."
    This claim is bogus. The developer that updated the package made perhaps a mistake in trusting that the upstream had not been tampered with. However, in lacking anything to verify against (the upstream apparently did not sign releases at that point), he couldn't have detected the backdoor except by manual inspection of all the code. He downloaded the package AFTER it had been tampered with (2009/11/11 I believe), so he never saw the tamper-free version either.

    The entire point of the Gentoo Manifests are to ensure that OUR mirrors are not the point where a compromise is introduced. We can detect upstream changes by this same mechanism, but they mostly tend to be upstream deciding to 'fix' something without bumping the version number. In this regard, they functioned perfectly.

    P.S. I'm not saying the existing Gentoo mirroring is perfect either, see my prior writings on tree-signing, and the "Attacks on Package Manager" papers by Cappos et al., which are blocked only with the full tree-signing system.

    robbat2: (Default)
    2010-03-30 12:24 pm
    Entry tags:

    On Google Summer of Code Applications

    (This post inspired by Petteri Räty (betelgeuse)'s similar post

    For this year's Gentoo GSoC projects, I'm a mentor on two of our suggested ideas (but also interested in totally new ideas that fit my fields):

    • upstart on Gentoo
    • Distfile Fetcher Intelligence
    Do you actually understand the project idea?
    This is actually a gap that I didn't expect to exist, but I have seen in previous years. This is mainly a difference of expectations between the proposal and what the potential student sees as what the idea really entails.
    Using Upstart as an example, it supports an existing init.d compatibility mode, but we're not interested in that. Instead we want our init.d scripts to be treated just like upstart jobs (located in /etc/init/). The init.5 manpage shipped with upstart gives a good start...
    Code maintainability
    betelgeuse spoke about long-term maintenance, but you should think about it long ahead of that. Some degrees of abstraction, and avoiding difficult to understand logic should be prevalent here. betelgeuse mentioned spaghetti code, but it's important to realize that even well formatted code can impose a much larger mental workload if not well thought out.
    Timezones, Timezones!
    Most of your project should not be blocking on asking for mentor advice, as timezones and real world pressures often conspire to prevent easy real world communication. I may live in UTC-7, but my hours drift as needed by work but I tend to be online anywhere between 17h00 UTC and 10h00 UTC. If you're trying to communicate with me on a regular basis, this can be tough, so being able work on a problem independently, ask highly directed questions via email can go a long way.
    robbat2: (Default)
    2010-03-25 10:14 pm
    Entry tags:

    Advice for Google Summer of Code students

    Good advice for any prospective GSoC student, regardless of gender

    I'm also a mentor for Gentoo again this year, after taking a break last year.
    You can find our list of potential ideas here: Google Summer of Code 2010 ideas for Gentoo
    But don't limit yourself to them! Creative ideas can get you very far too :-)

    I'll also be the infrastructure contact for the accepted SoC students, for any issues you have with the source code repositories (we'll be offering Git again), your shell accounts, and a sounding board on deploying your successful project (for those that hosting or larger resources).

    robbat2: (Default)
    2010-02-06 06:11 am

    FOSDEM: Notes from MirrorBrain talk

    Sitting in the MirrorBrain talk at FOSDEM, taking notes.

    Actively used since ~2007.
    Split between the redirector and the tester, explicitly made separate.
    SourceForge helped with the ASN/Closest-Network side.
    Metalinks and P2P support.
    Scans mirrors for filelist to see what's present.
    Load limiting by making director support mirrors that are limited to a local network / AS / country etc.
    MetaLinks don't have Magnet links presently, but I noted that it should be possible to include it.

    Using GeoDNS directly for lookups can cause trouble with partial mirrors. Ideally need to put a MirrorBrain server on each continent/region, and GeoDNS to point to that. Also, from some countries, bandwidth to adjcaent countries that might have a mirror is MUCH worse than bandwidth to a well-connected country elsewhere. Past user experience noted with a user in Mozambique, for whom the fastest mirror was via satellite to Canada. Routing data IS needed to make that best choice.

    MirrorBrain mailing lists also have a generic non-project-specific "networkers" list for talk between content providers and mirror admins, non-specific to any app.

    robbat2: (Default)
    2010-02-01 12:25 pm
    Entry tags:

    Spamtrap addresses vs. list confirmation emails, or how to lose 2k list emails

    In the early hours of this morning, a spammer managed to get the IP of the Gentoo list server on the NiX Spam RBL... simply by spamming the subscribe address :-(. This caused approximately 2000 deliveries of normal list mail to be rejected while the server was present on the RBL.

    Log details )

    Why did this happen? I do agree on the importance of spamtrap accounts, but they MUST check the content of their messages. A list confirmation message MUST NOT be considered as spam.

    The original subscribe request came from what seems to be a compromised server in Secunderabad, India. So it wouldn't have been detected by RBL focused on modem/dialup addresses.

    Short of raising the bar to subscribe (with a specific token that needs to be included, and then it's only a matter of time till spammers include it too), there isn't much we can do to block stuff like this at the list-server level. There is no way to detect than an address is a spamtrap. There cannot be by definition, as the spammers would avoid it themselves otherwise.

    robbat2: (Default)
    2009-12-23 01:31 am
    Entry tags:

    Stolen Bike: Orange DeVinci St Tropez

    My bicycle was stolen earlier this evening. Sufficiently close to see the guy cycling away with it. Drove around a bit with Dave looking for it, but didn't find :-(

    • Orange DeVinci St Tropez (large)
    • Reward if you return it!
    • Extensive scuff damage to the handlebar ends
    • Panier Rack
    • Rain fenders
    • 2x front LED lights
    • 2x rear LED lights
    • Serial: SA…863

    Last time I had my bike stolen I was in the downtown eastside. This time it was stolen from outside my house, NOT visible from the street or alley, around 23h00 at night.