Move along, nothing to read

This is a slightly edited copy of an email I send to the mailing lists for my local hackspace, VHS. I run their mailing lists presently for historical reasons, but we're working on migrating them slowly.

---

Hi all,

Speaking as your email list administrator here. I've tried to keep the logs below as intact as possible, I've censored only one user's domain as being identifying information explicitly, and then two other recipient addresses.

There have been a lot of reports lately of bounce notices from the list, and users have correctly contacted me, wondering what's going on. The bounce messages are seen primarily by users on Gmail and hosted Google Apps, but the problems do ultimately affect everybody.

67.6% of the vhs-general list uses either gmail or google apps (347 subs of 513). For the vhs-members list it's 68.3% (both of these stats created by checking if the MX record for the user's domain points to Google).

Google deciding that a certain list message is too much like spam, because of two things:

• because of content
• because of DMARC policy

Content:

We CAN do something about the content.

Please don't send email that has one or twos, containing a URL and a short line of text. It's really suspicious and spam-like.

Include a better description (two or three lines) with the URL.

This gets an entry in the mailserver logs like:

delivery 47198: failure:
+173.194.79.26_failed_after_I_sent_the_message./Remote_host_said:_550-5.7.1_[66.196.40.251______12]_Our_system_has_detected_that_this_message_is/550-5.7.1_likely_unsolicited_mail._To_reduce_the_amount_of_spam_sent_to_Gmail,/550-5.7.1_this_message_has_been_blocked._Please_visit/550-5.7.1_http://support.google.com/m
+ail/bin/answer.py?hl=en&answer=188131_for/550_5.7.1_more_information._mu18si1139639pab.287_-_gsmtp/

That was triggered by this email earlier in the month:

> Subject: Kano OS for RasPi
> http://kano.me/downloads
> Apparently it's faster than Rasbian

DMARC policy:

TL;DR: If you work on an open-source mailing list app, please implement DMARC support ASAP!

Google and other big mail hosters have been working on an anti-spam measure called DMARC [1].

Unlike many prior attempts, it latches onto the From header as well as the SMTP envelope sender, and this unfortunately interferes with mailing lists [2], [3].

I do applaud the concept behind DMARC, but the rollout seems to be hurting lots of the small guys.

At least person (Eric Sachs) at Google is aware of this [4]. There is no useful workaround that I can enact as a list admin right now, other than asking the one present user to tweak his mailserver if possible.

There is also no completed open source support I can find for DMARC. Per the Google post above, the Mailman project is working on it [5], [6], but it's not yet available as of the last release. Our lists run on ezmlm-idx, and I run some other very large lists using mlmmj (gentoo.org) and sympa; none of them have DMARC support.

The problem is only triggering with a few conditions so far:

• Recpient is on a mail service that implements DMARC (and DKIM and SPF)
• Sender is on a domain that has a DMARC policy of reject

Of the 115 unique domains used by subscribers on this list, here are all the DMARC policies:

_dmarc.gmail.com.       600  IN TXT "v=DMARC1\; p=none\; rua=mailto:mailauth-reports@google.com"
_dmarc.USERDOMAIN.ca.   7200 IN TXT "v=DMARC1\; p=reject\; rua=mailto:azrxfkte@ag.dmarcian.com\; ruf=mailto:azrxfkte@fr.dmarcian.com\; adkim=s\; aspf=s"
_dmarc.icloud.com.      3600 IN TXT "v=DMARC1\; p=none\; rua=mailto:dmarc_agg@auth.returnpath.net, mailto:d@rua.agari.com\; ruf=mailto:d@ruf.agari.com, mailto:dmarc_afrf@auth.returnpath.net\;rf=afrf\;pct=100"
_dmarc.mac.com.         3600 IN TXT "v=DMARC1\; p=none\; rua=mailto:d@rua.agari.com\; ruf=mailto:d@ruf.agari.com\;"
_dmarc.me.com.          3600 IN TXT "v=DMARC1\; p=none\; rua=mailto:d@rua.agari.com\; ruf=mailto:d@ruf.agari.com\;"
_dmarc.yahoo.ca.        7200 IN TXT "v=DMARC1\; p=none\; pct=100\; rua=mailto:dmarc-yahoo-rua@yahoo-inc.com\;"
_dmarc.yahoo.com.       1800 IN TXT "v=DMARC1\; p=none\; pct=100\; rua=mailto:dmarc-yahoo-rua@yahoo-inc.com\;"
_dmarc.yahoo.co.uk.     1800 IN TXT "v=DMARC1\; p=none\; pct=100\; rua=mailto:dmarc-yahoo-rua@yahoo-inc.com\;"

Only one of those includes a reject policy, but I suspect it's a matter of time until more of them will include it. I'm going to use USERDOMAIN.ca here as the rest of the example, and that user is indirectly responsible for lots of the rejects we are seeing.

Step 1.

User sends this email.

From: A User <someuser@userdomain.ca>
To: vhs-general@lists.hackspace.ca

Delivered to list server via SMTP (these two addresses form the SMTP envelope)

MAIL FROM:<someuser@userdomain.ca>
RCPT TO:<vhs-general@lists.hackspace.ca>

Step 2.

If the MAIL-FROM envelope address is on the list of list subscribers, your message is accepted.

Step 3.0.

The list adjusts the mail to outgoing, and uses SMTP VERP [7] to get the mail server to send the new message. This means it hands off a single copy of the email, as well as a list of all recipients for the mail. Envelope from address in this case will encode the name of the list and the number of the mail in the archive.

If it was delivering to me (robbat2@orbis-terrarum.net), the outgoing SMTP connection would look roughly like:

MAIL FROM:<vhs-general-return-18094-robbat2=orbis-terrarum.net@lists.hackspace.ca>
RCPT TO:<robbat2@orbis-terrarum.net>

And the mail itself still looks like:

From: A User <someuser@userdomain.ca>
To: vhs-general@lists.hackspace.ca

Step 3.1.

I got this email, and if I open it I see this telling me about the SMTP details:

Return-Path: <vhs-general-return-18094-robbat2=orbis-terrarum.net@lists.hackspace.ca>

I don't implement DMARC on my domain. If my system bounced the email, it would have gone to that address, and the list app would know that message 18094 on list vhs-general bounced to user robbat2@orbis-terrarum.net.

Step 3.2.

Google DOES implement DMARC, so lets run through that.

The key part of DMARC is that it takes the domain from the From header.

_dmarc.USERDOMAIN.ca.   7200 IN TXT "v=DMARC1\; p=reject\; rua=mailto:azrxfkte@ag.dmarcian.com\; ruf=mailto:azrxfkte@fr.dmarcian.com\; adkim=s\; aspf=s"

The relevant parts to us are:

p=reject, aspf=s

The ASPF section applies strict mode, and says the mail with a From header of someuser@USERDOMAIN.ca, must have an exact match of the MAIL FROM transaction of @USERDOMAIN.ca.

It doesn't match, as the list changed the MAIL FROM address. The p=reject says to reject the mail if this happens.

This runs counter to the design principles of mailing lists, so DMARC has a bunch of options, all of which require changing the mail in some way.

Here's the logs from the above failure:

> 2014-03-19 11:19:50.783996500 new msg 98907
> 2014-03-19 11:19:50.783998500 info msg 98907: bytes 8864 from <vhs-general-return-18094-@lists.hackspace.ca-@[]> qp 32511 uid 89
> 2014-03-19 11:19:50.785359500 starting delivery 211352: msg 98907 to remote user1@gappsdomain.com
> 2014-03-19 11:19:50.785385500 status: local 1/10 remote 1/40
> 2014-03-19 11:19:50.785450500 starting delivery 211353: msg 98907 to remote user2@gmail.com
> ...
> 2014-03-19 11:19:58.713558500 delivery 211352: failure:
+74.125.25.27_failed_after_I_sent_the_message./Remote_host_said:_550-5.7.1_Unauthenticated_email_from_USERDOMAIN.ca_is_not_accepted_due_to_domain's/550-5.7.1_DMARC_policy._Please_contact_administrator_of_USERDOMAIN.ca_domain_if/550-5.7.1_this_was_a_legitimate_mail._Please_visit/550-5.7.1__http://support.google.com
+/mail/answer/2451690_to_learn_about_DMARC/550_5.7.1_initiative._ub8si9386628pac.133_-_gsmtp/
> 2014-03-19 11:19:59.053816500 delivery 211353: failure:
+173.194.79.26_failed_after_I_sent_the_message./Remote_host_said:_550-5.7.1_Unauthenticated_email_from_USERDOMAIN.ca_is_not_accepted_due_to_domain's/550-5.7.1_DMARC_policy._Please_contact_administrator_of_USERDOMAIN.ca_domain_if/550-5.7.1_this_was_a_legitimate_mail._Please_visit/550-5.7.1__http://support.google.co
+m/mail/answer/2451690_to_learn_about_DMARC/550_5.7.1_initiative._my2si9389106pab.76_-_gsmtp/

[1] http://dmarc.org/
[2] http://dmarc.org/faq.html#s_3
[3] http://dmarc.org/faq.html#r_2
[4] https://sites.google.com/site/oauthgoog/mlistsdkim
[5] http://www.marshut.com/qskkv/adding-dmarc-support-for-mailman-3.html
[6] https://code.launchpad.net/~jimpop/mailman/dmarc-reject
[7] http://en.wikipedia.org/wiki/Variable_envelope_return_path

Good advice for any prospective GSoC student, regardless of gender

I'm also a mentor for Gentoo again this year, after taking a break last year.
You can find our list of potential ideas here: Google Summer of Code 2010 ideas for Gentoo
But don't limit yourself to them! Creative ideas can get you very far too :-)

I'll also be the infrastructure contact for the accepted SoC students, for any issues you have with the source code repositories (we'll be offering Git again), your shell accounts, and a sounding board on deploying your successful project (for those that hosting or larger resources).