Autistici/Inventati

the R* Plan: a resilient network of distributed selfmanaged servers.

info@autistici.org

Autistici / Inventati

A/I is a collective of activists engaged with technology, privacy, digital rights.

• involves about 40 people
• low budget: financing comes from subscription campaigns aimed to the users
• promotes sensibilization campaigns
• technical competences are built on self-education

Problems

• Centralization
  • 5000 anonymous users and hundreds of mailing lists, all on a single server - BAD
• Legal background
  • growing control paranoia, in Italy like elsewhere
  • coming laws that will endanger the possibility of offering anonymous services (keeping logs, etc...)
• Tecnhical problems
  • server reaching its capacity limits
• June 2005: crackdown

The R* Plan

The so-called R* plan is the attempt to provide an answer to the problems just identified:

• private data delocalization (i.e. mailboxes)
• infrastructure redundancy (partly)
• almost linear scalability

Network structure

N functionally equivalent server each hosts a fraction 1/N of the users' private data servers are geographically distributed in different countries servers are interconnected through a VPN

VPN

This is the logical view of the "private" network. The various rings correspond to different levels of database access (different trust levels, indeed).

Encrypted file systems

• encrypted partitions:
  • are mounted on boot with manual passphrase insertion
  • contain the private keys of the various services that use asymmetric keys (SSL, cfengine, tinc...)
  • implemented using dmcrypt (no performance issues)
• we're evaluating the possibility of encrypting the partitions that contain private user data - relying on them however is a problem.

User database

• all user-related data (mailboxes, FTP accounts, web sites, etc...) is stored in an LDAP database
• such db is replicated on all servers
• the db operates in single-master mode, however switching the master node is a quick operation
• administration can be done either through normal LDAP tools (for easy tasks) or with our own custom-written administration software

Configuration management

• configuration of all servers is centralized
• the mechanism used (CFengine) allows both file distribution and server-specific customizations
• the main repository is under version control so that changes can be monitored and accounted for
• CFengine is a stable and convergent solution

Mail

multiple MX records, with equal priority antispam checks are always performed locally internal routing uses the VPN interconnection

Mail / 2

POP/IMAP services are deployed using the same scheme:

• mail.autistici.org uses round-robin DNS records
• the connection is then forwarded through the VPN towards its final destination (using perdition)
• for the moment, IMAP connections from the webmail are only local (you can read email only on the server that holds your mailbox) to reduce VPN traffic

Forwarding scheme (and some math)

• The forwarding scheme used for SMTP and POP/IMAP implies a certain amount of "wasted" bandwidth. This amount can be computed:
  • each connection comes to each of the N servers with equal probability 1 / N
  • the probability that it should be forwarded to the right server is (N - 1) / N, that tends to 1 for large N
  • every server then receives on average (N - 1) / N connections forwarded from other servers for every incoming external connection
  • the total number of incoming connections has an asymptotic behavior as N grows larger.

Web

• some sites (the ones we directly manage) are active on all servers using round-robin DNS and MySQL database replication
• problem: the scarcity of interconnection bandwidth make this approach valid only when database reads are overwhelming in respect to writes
• more so, the general complexity of the application configuration makes it unsuitable for users' websites (no control on the app. config)
• user sites (and their relative MySQL databases) are simply split up between the various servers.

Web / 2

• HTTP redirect:
  • for users' sites, clients are redirected to the server hosting the site
  • for distributed applications, clients are locked onto a specific server once a session is established
• the redirect maps (together with the rest of the Apache configuration) are periodically generated from the database.

Mailing Lists

• mailing list manager: Mailman (for historical reasons)
• Mailman stores its configuration on the filesystem (in binary format)
• since we don't have a distributed filesystem, each list needs to be active on a single server at a time
• transport maps direct the mail to the appropriate server
• configurations are copied periodically to every server so that we can rapidly switch servers for failover

Anonymity

A/I implements many mechanisms for user identity protection:

• system logs are anonymized directly at the syslog level (IPs and mail addresses, mostly)
• Apache logs are anonymized (except the error.log!)
• SMTP headers of outgoing messages are stripped of the information about the sender's IP address