Argentina.com : A Case Study

- - Argentina.com : A Case Study - - - CarlosHorowicz -

ch@argentina.com -

- - - - - &tm-attrib.freebsd; - &tm-attrib.intel; - &tm-attrib.redhat; - &tm-attrib.general; - - - $FreeBSD$ - - $FreeBSD$ - - - - Overview - - Argentina.Com is an Argentine ISP with a small infrastructure - of fewer than 15 employees and whose primary source of income - originates in the free dialup business. It began operation in the - year 2000 with barely one server for mail and chat. - - It has since grown to a market presence in the Argentine free - dialup market of 4.5 billion minutes annually. Its most popular - product provides nearly half a million users with free e-mail with - webmail, POP3 and SMTP access, and 300M disk space. Towards the - end of 2002 there were around 50,000 mail users. After two and a - half years of re-engineering and consistent technical improvements - this ISP has grown by a factor of 3 in terms of billing, and by a - factor of 10 with regard to the mail user base. - - Our competitors in the Argentine market of free dialup include - Fullzero which is owned by the Clarin Media Group, Alternativa - Gratis, and Tutopia which is funded by IFX and promoted by - Hotmail. Some of these large corporate competitors started their - free dialup business with multi-million dollar investments and - aggressive television and Internet ad campaigns. Argentina.Com - does not rely on advertising like these other larger corporations. - It has climbed to the fourth position and to an 8% market share - during the last two years thanks to superior quality of service. - - In Argentina and Latin America in general people who do not - have computers at home go to so called Locutorios - (Internet Centers), where for a few pesos they can use a computer - connected to the Internet and usually read and write emails - through popular webmails like Hotmail, Yahoo or - Argentina.Com. - - Due to limited financial resources, Argentina.Com made the - decision to invest in a new email system instead of publicity in - the media. This strategic decision opens the door to a future - business in the corporate and paid email arena. - - - - The Challenge - - The main challenge for Argentina.Com is to achieve a dialup - uptime of at least 99.95%, or less than 5 hours yearly - downtime. Due to the high rotation and volatility in this - business, things have to work correctly so the user does not switch - -voluntarily or not- the dialup provider or the number he calls to - connect. The dialup business involves a support structure to deal - with the Telcos about telephony problems and quality of service, - plus a technical structure where latency and packet-loss should be - minimized due to the UDP nature of Radius and DNS, and where - recursive DNS should always be available. - - This also implies having a high uptime in the POP3 and SMTP - services, and in the webmail. For POP3 and SMTP we estimated the - need for an uptime equal to the one for dialup, whereas for the - webmail we could live with 99.5% which means around two days of - yearly downtime. - - We decided to migrate the email to a proprietary, opensource - architecture which should be horizontally scalable, and whose - antivirus and antispam infrastructure should support more than - just one type of mailstore or back-end. - - The rough competition in the free email market, mostly due to - the recent improvements introduced by Hotmail, Yahoo and Gmail, - made it necessary to design the new system with at least 300M user - disk space, but at a cost lower than 3 US dollars per GB with some - degree of redundancy. Bear in mind that rackmountable hardware is - hard to find in Argentina, and is between 30 and 40% more - expensive than in the US. Our total budget for equipment - acquisition in two years was 75,000 USD, which is only a fraction - of our direct competitors' investments. - - With regard to the antispam service, it became necessary to - develop a product that could compete with the systems offered by - the big ones. Given the hostile conditions imposed by the - existence of spam (dictionary attacks, spams with high degree of - obfuscation and refinement, phishing, trojans, mail-bombs, etc.) - it becomes very difficult to achieve an excellent uptime while - repelling attacks. One must also be careful that the user does not - lose mails because of false positives in the classification - strategy, that he does not become flooded with spam or spam - notifications, and dangerous mails do not make it through to his - mailbox. In addition, the technical infrastructure for spam - classification should not introduce noticeable delays in the - delivery of mails. Finally, the mail system has to be protected - from spammers who might misuse it to send spam. - - The opensource paradigm tends to require hiring large teams of - system administrators, operators and programmers who apply - patches, correct bugs and integrate platforms. The opposed - paradigm is also costly because of expensive software licences, - the need for increasingly expensive hardware and a large support - staff. So the challenge was to find the right mixture for scarce - human and monetary resources, high stability and predictability, - and quick and reliable deployment. In Buenos Aires, well-trained - Computer Science professionals are hard to find, most of them live - and work abroad, while the remaining have stable jobs either at - the government or big companies. - - - - - The FreeBSD solution - - - Introduction - - At the beginning of 2003 we had a CriticalPath mail system - running on Solaris x86 plus a Red Hat box for SMTP, Radius and - DNS. The DNS and Radius services were constantly down and we - were struggling with huge mail queues. There was an attempt to - install CriticalPath for &linux; into Red Hat on an &intel; box with - a Megaraid card, but the disk latency was enormous and the mail - application never really worked. - - The first step depicted towards the "FreeBSD solution" - consisted in migrating this hardware and commercial software to - FreeBSD 4.8 with &linux; emulation. - - - - The choice of FreeBSD - - The FreeBSD operating system is well-known for its great - stability, plus its pragmatism and common sense to put - applications on-line thanks to its excellent Ports System. We - consider its release - engineering process to be easily understandable, while - the users' community at the official mailing lists keeps a - polite and civilized style when it comes to asking for support - or reading other people's problems and solutions. - - Another important feature is quick deployment. Fortunately, - we could state our OS install policy around FreeBSD's great - out-of-the-box capability. In a small company you sometimes need - to run to a Datacenter and quickly setup a server for some - service. In the last two years, Argentina.Com acquired around - forty servers, most of them Pentium IV but also several - double-Xeons and a few double-Opterons to be co-located in the - Datacenters where we have dialup and hosting operation - contracts. All of them run FreeBSD, ranging from 4.8 (there are - a couple with two years uptime and zero trouble) til currently - 6.0-BETA2. - - The general policy for the operating system is to try to - bring all servers periodically to the stable code branch by - using RELENG_4, RELENG_5 - and now RELENG_6. This regularity lets us be - more prepared regarding possible exploits at the operating - system or base software level, especially in web servers. - - - - - Basic re-engineering - - The first re-engineering step was to put in place two - FreeBSD 4.8 boxes whose unique task was to be authoritative DNS - for all our domains. The chosen software was Bind9. Those boxes - were co-located in different datacenters, taking care that there - was good latency between them to avoid zone transfer problems, - and making it possible to deal with TTLs between 60 and 600 - seconds to have quicker response in case of trouble. - - Second step was to deploy two more boxes of the same class, - again in different Datacenters, to only deal with Radius and - recursive DNS. The Network Access Servers at the Telcos were - configured to send Radius Authorization and Accounting to those - servers, and to assign these recursive DNSs to dialup users. - - The third golden rule never to put SMTP - incoming and outgoing in the same servers. We deployed separate - FreeBSD boxes with postfix for incoming and outgoing mail. - - - - - Email migration - - The email migration required careful planning due to the - fact that we were going to migrate both mail front and - back-ends. We first built a perimetral antispam and antivirus - system in FreeBSD 4.x and 5.x based on postfix, amavisd-new, - clamav and SpamAssassin. These systems were to deliver mails to - both the old and the new system until the new back-end was in - place. In the meantime, we added small FreeBSD NFS boxes to - increase CriticalPath's mailspool, without any problem. - - At the frontline of incoming mail, we put in place several - MXs of the Argentina.com domain to filter dictionary attacks - (attempts to forward mail to nonexistent users) as well as a - black-list derived from SURBL that resulted in almost no false - positives. The mails are then multiplexed to a cluster of - double-Xeons and double-Opterons where we run amavisd-new with - MySQL based white and black-listing. We discarded the use of - Bayes and Autowhitelisting at the global level because of great - quantities of false positives and false negatives. We instead - defined a few spam levels going from the least to the most - tolerant, each one with cutoff or discard levels. Every email - with a score below the one associated with the selected spam - tolerance goes to the user's Inbox. Emails between this level - and the cutoff level go to a user's folder named Spam, and those - above the cutoff level get discarded because it is a very obvious - spam. For the sake of simplicity, we transparently associated - the use of the Address Book with the antispam system, so that - every personal contact gets automatically whitelisted. - - With the introduction of Spamassassin 3.x, the DNS traffic - to query global blacklists grew considerably, so we signed - agreements with SpamCop, Spamhaus and SURBL to install public - mirrors of their databases in our FreeBSD equipment. Thanks to - these mirrors that cost us between 1 and 2Mbps in traffic, we - were able to dramatically cut down Spamassassin latency. - - At the 3rd level there is the delivery to the maildrops. As - soon as we started building a new Cyrus-Imap back-end with MySQL - authentication, we needed to multiplex incoming mail to users in - both old and new maildrop formats. Finally, we managed to - migrate hundreds of thousands of mailspools to the new Cyrus - architecture using a great tool named imapsync, which is - directly installable from ports. We also put perdition, a POP3 - and IMAP proxy, in the middle to assure a transparent migration - and distribution of mailboxes across several servers. Briefly, - all information of where a user's maildrop is located resides in - MySQL, and is being used by all software pieces in the - chain. - - With regard to the hardware for disk space, we currently use - seven Cyrus-Imap loaded FreeBSD boxes with diverse hardware. The - biggest are Pentium IV with 4G of RAM and 3ware cards in chassis - with 12 hotswappable bays, organized in 3 RAID-5 units of 1 - Terabyte each. The 3ware software sends you en email whenever - the RAID is degraded -mostly because of a failing disk- and lets - you rebuild the RAID with everything up and running. We use - smartmontools in the cases where we have less redundancy, to - have immediate alerts of disks with temperature problems or - failing selftests. - - As webmail software, we chose a commercial product named - Atmail, which is available with perl sources and utilizes - mod_perl. Under FreeBSD it is extremely easy to deal with perl - modules, you do not even need to use the CPAN shell, you just - have to choose the right port and run "make install". After - several months of integration work, we integrated the - Client-only version of Atmail that talks IMAP with our - back-ends. We had to modify some parts of the code to adapt the - product to our massive free environment, and to our antispam and - antivirus perimeter, in addition to our specific customizations - and translations. - - - - - Web migration - - With the adoption of FreeBSD, there was almost no additional - effort necessary to setup a working Apache, PHP and MySQL - environment in minutes. Even the upgrades from PHP4 to PHP5 were - painless. The ports system was again extremely useful in these - cases, and permitted us to do things like compress text and html - contents in Apache with just a few lines of documentation. In - addition, we have experienced excellent performance and - rock-solid stability and uptime. - - - - - - Results - - We managed to deploy a FreeBSD based email architecture that - is horizontally scalable, using 3 Terabyte &intel; based storage - servers at a current cost of 3 dollars per Gigabyte with - redundancy. - - The great stability achieved enabled Argentina.Com to explore - other fields like hosting for resellers and housing with presence - in three Argentine Datacenters. - - We offer now also corporate dialup for roaming users in - Argentina and Peru thanks to our presence and contracts with most - Telcos. Among our indirect customers, there are major American - companies like Ford, Exxon and Reuters. We now run the free dialup - business in Brazil, Chile, Colombia and Panama as well. - - -

On this site

All the documentation on this site can be downloaded in a variety of different formats (HTML, Postscript, PDF, and more) and compression schemes (BZip2, Zip) from the FreeBSD FTP site.

Archived copies of the &os; documentation (articles, books, and textinfo manuals) are also available online at http://docs.FreeBSD.org/doc/.

This documentation is provided and maintained by the FreeBSD Documentation Project, and we are always looking for people to contribute new documentation and maintain existing documentation.

Books

The FreeBSD FAQ (faq)
Frequently Asked Questions, and answers, covering all aspects of FreeBSD.

The FreeBSD Handbook (handbook)
A constantly evolving, comprehensive resource for FreeBSD users.

The FreeBSD Developers' Handbook (developers-handbook)
For people who want to develop software for FreeBSD (and not just people who are developing FreeBSD itself).

The FreeBSD Architecture Handbook (arch-handbook)
For FreeBSD system developers. This book covers the architectural details of many important FreeBSD kernel subsystems.

The Porter's Handbook (porters-handbook)
Essential reading if you plan on providing a port of a third party piece of software.

The PMake Tutorial (pmake)
A tutorial for the make utility. This book is essential reading for anyone who wants to understand all the details of using make of reading and writing makefiles.

Articles

The Committer's Guide (committers-guide)
Introductory information for FreeBSD committers.

Console Server Tutorial (console-server)
How to setup a FreeBSD based console server with a cheap multi-port serial card.

FreeBSD From Scratch (fbsd-from-scratch)
How to automatically compile, install and configure a system from scratch (i.e. to an empty file system), including your favorite ports.

Filtering Bridges (filtering-bridges)
Configuring firewalls and filtering on FreeBSD hosts acting as bridges rather than routers.

Fonts and FreeBSD (fonts)
A description of the various font technologies in FreeBSD, and how to use them with different programs.

Build Your Own FreeBSD Update Server (freebsd-update-server)
Using a FreeBSD Update server allows a system administrator to perform fast updates for a number of machines from a local mirror.

Writing a GEOM Class (geom-class)
A guide to GEOM internals, and writing your own class.

Mirroring FreeBSD (hubs)
The all in one guide for mirroring the FreeBSD website, FTP servers, and more.

LDAP Authentication (ldap-auth)
A practical guide about setting up an LDAP server on &os; and how to use it for authenticating users.

Linux emulation in &os; (linux-emulation)
A technical description about the internals of the Linux emulation layer in &os;.

Introduction to NanoBSD (nanobsd)
Information about the NanoBSD tools, which can be used to create FreeBSD system images for embedded applications, suitable for use on a Compact Flash card (or other mass storage medium).

FreeBSD First Steps (new-users)
For people coming to FreeBSD and &unix; for the first time.

Perforce in FreeBSD Development (p4-primer)
A guide to the Perforce version control system. It also describes how to manage experimental projects with the FreeBSD Perforce server.

Port Mentor Guidelines (port-mentor-guidelines)
Guidelines for new and/or potential port mentors and mentees.

FreeBSD as a greylist mail server (relaydelay)
Implementing a greylist mail server on FreeBSD using Sendmail, MySQL, Perl and the relaydelay software. This is an excellent method to use in the fight against spam.

FreeBSD Release Engineering (releng)
Describes the approach used by the FreeBSD release engineering team to make production quality releases of the FreeBSD Operating System. It describes the tools available for those interested in producing customized FreeBSD releases for corporate rollouts or commercial productization.

FreeBSD Release Engineering for Third Party Packages (releng-packages)
Describes the approach used by the FreeBSD ports management team to produce a high quality package set suitable for official FreeBSD release media. This document is a work in progress, but eventually it will cover the process used to build a clean package set on the FreeBSD.org "Ports Cluster", how to configure any other set of machines as a ports cluster, how to split up the packages for the release media, and how to verify that a package set is consistent.

Serial and UART devices (serial-uart)
Detailed information about the use of serial ports on FreeBSD, including several multi-port serial cards.

On other web sites

Various independent efforts have also produced a great deal of useful information about FreeBSD.