ISP Mail Server 2.x HowTo

From Alpine Linux
This material is retained only for historical reasons ...

Alpine Linux 2.x support ended in November 2015. Please upgrade
URLs on this page may not function.

A Full Service Mail Server

This document describes installation process for latest Alpine Linux 2.x platform. The goal of this document is to describe how to set up postfix, dovecot, clamav, dspam, roundecube, and postfixadmin for a full-featured "ISP" level mail server. We recommend to run server at Alpine Linux version not earlier than 2.2, since release 2.2 has a lot of important fixes for bugs and security issues. (See also previous installation on 1.10 platform. There are also notes on upgrading from v1 to v2.)

The server must provide:

  • multiple virtual domains
  • admins for each domain (to add/remove virtual accounts)
  • quota support per domain / account
  • downloading email via IMAP / IMAPS / POP3 / POP3S
  • relaying email for authenticated users with TLS or SSL (Submission / SMTPS protocol)
  • standard filters (virus/spam/rbl/etc)
  • web mail client
  • value add services

Set up Lighttpd + PHP

PostfixAdmin needs php pgpsql and imap modules, so we do it in this step. 

 apk add lighttpd php php-pgsql php-imap

Stop and remove mini_httpd, and move ACF to lighttpd; We are setting this up to be a multi-domain virtual web server (replace host.example.com with the actual domain): 

 rc-service mini_httpd stop
 apk del mini_httpd
 mkdir -p /var/www/domains/host.example.com/www
 ln -s /usr/share/acf/www /var/www/domains/host.example.com/www/acf

Edit /var/www/domains/host.example.com/www/index.html to put a simple redirection page: 

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<title>host.example.com Redirector</title>
</head>
<body>
<ul>
<li><a href="/acf">ACF</a></li>
<li><a href="/postfixadmin">PostfixAdmin</a></li>
<li><a href="/roundcube">Roundcube</a></li>
</ul>
</body>

Edit /etc/lighttpd/mod_cgi.conf to serve haserl files by adding a "" => "" cgi handler and to treat /acf/cgi-bin as a CGI directory (remove the '^'): 

$HTTP["url"] =~ "/cgi-bin/" {
    # disable directory listings
    dir-listing.activate = "disable"
    # only allow cgi's in this directory
    cgi.assign = (
		".pl"	=>	"/usr/bin/perl",
		".cgi"	=>	"/usr/bin/perl",
		"" => ""
	)
}

Get a web certificate, and install it. You have two options: 1. If you want to use a self-signed cert, you can use the instructions found at Generating SSL certs with ACF or Generating SSL certs with ACF 1.9 to generate it. 2. Use the certificate created with the setup-acf command. 3. Get certificate from trusted root certification centers.

Option 1: If you create your own self-signed certificate, you can create the "server-bundle.pem" and the "ca-crt.pem" file with these commands: 

 openssl pkcs12 -nokeys -cacerts -in certificate.pfx  -out /etc/lighttpd/ca-crt.pem
 openssl pkcs12 -nodes -in certificate.pfx -out /etc/lighttpd/server-bundle.pem
 chown root:root /etc/lighttpd/server-bundle.pem
 chmod 400 /etc/lighttpd/server-bundle.pem

Note: The server certificate and key are in the server-bundle.pem file, so it is critical that the file be read-only by user "root".

Option 2: If you prefer to just use the default certificate created with the setup-acf command, then you will need to do the following: 

 setup-acf

During the above process, mini_httpd will be started, if it isn't already, and a certificate will be created. Once you have completed the setup-acf steps, do the following to move the certificate files to the correct location for lighttpd to use. 

 mv /etc/ssl/mini_httpd/server.pem /etc/lighttpd/server-bundle.pem
 chown root:root /etc/lighttpd/server-bundle.pem
 chmod 400 /etc/lighttpd/server-bundle.pem

Option 3: You may decide to receive a certificate from trusted root certification centers. One good choice is to request free certificate from StartSSL at page: https://www.startssl.com/?app=33.

Add these lines to /etc/lighttpd/lighttpd.conf to point to the new document root, and set it up to listen on port 443 (replace host.example.com with the actual domain and ip_address_of_server with the actual IP address): 


simple-vhost.server-root   = "/var/www/domains/"
simple-vhost.default-host  = "/host.example.com/"
simple-vhost.document-root = "www/"

$SERVER["socket"] == "ip_address_of_server:443" {
ssl.engine    = "enable"
ssl.pemfile   = "/etc/lighttpd/server-bundle.pem"
}

If you went with Option 1 above, then add an additional line underneath the ssl.pem file line, so that the section appears as follows: 

 $SERVER["socket"] == "ip_address_of_server:443" {
 ssl.engine    = "enable"
 ssl.pemfile   = "/etc/lighttpd/server-bundle.pem"
 ssl.ca-file   = "/etc/lighttpd/ca-crt.pem"
 }

Ensure that the simple_vhosts module is loaded, as well as the cgi config scripts by uncommenting the following lines in /etc/lighttpd/lighttpd.conf: 

server.modules = (
    #  other modules may be listed
    "mod_simple_vhost", 
    #  other modules may be listed
 .
 .
 .
    include "mod_cgi.conf"
    include "mod_fastcgi.conf"

Stop and remove mini_httpd; start lighttpd, test: 

 rc-service mini_httpd stop
 rc-update del mini_httpd
 apk del mini_httpd
 rc-update add lighttpd
 rc-service lighttpd start

At this point you should be able to see ACF being served with lighttpd (Note: this will work well with alpine 1.10 and 2.x. With earlier versions there will be problems.) https://host.example.com/acf/

Install Postgresql

Add and configure postgresql: 

 apk add acf-postgresql postgresql-client
 rc-service postgresql setup
 rc-service postgresql start
 rc-update add postgresql

At this point any user can connect to the sql server with "trust" mechanism. If you want to enforce password authentication (you probably do) edit /var/lib/postgresql/9.0/data/pg_hba.conf. Since by default "trust" mechanism is for local connections only we assume using trust password-less access as safe.

Create the postfix database: 

 psql -U postgres
  create user postfix with password '******';
  create database postfix owner postfix;
  \q

(Of course, use your selected password where ******* is shown above.)

Install PostfixAdmin

We are going to install the postfix admin web front-end before we install the mail server. This just creates an interface to populate the SQL tables that postfix and dovecot will use.

Download PostfixAdmin from Sourceforge. When these instructions were written, 2.3 was the current release, so (replace host.example.com with the actual domain): 

wget https://downloads.sourceforge.net/project/postfixadmin/postfixadmin/postfixadmin-2.3.3/postfixadmin-2.3.3.tar.gz
tar zxvf postfixadmin-2.3.3.tar.gz
mkdir -p /var/www/domains/host.example.com/www/postfixadmin
mv  postfixadmin-2.3.3/* /var/www/domains/host.example.com/www/postfixadmin
rm -rf postfixadmin*

Edit /var/www/domains/host.example.com/www/postfixadmin/config.inc.php and modify at least these lines (replace host.example.com with the actual domain): 

$CONF['configured'] = true;
$CONF['setup_password'] = "";  << Don't change this yet
$CONF['database_type'] = 'pgsql';
$CONF['database_host'] = 'localhost';
$CONF['database_user'] = 'postfix';
$CONF['database_password'] = '*****';   << The password you chose above
$CONF['database_name'] = 'postfix';
$CONF['database_prefix'] = "";
$CONF['admin_email'] = 'you@some.email.com';  << Your email address 
$CONF['encrypt'] = 'md5crypt';
$CONF['authlib_default_flavor'] = 'md5raw';
$CONF['dovecotpw'] = "/usr/sbin/dovecotpw";
$CONF['domain_path'] = 'YES';
$CONF['domain_in_mailbox'] = 'NO';
$CONF['aliases'] = '10';                       
$CONF['mailboxes'] = '10';
$CONF['maxquota'] = '10';
$CONF['quota'] = 'YES';
$CONF['quota_multiplier'] = '1024000';
$CONF['vacation'] = 'NO'; 
$CONF['vacation_control'] ='NO';
$CONF['vacation_control_admin'] = 'NO';
$CONF['alias_control'] = 'YES';
$CONF['alias_control_admin'] = 'YES';
$CONF['special_alias_control'] = 'YES';
$CONF['fetchmail'] = 'NO';
$CONF['user_footer_link'] = "https://host.example.com/postfixadmin";
$CONF['footer_link'] = 'https://host.example.com/postfixadmin/main.php';
$CONF['create_mailbox_subdirs_prefix']="";  
$CONF['used_quotas'] = 'YES';   
$CONF['new_quota_table'] = 'YES';

You should further edit /var/www/domains/host.example.com/www/postfixadmin/config.inc.php and replace all instances of "change-this-to-your.domain.tld" with your actual mail domain. This can be done with busybox sed (replace example.com with your domain name): 

sed -i -e 's/change-this-to-your.domain.tld/example.com/g' /var/www/domains/host.example.com/www/postfixadmin/config.inc.php

Go to https://host.example.com/postfixadmin/setup.php

Create the password hash, add it to the config.inc.php file

Go back to https://host.example.com/postfixadmin/setup.php

Create superadmin account.

NOTE: Check https://sourceforge.net/tracker/index.php?func=detail&aid=2859165&group_id=191583&atid=937964 if you have bug on listing domains page.

Install Postfix

Create a user for the virtual mail delivery, and get its uid/gid (you'll need the numeric uid/gid for postfix): 

adduser vmail -H -D -s /bin/false
grep vmail /etc/passwd

(In examples below, we use 1006/1006 for the uid/gid)

Create the mail directory, and assign vmail as the owner: 

mkdir -p /var/mail/domains
chown -R vmail:vmail /var/mail/domains

Install postfix: 

apk add acf-postfix postfix-pgsql postfix-pcre

Edit the /etc/postfix/main.cf file. Here's an example (don't forget to replace the uid/gid): 

myhostname=host.example.com
mydomain=example.com

mydestination = localhost.$mydomain, localhost
mynetworks_style = subnet
mynetworks = 127.0.0.0/8

virtual_mailbox_domains = proxy:pgsql:/etc/postfix/sql/pgsql_virtual_domains_maps.cf
virtual_alias_maps = proxy:pgsql:/etc/postfix/sql/pgsql_virtual_alias_maps.cf,
       proxy:pgsql:/etc/postfix/sql/pgsql_virtual_alias_domain_maps.cf,
       proxy:pgsql:/etc/postfix/sql/pgsql_virtual_alias_domain_catchall_maps.cf

virtual_mailbox_maps = proxy:pgsql:/etc/postfix/sql/pgsql_virtual_mailbox_maps.cf,
       proxy:pgsql:/etc/postfix/sql/pgsql_virtual_alias_domain_mailbox_maps.cf

virtual_mailbox_base = /var/mail/domains/
virtual_gid_maps = static:1006
virtual_uid_maps = static:1006
virtual_minimum_uid = 100
virtual_transport = virtual


# This next command means you must create a virtual
# domain for the host itself - ALL mail goes through
# The virtual transport

mailbox_transport = virtual
local_transport = virtual
local_transport_maps = $virtual_mailbox_maps

smtpd_helo_required = yes
disable_vrfy_command = yes
message_size_limit = 10240000
queue_minfree = 51200000

smtpd_sender_restrictions =
       permit_mynetworks,
       reject_non_fqdn_sender,
       reject_unknown_sender_domain

smtpd_recipient_restrictions =
       reject_non_fqdn_recipient,
       reject_unknown_recipient_domain,
       permit_mynetworks,
       permit_sasl_authenticated,
       reject_unauth_destination,
       reject_rbl_client dnsbl.sorbs.net,
       reject_rbl_client zen.spamhaus.org,
       reject_rbl_client bl.spamcop.net

smtpd_data_restrictions = reject_unauth_pipelining

# we will use this later - This prevents cleartext authentication
# for relaying
smtpd_tls_auth_only = yes


Now we need to create a *bunch* of files so that postfix can get the delivery information out of sql. Here's a shell script to create the scripts. Change PGPW to the password for the postfix user of the postfix SQL database: 

cd /etc/postfix
mkdir sql
PGPW="ChangeMe"

cat - <<EOF >sql/pgsql_virtual_alias_domain_catchall_maps.cf
user=postfix
password = $PGPW
hosts = localhost
dbname = postfix
query = Select goto From alias,alias_domain where alias_domain.alias_domain = '%d' and alias.address = '@' ||  alias_domain.target_domain and alias.active = true and alias_domain.active= true 
EOF

cat - <<EOF >sql/pgsql_virtual_alias_domain_mailbox_maps.cf
user=postfix
password = $PGPW
hosts = localhost
dbname = postfix
query = Select maildir from mailbox,alias_domain where alias_domain.alias_domain = '%d' and mailbox.username = '%u' || '@' || alias_domain.target_domain and mailbox.active = true and alias_domain.active
EOF

cat - <<EOF >sql/pgsql_virtual_alias_domain_maps.cf
user=postfix
password = $PGPW
hosts = localhost
dbname = postfix
query = select goto from alias,alias_domain where alias_domain.alias_domain='%d' and alias.address = '%u' || '@' || alias_domain.target_domain and alias.active= true and alias_domain.active= true
EOF

cat - <<EOF >sql/pgsql_virtual_alias_maps.cf
user=postfix
password = $PGPW
hosts = localhost
dbname = postfix
query = Select goto From alias Where address='%s' and active ='1'
EOF

cat - <<EOF >sql/pgsql_virtual_domains_maps.cf
user=postfix
password = $PGPW
hosts = localhost
dbname = postfix
query = Select domain from domain where domain='%s' and active='1'
EOF

cat - <<EOF >sql/pgsql_virtual_mailbox_maps.cf
user=postfix
password = $PGPW
hosts = localhost
dbname = postfix
query = Select maildir from mailbox where username='%s' and active=true
EOF

chown -R postfix:postfix sql
chmod 640 sql/*

At this point you should be able to start up postfix: 

newaliases  # so postfix is happy...
rc-service postfix start
rc-update add postfix

Create a domain in PostfixAdmin and test

Go to https://host.example.com/postfixadmin/

Log in using the superadmin account, create a domain for the local box (e.g. example.com), and create a user mailbox (e.g. root).

From the machine, send a test message: 

sendmail -t root@example.com
subject: test
.
^d

In /var/log/mail.log (or /var/log/messages, if you still have busybox syslogd running) you should see the message queued. The message should be in /var/mail/domains/example.com/root/new

Install Dovecot

Dovecot is the POP3/IMAP server to retrieve mail.

As before, we install dovecot: 

apk add acf-dovecot dovecot-pgsql

Edit /etc/dovecot/dovecot.conf: 

auth_mechanisms = plain login
auth_username_format = %Lu
#auth_verbose = yes
#auth_debug = yes
#auth_debug_passwords = no

disable_plaintext_auth = no

mail_location = maildir:/var/mail/domains/%d/%n

first_valid_gid = 1000
first_valid_uid = 1000
last_valid_gid = 65535
last_valid_uid = 65535

log_timestamp = "%Y-%m-%d %H:%M:%S "
login_greeting = IMAP server ready

protocols = imap

service anvil {
  client_limit = 2100
}

ssl_cert = </etc/lighttpd/server-bundle.pem
ssl_key = </etc/lighttpd/server-bundle.pem

userdb {
  args = uid=1006 gid=1006 home=/var/mail/domains/%d/%n
  driver = static
}

passdb {
  args = /etc/dovecot/dovecot-sql.conf
  driver = sql
}

protocol imap {
  mail_plugins = autocreate
}

plugin {
  autocreate = Trash
  autocreate2 = Spam
  autocreate3 = Sent
  autosubscribe = Trash
  autosubscribe2 = Spam
  autosubscribe3 = Sent
}

Be sure to replace the uid and gid with the appropriate values for the vmail user.

We need a certificate for SSL/TLS authentication, so in the example above, we use the lighttpd cert. That way when the cert is renewed/replaced, Dovecot will have access to the new cert as well.

Create the /etc/dovecot/dovecot-sql.conf file: 

driver = pgsql
connect = host=localhost dbname=postfix user=postfix password=********
password_query = select username,password from mailbox where local_part = '%n' and domain = '%d'
default_pass_scheme =  MD5-CRYPT

Again, change the password above to your postfix user password, and protect the file from prying eyes: 

chown root:root /etc/dovecot/dovecot-sql.conf
chmod 600 /etc/dovecot/dovecot-sql.conf

Start dovecot 

rc-service dovecot start
rc-update add dovecot

Testing

Make sure your firewall allows in ports 25(SMTP) 110 (POP3), 995 (POP3S), 143(IMAP), 993(IMAPS), or whatever subset you support.

At this point, you should be able to: 

* Create a new domain and add users with PostfixAdmin
* Send mail to those users via SMTP to port 25
* Retrieve mail using the user's full email and password (e.g. username: user@example.com  password: ChangeMe)

Value Add Features

If you followed the guide above, you now have a functional mail server with many interconnected parts. The features below assume that the server is already running as described above. You should be able to add any or all of these features below to further enhance the mail service.

Virus Scanning

This procedure uses clamav and the postfix content_filter mechanism to scan inbound and outbound email for viruses. Infected emails are dropped. Clean emails are tagged with a "scanned by clamav" header.

  • Install clamav and clamsmtp:
apk add acf-clamav clamsmtp
  • Edit the /etc/clamav/clamd.conf file if desired (not necessary in most cases)
  • Edit /etc/clamsmtpd.conf and verify the following lines
OutAddress: 10026
Listen: 127.0.0.1:10025                                               
Header: X-Virus-Scanned: ClamAV using ClamSMTP
Action: drop
User: clamav
  • Start the daemons
rc-update add clamd
rc-update add clamsmtpd
rc-service clamd start
rc-service clamsmtpd start
  • Verify clamsmtp is listening on port 10025:
netstat -anp | grep clamsmtp

content_filter = scan:[127.0.0.1]:10025
    • edit /etc/postfix/master.cf and add
# AV scan filter (used by content_filter)
scan      unix  -       -       n       -       16      smtp
        -o smtp_send_xforward_command=yes
        -o smtp_enforce_tls=no
# For injecting mail back into postfix from the filter
127.0.0.1:10026 inet  n -       n       -       16      smtpd
        -o content_filter=
        -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks
        -o smtpd_helo_restrictions=
        -o smtpd_client_restrictions=
        -o smtpd_sender_restrictions=
        -o smtpd_recipient_restrictions=permit_mynetworks,reject
        -o mynetworks_style=host
        -o smtpd_authorized_xforward_hosts=127.0.0.0/8
  • postfix reload
  • Send and email into a local virtual domain - it should have the X-Virus-Scanned: ClamAV using ClamSMTP header.

Relay for Authenticated Users

As configured above, the mail server accepts email from the Internet, but it does not relay email. If it is a perimeter exchanger for a protected network, then you can add the protected networks to the mynetworks configuration line in /etc/postfix/main.cf

This configuration change allows remote users to authenticate against the mail server and relay through it. The rules for relaying are:

  • Only authenticated users can relay
  • Authentication Credentials must be encrypted with TLS or SSL
  • Allow Submission and SMTPS ports for relaying (many consumer networks block port 25 - SMTP by default)

The process uses the dovecot authentication mechanism (used with IMAPS) to authenticate users before they are allowed to relay through postfix.

  • Edit /etc/dovecot/dovecot.conf and add the following:
  1. this is for postfix SASL (authenticated users can relay through us)
service auth {
 unix_listener /var/spool/postfix/private/auth {
   group = postfix
   mode = 0660
   user = postfix
 }
 unix_listener /var/spool/postfix/auth-master {
   group = postfix
   mode = 0660
   user = vmail
 }
 user = root
}
  • Restart dovecot
rc-service dovecot restart
  • Edit /etc/postfix/main.cf and add:
# TLS Stuff -- since we allow SASL with tls *only*, we have to set up TLS first                    

smtpd_tls_cert_file = /etc/lighttpd/server-bundle.pem
smtpd_tls_key_file = /etc/lighttpd/server-bundle.pem
smtpd_tls_CAfile = /etc/lighttpd/ca-crt.pem
# If tls_security_level is set to "encrypt", then SMTP rejects 
# unencrypted email (e.g. normal mail) which is bad.
# By setting it to "may" you get TLS encrypted mail from google, slashdot, and other 
# interesting places.  Check your logs to see who
smtpd_tls_security_level = may
# Log info about the negotiated encryption levels
smtpd_tls_received_header = yes
smtpd_tls_loglevel = 1

# SASL - this allows senders to authenticiate themselves
# This along with "permit_sasl_authenticated" in smtpd_recipient_restrictions allows relaying
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
smtpd_sasl_authenticated_header = yes
broken_sasl_auth_clients = yes
smtpd_tls_auth_only = yes
  • Edit /etc/postfix/master.cf and enable the submission and smtps transports. They are probably already at the top of your master.cf file, just commented out:
submission inet n       -       n       -       -       smtpd
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING
smtps     inet  n       -       n       -       -       smtpd
  -o smtpd_tls_security_level=encrypt
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING
  • Verfiy submission and smtps are defined in /etc/services
grep "submission\|ssmtp" /etc/services
submission	587/tcp				# mail message submission
submission	587/udp
smtps		465/tcp		ssmtp		# smtp protocol over TLS/SSL
smtps		465/udp		ssmtp
  • Restart postfix
postfix reload

At this point, you should be able to set up a mail client to relay through the server with TLS (port 587) or SSL (port 465) Note that "plain" authentication is used because the underlying link is encrypted. For example, in Thunderbird leave "secure authentication" unchecked, and choose STARTTLS (or TLS) for the connection security.

Mailbox Quotas

In the default configuration, PostfixAdmin knows about quotas, but they are not enforced. Documentation on the web mentions the vda patch to postfix to enforce quotas. The only bad thing... its a patch. Postfix and Dovecot are both conservative systems, so if the patch isn't in the upstream source, we'll assume there's a good reason. There is a way of using quotas without patches - and it involves using dovecot's lda for local delivery.

  • Replace /etc/dovecot/dovecot.conf with the following:
auth_mechanisms = plain login
auth_username_format = %Lu
#auth_verbose = yes
#auth_debug = yes
#auth_debug_passwords = no

disable_plaintext_auth = no

info_log_path = /var/log/dovecot-info.log
log_path = /var/log/dovecot.log

mail_location = maildir:/var/mail/domains/%d/%n

first_valid_gid = 1000
first_valid_uid = 1000
last_valid_gid = 65535
last_valid_uid = 65535

log_timestamp = "%Y-%m-%d %H:%M:%S "
login_greeting = IMAP server ready

protocols = imap

service anvil {
  client_limit = 2100
}

service auth {
  unix_listener /var/spool/postfix/auth-master {
    group = postfix
    mode = 0660
    user = vmail
  }
  unix_listener /var/spool/postfix/private/auth {
    group = postfix
    mode = 0660
    user = postfix
  }
  user = root
}

service imap-login {
  inet_listener imap {
    address = 127.0.0.1
    port = 143
  }
  inet_listener imaps {
    address = *
    port = 993
  }
  process_limit = 1024
}

service pop3-login {
  process_limit = 1024
}

service dict {
  unix_listener dict {
    group =
    mode = 0600
    user = vmail
  }
}

ssl_ca = </etc/ssl/certs/<CA Certificate file>
ssl_cert = </etc/ssl/private/<Public part of certificate file>
ssl_key = </etc/ssl/private/<Private part of certificate file>

passdb {
  args = /etc/dovecot/dovecot-pgsql.conf
  driver = sql
}

userdb {
  driver = prefetch
}

userdb {
  args = /etc/dovecot/dovecot-pgsql.conf
  driver = sql
}

plugin {
  quota = dict:user::proxy::quotadict

  autocreate = Trash
  autocreate2 = Spam
  autocreate3 = Sent
  autosubscribe = Trash
  autosubscribe2 = Spam
  autosubscribe3 = Sent
}

protocol imap {
  mail_plugins = autocreate quota imap_quota
}

protocol pop3 {                                                               
         mail_plugins = quota                                                  
}   

dict {
  quotadict = pgsql:/etc/dovecot/dovecot-dict-quota.conf
}

protocol lda {
  auth_socket_path = /var/spool/postfix/auth-master
  mail_plugins = quota
  postmaster_address = postmaster@host.example.com
  sendmail_path = /usr/sbin/sendmail
}
  • edit /etc/dovecot/dovecot-sql.conf and replace the user and password queries with the following (you may not have a user_query yet - add it):
password_query = select username as user, password, 1006 as userdb_uid, 1006 as userdb_gid, '*:bytes=' || quota as userdb_quota_rule from mailbox  where local_part = '%n' and domain = '%d'
user_query = select '/var/mail/domains/' || maildir as home, 1006 as uid, 1006 as gid, '*:bytes=' || quota  as quota_rule from mailbox where local_part = '%n' and domain ='%d'
  • create /etc/dovecot/dovecot-dict-quota.conf
connect = host=localhost dbname=postfix user=postfix password=********

map {
        pattern = priv/quota/storage
        table = quota2
        username_field =username
        value_field = bytes
}

map {
       pattern= priv/quota/messages
       table = quota2
       username_field = username
       value_field = messages
}

Again, change the password above to your postfix user password, and protect the file from prying eyes: 

 chown dovecot:root /etc/dovecot/dovecot-sql.conf
 chmod 600 /etc/dovecot/dovecot-sql.conf
 chown dovecot:root /etc/dovecot/dovecot-dict-quota.conf
 chmod 600 /etc/dovecot/dovecot-dict-quota.conf

Side note: The Dovecot Quota Documentation mentions the need for a trigger with pgsql. This was created in the PostfixAdmin install, which is why you instantiated the pgsql language when creating the database. If not, you will need to create the trigger, to reference the quota2 table, not the quota table mentioned in the dovecot docs.


  • create a new transport for the dovecot lda. Add the following to /etc/postfix/master.cf:
# The dovecot deliver lda
dovecot   unix  -       n       n       -       -       pipe
  flags=DRhu user=vmail:vmail argv=/usr/libexec/dovecot/deliver -f ${sender} -d ${user}@${nexthop}
  • Edit the /etc/postfix/main.cf. Replace
virtual_transport = virtual

with 

virtual_transport = dovecot
dovecot_destination_recipient_limit = 1

Change permissions on the /var/log/dovecot* log files, so that the vmail user can write to them: 

chown vmail:vmail /var/log/dovecot*

Restart Postfix and Dovecot: 

rc-service postfix restart
rc-service dovecot restart

TODO This will cause over-quota emails to bounce. Which could be a source of backscatter. We need a way of checking quota limits after RBL checking but before the message is accepted in the queue.

WebMail (RoundCube)

RoundCube is an "ajax /Web2.0" web-mail client. These instructions are for the Alpine Linux 2.2 repository

  • Verify that you have at least the following in /etc/postfix/main.cf. Unless you have followed the Relay for Authenticated Users section above, set smtpd_tls_auth_only = no, otherwise leave it set to yes:
# SASL - this allows senders to authenticiate themselves
# This along with "permit_sasl_authenticated" in smtpd_recipient_restrictions allows relaying
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/dovecot-auth.sock
smtpd_sasl_auth_enable = yes
smtpd_sasl_authenticated_header = yes
# Set the next line to no if TLS auth is not configured 
smtpd_tls_auth_only = no
  • Ensure you have followed section Relay_for_Authenticated_Users.
  • Restart the relevant services:
rc-service postfix restart
rc-service dovecot restart
  • Add the package and related php modules:
apk add roundcubemail php-xml php-openssl php-mcrypt php-gd php-iconv php-dom php-intl
  • Link the roundcube application back into the docroot
ln -s /usr/share/webapps/roundcube /var/www/domains/host.example.com/www/roundcube
  • Install roundcubemail-install package
apk add roundcubemail-installer
  • Follow the instructions in /usr/share/webapps/roundcube/INSTALL:
cd /usr/share/webapps/roundcube
chown -R lighttpd:lighttpd /var/log/roundcube

su postgres
createuser roundcube
  Shall the new role be a superuser? (y/n) n
  Shall the new role be allowed to create databases? (y/n) n
  Shall the new role be allowed to create more new roles? (y/n) y
createdb -O roundcube -E UNICODE -T template0 roundcubemail
psql roundcubemail
  roundcubemail=# ALTER USER roundcube WITH PASSWORD 'the_new_password';
  roundcubemail=# \c - roundcube
  roundcubemail=> \i /usr/share/webapps/roundcube/SQL/postgres.initial.sql
  roundcubemail=> \q
exit
  • Edit /etc/php/php.ini and set date.timezone to your local timezone, or to UTC
  • Restart lighttpd to verify the new php libraries are used
rc-service lighttpd restart
  • Enable installer mode in /etc/roundcube/main.inc.php file:
$rcmail_config['enable_installer'] = true;
  • Point your browser to https://host.example.com/roundcube/installer
  • Start installation

For the specific configuration parameters in the install step:

Property Setting
enable_spellcheck disabled
identities_level one identity with possibility to edit all params but not email address
log driver syslog
sylog_id roundcube
syslog_facility mailsubsystem
db_dnsw pgsql properties, as described above
imap_host 127.0.0.1
auto_create_user enabled
smtp_server 127.0.0.1
smtp_port 25
smtp_user/smtp_pass enable Use Current IMAP username and password for SMTP authentication
smtp_log enable (optional, but gives additional log record)

The other items can be left at default settings, or adjusted if desired.

  • Follow the instructions in step 2 of the install to copy the files to the server
  • You should now be able to get to roundcube at https://host.example.com/roundcube
  • After its working, the INSTALL file recommends removing the install directory.
apk del roundcubemail-installer
  • Disable installer mode in /etc/roundcube/main.inc.php file:
$rcmail_config['enable_installer'] = false;
  • Change the ownership and permissions
cd /usr/share/webapps/roundcube
chown -R root:root LICENSE UPGRADING INSTALL README CHANGELOG
chmod -R 600 LICENSE UPGRADING INSTALL README CHANGELOG
  • If needed customize logos such as watermark.gif, roundcube_logo.gif, favicon.ico
  • If you would like to disable displaying of standard logos update template files accordingly
  • Comment all entries like <div ... img src="/images/roundcube_logo.png"... in files:
includes/header.html 
templates/error.html
templates/messageprint.html
templates/login.html
templates/printmessage.html
  • Comment all entries like <img src="/images/watermark.gif"... in files:
templates/identities.html
templates/messageerror.html
watermark.html

Enable Plug-ins

RoundCube has various useful plug-ins, which could be found in /usr/share/webapps/roundcube/plugins directory. For example you may want to enable password plug-in to let users change their passwords directly from RoundCube using an extra Password Tab added to User Settings.

  • Grant limited permissions for roundcube database role
psql -U postgres postfix
  postfix=# GRANT UPDATE (password,modified) ON mailbox TO roundcube;
  postfix=# GRANT SELECT (username) ON mailbox TO roundcube;
  postfix=# GRANT INSERT ON log TO roundcube;
  postfix=# \q
  • Setup password plug-in parameters in /usr/share/webapps/roundcube/plugins/password/config.inc.php
mv /usr/share/webapps/roundcube/plugins/password/config.inc.php.dist /usr/share/webapps/roundcube/plugins/password/config.inc.php
vi /usr/share/webapps/roundcube/plugins/password/config.inc.php

$rcmail_config['password_minimum_length'] = 7;
$rcmail_config['password_require_nonalpha'] = true;
...
$rcmail_config['password_db_dsn'] = 'pgsql://roundcube:<roundcube_password>@localhost/postfix';
...
$rcmail_config['password_query'] = "UPDATE mailbox set password = %c, modified = NOW() where username = %u; INSERT INTO log (timestamp,username,domain,action,data) VALUES (NOW(),%u || ' (' || %h || ')',%d,'edit_password',%u)";
  • Enable password plug-in
vi /usr/share/webapps/roundcube/config/main.inc.php

...
$rcmail_config['plugins'] = array('password');
  • Enable create_default_folders for RoundCube
vi /usr/share/webapps/roundcube/config/main.inc.php

...
$rcmail_config['create_default_folders'] = TRUE;
...

OpenLDAP based Address Book

This OpenLDAP configuration uses the SQL backend, which represents information stored in PostgreSQL as an LDAP subtree for Address Book functionality for email lookups, user authentication or even replication account information between sites. This procedure uses some metainformation to translate LDAP queries to SQL queries, leaving relational schema untouched, which allows SQL and LDAP applications to inter-operate without replication, and exchange data as needed. The SQL backend uses UnixODBC to connect to PostgresSQL.

  • Install OpenLDAP and ODBC
apk add openldap libldap openldap-back-sql php-ldap unixodbc psqlodbc ca-certificates
  • Update "postfix" database (it will add 'id' columns to mailbox and domain tables, also will create tables and views to represent LDAP metainformation)

Note: These instructions are for example domain example.com. So make sure you replaced all entries of 'example' and 'com' according to your domain name parts.

Put the following into a new file called script: 

ALTER TABLE domain ADD COLUMN id SERIAL; 
ALTER TABLE mailbox ADD COLUMN id SERIAL; 

CREATE TABLE ldap_entry_objclasses (
    entry_id integer NOT NULL,
    oc_name character varying(64)
);

CREATE TABLE ldap_oc_mappings (
    name character varying(64) NOT NULL,
    keytbl character varying(64) NOT NULL,
    keycol character varying(64) NOT NULL,
    create_proc character varying(255),
    delete_proc character varying(255),
    expect_return integer NOT NULL
);

ALTER TABLE ldap_oc_mappings ADD COLUMN id SERIAL;
ALTER TABLE ldap_oc_mappings ADD PRIMARY KEY (id);

CREATE TABLE ldap_attr_mappings (
    oc_map_id integer NOT NULL REFERENCES ldap_oc_mappings(id),
    name character varying(255) NOT NULL,
    sel_expr character varying(255) NOT NULL,
    sel_expr_u character varying(255),
    from_tbls character varying(255) NOT NULL,
    join_where character varying(255),
    add_proc character varying(255),
    delete_proc character varying(255),
    param_order integer NOT NULL,
    expect_return integer NOT NULL
);

ALTER TABLE ldap_attr_mappings ADD COLUMN id SERIAL;
ALTER TABLE ldap_attr_mappings ADD PRIMARY KEY (id);

CREATE VIEW ldap_dcs AS
    ((SELECT (domain.id + 100000) AS id,
            ('dc='::text || replace((domain.domain)::text, '.'::text, ',dc='::text)) AS dn,
            1 AS oc_map_id,
            100000 AS parent,
            0 AS keyval,
            domain.domain
     FROM domain
     WHERE domain.domain <> 'ALL')
      UNION
     (SELECT 100000 AS id,
           ('dc=' || regexp_replace((domain.domain)::text, '.*\\.', ''::text)) AS dn,
           1 AS oc_map_id,
           0 AS parent,
           0 AS keyval,
           (regexp_replace((domain.domain)::text, '.*\\.', ''::text)) AS domain
      FROM domain
      WHERE domain.domain <> 'ALL'
      LIMIT 1));

CREATE VIEW ldap_entries AS
    SELECT mailbox.id,
    ((('cn='::text || initcap(replace(split_part((mailbox.username)::text, '@'::text, 1), '.'::text, ' '::text))) || ',dc='::text) ||
             replace(regexp_replace((mailbox.username)::text, '.*@', ''::text), '.'::text, ',dc='::text)) AS dn,
          1 AS oc_map_id,
          (SELECT ldap_dcs.id
           FROM ldap_dcs
           WHERE ((ldap_dcs.domain)::text = (mailbox.domain)::text)) AS parent,
           mailbox.id AS keyval
           FROM mailbox
           UNION
           SELECT ldap_dcs.id,
                  ldap_dcs.dn,
                  ldap_dcs.oc_map_id,
                  ldap_dcs.parent,
                  ldap_dcs.keyval
           FROM ldap_dcs;

Question to experts: Is this normal to have in this script "WARNING: nonstandard use of \\ in a string literal"?

Finally, execute the commands in the file with: 

cat script | psql -U postfix postfix
rm script
  • Fill out LDAP tables according to following example (make sure to separate values with TABs):

Put the following into a new file called script: 

COPY ldap_oc_mappings (id, name, keytbl, keycol, create_proc, delete_proc, expect_return) FROM stdin;
1	exampleBox	mailbox	id	\N	\N	1
\.
COPY ldap_attr_mappings (id, oc_map_id, name, sel_expr, sel_expr_u, from_tbls, join_where, add_proc, delete_proc, param_order, expect_return) FROM stdin;
1	1	displayName	mailbox.name	\N	mailbox	\N	\N	\N	3	0
2	1	mail	mailbox.username	\N	mailbox	\N	\N	\N	3	0
3	1	cn	mailbox.name	\N	mailbox	\N	\N	\N	3	0
4	1	userPassword	'{CRYPT}'||mailbox.password	\N	mailbox	\N	\N	\N	3	0
\.

Finally, execute the commands in the file with: 

cat script | psql -U postfix postfix
rm script
  • Check that "ldap_dcs" view looks something like this:
echo 'select * from ldap_dcs' | psql -U postgres postfix

   id   |             dn              | oc_map_id | parent | keyval |       domain       
--------+-----------------------------+-----------+--------+--------+--------------------
 100000 | dc=com                      |         1 |      0 |      0 | com
 100001 | dc=example,dc=com           |         1 | 100000 |      0 | example.com
  • Check that "ldap_entries" view looks something like this:
echo 'select * from ldap_entries' | psql -U postgres postfix

   id   |                          dn                           | oc_map_id | parent | keyval 
--------+-------------------------------------------------------+-----------+--------+--------
    1   | cn=address1,dc=example,dc=com                         |         1 | 100001 |    1
...
   123  | cn=address123,dc=example,dc=com                       |         1 | 100001 |    1
 100000 | dc=com                                                |         1 |      0 |    0
 100001 | dc=example,dc=com                                     |         1 | 100000 |    0
  • Configure ODBC parameters

Edit /etc/odbc.ini: 

[PostgreSQL]
Description             = Connection to Postgres
Driver                  = PostgreSQL
Trace                   = Yes
TraceFile               = sql.log
Database                = postfix
Servername              = 127.0.0.1
UserName                =
Password                =
Port                    = 5432
Protocol                = 6.4
ReadOnly                = No
RowVersining            = No
ShowSystemTables        = No
ShowOidColumn           = No
FakeOidIndex            = No
ConnSettings            =

Edit /etc/odbcinst.ini: 

[PostgreSQL]
Description     = PostgreSQL driver for Linux
Driver          = /usr/lib/psqlodbcw.so
Setup           = /usr/lib/libodbcpsqlS.so
FileUsage       = 1
  • Test ODBC connection
echo "select * from domain;" | isql PostgreSQL postgres
  • Provide permission to certificate for LDAP server
chown ldap /etc/lighttpd/server-bundle.pem
  • Edit LDAP schema

Edit /etc/openldap/schema/example.com.schema: 

attributetype ( 0.9.2342.19200300.100.1.3
	NAME ( 'mail' 'rfc822Mailbox' )
	DESC 'RFC1274: RFC822 Mailbox'
        EQUALITY caseIgnoreIA5Match
        SUBSTR caseIgnoreIA5SubstringsMatch
        SYNTAX 1.3.6.1.4.1.1466.115.121.1.26{256} )

attributetype ( 2.16.840.1.113730.3.1.241
	NAME 'displayName'
	DESC 'RFC2798: preferred name to be used when displaying entries'
	EQUALITY caseIgnoreMatch
	SUBSTR caseIgnoreSubstringsMatch
	SYNTAX 1.3.6.1.4.1.1466.115.121.1.15
	SINGLE-VALUE )

objectclass   ( 2.16.840.1.113730.3.2.2
        NAME 'exampleBox'
	DESC 'example.com mailbox'
	MUST ( displayName $ mail $ userPassword )
	)

# RFC 1274 + RFC 2247
attributetype ( 0.9.2342.19200300.100.1.25
        NAME ( 'dc' 'domainComponent' )
        DESC 'RFC1274/2247: domain component'
        EQUALITY caseIgnoreIA5Match
        SUBSTR caseIgnoreIA5SubstringsMatch
        SYNTAX 1.3.6.1.4.1.1466.115.121.1.26 SINGLE-VALUE )

attributetype ( 2.5.4.46 NAME 'dnQualifier'
        DESC 'RFC2256: DN qualifier'
        EQUALITY caseIgnoreMatch
        ORDERING caseIgnoreOrderingMatch
        SUBSTR caseIgnoreSubstringsMatch
        SYNTAX 1.3.6.1.4.1.1466.115.121.1.44 )
  • Configure LDAP server

Edit /etc/openldap/slapd.conf: 

include         /etc/openldap/schema/example.com.schema
pidfile         /var/run/openldap/slapd.pid
argsfile        /var/run/openldap/slapd.args

# Uncomment next five TLS... lines if you want to use LDAPs (secured). Probably you don't want it...
#TLSCipherSuite HIGH
#TLSCACertificateFile /etc/lighttpd/ca-crt.pem
#TLSCertificateFile /etc/lighttpd/server-bundle.pem
#TLSCertificateKeyFile /etc/lighttpd/server-bundle.pem
#TLSVerifyClient never 

# This is needed for proper representation of MD5-CRYPT format stored in database
#  see more details in https://strugglers.net/~andy/blog/2010/01/23/openldap-and-md5crypt/
password-hash  {CRYPT}
password-crypt-salt-format "$1$%.8s"

loglevel        stats
moduleload	/usr/lib/openldap/back_sql.so
sizelimit 3000

database        sql

dbname		PostgreSQL
dbuser		postfix
dbpasswd	*****

suffix          "dc=example,dc=com"

upper_func      "upper"
strcast_func    "text"
concat_pattern  "?||?"
has_ldapinfo_dn_ru      no
lastmod         off

access to attrs=userPassword by * auth

access to * by peername.ip=127.0.0.1 read
#           by peername.ip=<IP>%<netmask> read
#           by peername.ip=<IP> read
	    by users read
  • Set permissions for slapd.conf
chown ldap:ldap /etc/openldap/slapd.conf
  • Configure startup parameters to make sure that LDAP server start AFTER PostgreSQL and listens on localhost with clear text and public IP with SSL. In case you uncommented TLS lines in slapd.conf use this string: OPTS="-h 'ldaps:// ldap://'"

Edit /etc/conf.d/slapd: 

rc_need="postgresql" 
OPTS="-h 'ldap://'"
  • Start LDAP server
rc-update add slapd default
rc-service slapd start
  • Configure LDAP client utilities. In case you uncommented TLS lines in slapd.conf replace ldap with ldaps

Edit /etc/openldap/ldap.conf 

BASE	dc=example,dc=com
URI	ldap://host.example.com

# Uncomment next three TLS... lines if you want to use LDAPs (secured). Probably you don't want it...
#TLS_CACERT /etc/lighttpd/ca-crt.pem
#TLS_CERT /etc/lighttpd/server-bundle.pem
#TLS_KEY /etc/lighttpd/server-bundle.pem
  • Test LDAP server
ldapsearch -z 3
ldapsearch -z 3 -x -W -D cn=admin,dc=example,dc=com
ldapsearch -z 3 -x -W -D cn=address1,dc=example,dc=com
  • Configure RoundCube webmail for email lookups

In order to enable php-ldap support you need to restart lighttpd server 

rc-service lighttpd restart

Edit /etc/roundcube/main.inc.php: 

$rcmail_config['ldap_debug'] = false;
...
$rcmail_config['address_book_type'] = 'sql';

$rcmail_config['ldap_public']['example.com'] = array(
  'name'          => 'example.com',
  'hosts'         => array('127.0.0.1'),
  'port'          => 389,
  'use_tls'         => false,
  'user_specific' => false,
  'base_dn'       => 'dc=example,dc=com',
  'bind_dn'       => '',
  'bind_pass'     => '',
  'writable'      => false,
  'LDAP_Object_Classes' => array("top", "exampleBox"),
  'required_fields'     => array("cn", "sn", "mail"),
  'LDAP_rdn'      => 'mail',
  'ldap_version'  => 3,
  'search_fields' => array('mail', 'cn', 'sn', 'givenName'),
  'name_field'    => 'cn',
  'email_field'   => 'mail',
  'surname_field' => 'sn',
  'firstname_field' => 'gn',
  'sort'          => 'cn',
  'scope'         => 'sub',
  'filter'        => '(objectClass=*)', // Construct here any filter you need
  'fuzzy_search'  => true);

$rcmail_config['autocomplete_addressbooks'] = array('sql','example.com');
  • Fix PostfixAdmin to work with the new table definition

Edit /var/www/domains/example.com/www/postfixadmin/list-domain.php. Replace the line: 

   SELECT domain.* , COUNT( DISTINCT mailbox.username ) AS mailbox_count

With the lines: 

   SELECT domain.domain, domain.description, domain.aliases, domain.mailboxes,
   domain.maxquota, domain.quota, domain.transport, domain.backupmx, domain.created,
   domain.modified, domain.active, COUNT( DISTINCT mailbox.username ) AS mailbox_count

log rotation

Ensure the busybox cron service is started and is configured to auto-start: 

rc-service cron start
rc-update add cron default

Add log rotate: 

apk add logrotate

Edit /etc/logrotate.conf as desired, but the defaults should be sufficient for most people.

Optional: Configure Web Server Virtual Domains

Note: These steps can be done in addition to the default lighttpd configuration above, which allows you to access the ACF, PostfixAdmin and Roundcube interfaces as subfolders of one web service.

Note: If you provide SSL access for multiple domain site you may need to follow https://redmine.lighttpd.net/projects/lighttpd/wiki/Docs:SSL#SSL-on-multiple-domains in order to provide multi-domain certificates. If you would like to redirect hosts to their secure equivalents use the following instructions https://redmine.lighttpd.net/projects/lighttpd/wiki/HowToRedirectHttpToHttps.

This server hosts three separate web applications, and these can be handled as three different virtual domains on the same web server. They will be distinguished by their DNS names, so you can choose domains for the three separate services (or at least the ones you want to publish):

  • ACF - Alpine Configuration Framework for managing the server
  • PostfixAdmin - for managing the postfix installation
  • RoundCube - for accessing individual mailboxes

Choose three different domains (from here on known as ACF_DOMAIN, POSTFIXADMIN_DOMAIN, and ROUNDCUBE_DOMAIN) and configure DNS for all three to point to the IP address of your host. These should be DNS A records.

Then, configure lighttpd to handle the three separate domains by editing /etc/lighttpd/lighttpd.conf: 

 $HTTP["host"] == "ACF_DOMAIN" {
	simple-vhost.server-root   = "/var/www/domains/"
	simple-vhost.default-host  = "/ACF_DOMAIN/"
	simple-vhost.document-root = "www/"
 }

 $HTTP["host"] == "POSTFIXADMIN_DOMAIN" {
	simple-vhost.server-root   = "/var/www/domains/"
	simple-vhost.default-host  = "/POSTFIXADMIN_DOMAIN/"
	simple-vhost.document-root = "www/"
 }

 $HTTP["host"] == "ROUNDCUBE_DOMAIN" {
	simple-vhost.server-root   = "/var/www/domains/"
	simple-vhost.default-host  = "/ROUNDCUBE_DOMAIN/"
	simple-vhost.document-root = "www/"
 }

And, then link the appropriate www directories. 

 mkdir -p /var/www/domains/ACF_DOMAIN
 ln -s /usr/share/acf/www /var/www/domains/ACF_DOMAIN/www

 mkdir -p /var/www/domains/POSTFIXADMIN_DOMAIN
 ln -s /var/www/domains/host.example.com/www/postfixadmin /var/www/domains/POSTFIXADMIN_DOMAIN/www

 mkdir -p /var/www/domains/ROUNDCUBE_DOMAIN
 ln -s /usr/share/webapps/roundcube /var/www/domains/ROUNDCUBE_DOMAIN/www

Optional: Enable compression in Lighttpd

  • Uncomment mod_compress and mod_setenv and modify website section as follows
mkdir -p /var/lib/lighttpd/cache
chown lighttpd:lighttpd  /var/lib/lighttpd/cache

vi /etc/lighttpd/lighttpd.conf

...
"mod_setenv",
"mod_compress",
...
$HTTP["host"] == "ROUNDCUBE_DOMAIN" {
 ...
 static-file.etags = "enable"
 etag.use-mtime = "enable"
 $HTTP["url"] =~ "^/(plugins|skins|program)" { setenv.add-response-header  = ( "Cache-Control" => "public, max-age=2592000") }
 compress.cache-dir   = var.statedir + "/cache/compress"
 compress.filetype = ("text/plain", "text/html", "text/javascript", "text/css", "text/xml", "image/gif", "image/png")
}
Retrieved from "https://wiki.alpinelinux.org/w/index.php?title=ISP_Mail_Server_2.x_HowTo&oldid=25937"