How to setup a Alpine Linux mirror: Difference between revisions

From Alpine Linux
(initial howto configure rsync and lighttpd for a mirror)
mNo edit summary
(31 intermediate revisions by 8 users not shown)
Line 1: Line 1:
== Introduction ==
This document describes how to set up an Alpine Linux mirror and make it available via http and rsync.
This document describes how to set up an Alpine Linux mirror and make it available via http and rsync.


Line 6: Line 7:
* set up lighttpd for http access
* set up lighttpd for http access
* set up rsync so other mirrors can rsync from you
* set up rsync so other mirrors can rsync from you
Make sure that you have enough disk space.
Current (2019-06-15) disk usage in GB:
{|class="wikitable"
!edge
!v2.4
!v2.5
!v2.6
!v2.7
!v3.0
!v3.1
!v3.2
!v3.3
!v3.4
!v3.5
!v3.6
!v3.7
!v3.8
!v3.9
!total
|-
|127.7
|18.9
|10.4
|13.0
|16.6
|16.5
|17.5
|14.5
|20.4
|24.3
|26.8
|48.3
|46.8
|74.0
|82.3
|'''558.2'''
|}
Script used to calculate the size:
<pre>
#!/usr/bin/env bash
total=0
dest="$(mktemp -d)"
for dir in edge v2.4 v2.5 v2.6 v2.7 v3.0 v3.1 v3.2 v3.3 v3.4 v3.5 v3.6 v3.7 v3.8 v3.9; do
    old_total="$total"
    src="rsync://rsync.alpinelinux.org/alpine/$dir/"
    size=`rsync -a -n --stats "$src" "$dest" | grep '^Total file size' | tr -d ',' | awk '{ print $4 }'`
    total=$(("$old_total" + "$size"))
    echo "$dir: $size" | awk '{ print $1 sprintf("%.1f", $2/1073741824) }'
done
echo "total: $total" | awk '{ print $1 sprintf("%.1f", $2/1073741824) }'
rm -r "$dest"
</pre>


== Setting up the cron job ==
== Setting up the cron job ==
Line 13: Line 74:
Save the following file as ''/etc/periodic/hourly/alpine-mirror''
Save the following file as ''/etc/periodic/hourly/alpine-mirror''
<pre>
<pre>
#!/bin/sh
#!/usr/bin/env sh


# make sure we never run 2 rsync at the same time
# make sure we never run 2 rsync at the same time
lockfile="/tmp/alpine-mirror.lock"
lockfile="/tmp/alpine-mirror.lock"
if [ -z "$flock" ] ; then
if [ -z "$flock" ] ; then
   exec env flock=1 flock -n $lockfile $0 "$@"
   exec env flock=1 flock -n $lockfile "$0" "$@"
fi
fi


Line 24: Line 85:
dest=/var/www/localhost/htdocs/alpine/
dest=/var/www/localhost/htdocs/alpine/


# uncomment this to only include v1.8 or newer
# uncomment this to exclude old v2.x branches
#exclude="--exclude v1.[1-7]"
#exclude="--exclude v2.*"


mkdir -p "$dest"
mkdir -p "$dest"
/usr/bin/rsync -prua \
/usr/bin/rsync \
         $exclude \
        --archive \
        --update \
         --hard-links \
         --delete \
         --delete \
        --delete-after \
        --delay-updates \
         --timeout=600 \
         --timeout=600 \
         --delay-updates \
         $exclude \
        --delete-after \
         "$src" "$dest"
         "$src" "$dest"


</pre>
</pre>
(or use [https://gist.github.com/jirutka/288c6fff7c0b8a835d143686207316be this script])


Make it executable:
Make it executable:
Line 44: Line 110:


== Setting up HTTP access via lighttpd ==
== Setting up HTTP access via lighttpd ==
Install the lighttpd server
Install the lighttpd server
{{Cmd|apk add lighttpd}}
{{Cmd|apk add lighttpd}}
Line 49: Line 116:
Enable dir listings by uncommenting the following line in ''/etc/lighttpd/lighttpd.conf'':
Enable dir listings by uncommenting the following line in ''/etc/lighttpd/lighttpd.conf'':
  dir-listing.activate      = "enable"
  dir-listing.activate      = "enable"
Also set cache-control to force cache revalidate every 30 mins. Uncomment mod_setenv in ''/etc/lighttpd/lighttpd.conf'':
"mod_setenv",
Add also the following lines to ''/etc/lighttpd/lighttpd.conf'':
setenv.add-response-header += (         
        "Cache-Control" => "must-revalidate"
)


Start lighttpd and make it start at boot:
Start lighttpd and make it start at boot:
{{Cmd|rc-service lighttpd start
{{Cmd|rc-service lighttpd start
rc-update add lighttpd}}
rc-update add lighttpd}}
{{Note|You may wish to consider [[Darkhttpd]] as an alternative to [[Lighttpd]]
If so, simply install, start and auto-start the webserver:
{{Cmd|apk add darkhttpd && rc-service darkhttpd start && rc-update add darkhttpd}}
Darkhttpd will, by default, offer directory listings and serve data from /var/www/localhost/htdocs/
See the main article on [[Darkhttpd]] for more configuration options}}


== Setting up rsyncd ==
== Setting up rsyncd ==
Line 67: Line 152:
</pre>
</pre>


== Mirror statistics ==
Simple bandwidth statistics can be generated with vnstat.
{{Cmd|apk add vnstat}}
edit /etc/vnstat.conf and replace the interface name with the appropriate one.
Start vnstatd
{{Cmd|/etc/init.d/vnstatd start }}
copy the following script to /etc/periodic/15min/stats and make sure your crond is running.
please not that heredoc should be tab indented or the script will fail. A working copy can be found here: http://tpaste.us/RrMv
<pre>
#!/usr/bin/env sh
output="/var/www/localhost/htdocs/.stats"
nic="eth0"
generate_index() {
    cat <<-EOF
    <!doctype html>
    <html lang="en">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="cache-control" content=no-cache">
        <meta http-equiv="refresh" content="3000">
        <title>Alpine Linux mirror statistics</title>
    </head>
    <body>
        <table border="0">
            <tr><td><img src="summary.png" alt="summary"></td><td><img src="hours.png" alt="hours"></td></tr>
            <tr><td rowspan="2"><img src="days.png" alt="days"></td><td><img src="top10.png" alt="top10"></td></tr>
            <tr><td><img src="months.png" alt="months"></td></tr>
        </table>
    </body>
    </html>
    EOF
}
if  [ ! -f "$output"/index.html ]; then
    mkdir -p $output
    generate_index > "$output"/index.html
fi
for type in hours days months top10 summary hsummary vsummary; do
    vnstati --${type} -i $nic -o $output/${type}.png
done
</pre>
== Update mirror from mqtt ==
If you want your mirror to be really uptodate compared to our master mirror you can subscribe to Alpine Linux message server "msg.alpinelinux.org" and check for upload messages.
Add mqtt-exec to be able to execute processes when specific topics are being send.
{{Cmd| apk add mqtt-exec}}
mqtt-exec supports running multiple time so we need to setup a specific config.
{{Cmd| ln -s /etc/init.d/mqtt-exec /etc/init.d/mqtt-exec.sync-mirror}}
{{Cmd| ln -s /etc/conf.d/mqtt-exec /etc/conf.d/mqtt-exec.sync-mirror}}
edit /etc/conf.d/mqtt-exec.sync-mirror
<pre>
mqtt_topics="rsync/rsync.alpinelinux.org/#"
exec_user="buildozer"
exec_command="/usr/local/bin/sync-mirror"
</pre>
Copy the following file to /usr/local/bin/sync-mirror and make it executable (dont forget to update the variables).
<pre>
#!/bin/sh
src="rsync://rsync.alpinelinux.org/alpine/"
dest="/var/www/localhost/htdocs/alpine/"
lock="/tmp/sync-mirror.lock"
topic="$1"
dir="$2"
[ -z "$flock" ] && exec env flock=1 flock $lock $0 "$@"
if [ -n "$dir" ] && [ -d "$dest/${dir%/*}" ]; then
    logger "Syncing directory: $dir"
    src="${src}${dir%/}/"
    dest="${dest}${dir%/}/"
else
    logger "Syncing all directories"
fi
/usr/bin/rsync \
    --archive \
    --update \
    --verbose \
    --progress \
    --timeout=600 \
    --delay-updates \
    --delete-after \
    "$src" \
    "$dest"
</pre>
And finally start mqtt-exec and let it listen on msg.alpinelinux.org
{{Cmd|/etc/init.d/mqtt-exec.sync-mirror start}}
To make sure you are not missing any packages (in case something goes wrong with MQTT subscription) you can periodically sync all directories by adding the script to cron.
{{Cmd|ln -s /usr/local/bin/sync-mirror /etc/periodic/hourly/sync-mirror}}


Now watch your syslog as it should tell you when it will update directories in your local mirror.


Reference:
[[Category:Server]]
[[File:sync.sh]]
[[Category:Package Manager]]

Revision as of 13:24, 15 June 2019

Introduction

This document describes how to set up an Alpine Linux mirror and make it available via http and rsync.

We will:

  • create the dir where we have the mirror
  • set up a cron job to sync with master mirror every hour
  • set up lighttpd for http access
  • set up rsync so other mirrors can rsync from you

Make sure that you have enough disk space.

Current (2019-06-15) disk usage in GB:

edge v2.4 v2.5 v2.6 v2.7 v3.0 v3.1 v3.2 v3.3 v3.4 v3.5 v3.6 v3.7 v3.8 v3.9 total
127.7 18.9 10.4 13.0 16.6 16.5 17.5 14.5 20.4 24.3 26.8 48.3 46.8 74.0 82.3 558.2

Script used to calculate the size:

#!/usr/bin/env bash

total=0
dest="$(mktemp -d)"

for dir in edge v2.4 v2.5 v2.6 v2.7 v3.0 v3.1 v3.2 v3.3 v3.4 v3.5 v3.6 v3.7 v3.8 v3.9; do
    old_total="$total"
    src="rsync://rsync.alpinelinux.org/alpine/$dir/"
    size=`rsync -a -n --stats "$src" "$dest" | grep '^Total file size' | tr -d ',' | awk '{ print $4 }'`
    total=$(("$old_total" + "$size"))
    echo "$dir: $size" | awk '{ print $1 sprintf("%.1f", $2/1073741824) }'
done

echo "total: $total" | awk '{ print $1 sprintf("%.1f", $2/1073741824) }'
rm -r "$dest"

Setting up the cron job

Install rsync which will be used to sync from the master mirror.

apk add rsync

Save the following file as /etc/periodic/hourly/alpine-mirror

#!/usr/bin/env sh

# make sure we never run 2 rsync at the same time
lockfile="/tmp/alpine-mirror.lock"
if [ -z "$flock" ] ; then
  exec env flock=1 flock -n $lockfile "$0" "$@"
fi

src=rsync://rsync.alpinelinux.org/alpine/ 
dest=/var/www/localhost/htdocs/alpine/

# uncomment this to exclude old v2.x branches
#exclude="--exclude v2.*"

mkdir -p "$dest"
/usr/bin/rsync \
        --archive \
        --update \
        --hard-links \
        --delete \
        --delete-after \
        --delay-updates \
        --timeout=600 \
        $exclude \
        "$src" "$dest"

(or use this script)

Make it executable:

chmod +x /etc/periodic/hourly/alpine-mirror

Now it will sync every hour. (given cron runs)

Setting up HTTP access via lighttpd

Install the lighttpd server

apk add lighttpd

Enable dir listings by uncommenting the following line in /etc/lighttpd/lighttpd.conf:

dir-listing.activate      = "enable"

Also set cache-control to force cache revalidate every 30 mins. Uncomment mod_setenv in /etc/lighttpd/lighttpd.conf:

"mod_setenv",

Add also the following lines to /etc/lighttpd/lighttpd.conf:

setenv.add-response-header += (           
        "Cache-Control" => "must-revalidate"
)

Start lighttpd and make it start at boot:

rc-service lighttpd start rc-update add lighttpd

Note: You may wish to consider Darkhttpd as an alternative to Lighttpd

If so, simply install, start and auto-start the webserver:

apk add darkhttpd && rc-service darkhttpd start && rc-update add darkhttpd

Darkhttpd will, by default, offer directory listings and serve data from /var/www/localhost/htdocs/

See the main article on Darkhttpd for more configuration options

Setting up rsyncd

Add the following lines to /etc/rsyncd.conf:

[alpine]
        path = /var/www/localhost/htdocs/alpine
        comment = My Alpine Linux Mirror

Optionally set a bandwidth limit in /etc/conf.d/rsyncd. In this example we limit to 500Kbytes/s (approx 5Mbit/s)

RSYNC_OPTS="--bwlimit=500"

Mirror statistics

Simple bandwidth statistics can be generated with vnstat.

apk add vnstat

edit /etc/vnstat.conf and replace the interface name with the appropriate one.

Start vnstatd

/etc/init.d/vnstatd start

copy the following script to /etc/periodic/15min/stats and make sure your crond is running. please not that heredoc should be tab indented or the script will fail. A working copy can be found here: http://tpaste.us/RrMv

#!/usr/bin/env sh

output="/var/www/localhost/htdocs/.stats"
nic="eth0"

generate_index() {
    cat <<-EOF
    <!doctype html>
    <html lang="en">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="cache-control" content=no-cache">
        <meta http-equiv="refresh" content="3000">
        <title>Alpine Linux mirror statistics</title>
    </head>
    <body>
        <table border="0">
            <tr><td><img src="summary.png" alt="summary"></td><td><img src="hours.png" alt="hours"></td></tr>
            <tr><td rowspan="2"><img src="days.png" alt="days"></td><td><img src="top10.png" alt="top10"></td></tr>
            <tr><td><img src="months.png" alt="months"></td></tr>
        </table>
    </body>
    </html>
    EOF
}

if  [ ! -f "$output"/index.html ]; then
    mkdir -p $output
    generate_index > "$output"/index.html
fi

for type in hours days months top10 summary hsummary vsummary; do
    vnstati --${type} -i $nic -o $output/${type}.png
done

Update mirror from mqtt

If you want your mirror to be really uptodate compared to our master mirror you can subscribe to Alpine Linux message server "msg.alpinelinux.org" and check for upload messages. Add mqtt-exec to be able to execute processes when specific topics are being send.

apk add mqtt-exec

mqtt-exec supports running multiple time so we need to setup a specific config.

ln -s /etc/init.d/mqtt-exec /etc/init.d/mqtt-exec.sync-mirror

ln -s /etc/conf.d/mqtt-exec /etc/conf.d/mqtt-exec.sync-mirror

edit /etc/conf.d/mqtt-exec.sync-mirror

mqtt_topics="rsync/rsync.alpinelinux.org/#"
exec_user="buildozer"
exec_command="/usr/local/bin/sync-mirror"

Copy the following file to /usr/local/bin/sync-mirror and make it executable (dont forget to update the variables).

#!/bin/sh

src="rsync://rsync.alpinelinux.org/alpine/"
dest="/var/www/localhost/htdocs/alpine/"
lock="/tmp/sync-mirror.lock"
topic="$1"
dir="$2"

[ -z "$flock" ] && exec env flock=1 flock $lock $0 "$@"

if [ -n "$dir" ] && [ -d "$dest/${dir%/*}" ]; then
    logger "Syncing directory: $dir"
    src="${src}${dir%/}/"
    dest="${dest}${dir%/}/"
else
    logger "Syncing all directories"
fi

/usr/bin/rsync \
    --archive \
    --update \
    --verbose \
    --progress \
    --timeout=600 \
    --delay-updates \
    --delete-after \
    "$src" \
    "$dest"

And finally start mqtt-exec and let it listen on msg.alpinelinux.org

/etc/init.d/mqtt-exec.sync-mirror start

To make sure you are not missing any packages (in case something goes wrong with MQTT subscription) you can periodically sync all directories by adding the script to cron.

ln -s /usr/local/bin/sync-mirror /etc/periodic/hourly/sync-mirror

Now watch your syslog as it should tell you when it will update directories in your local mirror.