Alpine Linux - User contributions [en]

How to setup a Alpine Linux mirror

2019-06-15T13:24:36Z

Tetsumaki:

== Introduction ==
This document describes how to set up an Alpine Linux mirror and make it available via http and rsync.

We will:
* create the dir where we have the mirror
* set up a cron job to sync with master mirror every hour
* set up lighttpd for http access
* set up rsync so other mirrors can rsync from you

Make sure that you have enough disk space.

Current (2019-06-15) disk usage in GB:

{|class="wikitable"
!edge
!v2.4
!v2.5
!v2.6
!v2.7
!v3.0
!v3.1
!v3.2
!v3.3
!v3.4
!v3.5
!v3.6
!v3.7
!v3.8
!v3.9
!total
|-
|127.7
|18.9
|10.4
|13.0
|16.6
|16.5
|17.5
|14.5
|20.4
|24.3
|26.8
|48.3
|46.8
|74.0
|82.3
|'''558.2'''
|}

Script used to calculate the size:

<pre>
#!/usr/bin/env bash

total=0
dest="$(mktemp -d)"

for dir in edge v2.4 v2.5 v2.6 v2.7 v3.0 v3.1 v3.2 v3.3 v3.4 v3.5 v3.6 v3.7 v3.8 v3.9; do
old_total="$total"
src="rsync://rsync.alpinelinux.org/alpine/$dir/"
size=`rsync -a -n --stats "$src" "$dest" | grep '^Total file size' | tr -d ',' | awk '{ print $4 }'`
total=$(("$old_total" + "$size"))
echo "$dir: $size" | awk '{ print $1 sprintf("%.1f", $2/1073741824) }'
done

echo "total: $total" | awk '{ print $1 sprintf("%.1f", $2/1073741824) }'
rm -r "$dest"
</pre>

== Setting up the cron job ==
Install rsync which will be used to sync from the master mirror.
{{Cmd|apk add rsync}}

Save the following file as ''/etc/periodic/hourly/alpine-mirror''
<pre>
#!/usr/bin/env sh

# make sure we never run 2 rsync at the same time
lockfile="/tmp/alpine-mirror.lock"
if [ -z "$flock" ] ; then
exec env flock=1 flock -n $lockfile "$0" "$@"
fi

src=rsync://rsync.alpinelinux.org/alpine/
dest=/var/www/localhost/htdocs/alpine/

# uncomment this to exclude old v2.x branches
#exclude="--exclude v2.*"

mkdir -p "$dest"
/usr/bin/rsync \
--archive \
--update \
--hard-links \
--delete \
--delete-after \
--delay-updates \
--timeout=600 \
$exclude \
"$src" "$dest"

</pre>

(or use [https://gist.github.com/jirutka/288c6fff7c0b8a835d143686207316be this script])

Make it executable:
{{Cmd|<nowiki>chmod +x /etc/periodic/hourly/alpine-mirror</nowiki>}}

Now it will sync every hour. (given cron runs)

== Setting up HTTP access via lighttpd ==

Install the lighttpd server
{{Cmd|apk add lighttpd}}

Enable dir listings by uncommenting the following line in ''/etc/lighttpd/lighttpd.conf'':
dir-listing.activate = "enable"

Also set cache-control to force cache revalidate every 30 mins. Uncomment mod_setenv in ''/etc/lighttpd/lighttpd.conf'':
"mod_setenv",

Add also the following lines to ''/etc/lighttpd/lighttpd.conf'':
setenv.add-response-header += (
"Cache-Control" => "must-revalidate"
)

Start lighttpd and make it start at boot:
{{Cmd|rc-service lighttpd start
rc-update add lighttpd}}

{{Note|You may wish to consider [[Darkhttpd]] as an alternative to [[Lighttpd]]

If so, simply install, start and auto-start the webserver:

{{Cmd|apk add darkhttpd && rc-service darkhttpd start && rc-update add darkhttpd}}

Darkhttpd will, by default, offer directory listings and serve data from /var/www/localhost/htdocs/

See the main article on [[Darkhttpd]] for more configuration options}}

== Setting up rsyncd ==
Add the following lines to ''/etc/rsyncd.conf'':
<pre>
[alpine]
path = /var/www/localhost/htdocs/alpine
comment = My Alpine Linux Mirror
</pre>

Optionally set a bandwidth limit in ''/etc/conf.d/rsyncd''. In this example we limit to 500Kbytes/s (approx 5Mbit/s)
<pre>
RSYNC_OPTS="--bwlimit=500"
</pre>

== Mirror statistics ==

Simple bandwidth statistics can be generated with vnstat.

{{Cmd|apk add vnstat}}

edit /etc/vnstat.conf and replace the interface name with the appropriate one.

Start vnstatd

{{Cmd|/etc/init.d/vnstatd start }}

copy the following script to /etc/periodic/15min/stats and make sure your crond is running.
please not that heredoc should be tab indented or the script will fail. A working copy can be found here: http://tpaste.us/RrMv

<pre>
#!/usr/bin/env sh

output="/var/www/localhost/htdocs/.stats"
nic="eth0"

generate_index() {
cat <<-EOF
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="cache-control" content=no-cache">
<meta http-equiv="refresh" content="3000">
<title>Alpine Linux mirror statistics</title>
</head>
<body>
<table border="0">
<tr><td><img src="summary.png" alt="summary"></td><td><img src="hours.png" alt="hours"></td></tr>
<tr><td rowspan="2"><img src="days.png" alt="days"></td><td><img src="top10.png" alt="top10"></td></tr>
<tr><td><img src="months.png" alt="months"></td></tr>
</table>
</body>
</html>
EOF
}

if [ ! -f "$output"/index.html ]; then
mkdir -p $output
generate_index > "$output"/index.html
fi

for type in hours days months top10 summary hsummary vsummary; do
vnstati --${type} -i $nic -o $output/${type}.png
done
</pre>

== Update mirror from mqtt ==

If you want your mirror to be really uptodate compared to our master mirror you can subscribe to Alpine Linux message server "msg.alpinelinux.org" and check for upload messages.
Add mqtt-exec to be able to execute processes when specific topics are being send.

{{Cmd| apk add mqtt-exec}}

mqtt-exec supports running multiple time so we need to setup a specific config.

{{Cmd| ln -s /etc/init.d/mqtt-exec /etc/init.d/mqtt-exec.sync-mirror}}

{{Cmd| ln -s /etc/conf.d/mqtt-exec /etc/conf.d/mqtt-exec.sync-mirror}}

edit /etc/conf.d/mqtt-exec.sync-mirror

<pre>
mqtt_topics="rsync/rsync.alpinelinux.org/#"
exec_user="buildozer"
exec_command="/usr/local/bin/sync-mirror"
</pre>

Copy the following file to /usr/local/bin/sync-mirror and make it executable (dont forget to update the variables).

<pre>
#!/bin/sh

src="rsync://rsync.alpinelinux.org/alpine/"
dest="/var/www/localhost/htdocs/alpine/"
lock="/tmp/sync-mirror.lock"
topic="$1"
dir="$2"

[ -z "$flock" ] && exec env flock=1 flock $lock $0 "$@"

if [ -n "$dir" ] && [ -d "$dest/${dir%/*}" ]; then
logger "Syncing directory: $dir"
src="${src}${dir%/}/"
dest="${dest}${dir%/}/"
else
logger "Syncing all directories"
fi

/usr/bin/rsync \
--archive \
--update \
--verbose \
--progress \
--timeout=600 \
--delay-updates \
--delete-after \
"$src" \
"$dest"
</pre>

And finally start mqtt-exec and let it listen on msg.alpinelinux.org

{{Cmd|/etc/init.d/mqtt-exec.sync-mirror start}}

To make sure you are not missing any packages (in case something goes wrong with MQTT subscription) you can periodically sync all directories by adding the script to cron.

{{Cmd|ln -s /usr/local/bin/sync-mirror /etc/periodic/hourly/sync-mirror}}

Now watch your syslog as it should tell you when it will update directories in your local mirror.

[[Category:Server]]
[[Category:Package Manager]]

How to setup a Alpine Linux mirror

2019-01-24T21:01:46Z

Tetsumaki:

== Introduction ==
This document describes how to set up an Alpine Linux mirror and make it available via http and rsync.

We will:
* create the dir where we have the mirror
* set up a cron job to sync with master mirror every hour
* set up lighttpd for http access
* set up rsync so other mirrors can rsync from you

Make sure that you have enough disk space.

Current (2019-01-24) disk usage in GB:

{|class="wikitable"
!edge
!v2.4
!v2.5
!v2.6
!v2.7
!v3.0
!v3.1
!v3.2
!v3.3
!v3.4
!v3.5
!v3.6
!v3.7
!v3.8
!v3.9
!total
|-
|98.1
|18.9
|10.4
|13.0
|16.6
|16.5
|17.5
|14.5
|20.4
|24.3
|33.8
|45.6
|43.8
|69.3
|66.8
|'''525.4'''
|}

Script used to calculate the size:

<pre>
#!/usr/bin/env bash

total=0
dest="$(mktemp -d)"

for dir in edge v2.4 v2.5 v2.6 v2.7 v3.0 v3.1 v3.2 v3.3 v3.4 v3.5 v3.6 v3.7 v3.8; do
old_total="$total"
src="rsync://rsync.alpinelinux.org/alpine/$dir/"
size=`rsync -a -n --stats "$src" "$dest" | grep '^Total file size' | tr -d ',' | awk '{ print $4 }'`
total=$(("$old_total" + "$size"))
echo "$dir: $size" | awk '{ print $1 sprintf("%.1f", $2/1073741824) }'
done

echo "total: $total" | awk '{ print $1 sprintf("%.1f", $2/1073741824) }'
rm -r "$dest"
</pre>

== Setting up the cron job ==
Install rsync which will be used to sync from the master mirror.
{{Cmd|apk add rsync}}

Save the following file as ''/etc/periodic/hourly/alpine-mirror''
<pre>
#!/usr/bin/env sh

# make sure we never run 2 rsync at the same time
lockfile="/tmp/alpine-mirror.lock"
if [ -z "$flock" ] ; then
exec env flock=1 flock -n $lockfile "$0" "$@"
fi

src=rsync://rsync.alpinelinux.org/alpine/
dest=/var/www/localhost/htdocs/alpine/

# uncomment this to exclude old v2.x branches
#exclude="--exclude v2.*"

mkdir -p "$dest"
/usr/bin/rsync \
--archive \
--update \
--hard-links \
--delete \
--delete-after \
--delay-updates \
--timeout=600 \
$exclude \
"$src" "$dest"

</pre>

(or use [https://gist.github.com/jirutka/288c6fff7c0b8a835d143686207316be this script])

Make it executable:
{{Cmd|<nowiki>chmod +x /etc/periodic/hourly/alpine-mirror</nowiki>}}

Now it will sync every hour. (given cron runs)

== Setting up HTTP access via lighttpd ==

Install the lighttpd server
{{Cmd|apk add lighttpd}}

Enable dir listings by uncommenting the following line in ''/etc/lighttpd/lighttpd.conf'':
dir-listing.activate = "enable"

Also set cache-control to force cache revalidate every 30 mins. Uncomment mod_setenv in ''/etc/lighttpd/lighttpd.conf'':
"mod_setenv",

Add also the following lines to ''/etc/lighttpd/lighttpd.conf'':
setenv.add-response-header += (
"Cache-Control" => "must-revalidate"
)

Start lighttpd and make it start at boot:
{{Cmd|rc-service lighttpd start
rc-update add lighttpd}}

{{Note|You may wish to consider [[Darkhttpd]] as an alternative to [[Lighttpd]]

If so, simply install, start and auto-start the webserver:

{{Cmd|apk add darkhttpd && rc-service darkhttpd start && rc-update add darkhttpd}}

Darkhttpd will, by default, offer directory listings and serve data from /var/www/localhost/htdocs/

See the main article on [[Darkhttpd]] for more configuration options}}

== Setting up rsyncd ==
Add the following lines to ''/etc/rsyncd.conf'':
<pre>
[alpine]
path = /var/www/localhost/htdocs/alpine
comment = My Alpine Linux Mirror
</pre>

Optionally set a bandwidth limit in ''/etc/conf.d/rsyncd''. In this example we limit to 500Kbytes/s (approx 5Mbit/s)
<pre>
RSYNC_OPTS="--bwlimit=500"
</pre>

== Mirror statistics ==

Simple bandwidth statistics can be generated with vnstat.

{{Cmd|apk add vnstat}}

edit /etc/vnstat.conf and replace the interface name with the appropriate one.

Start vnstatd

{{Cmd|/etc/init.d/vnstatd start }}

copy the following script to /etc/periodic/15min/stats and make sure your crond is running.
please not that heredoc should be tab indented or the script will fail. A working copy can be found here: http://tpaste.us/RrMv

<pre>
#!/usr/bin/env sh

output="/var/www/localhost/htdocs/.stats"
nic="eth0"

generate_index() {
cat <<-EOF
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="cache-control" content=no-cache">
<meta http-equiv="refresh" content="3000">
<title>Alpine Linux mirror statistics</title>
</head>
<body>
<table border="0">
<tr><td><img src="summary.png" alt="summary"></td><td><img src="hours.png" alt="hours"></td></tr>
<tr><td rowspan="2"><img src="days.png" alt="days"></td><td><img src="top10.png" alt="top10"></td></tr>
<tr><td><img src="months.png" alt="months"></td></tr>
</table>
</body>
</html>
EOF
}

if [ ! -f "$output"/index.html ]; then
mkdir -p $output
generate_index > "$output"/index.html
fi

for type in hours days months top10 summary hsummary vsummary; do
vnstati --${type} -i $nic -o $output/${type}.png
done
</pre>

== Update mirror from mqtt ==

If you want your mirror to be really uptodate compared to our master mirror you can subscribe to Alpine Linux message server "msg.alpinelinux.org" and check for upload messages.
Add mqtt-exec to be able to execute processes when specific topics are being send.

{{Cmd| apk add mqtt-exec}}

mqtt-exec supports running multiple time so we need to setup a specific config.

{{Cmd| ln -s /etc/init.d/mqtt-exec /etc/init.d/mqtt-exec.sync-mirror}}

{{Cmd| ln -s /etc/conf.d/mqtt-exec /etc/conf.d/mqtt-exec.sync-mirror}}

edit /etc/conf.d/mqtt-exec.sync-mirror

<pre>
mqtt_topics="rsync/rsync.alpinelinux.org/#"
exec_user="buildozer"
exec_command="/usr/local/bin/sync-mirror"
</pre>

Copy the following file to /usr/local/bin/sync-mirror and make it executable (dont forget to update the variables).

<pre>
#!/bin/sh

src="rsync://rsync.alpinelinux.org/alpine/"
dest="/var/www/localhost/htdocs/alpine/"
lock="/tmp/sync-mirror.lock"
topic="$1"
dir="$2"

[ -z "$flock" ] && exec env flock=1 flock $lock $0 "$@"

if [ -n "$dir" ] && [ -d "$dest/${dir%/*}" ]; then
logger "Syncing directory: $dir"
src="${src}${dir%/}/"
dest="${dest}${dir%/}/"
else
logger "Syncing all directories"
fi

/usr/bin/rsync \
--archive \
--update \
--verbose \
--progress \
--timeout=600 \
--delay-updates \
--delete-after \
"$src" \
"$dest"
</pre>

And finally start mqtt-exec and let it listen on msg.alpinelinux.org

{{Cmd|/etc/init.d/mqtt-exec.sync-mirror start}}

To make sure you are not missing any packages (in case something goes wrong with MQTT subscription) you can periodically sync all directories by adding the script to cron.

{{Cmd|ln -s /usr/local/bin/sync-mirror /etc/periodic/hourly/sync-mirror}}

Now watch your syslog as it should tell you when it will update directories in your local mirror.

[[Category:Server]]
[[Category:Package Manager]]

How to setup a Alpine Linux mirror

2019-01-24T20:59:54Z

Tetsumaki:

== Introduction ==
This document describes how to set up an Alpine Linux mirror and make it available via http and rsync.

We will:
* create the dir where we have the mirror
* set up a cron job to sync with master mirror every hour
* set up lighttpd for http access
* set up rsync so other mirrors can rsync from you

Make sure that you have enough disk space.

Current (2018-08-11) disk usage in GB:

{|class="wikitable"
!edge
!v2.4
!v2.5
!v2.6
!v2.7
!v3.0
!v3.1
!v3.2
!v3.3
!v3.4
!v3.5
!v3.6
!v3.7
!v3.8
!v3.9
!total
|-
|98.1
|18.9
|10.4
|13.0
|16.6
|16.5
|17.5
|14.5
|20.4
|24.3
|33.8
|45.6
|43.8
|69.3
|66.8
|'''525.4'''
|}

Script used to calculate the size:

<pre>
#!/usr/bin/env sh

total=0
dest="$(mktemp -d)"

for dir in edge v2.4 v2.5 v2.6 v2.7 v3.0 v3.1 v3.2 v3.3 v3.4 v3.5 v3.6 v3.7 v3.8; do
old_total="$total"
src="rsync://rsync.alpinelinux.org/alpine/$dir/"
size=`rsync -a -n --stats "$src" "$dest" | grep '^Total file size' | tr -d ',' | awk '{ print $4 }'`
total=$(("$old_total" + "$size"))
echo "$dir: $size" | awk '{ print $1 sprintf("%.1f", $2/1073741824) }'
done

echo "total: $total" | awk '{ print $1 sprintf("%.1f", $2/1073741824) }'
rm -r "$dest"
</pre>

== Setting up the cron job ==
Install rsync which will be used to sync from the master mirror.
{{Cmd|apk add rsync}}

Save the following file as ''/etc/periodic/hourly/alpine-mirror''
<pre>
#!/usr/bin/env sh

# make sure we never run 2 rsync at the same time
lockfile="/tmp/alpine-mirror.lock"
if [ -z "$flock" ] ; then
exec env flock=1 flock -n $lockfile "$0" "$@"
fi

src=rsync://rsync.alpinelinux.org/alpine/
dest=/var/www/localhost/htdocs/alpine/

# uncomment this to exclude old v2.x branches
#exclude="--exclude v2.*"

mkdir -p "$dest"
/usr/bin/rsync \
--archive \
--update \
--hard-links \
--delete \
--delete-after \
--delay-updates \
--timeout=600 \
$exclude \
"$src" "$dest"

</pre>

(or use [https://gist.github.com/jirutka/288c6fff7c0b8a835d143686207316be this script])

Make it executable:
{{Cmd|<nowiki>chmod +x /etc/periodic/hourly/alpine-mirror</nowiki>}}

Now it will sync every hour. (given cron runs)

== Setting up HTTP access via lighttpd ==

Install the lighttpd server
{{Cmd|apk add lighttpd}}

Enable dir listings by uncommenting the following line in ''/etc/lighttpd/lighttpd.conf'':
dir-listing.activate = "enable"

Also set cache-control to force cache revalidate every 30 mins. Uncomment mod_setenv in ''/etc/lighttpd/lighttpd.conf'':
"mod_setenv",

Add also the following lines to ''/etc/lighttpd/lighttpd.conf'':
setenv.add-response-header += (
"Cache-Control" => "must-revalidate"
)

Start lighttpd and make it start at boot:
{{Cmd|rc-service lighttpd start
rc-update add lighttpd}}

{{Note|You may wish to consider [[Darkhttpd]] as an alternative to [[Lighttpd]]

If so, simply install, start and auto-start the webserver:

{{Cmd|apk add darkhttpd && rc-service darkhttpd start && rc-update add darkhttpd}}

Darkhttpd will, by default, offer directory listings and serve data from /var/www/localhost/htdocs/

See the main article on [[Darkhttpd]] for more configuration options}}

== Setting up rsyncd ==
Add the following lines to ''/etc/rsyncd.conf'':
<pre>
[alpine]
path = /var/www/localhost/htdocs/alpine
comment = My Alpine Linux Mirror
</pre>

Optionally set a bandwidth limit in ''/etc/conf.d/rsyncd''. In this example we limit to 500Kbytes/s (approx 5Mbit/s)
<pre>
RSYNC_OPTS="--bwlimit=500"
</pre>

== Mirror statistics ==

Simple bandwidth statistics can be generated with vnstat.

{{Cmd|apk add vnstat}}

edit /etc/vnstat.conf and replace the interface name with the appropriate one.

Start vnstatd

{{Cmd|/etc/init.d/vnstatd start }}

copy the following script to /etc/periodic/15min/stats and make sure your crond is running.
please not that heredoc should be tab indented or the script will fail. A working copy can be found here: http://tpaste.us/RrMv

<pre>
#!/usr/bin/env sh

output="/var/www/localhost/htdocs/.stats"
nic="eth0"

generate_index() {
cat <<-EOF
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="cache-control" content=no-cache">
<meta http-equiv="refresh" content="3000">
<title>Alpine Linux mirror statistics</title>
</head>
<body>
<table border="0">
<tr><td><img src="summary.png" alt="summary"></td><td><img src="hours.png" alt="hours"></td></tr>
<tr><td rowspan="2"><img src="days.png" alt="days"></td><td><img src="top10.png" alt="top10"></td></tr>
<tr><td><img src="months.png" alt="months"></td></tr>
</table>
</body>
</html>
EOF
}

if [ ! -f "$output"/index.html ]; then
mkdir -p $output
generate_index > "$output"/index.html
fi

for type in hours days months top10 summary hsummary vsummary; do
vnstati --${type} -i $nic -o $output/${type}.png
done
</pre>

== Update mirror from mqtt ==

If you want your mirror to be really uptodate compared to our master mirror you can subscribe to Alpine Linux message server "msg.alpinelinux.org" and check for upload messages.
Add mqtt-exec to be able to execute processes when specific topics are being send.

{{Cmd| apk add mqtt-exec}}

mqtt-exec supports running multiple time so we need to setup a specific config.

{{Cmd| ln -s /etc/init.d/mqtt-exec /etc/init.d/mqtt-exec.sync-mirror}}

{{Cmd| ln -s /etc/conf.d/mqtt-exec /etc/conf.d/mqtt-exec.sync-mirror}}

edit /etc/conf.d/mqtt-exec.sync-mirror

<pre>
mqtt_topics="rsync/rsync.alpinelinux.org/#"
exec_user="buildozer"
exec_command="/usr/local/bin/sync-mirror"
</pre>

Copy the following file to /usr/local/bin/sync-mirror and make it executable (dont forget to update the variables).

<pre>
#!/bin/sh

src="rsync://rsync.alpinelinux.org/alpine/"
dest="/var/www/localhost/htdocs/alpine/"
lock="/tmp/sync-mirror.lock"
topic="$1"
dir="$2"

[ -z "$flock" ] && exec env flock=1 flock $lock $0 "$@"

if [ -n "$dir" ] && [ -d "$dest/${dir%/*}" ]; then
logger "Syncing directory: $dir"
src="${src}${dir%/}/"
dest="${dest}${dir%/}/"
else
logger "Syncing all directories"
fi

/usr/bin/rsync \
--archive \
--update \
--verbose \
--progress \
--timeout=600 \
--delay-updates \
--delete-after \
"$src" \
"$dest"
</pre>

And finally start mqtt-exec and let it listen on msg.alpinelinux.org

{{Cmd|/etc/init.d/mqtt-exec.sync-mirror start}}

To make sure you are not missing any packages (in case something goes wrong with MQTT subscription) you can periodically sync all directories by adding the script to cron.

{{Cmd|ln -s /usr/local/bin/sync-mirror /etc/periodic/hourly/sync-mirror}}

Now watch your syslog as it should tell you when it will update directories in your local mirror.

[[Category:Server]]
[[Category:Package Manager]]

How to setup a Alpine Linux mirror

2018-08-11T19:21:53Z

Tetsumaki: /* Introduction */

== Introduction ==
This document describes how to set up an Alpine Linux mirror and make it available via http and rsync.

We will:
* create the dir where we have the mirror
* set up a cron job to sync with master mirror every hour
* set up lighttpd for http access
* set up rsync so other mirrors can rsync from you

Make sure that you have enough disk space.

Current (2018-08-11) disk usage in GB:

{|class="wikitable"
!edge
!v2.4
!v2.5
!v2.6
!v2.7
!v3.0
!v3.1
!v3.2
!v3.3
!v3.4
!v3.5
!v3.6
!v3.7
!v3.8
!total
|-
|98.1
|18.9
|10.4
|13.0
|16.6
|16.5
|17.5
|14.5
|20.4
|24.3
|25.7
|40.2
|38.1
|64.8
|'''419.1'''
|}

Script used to calculate the size:

<pre>
#!/usr/bin/env sh

total=0
dest="$(mktemp -d)"

for dir in edge v2.4 v2.5 v2.6 v2.7 v3.0 v3.1 v3.2 v3.3 v3.4 v3.5 v3.6 v3.7 v3.8; do
old_total="$total"
src="rsync://rsync.alpinelinux.org/alpine/$dir/"
size=`rsync -a -n --stats "$src" "$dest" | grep '^Total file size' | tr -d ',' | awk '{ print $4 }'`
total=$(("$old_total" + "$size"))
echo "$dir: $size" | awk '{ print $1 sprintf("%.1f", $2/1073741824) }'
done

echo "total: $total" | awk '{ print $1 sprintf("%.1f", $2/1073741824) }'
rm -r "$dest"
</pre>

== Setting up the cron job ==
Install rsync which will be used to sync from the master mirror.
{{Cmd|apk add rsync}}

Save the following file as ''/etc/periodic/hourly/alpine-mirror''
<pre>
#!/usr/bin/env sh

# make sure we never run 2 rsync at the same time
lockfile="/tmp/alpine-mirror.lock"
if [ -z "$flock" ] ; then
exec env flock=1 flock -n $lockfile "$0" "$@"
fi

src=rsync://rsync.alpinelinux.org/alpine/
dest=/var/www/localhost/htdocs/alpine/

# uncomment this to exclude old v2.x branches
#exclude="--exclude v2.*"

mkdir -p "$dest"
/usr/bin/rsync \
--archive \
--update \
--hard-links \
--delete \
--delete-after \
--delay-updates \
--timeout=600 \
$exclude \
"$src" "$dest"

</pre>

(or use [https://gist.github.com/jirutka/288c6fff7c0b8a835d143686207316be this script])

Make it executable:
{{Cmd|<nowiki>chmod +x /etc/periodic/hourly/alpine-mirror</nowiki>}}

Now it will sync every hour. (given cron runs)

== Setting up HTTP access via lighttpd ==

Install the lighttpd server
{{Cmd|apk add lighttpd}}

Enable dir listings by uncommenting the following line in ''/etc/lighttpd/lighttpd.conf'':
dir-listing.activate = "enable"

Also set cache-control to force cache revalidate every 30 mins. Uncomment mod_setenv in ''/etc/lighttpd/lighttpd.conf'':
"mod_setenv",

Add also the following lines to ''/etc/lighttpd/lighttpd.conf'':
setenv.add-response-header += (
"Cache-Control" => "must-revalidate"
)

Start lighttpd and make it start at boot:
{{Cmd|rc-service lighttpd start
rc-update add lighttpd}}

{{Note|You may wish to consider [[Darkhttpd]] as an alternative to [[Lighttpd]]

If so, simply install, start and auto-start the webserver:

{{Cmd|apk add darkhttpd && rc-service darkhttpd start && rc-update add darkhttpd}}

Darkhttpd will, by default, offer directory listings and serve data from /var/www/localhost/htdocs/

See the main article on [[Darkhttpd]] for more configuration options}}

== Setting up rsyncd ==
Add the following lines to ''/etc/rsyncd.conf'':
<pre>
[alpine]
path = /var/www/localhost/htdocs/alpine
comment = My Alpine Linux Mirror
</pre>

Optionally set a bandwidth limit in ''/etc/conf.d/rsyncd''. In this example we limit to 500Kbytes/s (approx 5Mbit/s)
<pre>
RSYNC_OPTS="--bwlimit=500"
</pre>

== Mirror statistics ==

Simple bandwidth statistics can be generated with vnstat.

{{Cmd|apk add vnstat}}

edit /etc/vnstat.conf and replace the interface name with the appropriate one.

Start vnstatd

{{Cmd|/etc/init.d/vnstatd start }}

copy the following script to /etc/periodic/15min/stats and make sure your crond is running.
please not that heredoc should be tab indented or the script will fail. A working copy can be found here: http://tpaste.us/RrMv

<pre>
#!/usr/bin/env sh

output="/var/www/localhost/htdocs/.stats"
nic="eth0"

generate_index() {
cat <<-EOF
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="cache-control" content=no-cache">
<meta http-equiv="refresh" content="3000">
<title>Alpine Linux mirror statistics</title>
</head>
<body>
<table border="0">
<tr><td><img src="summary.png" alt="summary"></td><td><img src="hours.png" alt="hours"></td></tr>
<tr><td rowspan="2"><img src="days.png" alt="days"></td><td><img src="top10.png" alt="top10"></td></tr>
<tr><td><img src="months.png" alt="months"></td></tr>
</table>
</body>
</html>
EOF
}

if [ ! -f "$output"/index.html ]; then
mkdir -p $output
generate_index > "$output"/index.html
fi

for type in hours days months top10 summary hsummary vsummary; do
vnstati --${type} -i $nic -o $output/${type}.png
done
</pre>

== Update mirror from mqtt ==

If you want your mirror to be really uptodate compared to our master mirror you can subscribe to Alpine Linux message server "msg.alpinelinux.org" and check for upload messages.
Add mqtt-exec to be able to execute processes when specific topics are being send.

{{Cmd| apk add mqtt-exec}}

mqtt-exec supports running multiple time so we need to setup a specific config.

{{Cmd| ln -s /etc/init.d/mqtt-exec /etc/init.d/mqtt-exec.sync-mirror}}

{{Cmd| ln -s /etc/conf.d/mqtt-exec /etc/conf.d/mqtt-exec.sync-mirror}}

edit /etc/conf.d/mqtt-exec.sync-mirror

<pre>
mqtt_topics="rsync/rsync.alpinelinux.org/#"
exec_user="buildozer"
exec_command="/usr/local/bin/sync-mirror"
</pre>

Copy the following file to /usr/local/bin/sync-mirror and make it executable (dont forget to update the variables).

<pre>
#!/bin/sh

src="rsync://rsync.alpinelinux.org/alpine/"
dest="/var/www/localhost/htdocs/alpine/"
lock="/tmp/sync-mirror.lock"
topic="$1"
dir="$2"

[ -z "$flock" ] && exec env flock=1 flock $lock $0 "$@"

if [ -n "$dir" ] && [ -d "$dest/${dir%/*}" ]; then
logger "Syncing directory: $dir"
src="${src}${dir%/}/"
dest="${dest}${dir%/}/"
else
logger "Syncing all directories"
fi

/usr/bin/rsync \
--archive \
--update \
--verbose \
--progress \
--timeout=600 \
--delay-updates \
--delete-after \
"$src" \
"$dest"
</pre>

And finally start mqtt-exec and let it listen on msg.alpinelinux.org

{{Cmd|/etc/init.d/mqtt-exec.sync-mirror start}}

To make sure you are not missing any packages (in case something goes wrong with MQTT subscription) you can periodically sync all directories by adding the script to cron.

{{Cmd|ln -s /usr/local/bin/sync-mirror /etc/periodic/hourly/sync-mirror}}

Now watch your syslog as it should tell you when it will update directories in your local mirror.

[[Category:Server]]
[[Category:Package Manager]]

How to setup a Alpine Linux mirror

2018-08-11T18:53:31Z

Tetsumaki: /* Setting up the cron job */

== Introduction ==
This document describes how to set up an Alpine Linux mirror and make it available via http and rsync.

We will:
* create the dir where we have the mirror
* set up a cron job to sync with master mirror every hour
* set up lighttpd for http access
* set up rsync so other mirrors can rsync from you

Make sure that you have enough disk space; each v3.x branch has around 20 GiB.

Current (2018-01-03) disk usage:

{|class="wikitable"
!edge
!v2.4
!v2.5
!v2.6
!v2.7
!v3.0
!v3.1
!v3.2
!v3.3
!v3.4
!v3.5
!v3.6
!v3.7
!Total
|-
|71.3G
|18.8G
|10.4G
|13.0G
|16.5G
|16.5G
|17.5G
|14.5G
|19.0G
|23.2G
|32.5G
|34.4G
|71.3G
|'''358.9G'''
|}

== Setting up the cron job ==
Install rsync which will be used to sync from the master mirror.
{{Cmd|apk add rsync}}

Save the following file as ''/etc/periodic/hourly/alpine-mirror''
<pre>
#!/usr/bin/env sh

# make sure we never run 2 rsync at the same time
lockfile="/tmp/alpine-mirror.lock"
if [ -z "$flock" ] ; then
exec env flock=1 flock -n $lockfile "$0" "$@"
fi

src=rsync://rsync.alpinelinux.org/alpine/
dest=/var/www/localhost/htdocs/alpine/

# uncomment this to exclude old v2.x branches
#exclude="--exclude v2.*"

mkdir -p "$dest"
/usr/bin/rsync \
--archive \
--update \
--hard-links \
--delete \
--delete-after \
--delay-updates \
--timeout=600 \
$exclude \
"$src" "$dest"

</pre>

(or use [https://gist.github.com/jirutka/288c6fff7c0b8a835d143686207316be this script])

Make it executable:
{{Cmd|<nowiki>chmod +x /etc/periodic/hourly/alpine-mirror</nowiki>}}

Now it will sync every hour. (given cron runs)

== Setting up HTTP access via lighttpd ==

Install the lighttpd server
{{Cmd|apk add lighttpd}}

Enable dir listings by uncommenting the following line in ''/etc/lighttpd/lighttpd.conf'':
dir-listing.activate = "enable"

Also set cache-control to force cache revalidate every 30 mins. Uncomment mod_setenv in ''/etc/lighttpd/lighttpd.conf'':
"mod_setenv",

Add also the following lines to ''/etc/lighttpd/lighttpd.conf'':
setenv.add-response-header += (
"Cache-Control" => "must-revalidate"
)

Start lighttpd and make it start at boot:
{{Cmd|rc-service lighttpd start
rc-update add lighttpd}}

{{Note|You may wish to consider [[Darkhttpd]] as an alternative to [[Lighttpd]]

If so, simply install, start and auto-start the webserver:

{{Cmd|apk add darkhttpd && rc-service darkhttpd start && rc-update add darkhttpd}}

Darkhttpd will, by default, offer directory listings and serve data from /var/www/localhost/htdocs/

See the main article on [[Darkhttpd]] for more configuration options}}

== Setting up rsyncd ==
Add the following lines to ''/etc/rsyncd.conf'':
<pre>
[alpine]
path = /var/www/localhost/htdocs/alpine
comment = My Alpine Linux Mirror
</pre>

Optionally set a bandwidth limit in ''/etc/conf.d/rsyncd''. In this example we limit to 500Kbytes/s (approx 5Mbit/s)
<pre>
RSYNC_OPTS="--bwlimit=500"
</pre>

== Mirror statistics ==

Simple bandwidth statistics can be generated with vnstat.

{{Cmd|apk add vnstat}}

edit /etc/vnstat.conf and replace the interface name with the appropriate one.

Start vnstatd

{{Cmd|/etc/init.d/vnstatd start }}

copy the following script to /etc/periodic/15min/stats and make sure your crond is running.
please not that heredoc should be tab indented or the script will fail. A working copy can be found here: http://tpaste.us/RrMv

<pre>
#!/usr/bin/env sh

output="/var/www/localhost/htdocs/.stats"
nic="eth0"

generate_index() {
cat <<-EOF
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="cache-control" content=no-cache">
<meta http-equiv="refresh" content="3000">
<title>Alpine Linux mirror statistics</title>
</head>
<body>
<table border="0">
<tr><td><img src="summary.png" alt="summary"></td><td><img src="hours.png" alt="hours"></td></tr>
<tr><td rowspan="2"><img src="days.png" alt="days"></td><td><img src="top10.png" alt="top10"></td></tr>
<tr><td><img src="months.png" alt="months"></td></tr>
</table>
</body>
</html>
EOF
}

if [ ! -f "$output"/index.html ]; then
mkdir -p $output
generate_index > "$output"/index.html
fi

for type in hours days months top10 summary hsummary vsummary; do
vnstati --${type} -i $nic -o $output/${type}.png
done
</pre>

== Update mirror from mqtt ==

If you want your mirror to be really uptodate compared to our master mirror you can subscribe to Alpine Linux message server "msg.alpinelinux.org" and check for upload messages.
Add mqtt-exec to be able to execute processes when specific topics are being send.

{{Cmd| apk add mqtt-exec}}

mqtt-exec supports running multiple time so we need to setup a specific config.

{{Cmd| ln -s /etc/init.d/mqtt-exec /etc/init.d/mqtt-exec.sync-mirror}}

{{Cmd| ln -s /etc/conf.d/mqtt-exec /etc/conf.d/mqtt-exec.sync-mirror}}

edit /etc/conf.d/mqtt-exec.sync-mirror

<pre>
mqtt_topics="rsync/rsync.alpinelinux.org/#"
exec_user="buildozer"
exec_command="/usr/local/bin/sync-mirror"
</pre>

Copy the following file to /usr/local/bin/sync-mirror and make it executable (dont forget to update the variables).

<pre>
#!/bin/sh

src="rsync://rsync.alpinelinux.org/alpine/"
dest="/var/www/localhost/htdocs/alpine/"
lock="/tmp/sync-mirror.lock"
topic="$1"
dir="$2"

[ -z "$flock" ] && exec env flock=1 flock $lock $0 "$@"

if [ -n "$dir" ] && [ -d "$dest/${dir%/*}" ]; then
logger "Syncing directory: $dir"
src="${src}${dir%/}/"
dest="${dest}${dir%/}/"
else
logger "Syncing all directories"
fi

/usr/bin/rsync \
--archive \
--update \
--verbose \
--progress \
--timeout=600 \
--delay-updates \
--delete-after \
"$src" \
"$dest"
</pre>

And finally start mqtt-exec and let it listen on msg.alpinelinux.org

{{Cmd|/etc/init.d/mqtt-exec.sync-mirror start}}

To make sure you are not missing any packages (in case something goes wrong with MQTT subscription) you can periodically sync all directories by adding the script to cron.

{{Cmd|ln -s /usr/local/bin/sync-mirror /etc/periodic/hourly/sync-mirror}}

Now watch your syslog as it should tell you when it will update directories in your local mirror.

[[Category:Server]]
[[Category:Package Manager]]

How to setup a Alpine Linux mirror

2018-08-11T18:44:21Z

Tetsumaki: /* Mirror statistics */

== Introduction ==
This document describes how to set up an Alpine Linux mirror and make it available via http and rsync.

We will:
* create the dir where we have the mirror
* set up a cron job to sync with master mirror every hour
* set up lighttpd for http access
* set up rsync so other mirrors can rsync from you

Make sure that you have enough disk space; each v3.x branch has around 20 GiB.

Current (2018-01-03) disk usage:

{|class="wikitable"
!edge
!v2.4
!v2.5
!v2.6
!v2.7
!v3.0
!v3.1
!v3.2
!v3.3
!v3.4
!v3.5
!v3.6
!v3.7
!Total
|-
|71.3G
|18.8G
|10.4G
|13.0G
|16.5G
|16.5G
|17.5G
|14.5G
|19.0G
|23.2G
|32.5G
|34.4G
|71.3G
|'''358.9G'''
|}

== Setting up the cron job ==
Install rsync which will be used to sync from the master mirror.
{{Cmd|apk add rsync}}

Save the following file as ''/etc/periodic/hourly/alpine-mirror''
<pre>
#!/bin/sh

# make sure we never run 2 rsync at the same time
lockfile="/tmp/alpine-mirror.lock"
if [ -z "$flock" ] ; then
exec env flock=1 flock -n $lockfile "$0" "$@"
fi

src=rsync://rsync.alpinelinux.org/alpine/
dest=/var/www/localhost/htdocs/alpine/

# uncomment this to exclude old v2.x branches
#exclude="--exclude v2.*"

mkdir -p "$dest"
/usr/bin/rsync \
--archive \
--update \
--hard-links \
--delete \
--delete-after \
--delay-updates \
--timeout=600 \
$exclude \
"$src" "$dest"

</pre>

(or use [https://gist.github.com/jirutka/288c6fff7c0b8a835d143686207316be this script])

Make it executable:
{{Cmd|<nowiki>chmod +x /etc/periodic/hourly/alpine-mirror</nowiki>}}

Now it will sync every hour. (given cron runs)

== Setting up HTTP access via lighttpd ==

Install the lighttpd server
{{Cmd|apk add lighttpd}}

Enable dir listings by uncommenting the following line in ''/etc/lighttpd/lighttpd.conf'':
dir-listing.activate = "enable"

Also set cache-control to force cache revalidate every 30 mins. Uncomment mod_setenv in ''/etc/lighttpd/lighttpd.conf'':
"mod_setenv",

Add also the following lines to ''/etc/lighttpd/lighttpd.conf'':
setenv.add-response-header += (
"Cache-Control" => "must-revalidate"
)

Start lighttpd and make it start at boot:
{{Cmd|rc-service lighttpd start
rc-update add lighttpd}}

{{Note|You may wish to consider [[Darkhttpd]] as an alternative to [[Lighttpd]]

If so, simply install, start and auto-start the webserver:

{{Cmd|apk add darkhttpd && rc-service darkhttpd start && rc-update add darkhttpd}}

Darkhttpd will, by default, offer directory listings and serve data from /var/www/localhost/htdocs/

See the main article on [[Darkhttpd]] for more configuration options}}

== Setting up rsyncd ==
Add the following lines to ''/etc/rsyncd.conf'':
<pre>
[alpine]
path = /var/www/localhost/htdocs/alpine
comment = My Alpine Linux Mirror
</pre>

Optionally set a bandwidth limit in ''/etc/conf.d/rsyncd''. In this example we limit to 500Kbytes/s (approx 5Mbit/s)
<pre>
RSYNC_OPTS="--bwlimit=500"
</pre>

== Mirror statistics ==

Simple bandwidth statistics can be generated with vnstat.

{{Cmd|apk add vnstat}}

edit /etc/vnstat.conf and replace the interface name with the appropriate one.

Start vnstatd

{{Cmd|/etc/init.d/vnstatd start }}

copy the following script to /etc/periodic/15min/stats and make sure your crond is running.
please not that heredoc should be tab indented or the script will fail. A working copy can be found here: http://tpaste.us/RrMv

<pre>
#!/usr/bin/env sh

output="/var/www/localhost/htdocs/.stats"
nic="eth0"

generate_index() {
cat <<-EOF
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="cache-control" content=no-cache">
<meta http-equiv="refresh" content="3000">
<title>Alpine Linux mirror statistics</title>
</head>
<body>
<table border="0">
<tr><td><img src="summary.png" alt="summary"></td><td><img src="hours.png" alt="hours"></td></tr>
<tr><td rowspan="2"><img src="days.png" alt="days"></td><td><img src="top10.png" alt="top10"></td></tr>
<tr><td><img src="months.png" alt="months"></td></tr>
</table>
</body>
</html>
EOF
}

if [ ! -f "$output"/index.html ]; then
mkdir -p $output
generate_index > "$output"/index.html
fi

for type in hours days months top10 summary hsummary vsummary; do
vnstati --${type} -i $nic -o $output/${type}.png
done
</pre>

== Update mirror from mqtt ==

If you want your mirror to be really uptodate compared to our master mirror you can subscribe to Alpine Linux message server "msg.alpinelinux.org" and check for upload messages.
Add mqtt-exec to be able to execute processes when specific topics are being send.

{{Cmd| apk add mqtt-exec}}

mqtt-exec supports running multiple time so we need to setup a specific config.

{{Cmd| ln -s /etc/init.d/mqtt-exec /etc/init.d/mqtt-exec.sync-mirror}}

{{Cmd| ln -s /etc/conf.d/mqtt-exec /etc/conf.d/mqtt-exec.sync-mirror}}

edit /etc/conf.d/mqtt-exec.sync-mirror

<pre>
mqtt_topics="rsync/rsync.alpinelinux.org/#"
exec_user="buildozer"
exec_command="/usr/local/bin/sync-mirror"
</pre>

Copy the following file to /usr/local/bin/sync-mirror and make it executable (dont forget to update the variables).

<pre>
#!/bin/sh

src="rsync://rsync.alpinelinux.org/alpine/"
dest="/var/www/localhost/htdocs/alpine/"
lock="/tmp/sync-mirror.lock"
topic="$1"
dir="$2"

[ -z "$flock" ] && exec env flock=1 flock $lock $0 "$@"

if [ -n "$dir" ] && [ -d "$dest/${dir%/*}" ]; then
logger "Syncing directory: $dir"
src="${src}${dir%/}/"
dest="${dest}${dir%/}/"
else
logger "Syncing all directories"
fi

/usr/bin/rsync \
--archive \
--update \
--verbose \
--progress \
--timeout=600 \
--delay-updates \
--delete-after \
"$src" \
"$dest"
</pre>

And finally start mqtt-exec and let it listen on msg.alpinelinux.org

{{Cmd|/etc/init.d/mqtt-exec.sync-mirror start}}

To make sure you are not missing any packages (in case something goes wrong with MQTT subscription) you can periodically sync all directories by adding the script to cron.

{{Cmd|ln -s /usr/local/bin/sync-mirror /etc/periodic/hourly/sync-mirror}}

Now watch your syslog as it should tell you when it will update directories in your local mirror.

[[Category:Server]]
[[Category:Package Manager]]