User talk:Prabuanand: Difference between revisions

From Alpine Linux
(added other references man page whynothugo blogpost. need to refactor page to make it concise and useful. added draft note)
(updated apk page)
Tag: Replaced
 
Line 1: Line 1:
{{Draft|1. Need to Refactor the page to organize the information. 2. Need to identify the page to host the information 3. section heading world or /etc/apk/world(?) }}
Why apk-tools is different than other package managers
apk add and apk del manipulate the desired state
In traditional package managers like dnf and apt, requesting the installation or removal of packages causes those packages to be directly installed or removed, after a consistency check.
In apk, when you do apk add foo or apk del bar, it adds foo or bar as a dependency constraint in /etc/apk/world which describes the desired system state.  Package installation or removal is done as a side effect of modifying this system state.  It is also possible to edit /etc/apk/world with the text editor of your choice and then use apk fix to synchronize the installed packages with the desired system state.
Because of this design, you can also add conflicts to the desired system state.  For example, we recently had a bug in Alpine where pipewire-pulse was preferred over pulseaudio due to having a simpler dependency graph.  This was not a problem though, because users could simply add a conflict against pipewire-pulse by doing apk add !pipewire-pulse.
Another result of this design is that apk will never commit a change to the system that leaves it unbootable.  If it cannot verify the correctness of the requested change, it will back out adding the constraint before attempting to change what packages are actually installed on the system.  This allows our dependency solver to be rigid: there is no way to override or defeat the solver other than providing a scenario that results in a valid solution.
Verification and unpacking is done in parallel to package fetching
Unlike other package managers, when installing or upgrading packages, apk is completely driven by the package fetching I/O.  When the package data is fetched, it is verified and unpacked on the fly.  This allows package installations and upgrades to be extremely fast.
To make this safe, package contents are initially unpacked to temporary files and then atomically renamed once the verification steps are complete and the package is ready to be committed to disk.
apk does not use a particularly advanced solver
Lately, traditional package managers have bragged about having advanced SAT solvers for resolving complicated constraint issues automatically.  For example, aptitude is capable of solving sudoku puzzles.  apk is definitely not capable of that, and I consider that a feature.
While it is true that apk does have a deductive dependency solver, it does not perform backtracking.  The solver is also constrained: it is not allowed to make changes to the /etc/apk/world file.  This ensures that the solver cannot propose a solution that will leave your system in an inconsistent state.
Personally, I think that trying to make a smart solver instead of appropriately constraining the problem is a poor design choice.  I believe the fact that apt, aptitude and dnf have all written code to constrain their SAT solvers in various ways proves this point.
To conclude, package managers can be made to go fast, and be safe while doing it, but require a careful design that is well-constrained.  apk makes its own tradeoffs: a less powerful but easy to audit solver, trickier parallel execution instead of phase-based execution.  These were the right decisions for us, but may not be the right decisions for other distributions.
Source: https://ariadne.space/2021/04/25/why-apk-tools-is-different-than-other-package-managers/
A high level view of the moving parts
Our adventure begins at the /etc/apk/world file. This file contains the basic set of constraints imposed on the system: every constraint listed here must be solvable in order for the system to be considered correct, and no transaction may be committed that is incorrect. In other words, the package management system can be proven to be in a correct state every time a constraint is added or removed with the apk add/del commands.
Note I used the word transaction there: at its core, apk is a transactional package manager, though we have not fully exploited the transactional capabilities yet. A transaction is created by copying the current constraint list (db->world), manipulating it with apk_deps_add and then committing it with apk_solver_commit. The commitment phase does pre-flight checks directly and returns an error if the transaction fails to pass.
This means that removing packages works the same way: you copy the current constraint set, remove the desired constraint, and then commit the result, which either errors out or updates the installed constraint set after the transaction is committed.
A deeper look into the solver itself
As noted above, the primary entry point into the solver is to call the apk_solver_commit function, which at the time that I am writing this, is located in the apk-tools source code at src/commit.c:679. This function does a few pre-flight checks and then calls into the solver itself, using apk_solver_solve, which generates the actual transaction to be committed. If there are errors, the generated transaction is discarded and a report is printed instead, otherwise the generated transaction is committed using apk_solver_commit_changeset.
In essence, the code in src/commit.c can be thought of as the middle layer between the applets and the core solver. The core solver itself lives in src/solver.c and as previously noted, the main entry point is apk_solver_solve, which generates a proposed transaction to satisfy the requested constraints. This function lives at src/solver.c:1021, and is the only entry point into the solver itself.
The first thing the solver does is alphabetically sort the constraint set. If you’ve noticed that /etc/apk/world is always in alphabetical order, this is a side effect of that sorting.
Once the world constraints (the ones in /etc/apk/world) are alphabetically ordered, the next step is to figure out what package, if any, presently satisfies the constraint. This is handled by the discover_name function, which is called recursively on every constraint applicable to the system, starting with the world constraint.
The next step is to generate a fuzzy solution. This is done by walking the dependency graph again, calling the apply_constraint function. This step does basic dependency resolution, removing possible solutions which explicitly conflict. Reverse dependencies (install_if) are partially evaluated in this phase, but complex constraints (such as those involving a version constraint or multiple solutions) are not evaluated yet.
Once basic constraints are applied to the proposed updated world, the next step is to walk the dependency graph again, reconsidering the fuzzy solution generated in the step above. This step is done by the reconsider_name function, which walks over parts of the dependency graph that are still ambiguous. Finally, packages are selected to resolve these ambiguities using the select_package function. Afterwards, the final changeset is emitted by the generate_changeset function.
A deep dive into reconsider_name and select_package
As should hopefully be obvious by now, the really complicated cases are handled by the reconsider_name function. These cases include scenarios such as virtual providers, situations where more than one package satisfies the constraint set, and so on. For these scenarios, it is the responsibility of the reconsider_name function to select the most optimal package. Similarly, it is the responsibility of the select_package function to check the work done by reconsider_name and finalize the package selection if appropriate by removing the constraint from the ambiguous list.
The primary purpose of the reconsider_name function is to use discover_name and apply_constraint to move more specific constraints upwards and downwards through the dependency graph, narrowing the possible set of packages which can satisfy a given restraint, ideally to one package or less. These simplified dependency nodes are then fed into select_package to deduce the best package selection to make.
The select_package function checks each constraint, and the list of remaining candidate packages, and then picks the best package for each constraint. This is done by calling compare_providers for each possible package and until the best one is found. The heuristics checked by compare_providers are, in order:
    The packages are checked to see if they are NULL or not. The one that isn’t NULL wins. This is mostly as a safety check.
    We check to see if the user is using --latest or not. If they are, then the behavior changes a little bit. The details aren’t so important, you can read the source if you really want to know. Basically, in this step, we determine how fresh a package is, in alignment with what the user’s likely opinion on freshness would be.
    The provider versions are compared, if applicable. Highest version wins.
    The package versions themselves are compared. Highest version wins.
    The already installed package is preferred if the version is the same (this is helpful in upgrade transactions to make them less noisy).
    The provider_priority field is compared. Highest priority wins. This means that provider_priority is only checked for unversioned providers.
    Finally, the earliest repository in /etc/apk/repositories is preferred if all else is the same.
Hopefully, this demystifies some of the common misconceptions around how the solver works, especially how provider_priority works. Personally, I think in retrospect, despite working on the spec and implementing it in apk-tools, that provider_priority was a mistake, and the preferred solution should be to always use versioned providers (e.g. provides="foo=100") instead. The fact that we have moved to versioning cmd: providers in this way demonstrates that provider_priority isn’t really a good design.
Source: https://ariadne.space/2021/10/31/spelunking-through-the-apk-tools-dependency-solver/
“keeping a list of the list of packages installed”. The main goal is
to be able to quickly recreate my installation, and also understand
how it changed over time (e.g.: being able to roll back to a previous
state with just the list of packages), but I also very much prefer a
declarative approach to installed packages (rather than an imperative
approach).
The idea behind a declarative approach is to basically have a list of
packages I want installed, and let the package manager install/remove
anything so that installed package match exactly that list (plus any
required dependencies, of course).
apk and /etc/apk/world
apk (literally “alpine package keeper) is Alpine’s package manager. I
was very pleased to learn about apk’s /etc/apk/world.
In essence, this file is a list of desired packages, and apk will
install and uninstall packages so that the system state converges to
the list. Exactly what I’d been looking for, but built right into the
package manager. Actually it’s not just built into the package
manager: it’s the very foundation of how it works. apk add actually
adds a package to that list, and then installs any that are not
present. apk del removes a package from that list, and only uninstalls
it if it’s not a dependency for anything else.
Understanding this also explains why apk uses add and del rather than
the more traditional install and uninstall. I extremely pleased by
this design and how well it works. Additionally, one can edit the
world file manually and run apk fix, which will install/uninstall any
packages required to converge to the declared list.
I also admit it’s a breath of fresh air to have “normal” verbs as
subcommands rather than pacman -Syu, even if I already know of most
pacman’s flags.
Things like the /etc/apk/world file makes management machines
easier. It's basically the requirements.txt file for your Linux
installation and you don't even need to use any extra configuration
management tools to get that functionality.
There's also some downsides to apk though. Things I'm missing is
optional packages and when things go wrong it has some of the most
useless error messages I've encountered in software: temporary error
(try again later) . Throwing away the original error and showing "user
friendly" messages usually does not improve the situation.
apk manages packages installed on the system. The set of top level
constraints on system packages is called the world (see apk-world(5)).
apk supports various sub-commands to query and manipulate world and
package repositories.
By default apk is non-interactive. See FILES or --interactive on
changing this default to be interactive.
NAME
/etc/apk/world - list of constraints for package selection.
DESCRIPTION
At /etc/apk/world, apk maintains the world, that is, a list of
constraints the package selection needs to fulfill.
If you edit this file manually, you should run apk-fix(8) to apply the
changes.
PACKAGE SPECIFICATION
This is a plaintext file with one constraint using dependency notation
per line. Each line has the format: name{@tag}{[<>~=]version}.
When modifying existing installation, the installed version is
preferred unless an upgrade is requested or a world constraint or
package dependency requires an alternate version.
To enable selection from a tagged repository, use the format name@tag,
e.g. busybox@edge. See apk-repositories(5) for details on tagged
package repositories. Untagged repositories are also considered for
constraints with a tag. The tag is inherited to all dependencies as an
allowed repository. That is, the dependencies are selected from the
tagged repository if the name@tag has a dependency with version
constraint requiring a version available only from the tagged
repository. If the dependency can be satisfied from non-tagged
repository it will be preferred.
To constrain the acceptable versions of the package, use the =, <, >,
>=, ~, >~ or <~ operators. Respectively, these require the package is
equal to, less than, greater than, greater than or equal, prefix
match, greater than or prefix match, or less than or prefix match to
the specified version. The ~ operator constrains the package to the
prefix match of the version number.
  busybox
          Installs busybox from the untagged repository from which it is
          available.
  busybox@edge
          Allows installation of busybox and it's dependencies from a
          repository tagged with "edge". Tagged repositories will not be
          prioritized. If a version from an untagged repository is a better
          fit it will be used.
      busybox=1.6.1
          Install busybox version 1.6.1.
      busybox>1.6.1
          Install a busybox version greater than 1.6.1.
      busybox>=1.6.1
          Install a busybox version greater than or equal to 1.6.1.
      busybox<1.6.1
          Install a busybox version less than 1.6.1.
      busybox~1.6
          Install any busybox version starting with 1.6. Examples of match:
          1.6, 1.6.0_pre1, 1.6.0, 1.6.5, 1.6.9_p1.
      busybox>~1.6
          Install a busybox version greater than or prefix match of 1.6.
      busybox<~1.6
          Install a busybox version less than or prefix match of 1.6.
World - from docs.alpinelinux.org
The packages you want to have explicitly installed are listed in the
"world file", available in /etc/apk/world. It is safe to edit it by
hand. If you’ve edited it by hand, you may run apk add with no
arguments to bring the package selection to a consistent state.
Virtuals like cmd:, so: and pc: will appear as such in your world file
- this is why using so: is discouraged - the soname might get bumped!
Virtuals
While cmd:, so: and pc: packages are automatically created virtuals,
you can create your own as well. These allow for quick removal of
purpose-specific packages. See the following examples for details:
apk add a b c -t abc
apk del abc
apk add a b c --virtual abc
This will add the packages "a", "b" and "c" as the dependencies of a
virtual package "abc".  This will remove "abc" and all of its
components ("a", "b" and "c"), unless they are required elsewhere.
This is equivalent to the first example.  Swapping Repositories
When alpine has a new release, the repository path will change. Assuming you are going forward in time (e.g from 3.12 to 3.13), you can simply edit /etc/apk/repositories and run apk upgrade --available.
NAME
/etc/apk/repositories, /etc/apk/repositories.d/*.list - list of
      package repositories
DESCRIPTION
/etc/apk/repositories is the list of package repositories
apk(8) uses to retrieve package files for installation. Each line of
this file specifies the location of a package repository, and
optionally a tag.
The location may be an http://, https://, or ftp:// URL, or the path
to a directory on the local filesystem. A tagged repository is
prefixed with the @tag specifier, followed by a space and the
repository location. For more information about repository tags, see
apk-world(5).
REPOSITORY LAYOUT
      Each repository must store an index at
      $repository/$arch/APKINDEX.tar.gz. See apk-index(8) for information
      about generating this file. The packages themselves are stored at
      $repository/$arch/$pkgname-$pkgver-r$pkgrel.apk.
      apk(8) verifies that each of these files has a valid cryptographic
      signature unless explicitly told not to via the --allow-untrusted flag.
      See abuild-keygen(1) for information about generating keys, apk-keys(5)
      to add keys to the list of trusted keys, abuild-sign(1) for information
      about using these keys to sign files, and apk-verify(8) for information
      about verifying those signatures.
UPDATING INDICIES apk(8) fetches and stores the index for each package
      repository at /var/cache/apk. To fetch fresh indicies for all
      configured repositories, use apk-update(8).
References:
* [https://ariadne.space/2021/04/25/why-apk-tools-is-different-than-other-package-managers/  Why apk-tools is different than other package managers]
* [https://ariadne.space/2021/10/31/spelunking-through-the-apk-tools-dependency-solver/ apk-tools high level view of the moving parts]
* [https://blog.brixit.nl/alpine-linux-is-pretty-neat/ alpine linux is pretty neat]
* [https://man.archlinux.org/man/apk-world.5.en man 5 apk-world]
* [https://superuser.com/questions/1174021/why-does-alpine-apk-report-unsatisfiable-constraints-when-installing-an-older superuser query about apk constraints]
* [https://whynothugo.nl/journal/2023/11/19/setting-up-an-alpine-linux-workstation/ setting up an alpine linux workstation]
* [https://whynothugo.nl/journal/2023/02/18/in-praise-of-alpine-and-apk/ in praise of alpine and apk]
* [https://docs.alpinelinux.org/user-handbook/0.1a/Working/apk.html docs handbook]
<!--
* [https://drewdevault.com/2021/05/06/Praise-for-Alpine-Linux.html Praise for Alpine Linux]
* [https://drewdevault.com/2023/07/25/Alpine-does-not-make-news.html Alpine does not make news]
-->
URL for scripts to build alpine iso's.
URL for scripts to build alpine iso's.
* [https://git.alpinelinux.org/aports/tree/scripts iso building scripts]
* [https://git.alpinelinux.org/aports/tree/scripts iso building scripts]

Latest revision as of 09:54, 12 October 2024

URL for scripts to build alpine iso's.

Retrieved from "https://wiki.alpinelinux.org/w/index.php?title=User_talk:Prabuanand&oldid=27520"