Mercurial > ~samis > hgweb.cgi > blag

annotate posts/Gentoo_Repositories.md @ 37:915bddd6e7ed

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Ignore draft ikiwiki posts in the posts and index pages
author Samuel Hodgkins <samuel.hodgkins@sky.com>
date Tue, 20 Mar 2018 02:36:39 +0000
parents 84e17c0f39c2
children f49c9ba2f8e3
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
36
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
1 [[!meta title="Using eselect-repository with Gentoo" author="samis"]]
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
2 [[!tag tech linux gentoo portage]]
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
3 # Background
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
4 Some time after installing Gentoo, I moved away from syncing the portage tree via Git, because having history of the repository is nice and as a bonus, the repository can be synchronized from GitHub over HTTPS while rsync uses plain text.
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
5 Commits in the repository are also GPG-signed, but the current version of portage does not use this to validate the repository [(yet)](https://github.com/gentoo/portage/commit/7a507942b669b6a157612e8f7ce3fe362c39b38f)
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
6
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
7 However, the official Gentoo package repository is not enough, and even though I use [Bedrock Linux](https://bedrocklinux.org/), it's nice to have packages designed for Gentoo so I use a number of unofficial repositories or overlays to add more packages and the conventional tool to use is [Layman](https://wiki.gentoo.org/wiki/Layman) which has a decent-enough interface and support for a wide variety of methods to sync an overlay - all of the commonly-known OSS version control systems and some others.
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
8 These overlays are updated at the same time as the official repository via portage (which calls down to layman to do the actual sync).
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
9
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
10 When I switched to syncing the tree via Git, I used a repository that had pre-generated all the additional information not directly contained in the actual 'gentoo' repository like metadata and other components.
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
11 This has the advantage of being simpler to use, but as a consequence the repository's history and disk-usage is bloated because it has to track every version of the metadata and other things, when doing this is entirely unneeded because it can be generated or taken from elsewhere - and recently I found a package that resolves the problem with ease-of-use so I decided to try it out (I had other reasons, but that's for later)
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
12
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
13 At the same time, I thought that since most of my used overlays were using git, wouldn't it make sense to sync them the same way, rather than going through layman to it?
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
14 It'd certainly be simpler, a quick look showed that there was a package that provided a good interface for this, [eselect-repository](https://github.com/mgorny/eselect-repository) that also was much simpler than layman, being under 500 total lines in a file compared to layman's roughly 10kloc (estimated by cloc after removing autoconf and doc).
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
15 Any non-git overlays would stay managed by layman, though.
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
16
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
17 # Doing the move
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
18 (Note: Some parts left out or tweaked for the purposes of this post.)
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
19 1. I added the mv overlay to add the package with the scripts, and then installed both it and `eselect-repository` with emerge.
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
20 2. Ensure /etc/portage/repos.conf/gentoo.conf looks like this (change the clone-depth if you don't want a full clone):
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
21
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
22 [DEFAULT]
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
23 main-repo = gentoo
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
24
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
25 [gentoo]
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
26 location = /usr/portage
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
27 sync-type = git
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
28 sync-uri = https://github.com/gentoo/gentoo.git
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
29 auto-sync = yes
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
30 clone-depth = 0
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
31
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
32 3. Tweakk /etc/eselect/repository.conf so it uses the original sources, rather than the ones with metadata in the history.
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
33 4. Make a note of your layman-managed git overlays somewhere to re-add them shortly afterwards.
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
34 5. Remove (or disable) all the git overlays using layman (-d or -D)
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
35 6. Re-add the overlays using eselect repository (either add or enable, depending on exactly what)
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
36 7. Run your first sync command. Ignore the messages about non-existent directories, they'll go away as each overlay is created when synced.
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
37
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
38 # # Unrelated Benefits
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
39 While this section isn't related to either eselect-repository or layman, it is related to the changes and why I did them. Essentially, after finding out about a certain overlay which had a large number of packages, I noticed that eix-sync was taking a good amount of time to update the database for this large overlay - looking at another, I noticed that I never generated the metadata cache for any of my overlays. While googling about this, I found the complete set of shell scripts to update this for all overlays and repositories, along with other things (and also found a post pointing out the wastefulness of storing all the metadata in git for the Gentoo repository). The end-result was this post, and the generated metadata cache, while taking up-front CPU time, makes the eix updates faster overall and hopefully will also provide a nice speed boost for portage in general.
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
40
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
41 # Next?
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
42 While there are not many next steps, the most obvious one is to add support in portage for the sync methods that Layman provides but Portage itself does not. Even though I'm not an expert in Python, this shouldn't be difficult to implement.