annotate posts/Managing_dotfiles_with_vcsh_and_mr.md @ 19:db3648f34015

creating tag page tags/bedrock
author Samuel Hodgkins <samuel.hodgkins@sky.com>
date Fri, 01 Sep 2017 02:45:19 +0000
parents 2307e281b4b7
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
13
2307e281b4b7 tagify all the things
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 3
diff changeset
1 [[!tag tech linux dotfiles myrepos vcsh old]]
3
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
2 Over time, a Linux user may customize and configure his environment rather substantially.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
3 These modifications are stored in a collection of configuration files / data known as 'dotfiles' (because the first letter of many of them is '.').
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
4 For multiple reasons, it is very beneificial if you track, control and synchronise all of your personal dotfiles, a few example reasons include:
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
5 - Having an additional backup
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
6 - Being able to see their history, how they changed over time
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
7 - Being able to rollback changes if needed (I haven't needed this yet)
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
8 - Being able to use the same set of files accross multiple physical/virtual machines
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
9 - Being able to share you configuration with the world so people can learn from it just like you learn from other people's.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
10
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
11 However, there is not one single universal method for managing them, instead there are many tools and approaches that one can take.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
12 GitHub provides a decent list of programs [here](https://dotfiles.github.io/) but I intend to summarize the main approaches below.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
13 (It may be worth noting that while the methods may not be mutually exclusive, there is one 'main' approach/method per tool and that is what counts.)
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
14
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
15 1. Symlink-driven management involves moving the dotfiles away from their original location, instead creating symbolic links to one or more destinations.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
16 There are many ways/approaches of doing this, but the simplest is to just have a single directory be the destination for all the links.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
17 2. VC(Version Control)-driven management involves less management of actual dotfiles compared to the other two. Instead of copying or using symbolic links,
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
18 instead a version-control system is primarily used to track/manage dotfiles in groups. The original dotfiles are left in place, instead they can be treated
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
19 just like every other repository. There are multiple methods of implementing this approach with their own unique advantages and drawbacks.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
20 3. Configuration-driven management involves using explicit configuration file(s) to determine exactly what dotfiles are to be managed/tracked as well as how they are to be tracked among other things.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
21 The key difference between this method and the others is that rather than using interactive commands to manage and modify dotfiles, one or more configuration files are used.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
22 Typical formats for this information include YAML/JSON or a purpose-built configuration format. Typically but not exclusively uses symbolic links for dotfiles.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
23
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
24 I have been tracking my dotfiles for short-to-moderate period of time. I originally started when I read an article about using GNU Stow as the management tool.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
25 Stow has some features that make it just as useful for this as a dedicated too: It supports 'packages' so you can choose to install part of the dotfiles.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
26 It also doesn't make you specify specifically which files to symlink, it just symlinks the entire package.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
27 However, it's definitely not perfect: Symlinks can be overwritten, Moving dotfiles and replicating directory structures sucked, and you could only manage operations from the right directory.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
28 (I could also only easily have 1 VCS repo, which effectively meant private dotfiles couldn't be tracked)
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
29
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
30 One day, while inspecting my ~/dotfiles I noticed that the .git directory was missing. I could've seen this as a disaster, but I didn't.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
31 I had been thinking about migrating away from Stow for a while, but I never actually did anything - so I took this opportunity.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
32 After some reading/googling, I had made the decision to use `mr` and `vcsh`.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
33 `vcsh` would provide each individual repository, public and private while `mr` would be used for higher-level tasks.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
34 There are multiple guides to this pair of tools, such as:
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
35
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
36 * [This very short post on vcsh & mr](https://sumancluster.wordpress.com/2015/05/29/managing-dotfiles-using-vcsh-and-mr/)
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
37 * [This one which links to a more in-depth tutorial but also takes a look at the internals](https://www.kunxi.org/blog/2014/02/manage-dotfiles-using-vcsh-and-mr/)
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
38 * [This very useful and detailed one](http://srijanshetty.in/technical/vcsh-mr-dotfiles-nirvana/)
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
39
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
40 When I was migrating, I particularly found the latter link to be rather useful due to the detailed explanations of multiple common tasks.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
41 However, should you not want to read any of the above links I will attempt to give an overview of how it works in practice.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
42
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
43 # Creating a new repository
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
44
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
45 1. Clone/Initialize the local vcsh repository
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
46 2. Update the myrepos(mr) configuration to include that repository
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
47 3. Add the wanted stuff to the vcsh repository
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
48 4. Write/generate a .gitignore and modify as needed
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
49 5. Commit to the vcsh repository and push both sets of changes as needed.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
50
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
51 # Updating an existing repository
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
52
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
53 1. You can prefix git operations with vcsh and then the repo name to perform them on the repository.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
54 2. Alternatively, use 'vcsh enter' to go into an environment where git can be used normally.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
55
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
56 # Updating *all* the repositories
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
57
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
58 1. Use `mr up` and let myrepos do the job it was designed to do.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
59
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
60 # Bootstrapping the dotfiles
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
61 (presuming git is installed. If not, install it.)
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
62
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
63 1. Install myrepos and vcsh. If there's no distribution package, a manual install is easy (they're just standalone scripts)
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
64 2. Obtain your myrepos configuration.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
65 3. Use `mr up` and let myrepos obtain all your repositories as needed.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
66
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
67 If you think the above workflow looks interesting, I recommend you have a nice read of the above links - especially the last one
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
68 as I found it very useful. However, I have not moved my entire collection of dotfiles over and yet I still have some interesting problems/caveats to tackle.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
69
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
70 Firstly, while using a (private) Git repository to track my SSH/GPG data is useful, certain files have special filesystem permissions which Git does not preserve. While this can be solved with a chmod or two, it may grow
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
71 to be more difficult if I need more of these files in the future - plus I might be able to automate it using mr's 'fixups' functionality.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
72
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
73 Secondly, this is more of an observation than a problem: I'm currently using an Apache-style configuration involving both *'available.d'* and *'config.d'*. This works and is flexible, but it'd be simpler to only have a single directory and have equivalent information be part of the configuration itself.
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
74
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
75 Thirdly, bootstrapping from a completely clean slate is rather complicated. Certain repositories may depend on others to work / be in the correct location. Then there's the problem of access to private repositories, perhaps HTTP(s) could be used to download SSH keys using pre-entered cached credentials? A similar but lesser problem exists with GPG. Public repositories have no issues with this - if need be, they can have the master remote be changed afterwards.s
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
76
45362e07941c Add more old posts.
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
77 Anyway, that's all for now. If and when I solve the above issues I'll make sure to explain and blog about each my solutions. Until then, I don't expect this to come up again.