feat: module system in nixos / home-manager / nix-darwin

ryan4yin · Dec 18, 2023 · 883393e · 883393e
1 parent e486604
commit 883393e
Show file tree

Hide file tree

Showing 12 changed files with 547 additions and 31 deletions.
diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts
@@ -252,8 +252,8 @@ function themeConfigEnglish() {
               link: "/other-usage-of-flakes/the-new-cli.md",
             },
             {
-              text: "[WIP]Nix Options",
-              link: "/other-usage-of-flakes/options.md",
+              text: "Module System & Custom Options",
+              link: "/other-usage-of-flakes/module-system.md",
             },
             {
               text: "[WIP]Testing",
@@ -442,8 +442,8 @@ function themeConfigChinese() {
               link: "/zh/other-usage-of-flakes/the-new-cli.md",
             },
             {
-              text: "[WIP]Nix Options",
-              link: "/zh/other-usage-of-flakes/options.md",
+              text: "模块系统与自定义 options",
+              link: "/zh/other-usage-of-flakes/module-system.md",
             },
             {
               text: "[WIP]Testing",

diff --git a/docs/introduction/advantages-and-disadvantages.md b/docs/introduction/advantages-and-disadvantages.md
@@ -33,7 +33,7 @@
 - **High Learning Curve**:
   - Achieving complete reproducibility and avoiding pitfalls associated with improper usage requires learning about Nix's entire design and managing the system declaratively, rather than blindly using commands like `nix-env -i` (similar to `apt-get install`).
 - **Disorganized Documentation**:
-  - Currently, Nix Flakes remains an experimental feature, and there is limited documentation specifically focused on it. Most Nix community documentation primarily covers the older `nix-env`/`nix-channel` approach. If you want to start learning directly from Nix Flakes, you need to refer to a significant amount of outdated documentation and extract the relevant information. Additionally, some core features of Nix, such as `imports` and the Nix Module System, lack detailed official documentation, requiring resorting to source code analysis.
+  - Currently, Nix Flakes remains an experimental feature, and there is limited documentation specifically focused on it. Most Nix community documentation primarily covers the older `nix-env`/`nix-channel` approach. If you want to start learning directly from Nix Flakes, you need to refer to a significant amount of outdated documentation and extract the relevant information. Additionally, some core features of Nix, such as `imports` and the Nixpkgs Module System, lack detailed official documentation, requiring resorting to source code analysis.
 - **Increased Disk Space Usage**:
   - To ensure the ability to roll back the system at any time, Nix retains all historical environments by default, resulting in increased disk space usage.
   - While this additional space usage may not be a concern on desktop computers, it can become problematic on resource-constrained cloud servers.

diff --git a/docs/nixos-with-flakes/modularize-the-configuration.md b/docs/nixos-with-flakes/modularize-the-configuration.md
@@ -274,6 +274,8 @@ As you can see, the order of `systemPackages` is `git -> curl -> default package
 
 > Although adjusting the order of `systemPackages` may not be useful in practice, it can be helpful in other scenarios.
 
+> For a deeper introduction to the module system, see [Module System & Custom Options](../other-usage-of-flakes/module-system.md).
+
 ## References
 
 - [Nix modules: Improving Nix's discoverability and usability](https://cfp.nixcon.org/nixcon2020/talk/K89WJY/)

diff --git a/docs/nixos-with-flakes/nixos-with-flakes-enabled.md b/docs/nixos-with-flakes/nixos-with-flakes-enabled.md
@@ -116,15 +116,15 @@ Note that the copied template cannot be used directly. You need to modify it to
         # The Nix module system can modularize configuration,
         # improving the maintainability of configuration.
         #
-        # Each parameter in the `modules` is a Nix Module, and
+        # Each parameter in the `modules` is a Nixpkgs Module, and
         # there is a partial introduction to it in the nixpkgs manual:
         #    <https://nixos.org/manual/nixpkgs/unstable/#module-system-introduction>
         # It is said to be partial because the documentation is not
         # complete, only some simple introductions.
         # such is the current state of Nix documentation...
         #
-        # A Nix Module can be an attribute set, or a function that
-        # returns an attribute set. By default, if a Nix Module is a
+        # A Nixpkgs Module can be an attribute set, or a function that
+        # returns an attribute set. By default, if a Nixpkgs Module is a
         # function, this function have the following default parameters:
         #
         #  lib:     the nixpkgs function library, which provides many
@@ -155,7 +155,7 @@ Note that the copied template cannot be used directly. You need to modify it to
         modules = [
           # Import the configuration.nix here, so that the
           # old configuration file can still take effect.
-          # Note: configuration.nix itself is also a Nix Module,
+          # Note: configuration.nix itself is also a Nixpkgs Module,
           ./configuration.nix
         ];
       };

diff --git a/docs/other-usage-of-flakes/module-system.md b/docs/other-usage-of-flakes/module-system.md
@@ -0,0 +1,259 @@
+# Module System and Custom Options
+
+In our previous NixOS configurations, we set various values for `options` to configure NixOS or Home Manager. These `options` are actually defined in two locations:
+
+> If you are still using nix-darwin, its configuration is similar, and its module system is implemented in [nix-darwin/modules](https://github.com/LnL7/nix-darwin/tree/master/modules).
+
+- NixOS: [nixpkgs/nixos/modules](https://github.com/NixOS/nixpkgs/tree/23.11/nixos/modules), where all NixOS options visible on <https://search.nixos.org/options> are defined.
+- Home Manager: [home-manager/modules](https://github.com/nix-community/home-manager/blob/release-23.11/modules), where you can find all its options at <https://nix-community.github.io/home-manager/options.xhtml>.
+
+The foundation of the aforementioned NixOS Modules and Home Manager Modules is a universal module system implemented in Nixpkgs, found in [lib/modules.nix](lib/modules.nix). The official documentation for this module system is provided below (even for experienced NixOS users, understanding this can be a challenging task):
+
+- [Module System - Nixpkgs](Module System - Nixpkgs)
+
+Because the documentation for Nixpkgs' module system is lacking, it directly recommends reading another writing guide specifically for NixOS module system, which is clearer but might still be challenging for newcomers:
+
+- [Writing NixOS Modules - Nixpkgs](Writing NixOS Modules - Nixpkgs)
+
+In summary, the module system is implemented by Nixpkgs and is not part of the Nix package manager. Therefore, its documentation is not included in the Nix package manager's documentation. Additionally, both NixOS and Home Manager are based on Nixpkgs' module system implementation.
+
+## What is the Purpose of the Module System?
+
+As ordinary users, using various options implemented by NixOS and Home Manager based on the module system is sufficient to meet most of our needs. So, what are the benefits of delving into the module system for us?
+
+In the earlier discussion on modular configuration, the core idea was to split the configuration into multiple modules and then import these modules using `imports = [ ... ];`. This is the most basic usage of the module system. However, using only `imports = [ ... ];` allows us to import configurations defined in the module as they are without any customization, which limits flexibility. In simple configurations, this method is sufficient, but if the configuration is more complex, it becomes inadequate.
+
+To illustrate the drawback, let's consider an example. Suppose I manage four NixOS hosts, A, B, C, and D. I want to achieve the following goals while minimizing configuration repetition:
+
+- All hosts (A, B, C, and D) need to enable the Docker service and set it to start at boot.
+- Host A should change the Docker storage driver to `btrfs` while keeping other settings the same.
+- Hosts B and C, located in China, need to set a domestic mirror in Docker configuration.
+- Host C, located in the United States, has no special requirements.
+- Host D, a desktop machine, needs to set an HTTP proxy to accelerate Docker downloads.
+
+If we purely use `imports`, we might have to split the configuration into several modules like this and then import different modules for each host:
+
+```bash
+› tree
+.
+├── docker-default.nix  # Basic Docker configuration, including starting at boot
+├── docker-btrfs.nix    # Imports docker-default.nix and changes the storage driver to btrfs
+├── docker-china.nix    # Imports docker-default.nix and sets a domestic mirror
+└── docker-proxy.nix    # Imports docker-default.nix and sets an HTTP proxy
+```
+
+Doesn't this configuration seem redundant? This is still a simple example; if we have more machines with greater configuration differences, the redundancy becomes even more apparent.
+
+Clearly, we need other means to address this redundant configuration issue, and customizing some of our own `options` is an excellent choice.
+
+Before delving into the study of the module system, I emphasize once again that the following content is not necessary to learn and use. Many NixOS users have not customized any `options` and are satisfied with simply using `imports` to meet their needs. If you are a newcomer, consider learning this part when you encounter problems that `imports` cannot solve. That's completely okay.
+
+## Basic Structure and Usage
+
+The basic structure of modules defined in Nixpkgs is as follows:
+
+```nix
+{ config, pkgs, ... }:
+
+{
+  imports =
+    [ # import other modules here
+    ];
+
+  options = {
+    # ...
+  };
+
+  config = {
+    # ...
+  };
+}
+```
+
+Among these, we are already familiar with `imports = [ ... ];`, but the other two parts are yet to be explored. Let's have a brief introduction here:
+
+- `options = { ... };`: Similar to variable declarations in programming languages, it is used to declare configurable options.
+- `config = { ... };`: Similar to variable assignments in programming languages, it is used to assign values to the options declared in `options`.
+
+The most typical usage is to, within the same Nixpkgs module, set values for other `options` in `config = { .. };` based on the current values declared in `options = { ... };`. This achieves the functionality of parameterized configuration.
+
+It's easier to understand with a direct example:
+
+```nix
+{ config, lib, pkgs, ... }:
+
+with lib;
+
+let
+  cfg = config.programs.foo;
+in {
+  options.programs.foo = {
+    enable = mkEnableOption "the foo program";
+
+    package = mkOption {
+      type = types.package;
+      default = pkgs.jq;
+      defaultText = literalExpression "pkgs.foo";
+      description = "foo package to use.";
+    };
+
+    extraConfig = mkOption {
+      default = "";
+      example = ''
+        foo bar
+      '';
+      type = types.lines;
+      description = ''
+        Extra settings for foo.
+      '';
+    };
+  };
+
+  config = mkIf cfg.enable {
+    home.packages = [ cfg.package ];
+    xdg.configFile."foo/foorc" = mkIf (cfg.extraConfig != "") {
+      text = ''
+        # Generated by Home Manager.
+
+        ${cfg.extraConfig}
+      '';
+    };
+  };
+}
+```
+
+The module defined above introduces three `options`:
+
+- `programs.foo.enable`: Used to control whether to enable this module.
+- `programs.foo.package`: Allows customization of the `foo` package, such as using different versions, setting different compilation parameters, and so on.
+- `programs.foo.extraConfig`: Used for customizing the configuration file of `foo`.
+
+Then, in the `config` section, based on the values declared in these three variables in `options`, different settings are applied:
+
+- If `programs.foo.enable` is `false` or undefined, no settings are applied.
+  - This is achieved using `lib.mkIf`.
+- Otherwise,
+  - Add `programs.foo.package` to `home.packages` to install it in the user environment.
+    - Write the value of `programs.foo.extraConfig` to `~/.config/foo/foorc`.
+
+This way, we can import this module in another Nix file and achieve custom configuration for `foo` by setting the `options` defined here. For example:
+
+```nix
+{ config, lib, pkgs, ... }:
+
+{
+  imports = [
+    ./foo.nix
+  ];
+
+  programs.foo ={
+    enable = true;
+    package = pkgs.foo;
+    extraConfig = ''
+      foo baz
+    '';
+  };
+}
+```
+
+In the example above, the way we assign values to `options` is actually a kind of **abbreviation**. When a module declares only `options` without `config` (and other special parameters of the module system), we can omit the `config` prefix and directly use the name of `options` for assignment.
+
+## Assignment and Lazy Evaluation in the Module System
+
+The module system takes full advantage of Nix's lazy evaluation feature, which is crucial for achieving parameterized configuration.
+
+Let's start with a simple example:
+
+```nix
+{
+  description = "NixOS Flake for Test";
+
+  inputs = {
+    nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.11";
+    home-manager = {
+      url = "github:nix-community/home-manager/release-23.11";
+      inputs.nixpkgs.follows = "nixpkgs";
+    };
+  };
+
+  outputs = {nixpkgs, ...}: {
+    nixosConfigurations = {
+      "test" = nixpkgs.lib.nixosSystem {
+        system = "x86_64-linux";
+        modules = [
+          ({
+            config,
+            lib,
+            ...
+          }: {
+            options = {
+              foo = lib.mkOption {
+                default = false;
+                type = lib.types.bool;
+              };
+            };
+
+            # Scenario 1 (works fine)
+            config.warnings = if config.foo then ["foo"] else [];
+
+            # Scenario 2 (error: infinite recursion encountered)
+            # config = if config.foo then { warnings = ["foo"];} else {};
+
+            # Scenario 3 (works fine)
+            # config = lib.mkIf config.foo {warnings = ["foo"];};
+          })
+        ];
+      };
+    };
+  };
+}
+```
+
+In the examples 1, 2, and 3 of the above configuration, the value of `config.warnings` depends on the value of `config.foo`, but their implementation methods are different. Save the above configuration as `flake.nix`, and then use the command `nix eval .#nixosConfigurations.test.config.warnings` to test examples 1, 2, and 3 separately. You will find that examples 1 and 3 work correctly, while example 2 results in an error: `error: infinite recursion encountered`.
+
+Let's explain each case:
+
+1. Example 1 evaluation flow: `config.warnings` => `config.foo` => `config`
+    1. First, Nix attempts to compute the value of `config.warnings` but finds that it depends on `config.foo`.
+    2. Next, Nix tries to compute the value of `config.foo`, which depends on its outer `config`.
+    3. Nix attempts to compute the value of `config`, and since the contents not genuinely used by `config.foo` are lazily evaluated by Nix, there is no recursive dependency on `config.warnings` at this point.
+    4. The evaluation of `config.foo` is completed, followed by the assignment of `config.warnings`, and the computation ends.
+
+2. Example 2: `config` => `config.foo` => `config`
+    1. Initially, Nix tries to compute the value of `config` but finds that it depends on `config.foo`.
+    2. Next, Nix attempts to compute the value of `config.foo`, which depends on its outer `config`.
+    3. Nix tries to compute the value of `config`, and this loops back to step 1, leading to an infinite recursion and eventually an error.
+
+3. Example 3: The only difference from example 2 is the use of `lib.mkIf` to address the infinite recursion issue.
+
+The key lies in the function `lib.mkIf`. When using `lib.mkIf` to define `config`, it will be lazily evaluated by Nix. This means that the calculation of `config = lib.mkIf ...` will only occur after the evaluation of `config.foo` is completed.
+
+The Nixpkgs module system provides a series of functions similar to `lib.mkIf` for parameterized configuration and intelligent module merging:
+
+1. `lib.mkIf`: Already introduced.
+2. `lib.mkOverride` / `lib.mkDefault` / `lib.mkForce`: Previously discussed in [Modularizing NixOS Configuration](../nixos-with-flakes/modularize-the-configuration.md).
+3. `lib.mkOrder`, `lib.mkBefore`, and `lib.mkAfter`: As mentioned above.
+4. Check [Option Definitions - NixOS][Option Definitions - NixOS] for more functions related to option assignment (definition).
+
+## Option Declaration and Type Checking
+
+While assignment is the most commonly used feature of the module system, if you need to customize some `options`, you also need to delve into option declaration and type checking. I find this part relatively straightforward; it's much simpler than assignment, and you can understand the basics by directly referring to the official documentation. I won't go into detail here.
+
+- [Option Declarations - NixOS][Option Declarations - NixOS]
+- [Options Types - NixOS][Options Types - NixOS]
+
+## References
+
+- [Best resources for learning about the NixOS module system? - Discourse](https://discourse.nixos.org/t/best-resources-for-learning-about-the-nixos-module-system/1177/4)
+- [NixOS modules - NixOS Wiki](https://nixos.wiki/wiki/NixOS_modules)
+- [NixOS: config argument - NixOS Wiki](https://nixos.wiki/wiki/NixOS:config_argument)
+- [Module System - Nixpkgs][Module System - Nixpkgs]
+- [Writing NixOS Modules - Nixpkgs][Writing NixOS Modules - Nixpkgs]
+
+
+[lib/modules.nix]: https://github.com/NixOS/nixpkgs/blob/23.11/lib/modules.nix#L995
+[Module System - Nixpkgs]: https://github.com/NixOS/nixpkgs/blob/23.11/doc/module-system/module-system.chapter.md
+[Writing NixOS Modules - Nixpkgs]: https://github.com/NixOS/nixpkgs/blob/nixos-23.11/nixos/doc/manual/development/writing-modules.chapter.md
+[Option Definitions - NixOS]: https://github.com/NixOS/nixpkgs/blob/nixos-23.11/nixos/doc/manual/development/option-def.section.md
+[Option Declarations - NixOS]: https://github.com/NixOS/nixpkgs/blob/nixos-23.11/nixos/doc/manual/development/option-declarations.section.md
+[Options Types - NixOS]: https://github.com/NixOS/nixpkgs/blob/nixos-23.11/nixos/doc/manual/development/option-types.section.md
diff --git a/docs/other-usage-of-flakes/options.md b/docs/other-usage-of-flakes/options.md
diff --git a/docs/zh/introduction/advantages-and-disadvantages.md b/docs/zh/introduction/advantages-and-disadvantages.md
@@ -32,7 +32,7 @@
 - **学习成本高**:
   - 如果你希望系统完全可复现，并且避免各种不当使用导致的坑，那就需要学习了解 Nix 的整个设计，并以声明式的方式管理系统，不能无脑 `nix-env -i`（这类似 `apt-get install`）。
 - **文档混乱**:
-  - 首先 Nix Flakes 目前仍然是实验性特性，介绍它本身的文档目前比较匮乏, Nix 社区绝大多数文档都只介绍了旧的 `nix-env`/`nix-channel`，想直接从 Nix Flakes 开始学习的话，需要参考大量旧文档，从中提取出自己需要的内容。另外一些 Nix 当前的核心功能，官方文档都语焉不详（比如 `imports` 跟 Nix Module System），想搞明白基本只能看源码了...
+  - 首先 Nix Flakes 目前仍然是实验性特性，介绍它本身的文档目前比较匮乏, Nix 社区绝大多数文档都只介绍了旧的 `nix-env`/`nix-channel`，想直接从 Nix Flakes 开始学习的话，需要参考大量旧文档，从中提取出自己需要的内容。另外一些 Nix 当前的核心功能，官方文档都语焉不详（比如 `imports` 跟 Nixpkgs Module System），想搞明白基本只能看源码了...
 - **比较吃硬盘空间**:
   - 为了保证系统可以随时回退，nix 默认总是保留所有历史环境，这会使用更多的硬盘空间。
   - 多使用的这这些空间，在桌面电脑上可能不是啥事，但是在资源受限的云服务器上可能会是个问题。

diff --git a/docs/zh/nixos-with-flakes/modularize-the-configuration.md b/docs/zh/nixos-with-flakes/modularize-the-configuration.md
@@ -251,6 +251,8 @@ nix-repl> outputs.nixosConfigurations.nixos-test.config.environment.systemPackag
 
 > 虽然单纯调整 `systemPackages` 的顺序没什么用，但是在其他地方可能会有用...
 
+> 对模块系统更深入的介绍，参见 [模块系统与自定义 options](../other-usage-of-flakes/module-system.md).
+
 ## References
 
 - [Nix modules: Improving Nix's discoverability and usability ](https://cfp.nixcon.org/nixcon2020/talk/K89WJY/)