-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy path0 - Erase your darlings_ immutable infrastructure for mutable systems - Graham Christensen (3_24_2022 1_48_47 PM).html
86 lines (86 loc) · 31.5 KB
/
0 - Erase your darlings_ immutable infrastructure for mutable systems - Graham Christensen (3_24_2022 1_48_47 PM).html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
<!DOCTYPE html> <html><!--
Page saved with SingleFile
url: https://grahamc.com/blog/erase-your-darlings
saved date: Thu Mar 24 2022 13:48:47 GMT+0530 (India Standard Time)
--><meta charset=utf-8> <title>Erase your darlings: immutable infrastructure for mutable systems - Graham Christensen</title> <link rel="shortcut icon" href=data:image/x-icon;, type=image/x-icon>
<meta name=google-site-verification content=HvOMOXBefNLvJCYEGHm9ulUQsdgB59PYXjlsQe5H844>
<meta name=robots content="index, follow"> <meta name=viewport content="width=device-width, initial-scale=1.0"> <link rel=alternate type=application/rss+xml title="RSS 2.0" href=https://grahamc.com/feed> <style>body{background-color:rgb(251,251,251);color:rgb(84,84,84);max-width:660px;margin:22px auto 0 auto;padding-left:15px;padding-right:15px;font-family:Georgia,Palatino,Times,'Times New Roman',serif;font-size:18px;line-height:28px}code{font-size:.9em}h1,h2,h3,h4{color:rgb(75,75,75)}h1{line-height:1.5em;font-size:1.25em;font-weight:normal}h2{line-height:1.25em;font-size:1.5em;font-weight:normal}article header{margin-left:2rem;line-height:1.2}header{padding-bottom:0.5rem}header p.subtitle{margin-right:2em;font-style:italic}a,a:visited{color:rgb(65,110,210)}pre>code{display:block;overflow-x:auto}.p{font-weight:bold}</style> <link rel=canonical href=https://grahamc.com/blog/erase-your-darlings><meta http-equiv=content-security-policy content="default-src 'none'; font-src 'self' data:; img-src 'self' data:; style-src 'unsafe-inline'; media-src 'self' data:; script-src 'unsafe-inline' data:;"></head> <body> <header> <h1>Graham Christensen</h1> </header> <article> <header> <h2> Erase your darlings </h2> <p class=subtitle> immutable infrastructure for mutable systems </p> <p class=date>posted on April 13 2020</p> </header> <p>I erase my systems at every boot.</p> <p>Over time, a system collects state on its root partition. This state lives in assorted directories like <code class="language-plaintext highlighter-rouge">/etc</code> and <code class="language-plaintext highlighter-rouge">/var</code>, and represents every under-documented or out-of-order step in bringing up the services.</p> <blockquote> <p>“Right, run <code class="language-plaintext highlighter-rouge">myapp-init</code>.”</p> </blockquote> <p>These small, inconsequential “oh, oops” steps are the pieces that get lost and don’t appear in your runbooks.</p> <blockquote> <p>“Just download ca-certificates to … to fix …”</p> </blockquote> <p>Each of these quick fixes leaves you doomed to repeat history in three years when you’re finally doing that dreaded RHEL 7 to RHEL 8 upgrade.</p> <blockquote> <p>“Oh, <code class="language-plaintext highlighter-rouge">touch /etc/ipsec.secrets</code> or the l2tp tunnel won’t work.”</p> </blockquote> <h3 id=immutable-infrastructure-gets-us-so-close>Immutable infrastructure gets us <em>so</em> close</h3> <p>Immutable infrastructure is a wonderfully effective method of eliminating so many of these forgotten steps. Leaning in to the pain by deleting and replacing your servers on a weekly or monthly basis means you are constantly testing and exercising your automation and runbooks.</p> <p>The nugget here is the regular and indiscriminate removal of system state. Destroying the whole server doesn’t leave you much room to forget the little tweaks you made along the way.</p> <p>These techniques work great when you meet two requirements:</p> <ul> <li>you can provision and destroy servers with an API call</li> <li>the servers aren’t inherently stateful</li> </ul> <h4 id=long-running-servers>Long running servers</h4> <p>There are lots of cases in which immutable infrastructure <em>doesn’t</em> work, and the dirty secret is <strong>those servers need good tools the most.</strong></p> <p>Long-running servers cause long outages. Their runbooks are outdated and incomplete. They accrete tweaks and turn in to an ossified, brittle snowflake — except its arms are load-bearing.</p> <p>Let’s bring the ideas of immutable infrastructure to these systems too. Whether this system is embedded in a stadium’s jumbotron, in a datacenter, or under your desk, we <em>can</em> keep the state under control.</p> <h4 id=fhs-isnt-enough>FHS isn’t enough</h4> <p>The hard part about applying immutable techniques to long running servers is knowing exactly where your application state ends and the operating system, software, and configuration begin.</p> <p>This is hard because legacy operating systems and the Filesystem Hierarchy Standard poorly separate these areas of concern. For example, <code class="language-plaintext highlighter-rouge">/var/lib</code> is for state information, but how much of this do you actually care about tracking? What did you configure in <code class="language-plaintext highlighter-rouge">/etc</code> on purpose?</p> <p>The answer is probably not a lot.</p> <p>You may not care, but all of this accumulation of junk is a tarpit. Everything becomes harder: replicating production, testing changes, undoing mistakes.</p> <h3 id=new-computer-smell>New computer smell</h3> <p>Getting a new computer is this moment of cleanliness. The keycaps don’t have oils on them, the screen is perfect, and the hard drive is fresh and unspoiled — for about an hour or so.</p> <p>Let’s get back to that.</p> <h2 id=how-is-this-possible>How is this possible?</h2> <p>NixOS can boot with only two directories: <code class="language-plaintext highlighter-rouge">/boot</code>, and <code class="language-plaintext highlighter-rouge">/nix</code>.</p> <p><code class="language-plaintext highlighter-rouge">/nix</code> contains read-only system configurations, which are specified by your <code class="language-plaintext highlighter-rouge">configuration.nix</code> and are built and tracked as system generations. These never change. Once the files are created in <code class="language-plaintext highlighter-rouge">/nix</code>, the only way to change the config’s contents is to build a new system configuration with the contents you want.</p> <p>Any configuration or files created on the drive outside of <code class="language-plaintext highlighter-rouge">/nix</code> is state and cruft. We can lose everything outside of <code class="language-plaintext highlighter-rouge">/nix</code> and <code class="language-plaintext highlighter-rouge">/boot</code> and have a healthy system. My technique is to explicitly opt in and <em>choose</em> which state is important, and only keep that.</p> <p>How this is possible comes down to the boot sequence.</p> <p>For NixOS, the bootloader follows the same basic steps as a standard Linux distribution: the kernel starts with an initial ramdisk, and the initial ramdisk mounts the system disks.</p> <p>And here is where the similarities end.</p> <h3 id=nixoss-early-startup>NixOS’s early startup</h3> <p>NixOS configures the bootloader to pass some extra information: a specific system configuration. This is the secret to NixOS’s bootloader rollbacks, and also the key to erasing our disk on each boot. The parameter is named <code class="language-plaintext highlighter-rouge">systemConfig</code>.</p> <p>On every startup the very early boot stage knows what the system’s configuration should be: the entire system configuration is stored in the read-only <code class="language-plaintext highlighter-rouge">/nix/store</code>, and the directory passed through <code class="language-plaintext highlighter-rouge">systemConfig</code> has a reference to the config. Early boot then manipulates <code class="language-plaintext highlighter-rouge">/etc</code> and <code class="language-plaintext highlighter-rouge">/run</code> to match the chosen setup. Usually this involves swapping out a few symlinks.</p> <p>If <code class="language-plaintext highlighter-rouge">/etc</code> simply doesn’t exist, however, early boot <em>creates</em> <code class="language-plaintext highlighter-rouge">/etc</code> and moves on like it were any other boot. It also <em>creates</em> <code class="language-plaintext highlighter-rouge">/var</code>, <code class="language-plaintext highlighter-rouge">/dev</code>, <code class="language-plaintext highlighter-rouge">/home</code>, and any other core directories that must be present.</p> <p>Simply speaking, an empty <code class="language-plaintext highlighter-rouge">/</code> is <em>not surprising</em> to NixOS. In fact, the NixOS netboot, EC2, and installation media all start out this way.</p> <h2 id=opting-out>Opting out</h2> <p>Before we can opt in to saving data, we must opt out of saving data <em>by default</em>. I do this by setting up my filesystem in a way that lets me easily and safely erase the unwanted data, while preserving the data I do want to keep.</p> <p>My preferred method for this is using a ZFS dataset and rolling it back to a blank snapshot before it is mounted. A partition of any other filesystem would work just as well too, running <code class="language-plaintext highlighter-rouge">mkfs</code> at boot, or something similar. If you have a lot of RAM, you could skip the erase step and make <code class="language-plaintext highlighter-rouge">/</code> a tmpfs.</p> <h3 id=opting-out-with-zfs>Opting out with ZFS</h3> <p>When installing NixOS, I partition my disk with two partitions, one for the boot partition, and another for a ZFS pool. Then I create and mount a few datasets.</p> <p>My root dataset:</p> <div class="language-plaintext highlighter-rouge"><div class=highlight><pre class=highlight><code># zfs create -p -o mountpoint=legacy rpool/local/root
</code></pre></div></div> <p>Before I even mount it, I <strong>create a snapshot while it is totally blank</strong>:</p> <div class="language-plaintext highlighter-rouge"><div class=highlight><pre class=highlight><code># zfs snapshot rpool/local/root@blank
</code></pre></div></div> <p>And then mount it:</p> <div class="language-plaintext highlighter-rouge"><div class=highlight><pre class=highlight><code># mount -t zfs rpool/local/root /mnt
</code></pre></div></div> <p>Then I mount the partition I created for the <code class="language-plaintext highlighter-rouge">/boot</code>:</p> <div class="language-plaintext highlighter-rouge"><div class=highlight><pre class=highlight><code># mkdir /mnt/boot
# mount /dev/the-boot-partition /mnt/boot
</code></pre></div></div> <p>Create and mount a dataset for <code class="language-plaintext highlighter-rouge">/nix</code>:</p> <div class="language-plaintext highlighter-rouge"><div class=highlight><pre class=highlight><code># zfs create -p -o mountpoint=legacy rpool/local/nix
# mkdir /mnt/nix
# mount -t zfs rpool/local/nix /mnt/nix
</code></pre></div></div> <p>And a dataset for <code class="language-plaintext highlighter-rouge">/home</code>:</p> <div class="language-plaintext highlighter-rouge"><div class=highlight><pre class=highlight><code># zfs create -p -o mountpoint=legacy rpool/safe/home
# mkdir /mnt/home
# mount -t zfs rpool/safe/home /mnt/home
</code></pre></div></div> <p>And finally, a dataset explicitly for state I want to persist between boots:</p> <div class="language-plaintext highlighter-rouge"><div class=highlight><pre class=highlight><code># zfs create -p -o mountpoint=legacy rpool/safe/persist
# mkdir /mnt/persist
# mount -t zfs rpool/safe/persist /mnt/persist
</code></pre></div></div> <blockquote> <p><em>Note:</em> in my systems, datasets under <code class="language-plaintext highlighter-rouge">rpool/local</code> are never backed up, and datasets under <code class="language-plaintext highlighter-rouge">rpool/safe</code> are.</p> </blockquote> <p>And now safely erasing the root dataset on each boot is very easy: after devices are made available, roll back to the blank snapshot:</p> <div class="language-nix highlighter-rouge"><div class=highlight><pre class=highlight><code><span class=p>{</span>
<span class=nv>boot</span><span class=o>.</span><span class=nv>initrd</span><span class=o>.</span><span class=nv>postDeviceCommands</span> <span class=o>=</span> <span class=nv>lib</span><span class=o>.</span><span class=nv>mkAfter</span> <span class=s2>''</span><span class=err>
</span><span class=s2> zfs rollback -r rpool/local/root@blank</span><span class=err>
</span><span class=s2> ''</span><span class=p>;</span>
<span class=p>}</span>
</code></pre></div></div> <p>I then finish the installation as normal. If all goes well, your next boot will start with an empty root partition but otherwise be configured exactly as you specified.</p> <h2 id=opting-in>Opting in</h2> <p>Now that I’m keeping no state, it is time to specify what I do want to keep. My choices here are different based on the role of the system: a laptop has different state than a server.</p> <p>Here are some different pieces of state and how I preserve them. These examples largely use reconfiguration or symlinks, but using ZFS datasets and mount points would work too.</p> <h4 id=wireguard-private-keys>Wireguard private keys</h4> <p>Create a directory under <code class="language-plaintext highlighter-rouge">/persist</code> for the key:</p> <div class="language-plaintext highlighter-rouge"><div class=highlight><pre class=highlight><code># mkdir -p /persist/etc/wireguard/
</code></pre></div></div> <p>And use Nix’s wireguard module to generate the key there:</p> <div class="language-nix highlighter-rouge"><div class=highlight><pre class=highlight><code><span class=p>{</span>
<span class=nv>networking</span><span class=o>.</span><span class=nv>wireguard</span><span class=o>.</span><span class=nv>interfaces</span><span class=o>.</span><span class=nv>wg0</span> <span class=o>=</span> <span class=p>{</span>
<span class=nv>generatePrivateKeyFile</span> <span class=o>=</span> <span class=kc>true</span><span class=p>;</span>
<span class=nv>privateKeyFile</span> <span class=o>=</span> <span class=s2>"/persist/etc/wireguard/wg0"</span><span class=p>;</span>
<span class=p>};</span>
<span class=p>}</span>
</code></pre></div></div> <h4 id=networkmanager-connections>NetworkManager connections</h4> <p>Create a directory under <code class="language-plaintext highlighter-rouge">/persist</code>, mirroring the <code class="language-plaintext highlighter-rouge">/etc</code> structure:</p> <div class="language-plaintext highlighter-rouge"><div class=highlight><pre class=highlight><code># mkdir -p /persist/etc/NetworkManager/system-connections
</code></pre></div></div> <p>And use Nix’s <code class="language-plaintext highlighter-rouge">etc</code> module to set up the symlink:</p> <div class="language-nix highlighter-rouge"><div class=highlight><pre class=highlight><code><span class=p>{</span>
<span class=nv>etc</span><span class=o>.</span><span class=s2>"NetworkManager/system-connections"</span> <span class=o>=</span> <span class=p>{</span>
<span class=nv>source</span> <span class=o>=</span> <span class=s2>"/persist/etc/NetworkManager/system-connections/"</span><span class=p>;</span>
<span class=p>};</span>
<span class=p>}</span>
</code></pre></div></div> <h4 id=bluetooth-devices>Bluetooth devices</h4> <p>Create a directory under <code class="language-plaintext highlighter-rouge">/persist</code>, mirroring the <code class="language-plaintext highlighter-rouge">/var</code> structure:</p> <div class="language-plaintext highlighter-rouge"><div class=highlight><pre class=highlight><code># mkdir -p /persist/var/lib/bluetooth
</code></pre></div></div> <p>And then use systemd’s tmpfiles.d rules to create a symlink from <code class="language-plaintext highlighter-rouge">/var/lib/bluetooth</code> to my persisted directory:</p> <div class="language-nix highlighter-rouge"><div class=highlight><pre class=highlight><code><span class=p>{</span>
<span class=nv>systemd</span><span class=o>.</span><span class=nv>tmpfiles</span><span class=o>.</span><span class=nv>rules</span> <span class=o>=</span> <span class=p>[</span>
<span class=s2>"L /var/lib/bluetooth - - - - /persist/var/lib/bluetooth"</span>
<span class=p>];</span>
<span class=p>}</span>
</code></pre></div></div> <h4 id=ssh-host-keys>SSH host keys</h4> <p>Create a directory under <code class="language-plaintext highlighter-rouge">/persist</code>, mirroring the <code class="language-plaintext highlighter-rouge">/etc</code> structure:</p> <div class="language-plaintext highlighter-rouge"><div class=highlight><pre class=highlight><code># mkdir -p /persist/etc/ssh
</code></pre></div></div> <p>And use Nix’s openssh module to create and use the keys in that directory:</p> <div class="language-nix highlighter-rouge"><div class=highlight><pre class=highlight><code><span class=p>{</span>
<span class=nv>services</span><span class=o>.</span><span class=nv>openssh</span> <span class=o>=</span> <span class=p>{</span>
<span class=nv>enable</span> <span class=o>=</span> <span class=kc>true</span><span class=p>;</span>
<span class=nv>hostKeys</span> <span class=o>=</span> <span class=p>[</span>
<span class=p>{</span>
<span class=nv>path</span> <span class=o>=</span> <span class=s2>"/persist/ssh/ssh_host_ed25519_key"</span><span class=p>;</span>
<span class=nv>type</span> <span class=o>=</span> <span class=s2>"ed25519"</span><span class=p>;</span>
<span class=p>}</span>
<span class=p>{</span>
<span class=nv>path</span> <span class=o>=</span> <span class=s2>"/persist/ssh/ssh_host_rsa_key"</span><span class=p>;</span>
<span class=nv>type</span> <span class=o>=</span> <span class=s2>"rsa"</span><span class=p>;</span>
<span class=nv>bits</span> <span class=o>=</span> <span class=mi>4096</span><span class=p>;</span>
<span class=p>}</span>
<span class=p>];</span>
<span class=p>};</span>
<span class=p>}</span>
</code></pre></div></div> <h4 id=acme-certificates>ACME certificates</h4> <p>Create a directory under <code class="language-plaintext highlighter-rouge">/persist</code>, mirroring the <code class="language-plaintext highlighter-rouge">/var</code> structure:</p> <div class="language-plaintext highlighter-rouge"><div class=highlight><pre class=highlight><code># mkdir -p /persist/var/lib/acme
</code></pre></div></div> <p>And then use systemd’s tmpfiles.d rules to create a symlink from <code class="language-plaintext highlighter-rouge">/var/lib/acme</code> to my persisted directory:</p> <div class="language-nix highlighter-rouge"><div class=highlight><pre class=highlight><code><span class=p>{</span>
<span class=nv>systemd</span><span class=o>.</span><span class=nv>tmpfiles</span><span class=o>.</span><span class=nv>rules</span> <span class=o>=</span> <span class=p>[</span>
<span class=s2>"L /var/lib/acme - - - - /persist/var/lib/acme"</span>
<span class=p>];</span>
<span class=p>}</span>
</code></pre></div></div> <h3 id=answering-the-question-what-am-i-about-to-lose>Answering the question “what am I about to lose?”</h3> <p>I found this process a bit scary for the first few weeks: was I losing important data each reboot? No, I wasn’t.</p> <p>If you’re worried and want to know what state you’ll lose on the next boot, you can list the files on your root filesystem and see if you’re missing something important:</p> <div class="language-plaintext highlighter-rouge"><div class=highlight><pre class=highlight><code># tree -x /
├── bin
│ └── sh -> /nix/store/97zzcs494vn5k2yw-dash-0.5.10.2/bin/dash
├── boot
├── dev
├── etc
│ ├── asound.conf -> /etc/static/asound.conf
... snip ...
</code></pre></div></div> <p>ZFS can give you a similar answer:</p> <div class="language-plaintext highlighter-rouge"><div class=highlight><pre class=highlight><code># zfs diff rpool/local/root@blank
M /
+ /nix
+ /etc
+ /root
+ /var/lib/is-nix-channel-up-to-date
+ /etc/pki/fwupd
+ /etc/pki/fwupd-metadata
... snip ...
</code></pre></div></div> <h2 id=your-stateless-future>Your stateless future</h2> <p>You may bump in to new state you meant to be preserving. When I’m adding new services, I think about the state it is writing and whether I care about it or not. If I care, I find a way to redirect its state to <code class="language-plaintext highlighter-rouge">/persist</code>.</p> <p>Take care to reboot these machines on a somewhat regular basis. It will keep things agile, proving your system state is tracked correctly.</p> <p>This technique has given me the “new computer smell” on every boot without the datacenter full of hardware, and even on systems that do carry important state. I have deployed this strategy to systems in the large and small: build farm servers, database servers, my NAS and home server, my raspberry pi garage door opener, and laptops.</p> <p>NixOS enables powerful new deployment models in so many ways, allowing for systems of all shapes and sizes to be managed properly and consistently. I think this model of ephemeral roots is yet another example of this flexibility and power. I would like to see this partitioning scheme become a reference architecture and take us out of this eternal tarpit of legacy.</p> </article> <hr> <nav> <h2>Posts</h2> <ul> <li> <a href=https://grahamc.com/blog/nixos-on-framework class=homelink> NixOS on the Framework </a> </li> <li> <a href=https://grahamc.com/blog/flakes-are-an-obviously-good-thing class=homelink> Flakes are such an obviously good thing </a> </li> <li> <a href=https://grahamc.com/blog/erase-your-darlings class=homelink> Erase your darlings </a> </li> <li> <a href=https://grahamc.com/blog/nixos-on-zfs class=homelink> ZFS Datasets for NixOS </a> </li> <li> <a href=https://grahamc.com/blog/nix-and-layered-docker-images class=homelink> Optimising Docker Layers for Better Caching with Nix </a> </li> <li> <a href=https://grahamc.com/blog/an-epyc-nixos-build-farm class=homelink> an EPYC NixOS build farm </a> </li> <li> <a href=https://grahamc.com/blog/cache-nixos-org-now-more-local class=homelink> cache.nixos.org, now more local! </a> </li> <li> <a href=https://grahamc.com/blog/nixos-system-version-prometheus class=homelink> Prometheus and the NixOS System Version </a> </li> <li> <a href=https://grahamc.com/blog/nixos-on-dell-9560 class=homelink> NixOS on a Dell 9560 </a> </li> <li> <a href=https://grahamc.com/blog/timemachine-backups-linux-nixos class=homelink> How to use a NixOS Linux Server for Time Machine Backups </a> </li> <li> <a href=https://grahamc.com/blog/ascii-decode-error-pip-install-docker class=homelink> Pip Install with Docker and Fixing the ascii decode error </a> </li> <li> <a href=https://grahamc.com/blog/packer-ami-device-volume-types class=homelink> Packer - Create AMI with EBS Volumes with VolumeType </a> </li> <li> <a href=https://grahamc.com/blog/enable-certificate-revocation class=homelink> Enable certificate revocation in Chrome </a> </li> <li> <a href=https://grahamc.com/blog/clear-all-beanstalk-jobs-in-a-tube class=homelink> How to delete all (or most) jobs from a beanstalk tube from the shell </a> </li> <li> <a href=https://grahamc.com/blog/mysql-slave-broken-lvm-snapshot-table-corrupt class=homelink> Why a MySQL Slave Created from an LVM Snapshot Would Mark Tables Corrupt </a> </li> <li> <a href=https://grahamc.com/blog/getting-list-of-users-with-access-to-a-database class=homelink> Listing Users with Database Access </a> </li> </ul> </nav> <section> <h2>About</h2> <p> Graham works on <a href=https://nixos.org/>NixOS</a>. </p> <ul> <li> <strong>E-Mail:</strong> graham<span>@</span>grahamc.com<br> </li> <li> <strong>Phone:</strong> +1-407-670-9980 </li> <li> <strong>GitHub:</strong> <a href=https://github.com/grahamc>github.com/grahamc</a> </li> <li> <strong>Twitter:</strong> <a href=https://twitter.com/grhmc>@grhmc</a> </li> </ul> </section> <footer> <p> © 2022 Graham Christensen. All Rights Reserved. · <a href=https://grahamc.com/feed class=rss>Subscribe to RSS</a> </p> </footer>
<script>document.currentScript.remove();!function(){"use strict";(t=>{const n="singlefile-infobar",e="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEAAAABAAgMAAADXB5lNAAABhmlDQ1BJQ0MgcHJvZmlsZQAAKJF9kj1Iw0AYht+mSkUrDnYQcchQnSyIijqWKhbBQmkrtOpgcukfNGlIUlwcBdeCgz+LVQcXZ10dXAVB8AfEydFJ0UVK/C4ptIjx4LiH9+59+e67A4RGhalm1wSgapaRisfEbG5VDLyiDwEAvZiVmKkn0osZeI6ve/j4ehfhWd7n/hz9St5kgE8kjjLdsIg3iGc2LZ3zPnGIlSSF+Jx43KACiR+5Lrv8xrnosMAzQ0YmNU8cIhaLHSx3MCsZKvE0cVhRNcoXsi4rnLc4q5Uaa9XJbxjMaytprtMcQRxLSCAJETJqKKMCCxFaNVJMpGg/5uEfdvxJcsnkKoORYwFVqJAcP/gb/O6tWZiadJOCMaD7xbY/RoHALtCs2/b3sW03TwD/M3Cltf3VBjD3SXq9rYWPgIFt4OK6rcl7wOUOMPSkS4bkSH6aQqEAvJ/RM+WAwVv6EGtu31r7OH0AMtSr5Rvg4BAYK1L2use9ezr79u+ZVv9+AFlNcp0UUpiqAAAACXBIWXMAAC4jAAAuIwF4pT92AAAAB3RJTUUH5AsHAB8H+DhhoQAAABl0RVh0Q29tbWVudABDcmVhdGVkIHdpdGggR0lNUFeBDhcAAAAJUExURQAAAICHi4qKioTuJAkAAAABdFJOUwBA5thmAAAAAWJLR0QCZgt8ZAAAAJJJREFUOI3t070NRCEMA2CnYAOyDyPwpHj/Va7hJ3FzV7zy3ET5JIwoAF6Jk4wzAJAkzxAYG9YRTgB+24wBgKmfrGAKTcEfAY4KRlRoIeBTgKOCERVaCPgU4Khge2GqKOBTgKOCERVaAEC/4PNcnyoSWHpjqkhwKxbcig0Q6AorXYF/+A6eIYD1lVbwG/jdA6/kA2THRAURVubcAAAAAElFTkSuQmCC",o="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEAAAABAAgMAAADXB5lNAAABhmlDQ1BJQ0MgcHJvZmlsZQAAKJF9kj1Iw0AYht+mSkUrDnYQcchQnSyIijqWKhbBQmkrtOpgcukfNGlIUlwcBdeCgz+LVQcXZ10dXAVB8AfEydFJ0UVK/C4ptIjx4LiH9+59+e67A4RGhalm1wSgapaRisfEbG5VDLyiDwEAvZiVmKkn0osZeI6ve/j4ehfhWd7n/hz9St5kgE8kjjLdsIg3iGc2LZ3zPnGIlSSF+Jx43KACiR+5Lrv8xrnosMAzQ0YmNU8cIhaLHSx3MCsZKvE0cVhRNcoXsi4rnLc4q5Uaa9XJbxjMaytprtMcQRxLSCAJETJqKKMCCxFaNVJMpGg/5uEfdvxJcsnkKoORYwFVqJAcP/gb/O6tWZiadJOCMaD7xbY/RoHALtCs2/b3sW03TwD/M3Cltf3VBjD3SXq9rYWPgIFt4OK6rcl7wOUOMPSkS4bkSH6aQqEAvJ/RM+WAwVv6EGtu31r7OH0AMtSr5Rvg4BAYK1L2use9ezr79u+ZVv9+AFlNcp0UUpiqAAAACXBIWXMAAC4jAAAuIwF4pT92AAAAB3RJTUUH5AsHAB8VC4EQ6QAAABl0RVh0Q29tbWVudABDcmVhdGVkIHdpdGggR0lNUFeBDhcAAAAJUExURQAAAICHi4qKioTuJAkAAAABdFJOUwBA5thmAAAAAWJLR0QCZgt8ZAAAAJtJREFUOI3NkrsBgCAMRLFwBPdxBArcfxXFkO8rbKWAAJfHJ9faf9vuYX/749T5NmShm3bEwbe2SxeuM4+2oxDL1cDoKtVUjRy+tH78Cv2CS+wIiQNC1AEhk4AQeUTMWUJMfUJMSEJMSEY8kIx4IONroaYAimNxsXp1PA7PxwfVL8QnowwoVC0lig07wDDVUjAdbAnjwtow/z/bDW7eI4M2KruJAAAAAElFTkSuQmCC",i="SingleFile",A="single-file-ui-element",r="\n\t.infobar {\n\t\tbackground-color: #737373;\n\t\tcolor: white;\n\t\tdisplay: flex;\n\t\tposition: fixed;\n\t\ttop: 16px;\n\t\tright: 16px;\n\t\theight: auto;\n\t\twidth: auto;\n\t\tmin-height: 24px;\n\t\tmin-width: 24px;\n\t\tbackground-position: center;\n\t\tbackground-repeat: no-repeat;\n\t\tz-index: 2147483647;\n\t\tmargin: 0 0 0 16px;\n\t\tbackground-image: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEAAAABABAMAAABYR2ztAAABhmlDQ1BJQ0MgcHJvZmlsZQAAKJF9kj1Iw0AYht+mSkUrDnYQcchQnSyIijqWKhbBQmkrtOpgcukfNGlIUlwcBdeCgz+LVQcXZ10dXAVB8AfEydFJ0UVK/C4ptIjx4LiH9+59+e67A4RGhalm1wSgapaRisfEbG5VDLyiDwEAvZiVmKkn0osZeI6ve/j4ehfhWd7n/hz9St5kgE8kjjLdsIg3iGc2LZ3zPnGIlSSF+Jx43KACiR+5Lrv8xrnosMAzQ0YmNU8cIhaLHSx3MCsZKvE0cVhRNcoXsi4rnLc4q5Uaa9XJbxjMaytprtMcQRxLSCAJETJqKKMCCxFaNVJMpGg/5uEfdvxJcsnkKoORYwFVqJAcP/gb/O6tWZiadJOCMaD7xbY/RoHALtCs2/b3sW03TwD/M3Cltf3VBjD3SXq9rYWPgIFt4OK6rcl7wOUOMPSkS4bkSH6aQqEAvJ/RM+WAwVv6EGtu31r7OH0AMtSr5Rvg4BAYK1L2use9ezr79u+ZVv9+AFlNcp0UUpiqAAAACXBIWXMAAC4jAAAuIwF4pT92AAAAB3RJTUUH5AsHADIRLMaOHwAAABl0RVh0Q29tbWVudABDcmVhdGVkIHdpdGggR0lNUFeBDhcAAAAPUExURQAAAIqKioyNjY2OjvDw8L2y1DEAAAABdFJOUwBA5thmAAAAAWJLR0QB/wIt3gAAAGNJREFUSMdjYCAJsLi4OBCQx6/CBQwIGIDPCBcXAkYQUsACU+AwlBVQHg6Eg5pgZBGOboIJZugDFwRwoJECJCUOhJI1wZwzqmBUwagCuipgIqTABG9h7YIKaKGAURAFEF/6AQAO4HqSoDP8bgAAAABJRU5ErkJggg==);\n\t\tborder-radius: 16px;\n\t\tuser-select: none;\n\t\t-moz-user-select: none;\n\t\topacity: .7;\n\t\tcursor: pointer;\n\t\tpadding-left: 0;\n\t\tpadding-right: 0;\n\t\tpadding-top: 0;\n\t\tpadding-bottom: 0;\n\t\tborder: 2px solid #eee;\n\t\tbackground-size: 70% 70%;\n\t\ttransition: all 250ms;\n\t\tfont-size: 13px;\n\t}\n\t.infobar:hover {\n\t\topacity: 1;\n\t}\n\t.infobar-open {\n\t\topacity: 1;\n\t\tbackground-color: #f9f9f9;\n\t\tcursor: auto;\n\t\tcolor: #2d2d2d;\n\t\tpadding-top: 2px;\n\t\tpadding-bottom: 2px;\n\t\tborder: 2px solid #878787;\n\t\tbackground-image: none;\n\t\tborder-radius: 8px;\n\t\tuser-select: initial;\n\t\t-moz-user-select: initial;\n\t}\n\t.infobar-close-button {\n\t\tdisplay: none;\n\t\topacity: .7;\n\t\tpadding-top: 4px;\n\t\tpadding-left: 8px;\n\t\tpadding-right: 8px;\n\t\tcursor: pointer;\n\t\ttransition: opacity 250ms;\n\t\theight: 16px;\n\t}\n\t.infobar-close-button:hover {\n\t\topacity: 1;\n\t}\n\t.infobar-content {\n\t\tdisplay: none;\n\t\tfont-family: Arial;\n\t\tfont-size: 14px;\n\t\tline-height: 22px;\n\t\tword-break: break-word;\n\t\twhite-space: pre-wrap;\n\t\tposition: relative;\n\t\ttop: 1px;\n\t\ttext-align: left;\n\t}\n\t.infobar-link {\n\t\tdisplay: none;\n\t\tpadding-left: 8px;\n\t\tpadding-right: 8px;\n\t\tline-height: 11px;\n\t\tcursor: pointer;\n\t\tuser-select: none;\n\t\toutline: 0;\n\t}\n\t.infobar-link-icon {\n\t\tpadding-top: 4px;\n\t\tpadding-left: 2px;\n\t\tcursor: pointer;\n\t\topacity: .7;\n\t\ttransition: opacity 250ms;\n\t\theight: 16px;\n\t}\n\t.infobar-link-icon:hover {\n\t\topacity: 1;\n\t}\n\t.infobar-open .infobar-close-button, .infobar-open .infobar-content, .infobar-open .infobar-link {\n\t\tdisplay: inline-block;\n\t}";let a=!0;const c=t.browser;async function s(){const t=document.evaluate("//comment()",document,null,XPathResult.FIRST_ORDERED_NODE_TYPE,null);let s=t&&t.singleNodeValue;if(s&&((p=s).nodeType==Node.COMMENT_NODE&&p.textContent.includes(i))){const t=s.textContent.split("\n"),[,,i,p,...g]=t;if(i&&p){let t;t=c&&c.runtime&&c.runtime.sendMessage?await c.runtime.sendMessage({method:"tabs.getOptions",url:i}):{displayInfobar:!0},t.displayInfobar&&await async function(t,i,c){let s=document.querySelector(n);if(!s){if(t=t.split("url: ")[1],i=i.split("saved date: ")[1],c&&c.length>1){let t=c[0].split("info: ")[1].trim();for(let n=1;n<c.length-1;n++)t+="\n"+c[n].trim();c=t.trim()}else c=i;s=d(n,document.body),s.className=A;const p=await async function(t){if(t.attachShadow)return t.attachShadow({mode:"open"});{a=!1;const n=d("iframe",t);return n.style.setProperty("background-color","transparent","important"),n.style.setProperty("position","fixed","important"),n.style.setProperty("top",0,"important"),n.style.setProperty("right",0,"important"),n.style.setProperty("width","44px","important"),n.style.setProperty("height","48px","important"),n.style.setProperty("z-index",2147483647,"important"),new Promise((t=>{n.contentDocument.body.style.setProperty("margin",0),n.onload=()=>t(n.contentDocument.body)}))}}(s),g=document.createElement("style");g.textContent=r,p.appendChild(g);const u=document.createElement("div");u.classList.add("infobar"),p.appendChild(u);const h=document.createElement("img");h.classList.add("infobar-close-button"),u.appendChild(h),h.src=o,h.onclick=t=>{0===t.button&&s.remove()};const m=document.createElement("span");u.appendChild(m),m.classList.add("infobar-content"),m.textContent=c;const b=document.createElement("a");b.classList.add("infobar-link"),u.appendChild(b),b.target="_blank",b.rel="noopener noreferrer",b.title="Open source URL: "+t,b.href=t;const f=document.createElement("img");f.classList.add("infobar-link-icon"),b.appendChild(f),f.src=e,l(u),document.addEventListener("click",(t=>{if(0===t.button){let n=t.target;for(;n&&n!=s;)n=n.parentElement;n!=s&&l(u)}}))}}(i,p,g)}}var p}function l(t){if(t.classList.remove("infobar-open"),t.onclick=e=>{if(0===e.button)return function(t){a||document.querySelector(n).childNodes[0].contentWindow.getSelection().removeAllRanges();if(t.classList.add("infobar-open"),t.onclick=null,t.onmouseout=null,!a){const e=document.querySelector(n).childNodes[0];e.style.setProperty("width","100vw","important"),e.style.setProperty("height","100vh","important"),e.style.setProperty("width",t.getBoundingClientRect().width+33+"px","important"),e.style.setProperty("height",t.getBoundingClientRect().height+21+"px","important")}}(t),!1},!a){const t=document.querySelector(n).childNodes[0];t.style.setProperty("width","44px","important"),t.style.setProperty("height","48px","important")}}function d(t,n){const e=document.createElement(t);return n.appendChild(e),Array.from(getComputedStyle(e)).forEach((t=>e.style.setProperty(t,"initial","important"))),e}t.window==t.top&&("loading"==document.readyState?document.addEventListener("DOMContentLoaded",s,!1):s())})("object"==typeof globalThis?globalThis:window)}();</script>