Updates from website repo

ryapric · ryapric · commit 05681cb0b864 · 2022-06-20T15:23:08.000-05:00
diff --git a/blog/index.html b/blog/index.html
@@ -26,6 +26,14 @@ <h2>OpenSourceCorp Blog <a href="/blog/index.xml"><img alt="rss-feed" src="/imag
 
 <ul>
   
+    <li>
+      <h4><a href="https://www.opensourcecorp.org/blog/posts/2022-06-20-progress-report/">Progress Report 2022-06-20</a></h4>
+      <p><em>Ryan J. Price, 2022-06-20</em></p>
+      Some heavy updates in this one, so buckle up for what&rsquo;s changed!
+Secrets management Up until now, OSC has been strictly leveraging an implementation of Salt&rsquo;s Pillar functionality for secrets management &ndash; and that implementation is the default, which is &ldquo;store all secrets on the configmgmt disk&rdquo;. While this has worked fine (and can be a valid strategy in some enterprise environments), it&rsquo;s not what you would call &ldquo;best-practice&rdquo;. If the configmgmt subsystem itself is ever compromised, every single piece of sensitive data is accessible to the attacker at once &ndash; default Pillar data is just stored in plaintext on disk. <a href="https://www.opensourcecorp.org/blog/posts/2022-06-20-progress-report/">(more)</a>
+      <br/><br/>
+    </li>
+  
     <li>
       <h4><a href="https://www.opensourcecorp.org/blog/posts/2022-06-05-progress-report/">Progress Report 2022-06-05</a></h4>
       <p><em>Ryan J. Price, 2022-06-05</em></p>
diff --git a/blog/index.xml b/blog/index.xml
@@ -7,6 +7,16 @@
     <generator>Hugo -- gohugo.io</generator>
     <language>en-us</language>
     <lastBuildDate>Wed, 01 Jun 2022 21:17:46 -0500</lastBuildDate><atom:link href="https://www.opensourcecorp.org/blog/index.xml" rel="self" type="application/rss+xml" />
+    <item>
+      <title>Progress Report 2022-06-20</title>
+      <link>https://www.opensourcecorp.org/blog/posts/2022-06-20-progress-report/</link>
+      <pubDate>Mon, 20 Jun 2022 15:00:00 -0500</pubDate>
+      
+      <guid>https://www.opensourcecorp.org/blog/posts/2022-06-20-progress-report/</guid>
+      <description>Some heavy updates in this one, so buckle up for what&amp;rsquo;s changed!
+Secrets management Up until now, OSC has been strictly leveraging an implementation of Salt&amp;rsquo;s Pillar functionality for secrets management &amp;ndash; and that implementation is the default, which is &amp;ldquo;store all secrets on the configmgmt disk&amp;rdquo;. While this has worked fine (and can be a valid strategy in some enterprise environments), it&amp;rsquo;s not what you would call &amp;ldquo;best-practice&amp;rdquo;. If the configmgmt subsystem itself is ever compromised, every single piece of sensitive data is accessible to the attacker at once &amp;ndash; default Pillar data is just stored in plaintext on disk.</description>
+    </item>
+    
     <item>
       <title>Progress Report 2022-06-05</title>
       <link>https://www.opensourcecorp.org/blog/posts/2022-06-05-progress-report/</link>
diff --git a/blog/posts/2021-07-05-announcing-opensourcecorp/index.html b/blog/posts/2021-07-05-announcing-opensourcecorp/index.html
@@ -23,8 +23,7 @@ <h3>Navigation</h3>
   <h2>Announcing OpenSourceCorp!</h2>
 
   <em>By Ryan J. Price @ 2021-07-05 23:50:00 -0500 CDT; reading time 5m</em>
-
-  <br/><br/><br/>
+  <hr/>
 
   <p>Today, I&rsquo;m very excited to finally announce the public launch of
 <a href="https://github.com/opensourcecorp">OpenSourceCorp</a> &ndash; the free and open-source
diff --git a/blog/posts/2021-07-25-new-arch-diagram/index.html b/blog/posts/2021-07-25-new-arch-diagram/index.html
@@ -23,8 +23,7 @@ <h3>Navigation</h3>
   <h2>New architecture diagram!</h2>
 
   <em>By Ryan J. Price @ 2021-07-25 20:00:00 -0600 -0600; reading time 1m</em>
-
-  <br/><br/><br/>
+  <hr/>
 
   <p><a href="https://github.com/opensourcecorp/opensourcecorp">OpenSourceCorp&rsquo;s home repo</a>
 has <a href="https://github.com/opensourcecorp/opensourcecorp/blob/main/docs/architecture.md">a new architecture
diff --git a/blog/posts/2021-10-04-deployment-update/index.html b/blog/posts/2021-10-04-deployment-update/index.html
@@ -23,8 +23,7 @@ <h3>Navigation</h3>
   <h2>OSC updates: hosting decision &amp; live services!</h2>
 
   <em>By Ryan J. Price @ 2021-10-04 21:15:00 -0500 CDT; reading time 2m</em>
-
-  <br/><br/><br/>
+  <hr/>
 
   <p><a href="https://github.com/opensourcecorp/opensourcecorp">OpenSourceCorp</a> (OSC)
 development has slowed down in recent months as
diff --git a/blog/posts/2021-12-11-local-bootstrapper-available/index.html b/blog/posts/2021-12-11-local-bootstrapper-available/index.html
@@ -23,8 +23,7 @@ <h3>Navigation</h3>
   <h2>OSC local bootstrapper available</h2>
 
   <em>By Ryan J. Price @ 2021-12-11 16:00:00 -0500 -0500; reading time 2m</em>
-
-  <br/><br/><br/>
+  <hr/>
 
   <p><a href="https://github.com/opensourcecorp/opensourcecorp">OpenSourceCorp</a> (OSC)
 development continues to chug along. Below are some of the updates we&rsquo;ve been
diff --git a/blog/posts/2022-01-02-progress-report/index.html b/blog/posts/2022-01-02-progress-report/index.html
@@ -23,8 +23,7 @@ <h3>Navigation</h3>
   <h2>Progress Report 2022-01-02</h2>
 
   <em>By Ryan J. Price @ 2022-01-02 19:30:00 -0500 -0500; reading time 4m</em>
-
-  <br/><br/><br/>
+  <hr/>
 
   <p>Happy New Year!</p>
 <p>Over the holiday season, I&rsquo;ve had about a week of dedicated time to do a lot of
diff --git a/blog/posts/2022-02-27-progress-report/index.html b/blog/posts/2022-02-27-progress-report/index.html
@@ -23,8 +23,7 @@ <h3>Navigation</h3>
   <h2>Progress Report 2022-02-27</h2>
 
   <em>By Ryan J. Price @ 2022-02-27 12:00:00 -0600 CST; reading time 3m</em>
-
-  <br/><br/><br/>
+  <hr/>
 
   <p>More radio silence as ever more work has been done across OSC. As you&rsquo;ll see
 from the list below though, this is a really big set of changes/new stuff!</p>
diff --git a/blog/posts/2022-05-15-progress-report/index.html b/blog/posts/2022-05-15-progress-report/index.html
@@ -23,8 +23,7 @@ <h3>Navigation</h3>
   <h2>Progress Report 2022-05-15</h2>
 
   <em>By Ryan J. Price @ 2022-05-15 12:00:00 -0600 -0600; reading time 4m</em>
-
-  <br/><br/><br/>
+  <hr/>
 
   <p>(<em>Checks date of last post, sweats profusely</em>)</p>
 <p>Not a ton of changes in this update, I&rsquo;ve had a lot of things come up these past
diff --git a/blog/posts/2022-06-05-progress-report/index.html b/blog/posts/2022-06-05-progress-report/index.html
@@ -23,8 +23,7 @@ <h3>Navigation</h3>
   <h2>Progress Report 2022-06-05</h2>
 
   <em>By Ryan J. Price @ 2022-06-05 21:20:16 -0500 CDT; reading time 3m</em>
-
-  <br/><br/><br/>
+  <hr/>
 
   <p>Not too long between updates this time, but I&rsquo;ve got a few neat pieces of news
 to share.</p>
diff --git a/blog/posts/2022-06-20-progress-report/index.html b/blog/posts/2022-06-20-progress-report/index.html
@@ -0,0 +1,213 @@
+<!DOCTYPE html>
+<html><head>
+  <meta lang="en" charset="UTF-8"/>
+  <meta name="viewport" content="width=device-width, initial-scale=1"/>
+  <link rel="icon" type="image/x-icon" href="/images/osc-logo.png">
+  <link rel="stylesheet" href="/css/main.css"/>
+  <title>OpenSourceCorp</title>
+</head>
+<body><div class="sidenav">
+  <a href="/">
+    <img src="/images/osc-logo.png" style="width:120px;height:100px;" alt="OSC Logo"/>
+  </a>
+  <h3>Navigation</h3>
+  <ul>
+    <li><a href="/index.html">Home</a></li>
+    <li><a href="/wiki/index.html">Wiki</a></li>
+    <li><a href="/blog/index.html">Blog</a></li>
+    <li><a href="https://github.com/opensourcecorp">Project SCM</a></li>
+  </ul>
+</div>
+<div id="content" class="content">
+      
+  <h2>Progress Report 2022-06-20</h2>
+
+  <em>By Ryan J. Price @ 2022-06-20 15:00:00 -0500 CDT; reading time 8m</em>
+  <hr/>
+
+  <p>Some heavy updates in this one, so buckle up for what&rsquo;s changed!</p>
+<h3 id="secrets-management">Secrets management</h3>
+<p>Up until now, OSC has been strictly leveraging an implementation of Salt&rsquo;s
+Pillar functionality for secrets management &ndash; and that implementation is the
+default, which is &ldquo;store all secrets on the <code>configmgmt</code> disk&rdquo;. While this has
+worked fine (and can be a valid strategy in some enterprise environments), it&rsquo;s
+not what you would call &ldquo;best-practice&rdquo;. If the <code>configmgmt</code> subsystem itself is
+ever compromised, <em>every single piece of sensitive data</em> is accessible to the
+attacker at once &ndash; default Pillar data is just stored in plaintext on disk.</p>
+<p>I&rsquo;ve had a dedicated secrets-management solution on the roadmap since the
+beginning at OSC, but have spent all time since getting the other subsystems up
+&amp; running first. And now, that solution is successfully set up as another OSC
+subsystem. Named simply <code>secretsmgmt</code>, OSC&rsquo;s implementation of <a href="https://vaultproject.io">HashiCorp
+Vault</a> is now successfully building as part of the
+bootstrapper!</p>
+<p>There&rsquo;s still a lot of work to do here though:</p>
+<ul>
+<li>
+<p>There&rsquo;s no actual secrets in Vault right now. The deployment &amp; unsealing of
+the subsystem is live, but it has no secrets, nor are any other subsystems
+currently using it. The good news here though is that Salt Pillar supports
+adding Vault as an &ldquo;external Pillar&rdquo;, which means the other subsystems may not
+even need to know anything about Vault and just continue using Pillar data as
+usual. I&rsquo;ll explore more what those security implications are though.</p>
+</li>
+<li>
+<p>In order to get a Vault cluster actually ready for use, it must first be
+&ldquo;unsealed&rdquo;. Until a cluster is unsealed, Vault cannot encrypt/decrypt any
+secrets, and indeed is unusable as a consequence. While the deployment does
+currently handle the unseal operation, the unseal key shards used are
+discarded post-setup. This is OK for now, but if the cluster ever goes down or
+needs to be recreated, the original unseal key shards are the only way to
+reactivate the cluster to a working state. This is flagged as a <code>TODO:</code>, so
+we&rsquo;ll revisit in time.</p>
+</li>
+<li>
+<p>The cluster is not currently highly-available (HA) &ndash; it&rsquo;s just a single node.
+Vault has native facilities for multi-node use, I just haven&rsquo;t set them up
+yet.</p>
+</li>
+</ul>
+<p>Overall, this is still a great start, and having a dedicated secrets manager is
+a good thing to have, and something you will see in mature enterprise
+environments.</p>
+<h3 id="cicd-replatform">CI/CD replatform</h3>
+<p>Being the very first subsystem worked on after the <code>imgbuilder</code> &amp; <code>configmgmt</code>
+bases, the <code>cicd</code> subsystem is the oldest in OSC&rsquo;s infrastructure. Originally, I
+wanted to find a tool that:</p>
+<p>(a) Not &ldquo;cloud-native&rdquo; (specifically: not requiring a Kubernetes cluster)</p>
+<p>(b) did not have any kind of commercial tier available to it (so, no
+&ldquo;open-core&rdquo;)</p>
+<p>(c) have centralizable pipeline definitions so every repo need not bring its own
+def</p>
+<p>(d) have pipeline jobs defined in something readable/maintainable and not
+pseudo-script YAML.</p>
+<p>There was also an implicit (e) requirement of not requiring the JVM, because I
+have a visceral reaction to needing to extend the subsystem using Java (I don&rsquo;t
+like it lol). So that removed from consideration a number of tools, such as
+<a href="https://spinnaker.io">Spinnaker</a>, <a href="https://jenkins.io">Jenkins</a>, and others.</p>
+<p>So, we went with <a href="https://concourse-ci.org/">Concourse CI</a>. It&rsquo;s a really slick
+CI/CD system that hits every single one of the above requirements. As time went
+on however &ndash; whether due to warts of my own doing in defining the config or
+Concourse&rsquo;s own quirks &ndash; I kept running into more &amp; more issues with actually
+running the services. Sometimes it was shoddy container DNS that wasn&rsquo;t agreeing
+with <code>netsvc</code> (or with anything), sometimes the service would say it was running
+but it really wasn&rsquo;t, sometimes the worker nodes would just&hellip; hang for no
+reason.</p>
+<p>Overall, Concourse is super cool and I have nothing bad to say about it; I just
+ran into too many issues with the rest of our stack, and in a fit of frustration
+I decided to completely rip it out and choose a new platform. And that platform
+is&hellip;</p>
+<p><em><strong>Jenkins!</strong></em> It&rsquo;s not a change I ever thought I would make, but hear me out.
+Jenkins is a <em>very</em> mature piece of software, and its failure modes are very
+well-understood in general. In addition, Jenkins is probably the most common
+CI/CD system you will run into in enterprise environments (based on my own
+experiences in consulting). Those two points alone serve the broader mission of
+OSC as a teaching instrument.</p>
+<p>But, Jenkins also has some very nice things about it &ndash; it has a very
+straightforward (albeit &ldquo;ugly&rdquo;, which I like) web interface, it has a <em>wealth</em>
+of plugins available to extend its functionality to varying degrees, jobs can be
+run in any kind of environment you configure (Concourse requires all jobs to run
+in containers), and more that I don&rsquo;t have time to list. Additionally, Jenkins
+has a &ldquo;configuration as code&rdquo; plugin that can not only itself be installed
+automatically, but can then be used to define an entire Jenkins server exactly
+the way you want it to be &ndash; as, well, code. It&rsquo;s something I&rsquo;ve known about for
+many years but never actually had the chance to try out myself, and I&rsquo;m
+pleasantly surprised that it works as well as it does.</p>
+<p>The only real thing keeping me away from Jenkins initially was that it&rsquo;s written
+in Java. But I&rsquo;ve used Jenkins <em><strong>a lot</strong></em> in my career, and despite it&rsquo;s not
+just Stockholm syndrome &ndash; it&rsquo;s a tool that does its job well, and has been
+doing that job for a very long time.</p>
+<p>So, that changeover is now complete, and Jenkins is the new underpinning of the
+<code>cicd</code> subsystem. I&rsquo;m unironically excited for the change, because it&rsquo;s already
+working even better than I&rsquo;d hoped. Check it out!</p>
+<h3 id="rhad-changes"><code>rhad</code> changes</h3>
+<p>As mentioned before, <code>rhad</code> is our CI/CD lifecycle management tool. It started
+out as a linter aggregator, and then we decided to roll in full lifecycle
+functionality once we revisited it (and rewrote it in Go, from Bash).</p>
+<p>However, despite <code>rhad</code> having a generally straightforward way to add new
+linters to the codebase, that list would inevitably grow to be <em><strong>absolutely
+massive</strong></em> to support every possible linting case. We only had 6 or so linters,
+and already the copy-paste was showing. Plus, we&rsquo;d need to understand every
+single linter in use, how to manage around them, how to provide them config
+overrides, etc. This was already tiresome once I introduced <code>staticcheck</code> for Go
+linting, and when I tried to add <code>tflint</code> for Terraform (you wanna talk about a
+frustrating tool to configure, WOOO BOY).</p>
+<p>So, I took a step back and revisited <a href="https://github.com/github/super-linter">GitHub&rsquo;s
+Super-Linter</a> for inspiration &ndash; and
+walked away deciding to just use Super-Linter for most of our linting purposes
+at OSC. The developers of that tool have already done all the hard work solving
+for cases mentioned above &amp; more, plus it has <em><strong>so many included linters</strong></em> out
+of the box.</p>
+<p>However, Super-Linter currently has some outstanding bugs that prevent a
+successful linting experience &ndash; specifically, a bug where <a href="https://github.com/github/super-linter/issues/1599"><code>golangci-lint</code>
+isn&rsquo;t configured correctly to scan packages vs. individual
+files</a>, and so fails
+incorrectly when linting a Go repo.</p>
+<p>So, what I&rsquo;ve decided to do is first run Super-Linter with buggy languages
+disabled, and then run <code>rhad</code>&rsquo;s linters for those cases that we can better
+control. At this moment, this feels like a good compromise, doesn&rsquo;t require me
+to trash all the linter code I&rsquo;ve already written, and still allows for <code>rhad</code>
+to be used as a lifecycle manager post-lint.</p>
+<h3 id="website-updates">Website updates</h3>
+<ul>
+<li>
+<p>Blog has RSS feed link (but the feed itself had existed already)</p>
+</li>
+<li>
+<p>Installing Docker (or rather, packages from the <code>docker-rootless-extras</code> repo)
+on the subsystems has been throwing hash-sum mismatch errors a lot lately &ndash;
+sporadically it works, but I want to explore workarounds/solutions.</p>
+</li>
+</ul>
+<h3 id="miscellaneous">Miscellaneous</h3>
+<ul>
+<li>
+<p>For some time, at least one of the subsystem <code>Vagrantfiles</code> has specified a
+larger-than-default disk size at runtime. This works as expected (in that a
+disk that was 10GB would then be 20GB), but I was reminded the hard way that
+<em><strong>simply increasing disk size does not increase available space</strong></em>. You need
+to also expand a partition to fill the new unused space, and then resize the
+filesystem to take advantage of it. I noticed this when trying to pull very
+large container images for <code>cicd</code> jobs, and the disk filled up when it
+&ldquo;shouldn&rsquo;t have&rdquo;. So, in <code>imgbuilder/scripts/run/main.sh</code>, there is now a
+block that handles the two-step process of expanding the root partition and
+resizing the filesystem. It&rsquo;s working as expected, which is a pleasant
+surprise.</p>
+</li>
+<li>
+<p><code>osc-infra</code> will start having tagged releases from now on. This should help
+others (and myself, frankly) keep cleaner track of when each commit was marked
+as a) buildable and b) of some notable milestone.</p>
+</li>
+<li>
+<p>I&rsquo;ve started to notice some really frustrating issues in the local VM
+bootstrapper where builds fail for spraodic (but usually consistent) reasons.
+Most commonly, it&rsquo;s when trying to do and <code>apt-get update</code> anywhere at all, at
+any stage of the lifecycle install. I first noticed it when trying to install
+Docker on a subsystem node (via
+<code>osc-infra/configmgmt/salt/salt/_common/docker.sls</code>) and APT errors out when
+it finds a hash sum mismatch for the Debian package repo. It&rsquo;s only sometimes,
+but it&rsquo;s been so far very difficult to isolate (let alone reproduce) because
+the acual repo that throws the mismatch error changes &ndash; sometimes it&rsquo;s the
+<code>containerd.io</code> repo, sometimes it&rsquo;s the <code>docker-ce-rootless-extras</code> one, etc.
+This has been making progress &amp; iteration on <code>ociregistry</code> and <code>cicd</code> very
+slow, since both subsystems use Docker. But additionally, this has started to
+happen during the base-image builds too (without even running Salt calls). I&rsquo;m
+beginning to suspect it&rsquo;s either a Debian repo, OR VirtualBox bug. Both of
+which are not easy fixes on my end.</p>
+<ul>
+<li>UPDATE: restarting my host machine seems to have gotten things to run
+smoothly again at least once &ndash; so I&rsquo;m now wondering if it is indeed a
+VirtualBox bug.</li>
+</ul>
+<p>Another strange error is a sporadic failure during the internal TLS cert
+generation (<code>osc-infra/configmgmt/salt/salt/_common/internal_tls_certs.sls</code>).
+Sometimes (with no discernable pattern), the whole script fails at the very
+first command (<code>openssl genrsa ...</code>). I can&rsquo;t yet tell if it&rsquo;s a race
+condition thing, or something more opaque, but that&rsquo;s another thing to explore
+more.</p>
+</li>
+</ul>
+
+
+    </div></body>
+</html>
diff --git a/index.xml b/index.xml
@@ -6,7 +6,17 @@
     <description>Recent content in OpenSourceCorp - The Free and Open-Source Enterprise on OpenSourceCorp</description>
     <generator>Hugo -- gohugo.io</generator>
     <language>en-us</language>
-    <lastBuildDate>Mon, 06 Jun 2022 10:39:00 -0500</lastBuildDate><atom:link href="https://www.opensourcecorp.org/index.xml" rel="self" type="application/rss+xml" />
+    <lastBuildDate>Mon, 20 Jun 2022 15:00:00 -0500</lastBuildDate><atom:link href="https://www.opensourcecorp.org/index.xml" rel="self" type="application/rss+xml" />
+    <item>
+      <title>Progress Report 2022-06-20</title>
+      <link>https://www.opensourcecorp.org/blog/posts/2022-06-20-progress-report/</link>
+      <pubDate>Mon, 20 Jun 2022 15:00:00 -0500</pubDate>
+      
+      <guid>https://www.opensourcecorp.org/blog/posts/2022-06-20-progress-report/</guid>
+      <description>Some heavy updates in this one, so buckle up for what&amp;rsquo;s changed!
+Secrets management Up until now, OSC has been strictly leveraging an implementation of Salt&amp;rsquo;s Pillar functionality for secrets management &amp;ndash; and that implementation is the default, which is &amp;ldquo;store all secrets on the configmgmt disk&amp;rdquo;. While this has worked fine (and can be a valid strategy in some enterprise environments), it&amp;rsquo;s not what you would call &amp;ldquo;best-practice&amp;rdquo;. If the configmgmt subsystem itself is ever compromised, every single piece of sensitive data is accessible to the attacker at once &amp;ndash; default Pillar data is just stored in plaintext on disk.</description>
+    </item>
+    
     <item>
       <title>About OSC</title>
       <link>https://www.opensourcecorp.org/wiki/about/</link>
diff --git a/sitemap.xml b/sitemap.xml
