ScottW/rites

I am unsure how to title this post, but it feels like a good idea to share what happened so that you have some steps to follow if you are in a bit of a panic.

First, here is what happened. Earlier today, I deployed a change that would add a column to a table. There was no backfill on the column. There are no required indexes. Nothing. It’s a blank column on a relatively small (20k rows) table. I even held the code that would use this table for a second deployment.

Our Heroku deploys have been taking quite a while recently (15 minutes or so). I haven’t had time to dig into why yet, so I wasn’t overly concerned about the duration initially. Then I got the first ping that something was offline (Sidekiq error queue). That one felt unrelated. Our monitoring is set to notify us anytime the error queue has more than 100 items. Shortly after, I received notifications that other services were unavailable (timing out). A quick check on the recent deployment shows that the release phase has been running for 20 minutes.

At this point, I decide it is time to kill the deployment. I commnd+D the terminal and checked the current processes on Heroku (heroku ps). I can see the release command is still running. The next thing to check is the database. I can console into the database and other apps that use this same database are all functioning as expected. In addition, as far as I can tell, all the background jobs for this app are still running as expected (we use Sidekiq+Redis, but ultimately all work is done against the PG database).

To be safe, I ran pg:diagnose and could see long-running queries against the table I was attempting to migrate.

Next, I focused on killing the release phase process. In nearly 12 years of using Heroku, I have never had to kill a running process. I find references to ps:stop and ps:kill. Both report they worked, but running us ps, I can see the process is still running. It turns out that you need to include the process type as well: heroku ps:kill release.2343. Better output here would have been helpful.

While this killed the process, the app’s state did not improve. I restarted the processes, which again did not fix the problem. Finally, I figured something was off with the app state, so I rolled back to a previous update (note: the new deploy was never fully deployed and unavailable). This appeared to fix things for a few seconds, but everything on the main app again began to time out.

I checked heroku pg:diagnose again and could see the same long-running queries were still there. There were about 40 or so of them, but I couldn’t get the output in a state where I could quickly grab the PIDs to kill each process, so I went ahead and ran heroku pg: killall. After this, I restarted the main app (and related apps), and everything appears to be working well.

So the takeaways:

1. Never deploy before coffee. The mind is not ready for this kind of stress.
2. My best guess is that the connection pool for the main web app somehow got into a bad state. Killing all the connections, I was able to reset it.

I still have to deploy again, but I assume this was a freak condition.

With my SQLite backup script, I mentioned you need to add execute permission after you deploy.

However, I cloned a Rails app off of GitHub today and noticed that the bin/setup worked as expected and had proper execute permissions. 👀

I eventually found my way to the git’s update-index command:

Modifies the index. Each file mentioned is updated into the index and any unmerged or needs updating state is cleared.

That description is clear as mud. 😛

But digging further is this option: --chmod=(+|-)x

Set the execute permissions on the updated files.

So here is how to use it.

1. Add a new script file or modify an executing one (even with just a comment). This is important because update-index will not take effect unless you commit to some change.
2. Add the change to git: git add bin/backup
3. Execute update-index: `git update-index –chmod=+x bin/backup
4. Commit the change: git commit -m "Now with execute permission"

A little Ruby distinction I had not seen (or remembered seeing) before.

In Ruby, both String#sub and String#gsub are methods used for string substitution, but they have a subtle difference:

String#sub: This method performs a substitution based on a regular expression pattern, replacing only the first occurrence that matches the pattern.

1
2
3

str = "hello world"
new_str = str.sub(/o/, "a")
puts new_str

Output: hella world

String#gsub: This method also performs a substitution based on a regular expression pattern, but it replaces all occurrences that match the pattern within the string.

1
2
3

str = "hello world"
new_str = str.gsub(/o/, "a")
puts new_str

Output: hella warld

Hat tip to ChatGPT, who answered this question for me.

Ruby’s debugging story has improved dramatically in 3.x (and Rails).

I figured the best way to understand what’s happening in Campfire would be to attach the debugger and step through some of the more interesting parts.

Unfortunately, I was greeted with a recurring error that often looked something like this:

1
2
3
4
5
6
7

<Thread:0x00000001276a7750@DEBUGGER__::Server::reader /Users/scott/.asdf/installs/ruby/3.3.0/lib/ruby/gems/3.3.0/gems/debug-1.9.1/lib/debug/server.rb:44 aborting> terminated with exception (report_on_exception is true):
/Users/scott/.asdf/installs/ruby/3.3.0/lib/ruby/3.3.0/socket.rb:1128:in `unlink': No such file or directory @ apply2files - /var/folders/6q/xz6r4tqd4sl9qpbqkjlqj3dr0000gn/T/rdbg-501/rdbg-29191 (Errno::ENOENT)
	from /Users/scott/.asdf/installs/ruby/3.3.0/lib/ruby/3.3.0/socket.rb:1128:in `ensure in unix_server_socket'
/Users/scott/.asdf/installs/ruby/3.3.0/lib/ruby/3.3.0/socket.rb:1128:in `unlink'	from /Users/scott/.asdf/installs/ruby/3.3.0/lib/ruby/3.3.0/socket.rb:1128:in `unix_server_socket'
	from /Users/scott/.asdf/installs/ruby/3.3.0/lib/ruby/3.3.0/socket.rb:1169:in `unix_server_loop'
	from /Users/scott/.asdf/installs/ruby/3.3.0/lib/ruby/gems/3.3.0/gems/debug-1.9.1/lib/debug/server.rb:502:in `accept'
	from /Users/scott/.asdf/installs/ruby/3.3.0/lib/ruby/gems/3.3.0/gems/debug-1.9.1/lib/debug/server.rb:49:in `block in activate'

At first, I was convinced that debugging was busted on my computer. But I spun up a new project, and everything worked as expected.

Then, I thought about the error above showing up multiple times and the problems with threads I had previously mentioned.

I restarted the process without cluster mode enabled for Puma (WEB_CONCURRENCY=0) and could connect the debugger as expected.

From here, I decided to compare the puma.rb file in Campfire to the one in the empty Rails 7.1 project I just spun up, and I found the problem.

In the Puma configuration file, cluster mode is enabled if workers is greater than 0.

In a fresh Rails 7.1 puma.rb file, the worker configuration looks like this:

1
2
3
4
5

if ENV["RAILS_ENV"] == "production"
  require "concurrent-ruby"
  worker_count = Integer(ENV.fetch("WEB_CONCURRENCY") { Concurrent.physical_processor_count })
  workers worker_count if worker_count > 1
end

However, in Campfire, the RAILS_ENV check is removed

1
2

worker_count = (Concurrent.processor_count * 0.666).ceil
workers ENV.fetch("WEB_CONCURRENCY") { worker_count }

My guess is that with Campfire being a chat app with lots of connectivity, they opted for Puma’s cluster mode by default.

The good news is you can disable cluster mode in Campfire without changing any source. Just set the WEB_CONCURRENCY ENV to 0.

Something like this should do the trick:

WEB_CONCURRENCY=0 rdbg -n --open=vscode -c -- bin/rails server -p 3000

David Kimura at Drifting Ruby has some good videos on setting Campfire up outside the Once installer. Outside of the 3.3 RC1 and stringio issues, I was running into another issue: I could not generate thumbnails when running on localhost. The thumbnails generated as expected when using Puma Dev. Still, on localhost, they were failing, and worse, I would typically end up with one broken thumbnail variant per thread pool worker.

First, here are my setup steps:

1. Download the source
2. Run bin/setup (I had to remove rbenv from this file since I use ASDF these days)
3. From the rails console, run, WebPush.generate_key and copy the keys into ENV variables VAPID_PUBLIC_KEY and VAPID_PRIVATE_KEY
4. Add the msgpack gem to my GemFile

With all of this in place, https://campfire.test worked as expected.

However, when starting the server via bundle exec rails server (on an M1 MBP with Sonoma 14.3), the thumbnails of images were missing. I could click on the thumbnails and see them in the lightbox but not in the chat window.

Digging into the database, I could see they were not being analyzed, but I had no idea why.

Then I saw the following in the logs:

1
2

objc[84578]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[84578]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

The thread running ActiveJob running the ActiveStorage Analyze job was crashing.

A little searching led me to this bug thread and a suggestion to set OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES. A quick server reboot, and everything works as expected.

Before I got here, I also tried disabling cluster mode in Puma, WEB_CONCURRENCY=0 bundle exec rails s, which solved the problem.

I submitted this as a bug to 37Signals. I am not sure if this is just something on my computer or not, so I would try it without these changes first.

Update: 37Signals confirmed this is a known issue. The env variable is actually set in the .pumadev file which is why I did. not see the issue when using Puma Dev.

20+ years of professionally building software, and here is how my last hour or so went.

heroku login -> zsh: killed heroku login

hmmm…why did ZSH kill Heroku? It turns out it is just reporting what happened.

Open the crash reports and get a bunch of text I barely understand.

I remember that I have a second brain that knows what this crap means. ChatGPT, can you help me?

It does a decent job of educating me on what the reports (segfault on the main thread implies not even the hand of God can save this process).

When in doubt, just re-install.

No dice.

Google shows me others with similar problems, but none related to Heroku, and nothing that looks promising.

I wonder what the (brew) Doctor thinks about this (Heroku was installed this way).

The doctor takes her good old time getting back to me on this older Mac. But when she does respond, she says, “Dawg, your shit is F’ed Up!”

I am paraphrasing, but there are unlinked things, kegs with no formula,, dry taps, an unbrewed dylibs, and likely most importantly, this:

This is weird because I had previously checked for an update after an OS update, and I had tried to re-install the tools just to be safe.

In both cases, it said, “there are no updates, go away”.

Anywho, I run the following:

sudo rm -rf /Library/Developer/CommandLineTools

sudo xcode-select --install

Eventually, notice the popup window (because why install them from the terminal), click OK to continue, and 10 minutes or so later, we are back in business.

The key takeaways:

1. This shit isn’t hard, it just takes patience.
2. Even those who have done it for a long time still get stuck.
3. There is nothing wrong with yelling profanities at a computer from the comfort of your basement.

I am working on a significant update/rewrite of part of KickoffLabs. I had a task to work through what we do in a Sidekiq worker. It was late on Friday when I got to this part of the code, and I decided to punt on it until Monday.

Monday morning, I grab a cup of coffee and sit down to finish this section off and see a comment I missed (and had missed for a long time).

Outside the screenshot, there is an if statement, ensuring this code was not executed in the last four years. Still, it was frustrating that we had missed removing it for so long.

I figured ChatGPT would have a good solution to stay on top of this, but it mostly just tried to explain to me how to use comments. :)

Next up, I asked on Twitter (and Ruby.Social):

Any suggestions (other than search) to ensure code like this gets cleaned up?

From there, I got a lot of good suggestions.

Two Ruby gems looked interesting:

• todo_or_die
• smart_todo

If I had to choose, I would go with todo_or_die since it would cause a failure/notification locally. I would rather not wait until there was a pull request/etc.

The rest of the suggestions included if statements and date checks. My favorite, and the one I will likely adopt going forward, is to wrap it in a test that fails after a specific date.

I am tired of guessing the future of AI. What I am sure of, something routine like this used to take significantly longer than it does today.

In the past:

1. Google a bit; maybe find someone asking a similar question somewhere (hopefully, do not get distracted by something else).
2. Goto the docs (which often took Google to find)
3. Try and find something related to what you were doing.
4. (Sometimes) Apply what you learn in step #3 to your problem.

Now, we are back to a quick search, which is kind of like Google in the old days.

This piece by Justin Searls has been making the rounds for a week or so.

The Fall of the 10X Developer

My quick take: AI is leveling the developer playing field a bit. You still need to ask it proper questions to get good results. We have known for quite a while that colleges do not prepare you to be a good developer in the real world. AI won’t solve the preparation problem, but it will likely make new hires much more effective sooner.

Still, like any other career, if you just mail it in, you will not truly grow.

We must remember that today’s AI is much more pattern recognition than thinking. Maybe that will change, but for now, I still believe it is more assistive for most use cases than a replacement.

Do you see RuboCop offenses after installing and configuring the Standard VS Code Extension?

To fix this issue, you need to disable RubyLSP’s diagnostics:

1
2
3

rubyLsp.enabledFeatures": {
    "diagnostics": false
}

Hat Tip

This is likely the best post gem install message I have seen (well, next to HTTParty). Well done Stanard team.

GPT-Migrate

If you’ve ever faced the pain of migrating a codebase to a new framework or language, this project is for you.

We have a bunch of CoffeeScript that I would love to migrate. It is hard to justify the time because it works, but with the improvements made to modern JavaScript, I cannot imagine purposely writing more CoffeeScript at this point.

If you want to develop/customize your Micro.blog theme locally, you will likely encounter an issue with a missing file called custom_footer.html. My guess is this file is dynamically added to my Micro.blog at build time.

To work around this, you can remove the use of {{ partial "custom_footer.html" . }} from your theme (either permanently or temporarily), or you can simply check that it exists first:

1
2
3

{{ if templates.Exists ( "partials/custom_footer.html" ) }}
  {{ partial "custom_footer.html" . }}
{{ end }}

I have been working on converting something from Next.js to Ruby. The Next.js version is still more visually appealing to me, but the benefits of working in Ruby far supior to me long term.

Before:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

    <Card as="article">
      <Card.Title href={`/articles/${article.slug}`}>
        {article.title}
      </Card.Title>
      <Card.Eyebrow as="time" dateTime={article.date} decorate>
        {formatDate(article.date)}
      </Card.Eyebrow>
      <Card.Description>{article.description}</Card.Description>
      <Card.Cta>Read article</Card.Cta>
    </Card>

After:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

    {%@ Shared::Card::Card as: :article do %}
      {%@ Shared::Card::Title as: :h2, href: article.relative_url do %}
        {{article.data.title}}
      {% end %}
      {%@ Shared::Card::Eyebrow as: :time, dateTime: article.data.date, decorate: true do %}
        {{article.data.date}}
      {% end %}
      {%@ Shared::Card::Description do %}
        {{article.data.description}}
      {% end %}
      {%@ Shared::Card::Cta do %}Read article{% end %}
    {% end %}

I have been experimenting with Serbea templates with Bridgetown and was not able to get TailwindCSS IntelliSense to work.

Serbea files using the file extension .serb, so my first attempt to configure it looked like this:

1
2
3

"tailwindCSS.includeLanguages": {
  "serb": "html"
},

However, what I needed to do was specify the file content type:

1
2
3

"tailwindCSS.includeLanguages": {
  "serbea": "html"
},

Lots of great little PostgreSQL date tricks by @robconery

Using PostgreSQL to Handle Calendar Data Like a Freak - YouTube

I cloned a repo with a couple of files that were duplicated because of case sensitivity. Git was nice enough to warn me about them, but I couldn’t remember which files where an issue.

Two of them became a problem and were fixed, but I was sure there were more than two when I cloned the repro.

I found this excellent script to highlight and fix the problem files: git-detect-case-change

I didn’t realize Bullet Train was now open source (it looks like you have to pay to use Stripe, but that feels very fair).

I have not dug too deeply into yet, so no opinion other than it is probably worth looking through it before you start your next project.

Migrating from a Postgres Cluster to Distributed SQLite with LiteFS

I love me some Postgres, but the SQLite with LiteFS (via @flydotio) just sounds fun.

Steps to use solargraph for Rails projects in VS Code (WIP)

This is quite handy. I am unsure how it would handle multiple versions of Ruby, so I skipped updating the shim location and instead just installed the gems in my main 2 Ruby versions.

A nice quality-of-life improvement in Ruby 3.2 Enumerator::product

How to validate the presence of a boolean field in a Rails model

There is a subtle bug that happens when validating the presence of boolean fields in Rails. Ensuring adequate test coverage helps us find those issues before they hit production.

Crontab.guru

The quick and simple editor for cron schedule expressions

I do not write crontabs often, but when I do, it is usually with Crontab.guru.

Active Record enum form select

I find myself using this on almost all of my rails projects. I am surprised something like this isn’t available out of the box yet.

Finally made the autocomplete for IRB in Ruby 3.0 readable with my Dracula color scheme. All it took was changing the terminal’s definition of Cyan. Hopefully, this will be configurable in the future.