Turns out I wrote this back in November, but on a different machine and I never pushed my changes up to git. So here it is now!

The last two times I’ve run apt update && apt upgrade -y on my web server and OpenResty has an update to push out, I will get some errors including Openresty is not setup correctly. There are a few other errors in the systemctl status openresty output such as process 223478 [nginx] is still running. I might be slightly paraphrasing the errors, but that’s roughly what I’m finding.

Like any debug session, I make sure nginx is disabling and not running (systemctl disable and systemctl stop), which I can confirm. Now, Openresty does use Nginx under the hood, so that errors makes me think it’s just conflicting services trying to run on top of each other.

The weirder part is when it warns me that OpenResty is not setup correctly. I didn’t change anything… so what is the update/upgrade trying to setup?

Next, I’ll look through my config files (i.e. nginx.conf) - no changes there either and nothing that stands out as out of the ordinary.

One of my bad habits is that when doing this sort of debugging and running something like journalctl --since 21:45:00, I’ll look at the logs more closely with each try and fail. I should just look more closely from the beginning! But I digress.

Since the nginx process and openresty setup errors are the most plentiful but yield nothing, I’ll look back through the logs for the single lines that I miss on my first few passes. That’s when I see it, buried between the other errors. A single line that says my pid file can’t be found.

My pid file for OpenResty and Nginx is stored in my /run directory, but for whatever reason, whenever OpenResty pushes out an update, it overwrites my systemd file and starts looking for the pid file under /usr/local/openresty/nginx/logs/nginx.pid. Why the file would be located under a logs directory is still beyond me, but a quick update to the systemd file, a systemctl daemon-reload and then systemctl start openresty and all my public facing services are back in action.

After I fixed it for the second time, I realized I hadn’t written this down in my documentation. So I’ve now recorded it and thought I’d share it here in case it helps anyone else. I also did a bit of research after and found that even in this Digital Ocean post, it has OpenResty/Nginx’s PID file in a different location too. I couldn’t find OpenResty’s documentation on this yet, but Nginx’s Official Docs have the PID file in the same location as I am storing it. Perhaps this is just a mis-config between nginx coming natively on my linux distro and then OpenResty being installed on top of it?

Either way - problem solved, and documentation captured. That’s a win for my day.