Troubleshooting

In “Corporate World” there is a lot of focus on documenting things, which I didn’t used to do much in college. And doing it made me realize it’s importance. While working on any project, now I’m in a habit of writing things down, what worked, what didn’t, everything, which most of the time involve troubleshooting.

And looking back last 2-3 year, I’ve spent countless hours on installing and fixing things, sometimes even on repeated stuff and other than StackOverflow, blogs of other people helped me more than anything. Because of these reasons I’ve started to write about breaking and fixing things publicly. Here is the list of the posts on same:

Any feedback is most welcome.

Breaking And Fixing Of Pacman With Openssl

I needed to install Mongodb for a project I was working on, as usual I did:

$ sudo pacman -S mongodb

Now starting the mongodb.service with systemctl was failing without any specific error logs:

$ journalctl -u mongodb.service -b
-- Logs begin at Thu 2017-06-01 13:13:02 IST, end at Sun 2018-01-07 12:14:08 IST. --
Jan 07 12:05:56 TheBlackPearl systemd[1]: Started High-performance, schema-free document-oriented da
Jan 07 12:05:56 TheBlackPearl systemd[1]: mongodb.service: Main process exited, code=exited, status=
Jan 07 12:05:56 TheBlackPearl systemd[1]: mongodb.service: Unit entered failed state.
Jan 07 12:05:56 TheBlackPearl systemd[1]: mongodb.service: Failed with result 'exit-code'.
Jan 07 12:09:46 TheBlackPearl systemd[1]: Started High-performance, schema-free document-oriented da
Jan 07 12:09:46 TheBlackPearl systemd[1]: mongodb.service: Main process exited, code=exited, status=
Jan 07 12:09:46 TheBlackPearl systemd[1]: mongodb.service: Unit entered failed state.
Jan 07 12:09:46 TheBlackPearl systemd[1]: mongodb.service: Failed with result 'exit-code'.
lines 1-9/9 (END)

And running mongo and mongodb was throwing following error:

mongo: error while loading shared libraries: libboost_program_options.so.1.65.1: cannot open shared object file: No such file or directory

I seached this a bit and then installed boost package.

$ sudo pacman -S boost

And now running mongo was throwing different error:

mongo: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory

I tries installing mongo-tools package which also installed openssl-1.0 along with it. And running mongo was still throwing same error. So I installed openssl which got me openssl-1.1. That’s where almost everything broke. Running sudo:

$ sudo pacman 
sudo: error in /etc/sudo.conf, line 0 while loading plugin "sudoers_policy"
sudo: unable to load /usr/lib/sudo/sudoers.so: libssl.so.1.0.0: cannot open shared object file: No such file or directory
sudo: fatal error, unable to load plugins

Since sudo was not an option now, I switched to root by su only to realize that even pacman is broken:

$ pacman -Ss openssl
pacman: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory

Now, this was scary. How do fix broken packages without my package manager?

Searching for this I found grism’s answer to similar problem. Turns out current version of sudo and pacman depends on openssl-1.0 and I upgraded it to 1.1. To downgrade it, I didn’t need to download the 1.0 version package from web as installing mongodb-tools also installed openssl-1.0, pkg.tar.xz of which can be found at /var/cache/pacman/pkg. As per the above link, I extracted it in /tmp and made a symlink for libssl-1.0.0 and libcrypto-1.0.0 at /var/lib/:

# cd /tmp && tar xf /var/cache/pacman/pkg/openssl-1.0-1.0.2.n-1-x86_64.pkg.tar.xz
# ln -s /tmp/usr/lib/libcrypto.so.1.0.0 /usr/lib/libcrypto.so.1.0.0
# ln -s /tmp/usr/lib/libssl.so.1.0.0 /usr/lib/libssl.so.1.0.0

Well this fixed the pacman. So first thing first, I need my sudo back, so I installed openssl-1.0 from downloaded .pkg.tar.xz:

# pacman -U /var/cache/pacman/pkg/openssl-1.0-1.0.2.n-1-x86_64.pkg.tar.xz

Exit root and try:

$ sudo ls
[sudo] password for nks:

That’s better. Back to MongoDB problem.