Monthly Archives: June 2012

FreeBSD Jails – SSH Problem

Posted by on June 17, 2012 2 comments

After some time being able to SSH into a certain jail I noticed it stopped working, then I noticed that when trying to SSH from the jail that also did not work. When trying to SSH to or from the jail this is usually what I saw:

[root@dev-j1 ~]# ssh mark@somehost.com
Host key verification failed.

It all comes down to this error here:

debug1: read_passphrase: can’t open /dev/tty: No such file or directory

If you can reproduce the same or similar from either looking in your SSHD logs or by running ssh -vvv from the jail, then chances are you are seeing the same problem as me. I haven’t found a permanent and still secure fix yet, but I have found somewhat of a fix(I have to do more research to see if the fix is secure or not, so I wouldn’t recommend implementing this on a production system without further research or knowledge on the impact of what you are doing.

To fix, stop the jail and open /etc/defaults/devfs.rules and at the end of the file you should see something like this:

# Devices usually found in a jail.
#
[devfsrules_jail=4]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login

When I look at devfsrules_unhide_login I see that there are some rule to unhide a bunch of tty’s so I add the following to  the end of the devfs_uhide_login area:

add path ‘tty’ unhide

I actually insert it between these two:

add path 'ttyO*' unhide
add path ptmx unhide

Now save that file and start your jail again – you should see that /dev on the jail shows tty, so you should be able to SSH from it. The condition here is that you will need to use keys, because SSH can’t actually read from tty, as seen now:

ebug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/id_rsa
debug3: no such identity: /root/.ssh/id_rsa
debug1: Trying private key: /root/.ssh/id_dsa
debug3: no such identity: /root/.ssh/id_dsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
debug1: read_passphrase: can't open /dev/tty: No such file or directory
debug3: packet_send2: adding 64 (len 49 padlen 15 extra_pad 64)
debug2: we sent a password packet, wait for reply
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such file or directory
debug3: packet_send2: adding 64 (len 49 padlen 15 extra_pad 64)
debug2: we sent a password packet, wait for reply
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such file or directory
debug3: packet_send2: adding 64 (len 49 padlen 15 extra_pad 64)
debug2: we sent a password packet, wait for reply
debug1: Authentications that can continue: publickey,password
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey,password).

Now the reason you are seeing the above error is probably because you are logged into the console of the jail through a jail console session and not through SSH, so obviously something is going on with how the console session plays with the tty’s. I use ezjail-admin console jailname and thats when I see the above problem.

If you only SSH into the jail and don’t use the console you will find the problem is gone – you can SSH in and out of the jail without having to use keys every time and it works fine. Let me know if you know of a better way to resolve this.

 

Mac OS X Memory Issues

Posted by on June 15, 2012 2 comments

I thought I would put up something for those searching for issues to do with Mac OS X memory management which is clearly broken right of the box with Lion. I should start by saying that my first use of Mac OS started with Lion and I have never used Snow Leopard, so I can safely say from my experience that memory management in Mac OS is seriously broken.

A bit of context, I have a 27″ iMac with the Core i5(2011), 12GB RAM(I have removed RAM to see if that helps!) and a 1TB Hard Drive, and I also have a MacBook Air which I purchased at the same time which has 4GB Ram and a 128GB SSD. Within a few weeks of purchasing both of these, I noticed that the iMac which is supposed to be the more powerful machine of the two is slower by a large amount than the MacBook Air. I don’t run very intense stuff, although I had hoped to get rid of one of my servers and just run most of my virtual machines off of my iMac. So a bit of a run down of what I run every day: About 30-40 Chrome Tabs, 10-5 Terminal Tabs, A fe instances of TextEdit and/or TextMate, and a single VMware Fusion VM running Windows 7(2GB RAM).

I have a fairly large background in FreeBSD and OpenBSD, and I know that Mac OS is loosely based off of BSD(that’s all I’ll say on the matter because how much it is based off BSD and what parts have been taken from where seem to be the subject of a lot of debate) – and one of the main reasons I use FreeBSD so much is because I find it performs so much better than linux, it is a lot less weighty, but is still very powerful. Because of this, I did assume that Mac OS would be similar, and in any case, should there be any problems, there should be a myriad of ways to diagnose and troubleshoot the issue. Well, I was wrong, very wrong.

I noticed the the reason that the iMac is considerably slower than the MBA is because of the SSD in the MBA. Essentially, memory management is so broken in OSX that it continually pages all memory out to disk even if there is free memory, and if there isn’t any free memory, it generally doesn’t bother to clear it out. You can view this through your activity monitor and through the command “top” in Terminal. Check out the ridiculous load levels in top whilst I write this blog post:

Load Avg: 11.45, 13.73, 13.69  CPU usage: 70.55% user, 29.44% sys, 0.0% idle   SharedLibs: 12M resident, 6168K data, 0B linkedit. MemRegions: 38353 total, 2187M resident, 104M private, 3865M shared.
PhysMem: 7708M wired, 441M active, 147M inactive, 8296M used, 3988M free. VM: 382G vsize, 1119M framework vsize, 5707517(40) pageins, 147944(0) pageouts.
Networks: packets: 12870819/12G in, 9624876/1816M out. Disks: 2804511/44G read, 5195808/88G written

According to top and the activity monitor, the CPU Usage is VMware Fusion(sys) and Finder(user) and that is all, everything else looks normal. Further investigation will reveal that it is because for some reason Finder is indexing, and Fusion has outlook open, which once again comes back to disk IO. It should be noted at this time, that the Hard Drive in the iMac has been replaced because even though I here tonnes of activity on it, and the system crawls, I can only see minimal reads and writes in Activity Monitor(replacing the hard drive did not help). What I found in this instance is that the more CPU intensive applications that I quit, the more CPU usage for finder went up as if they were preventing it from using more CPU. Unfortunately though this is the first time I have seen this issue with Finder so I can’t lay any blame there.

Anyway, essentially Mac OS does not know how to use memory, so it pages everything to disk which results in super slow systems(unless you have a super duper latest and greatest MBP). So I found a so called fix for this, I did the folllowing:

sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist

The above Terminal command disables the so called dynamic pager, which is supposed to result in much better performance. If you plan on doing this yourself, remember to reboot afterwards, and if you want to re-enable the pager, do the following command followed by a reboot:

sudo launchctl load -wF /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist

After disabling the dynamic pager and rebooting, I have seen almost zero performance increase, and what I have noticed is that even though the system no longer reads the pages back, it is still sending a lot of memory to the page file which can be noticed by the above top command output(notice though in the above output that some pages have been read back in, but that is because my system nearly crashed, and it must have turned it back on?). Essentially, for me, the above command does not work – not even if I just have Chrome open browsing websites(with nothing else open, not even Fusion).

What has helped me however, is when I notice my memory getting low, I run the command purge from Terminal – I have heard that purge clears only the inactive memory but from what I can see, it also clears the active memory that is no longer in use and hasn’t properly been listed as inactive memory(it seems as though perhaps applications aren’t letting Mac OS know that they stopped using the memory, or perhaps it is just too stupid to work things out like just about any other OS can).

I thought I would share something from what happened to me today though… This morning I noticed my machine was running like a dog again, so I pulled up the Activity Monitor and noticed that Free Memory was down to about 19MB. So I pulled up Terminal to run purge and free up some memory(I saw about 1.2GB inactive). After I ran purge, this is what I saw:

ERROR]	<CPPathWatcher.c:101>	Events were dropped; full rescan required.
[ERROR]	<CPDevice.c:7668>	Error #-6 (kCPNotLoaded): Unable to update device instances having loaded one or more new or changed records.
    o-+- [com.apple.CoreProfile.Devices] <CPDevice.c:4947>  _CPUpdateDeviceInstances: CPStatus -6 (kCPNotLoaded)
      |  Failed to [completely, at least] update device instances.

I saw about 2 pages of that text repeated after each other, but once the purge command had completed, I noticed I had about 4GB RAM free, and that it had cleared the inactive memory as well as some of the active memory(by the looks of it). Not only that, I also noticed in the Activity Monitor that I had started paging out, so it must be that when the system starts to run out of memory, that it re-enables the pager to some extent and start paging memory in and out again. So hopefully this provides some insight for those who were not sure what would happen to their system if it ran out of memory. Also, when running purge a second time after this I saw no errors at all and the system returned to “normal”(which is only annoyingly slow instead of horribly slow)

I must say that all the usage diagnostics tools such as iostat, vm_stat,  fs_usage, etc ,were all completely unhelpful when trying to diagnose memory and disk related issues. In fact lately when my disk has been hammered I have been running fs_usage and not seeing anything out of the ordinary by any processes, when there clearly is something going on in the background(by contrast for example, if I load up VMware Fusion and start a VM I can see fs_usage go crazy with hundreds of pages of info showing VMware Fusion using lots of disk).

I am going to make some statements here that will require further evidence to back up, but I’m going to say it to see if I get any responses and see if anyone else is having the same issues:

1) Memory management and VM management in Mac OS is broken. Seriously broken – even with the dynamic pager turned off.

2) VMware Fusion appears according to Mac OS to use far more disk than it should. I don’t know if the problem here lies with Fusion or Mac OS(both are up to date versions) because I run VMware Workstation on Linux and Windows machines separately and do not have any of these issues. Whenever I have Vmware Fusion open disk usage is stupidly high – for example right now I have Fusion open with a VM running FreeBSD that is doing absolutely nothing(and according to the FreeBSD iostat, vmstat, top, etc, NOTHING is using disk) – and yet fs_usage will show me about 2 pages a second of disk activity that is attributed to VMware Fusion.

3) Disk usage is broken in Mac OS and I think this is possibly a deliberate attempt to make you move to a better Mac with a SSD  - I cannot for the life of me work out why a normal 7200RPM 1TB disk drive cannot sustain <10Mbps reads and writes without bringing the system to a grinding halt.

What frustrates me the most is that from what I can see, neither Apple or anyone in the Apple community will acknowledge a fault – there is so many posts explaining similar problems to mine with a response along the lines of “I run 4000 chrome tabs and Virtual Box Virtual Machines and I never have seen anything like this – clear the icons off your desktop and disable anything worth while like spotlight and see how you go from there.”

I welcome co-whinges, any evidence that anyone can provide, and hopefully some fixes from someone…