FreeBSD Jails – SSH Problem

Posted by on June 17, 2012 2 comments

After some time being able to SSH into a certain jail I noticed it stopped working, then I noticed that when trying to SSH from the jail that also did not work. When trying to SSH to or from the jail this is usually what I saw:

[root@dev-j1 ~]# ssh mark@somehost.com
Host key verification failed.

It all comes down to this error here:

debug1: read_passphrase: can’t open /dev/tty: No such file or directory

If you can reproduce the same or similar from either looking in your SSHD logs or by running ssh -vvv from the jail, then chances are you are seeing the same problem as me. I haven’t found a permanent and still secure fix yet, but I have found somewhat of a fix(I have to do more research to see if the fix is secure or not, so I wouldn’t recommend implementing this on a production system without further research or knowledge on the impact of what you are doing.

To fix, stop the jail and open /etc/defaults/devfs.rules and at the end of the file you should see something like this:

# Devices usually found in a jail.
#
[devfsrules_jail=4]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login

When I look at devfsrules_unhide_login I see that there are some rule to unhide a bunch of tty’s so I add the following to  the end of the devfs_uhide_login area:

add path ‘tty’ unhide

I actually insert it between these two:

add path 'ttyO*' unhide
add path ptmx unhide

Now save that file and start your jail again – you should see that /dev on the jail shows tty, so you should be able to SSH from it. The condition here is that you will need to use keys, because SSH can’t actually read from tty, as seen now:

ebug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/id_rsa
debug3: no such identity: /root/.ssh/id_rsa
debug1: Trying private key: /root/.ssh/id_dsa
debug3: no such identity: /root/.ssh/id_dsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
debug1: read_passphrase: can't open /dev/tty: No such file or directory
debug3: packet_send2: adding 64 (len 49 padlen 15 extra_pad 64)
debug2: we sent a password packet, wait for reply
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such file or directory
debug3: packet_send2: adding 64 (len 49 padlen 15 extra_pad 64)
debug2: we sent a password packet, wait for reply
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such file or directory
debug3: packet_send2: adding 64 (len 49 padlen 15 extra_pad 64)
debug2: we sent a password packet, wait for reply
debug1: Authentications that can continue: publickey,password
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey,password).

Now the reason you are seeing the above error is probably because you are logged into the console of the jail through a jail console session and not through SSH, so obviously something is going on with how the console session plays with the tty’s. I use ezjail-admin console jailname and thats when I see the above problem.

If you only SSH into the jail and don’t use the console you will find the problem is gone – you can SSH in and out of the jail without having to use keys every time and it works fine. Let me know if you know of a better way to resolve this.

 

Mac OS X Memory Issues

Posted by on June 15, 2012 2 comments

I thought I would put up something for those searching for issues to do with Mac OS X memory management which is clearly broken right of the box with Lion. I should start by saying that my first use of Mac OS started with Lion and I have never used Snow Leopard, so I can safely say from my experience that memory management in Mac OS is seriously broken.

A bit of context, I have a 27″ iMac with the Core i5(2011), 12GB RAM(I have removed RAM to see if that helps!) and a 1TB Hard Drive, and I also have a MacBook Air which I purchased at the same time which has 4GB Ram and a 128GB SSD. Within a few weeks of purchasing both of these, I noticed that the iMac which is supposed to be the more powerful machine of the two is slower by a large amount than the MacBook Air. I don’t run very intense stuff, although I had hoped to get rid of one of my servers and just run most of my virtual machines off of my iMac. So a bit of a run down of what I run every day: About 30-40 Chrome Tabs, 10-5 Terminal Tabs, A fe instances of TextEdit and/or TextMate, and a single VMware Fusion VM running Windows 7(2GB RAM).

I have a fairly large background in FreeBSD and OpenBSD, and I know that Mac OS is loosely based off of BSD(that’s all I’ll say on the matter because how much it is based off BSD and what parts have been taken from where seem to be the subject of a lot of debate) – and one of the main reasons I use FreeBSD so much is because I find it performs so much better than linux, it is a lot less weighty, but is still very powerful. Because of this, I did assume that Mac OS would be similar, and in any case, should there be any problems, there should be a myriad of ways to diagnose and troubleshoot the issue. Well, I was wrong, very wrong.

I noticed the the reason that the iMac is considerably slower than the MBA is because of the SSD in the MBA. Essentially, memory management is so broken in OSX that it continually pages all memory out to disk even if there is free memory, and if there isn’t any free memory, it generally doesn’t bother to clear it out. You can view this through your activity monitor and through the command “top” in Terminal. Check out the ridiculous load levels in top whilst I write this blog post:

Load Avg: 11.45, 13.73, 13.69  CPU usage: 70.55% user, 29.44% sys, 0.0% idle   SharedLibs: 12M resident, 6168K data, 0B linkedit. MemRegions: 38353 total, 2187M resident, 104M private, 3865M shared.
PhysMem: 7708M wired, 441M active, 147M inactive, 8296M used, 3988M free. VM: 382G vsize, 1119M framework vsize, 5707517(40) pageins, 147944(0) pageouts.
Networks: packets: 12870819/12G in, 9624876/1816M out. Disks: 2804511/44G read, 5195808/88G written

According to top and the activity monitor, the CPU Usage is VMware Fusion(sys) and Finder(user) and that is all, everything else looks normal. Further investigation will reveal that it is because for some reason Finder is indexing, and Fusion has outlook open, which once again comes back to disk IO. It should be noted at this time, that the Hard Drive in the iMac has been replaced because even though I here tonnes of activity on it, and the system crawls, I can only see minimal reads and writes in Activity Monitor(replacing the hard drive did not help). What I found in this instance is that the more CPU intensive applications that I quit, the more CPU usage for finder went up as if they were preventing it from using more CPU. Unfortunately though this is the first time I have seen this issue with Finder so I can’t lay any blame there.

Anyway, essentially Mac OS does not know how to use memory, so it pages everything to disk which results in super slow systems(unless you have a super duper latest and greatest MBP). So I found a so called fix for this, I did the folllowing:

sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist

The above Terminal command disables the so called dynamic pager, which is supposed to result in much better performance. If you plan on doing this yourself, remember to reboot afterwards, and if you want to re-enable the pager, do the following command followed by a reboot:

sudo launchctl load -wF /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist

After disabling the dynamic pager and rebooting, I have seen almost zero performance increase, and what I have noticed is that even though the system no longer reads the pages back, it is still sending a lot of memory to the page file which can be noticed by the above top command output(notice though in the above output that some pages have been read back in, but that is because my system nearly crashed, and it must have turned it back on?). Essentially, for me, the above command does not work – not even if I just have Chrome open browsing websites(with nothing else open, not even Fusion).

What has helped me however, is when I notice my memory getting low, I run the command purge from Terminal – I have heard that purge clears only the inactive memory but from what I can see, it also clears the active memory that is no longer in use and hasn’t properly been listed as inactive memory(it seems as though perhaps applications aren’t letting Mac OS know that they stopped using the memory, or perhaps it is just too stupid to work things out like just about any other OS can).

I thought I would share something from what happened to me today though… This morning I noticed my machine was running like a dog again, so I pulled up the Activity Monitor and noticed that Free Memory was down to about 19MB. So I pulled up Terminal to run purge and free up some memory(I saw about 1.2GB inactive). After I ran purge, this is what I saw:

ERROR]	<CPPathWatcher.c:101>	Events were dropped; full rescan required.
[ERROR]	<CPDevice.c:7668>	Error #-6 (kCPNotLoaded): Unable to update device instances having loaded one or more new or changed records.
    o-+- [com.apple.CoreProfile.Devices] <CPDevice.c:4947>  _CPUpdateDeviceInstances: CPStatus -6 (kCPNotLoaded)
      |  Failed to [completely, at least] update device instances.

I saw about 2 pages of that text repeated after each other, but once the purge command had completed, I noticed I had about 4GB RAM free, and that it had cleared the inactive memory as well as some of the active memory(by the looks of it). Not only that, I also noticed in the Activity Monitor that I had started paging out, so it must be that when the system starts to run out of memory, that it re-enables the pager to some extent and start paging memory in and out again. So hopefully this provides some insight for those who were not sure what would happen to their system if it ran out of memory. Also, when running purge a second time after this I saw no errors at all and the system returned to “normal”(which is only annoyingly slow instead of horribly slow)

I must say that all the usage diagnostics tools such as iostat, vm_stat,  fs_usage, etc ,were all completely unhelpful when trying to diagnose memory and disk related issues. In fact lately when my disk has been hammered I have been running fs_usage and not seeing anything out of the ordinary by any processes, when there clearly is something going on in the background(by contrast for example, if I load up VMware Fusion and start a VM I can see fs_usage go crazy with hundreds of pages of info showing VMware Fusion using lots of disk).

I am going to make some statements here that will require further evidence to back up, but I’m going to say it to see if I get any responses and see if anyone else is having the same issues:

1) Memory management and VM management in Mac OS is broken. Seriously broken – even with the dynamic pager turned off.

2) VMware Fusion appears according to Mac OS to use far more disk than it should. I don’t know if the problem here lies with Fusion or Mac OS(both are up to date versions) because I run VMware Workstation on Linux and Windows machines separately and do not have any of these issues. Whenever I have Vmware Fusion open disk usage is stupidly high – for example right now I have Fusion open with a VM running FreeBSD that is doing absolutely nothing(and according to the FreeBSD iostat, vmstat, top, etc, NOTHING is using disk) – and yet fs_usage will show me about 2 pages a second of disk activity that is attributed to VMware Fusion.

3) Disk usage is broken in Mac OS and I think this is possibly a deliberate attempt to make you move to a better Mac with a SSD  - I cannot for the life of me work out why a normal 7200RPM 1TB disk drive cannot sustain <10Mbps reads and writes without bringing the system to a grinding halt.

What frustrates me the most is that from what I can see, neither Apple or anyone in the Apple community will acknowledge a fault – there is so many posts explaining similar problems to mine with a response along the lines of “I run 4000 chrome tabs and Virtual Box Virtual Machines and I never have seen anything like this – clear the icons off your desktop and disable anything worth while like spotlight and see how you go from there.”

I welcome co-whinges, any evidence that anyone can provide, and hopefully some fixes from someone…

A bit about CPUs & Registers

Posted by on July 14, 2011 No comments

So I have been learning a little about smashing the stack, CPU registers, etc. Can’t say I’m even close to becoming an expert, but thought I would put up some notes anyways. There are a number of general purpose registers, and the first four we are starting with are EAX, ECX, EDX and EBX. The names of each of these registers are as follows:

EAX = Accumulator

ECX = Counter

EDX = Data

EBX = Base

These have a variety of purposes but are mainly used to act as temporary variables for the CPU when it is executing machine instructions. The second four general purpose registers are ESP, EBP, ESI and EDI(these are sometimes known as pointers and indexes).

ESP = Stack Pointer

EBP = Base Pointer

ESI = Source Index

EDI = Destination Index

The first two(ESP, EBP) are called pointers because they store 32-bit addresses which essentially point to a location in memory. The last two registers(ESI, EDI) are technically pointers too. Commonly used to point to the source and destination when data needs to be read from or written to. There are some Load and Store Instructions that use these registers.

EIP = Instruction Pointer

The EIP points to the current instruction the processor is reading, or put more technically, it contains a memory address that points to the current instruction(in the main() in a C app for example) I get the feeling from my limited reading so far that the EIP is very important when looking into exploitation, and will be looked at a lot when debugging.

The EFLAGS register consists of several bit flags that are used for comparisons and memory segmentation.

The smallest amount that can be stored in memory is 1 bit, which can be represented as either a 1 or a 0. When you put 4 bits together, it is called a nibble, and this can represent values from 0000 to 1111. When you put two nibbles or 8 bits together you get a byte, which can represent values from 0 to (2^8 -1) which is 0 – 255 in decimal.

When you put two bytes together you get a WORD, which can represent values from 0 to (2^16 -1) or 0 – 65,535 in decimal. If you put two WORDs together you get a Double WORD or DWORD, which can represent values from 0 to (2^32 -1) or in decimal 0 – 4,294,967,297.

IPTables Summary

Posted by on July 11, 2011 No comments

What is IPTables? By definition it is a packet selection system that has been built on top of the Netfilter framework, so essentially IPTables is just a front end to the very powerful Netfilter. All IPTables does is allow you to make use of the Netfilter framework without having to do your own Kernel hacking – it lets you insert, update, list rules and create chains, etc. Essentially its a user friendly way of creating firewall rulesets by inserting kernel modules that perform certain actions on packets based on whether the packets meet certain criteria.

Netfilter on the other hand is the power behind IPTables. Netfilter framework is compiled into the kernel, and essentially it inserts a bunch of hooks in the kernel protocol stacks for IPv4, IPv6 and DECNET. That way when a packet passes through the Kernel, the Netfilter hook always gets called, and then actions can be taken if need be.

A table is an IPTables construct that delineates broad categories of functionality, such as packet filtering or Network Address Translation. There are four tables: filter, nat, mangle and raw.

Filtering rules are applied to the filter table, NAT rules are applied to the nat table, specialized rules that alter packet data are applied to the mangle table, and rules that should function independently of the Netfilter connection-tracking subsystem are applied to the raw table.

Each table has its own set of built in chains, but user-defined chains can also be created so that the user can build a set of rules that is related by a common tag. The most important built-in chains as far as I’m concerned are the INPUT, OUTPUT, and FORWARD chains in the filter table.

- The INPUT chain is traversed by packets that are destined for the local linux system after a routing calculation is made within the kernel(ie. Packet destined for the local socket).

- The OUTPUT chain is reserved for packets that are generated by the linux system itself.

- The FORWARD chain governs packets that are routed through the linux system.

Two additional chains that are important for any serious iptables deployment are the PREROUTING and POSTROUTING chains in the nat table, which are used to modify packet headers before and after an IP routing calculation is made within the kernel.

An iptables match is a condition that must be met by a packet in order for iptables to process the packet according to the action specified by the rule target. For example, to apply a rule only to TCP packets, you can use the –protocol match.

Important matches:

–source (-s)

–destination (-d)

–protocol (-p)

–in-interface (-i)

–out-interface (-o)

–state

–string

–comment

iptables supports a set of targets that trigger an action when a packet matches a rule. The most important targets I use are:

ACCEPT – Allows a packet through the firewall

DROP – Discards packet as though it was never received

LOG – Logs packet to syslog

REJECT – This drops a packet but then sends a response back to the source(eg. RST)

RETURN – Continues processing the packet so that it can be matched to other rules within the chain.

More IPTables coming soon!

Extended Exim logging on cPanel

Posted by on July 11, 2011 No comments
Exim’s logging isn’t too detailed by default, and in case of e-mail deliverability issues, provides insufficient information for proper diagnostics. So how do we extend Exim’s logging on cPanel systems?

 

This is a pure WHM action, so log into WHM as root, and navigate to Main >> Service Configuration >> Exim Configuration Editor

 

Scroll down to the bottom of the screen and enter the advanced editor mode by clicking on the Advanced Editor button.

 

When the Advanced Editor opens, put the following line into the first, topmost text box:

log_selector = +all -ident_timeout -host_lookup_failed -lost_incoming_connection

This will enable all extended logging attributes (+all), excluding the several ones that aren’t important but would fill logs with the excessive data (-ident_timeout -host_lookup_failed -lost_incoming_connection). While we are here, we will also close one security hole in Exim, that can enable a 3rd party to perform unauthorized relaying if left open.

 

If the following lines aren’t already in the first, topmost text box, add them as follows:

 

hostlist relay_hosts = 127.0.0.1

 

hostlist relay_from_hosts = 127.0.0.1
hostlist auth_relay_hosts = *

 

 

Scroll down to the bottom of the screen, and press the Save button. This will save your changes and restart Exim. Watch the output on the screen which, amongst other things, must read that Exim configuratin has passed test ie. that it is syntactically correct:

 

Configuration file passes test!  New configuration file was installed.
and that Exim was successfully restarted:
Exim MTA…
Waiting for exim to restart…………..finished.
exim (/usr/sbin/exim -bd -q60m) running as mailnull with PID 26404
exim started ok
…Done

But Wait! We are not done yet! Now navigate to Main >> Server Configuration >> Tweak Settings and make sure that Track the origin of messages sent though the mail server by adding the X-Source headers (exim 4.34+ required) is turned ON. If it isn’t, turn it on by clicking on the little checkbox at the right end of the aforementioned line, scroll down to the bottom of the screen and press Save.

 

Done, we have expanded the Exim’s logging facilities (and message headers for all future messages) and implemented the anti-relay protection.

 

Now, tailing Exim’s main log in shell using the command

 

tail -f /var/log/exim_mainlog

 

We will notice that logs have started to provide much more extensive information.

 

Quick Rundown of Linux Partitioning

Posted by on July 11, 2011 No comments

I have just put together a quick overview of the general partitioning information you will need when installing linux on your PC/server. Hope it helps if your just beginning :)

/ the root partition/volume is identified by a forward slash(/). All other directories are attached to this parent directory. It is somewhat equivalent to the system drive(C:\) in Windows.

/boot This contains almost everything required for the boot process. It stores data that is used before the kernel begins executing other programs. I usually set my boot partition to about 100MB in size.

/usr This is usually where all program files reside, somewhat like C:\Program Files on Windows.

/home This is where, by default all user home directories are stored. Somewhat equivalent to C:\Documents and Settings on Windows.

/var Logs are generally stored here so it’s a good idea to have a separate partition for /var so that log files don’t and can’t fill up all the space on your system.

/tmp This is where temporary files go, and its writable by any user – keep this on a separate partition to avoid malicious users filling it up with data, and make the partition NOEXEC if possible.

SWAP This is pretty much the same as the virtual memory feature on Windows. For a System with 1-4GB of RAM I usually allocate 2GB SWAP, whilst 4GB+ I usually allocate 4GB SWAP.

Hope this helps with any confusion you may have, and by the way – if your not sure what file system to use I usually recommend ext3. It’s a lot easier to troubleshoot and work with than Logical Volumes.

Redirects: Difference between the 301 and 302

Posted by on July 11, 2011 No comments

I see a lot of customers ask what the difference is between a 301 permanent redirection and a 302 temporary redirection. Both redirections are available to be used from a cPanel control panel and it can be confusing as to which one is the right choice. Use of common logic leads most people to believe that a temporary(302) redirection only lasts for a set period of time, and that after the period of time has expired the redirection no longer exists – this is entirely incorrect, so no if you have setup a temporary redirection you don’t have to worry about it disappearing one day.

Honestly I have never come across a situation where a 302 redirection is needed – how often is it that you only need to move a page temporarily? For me that has been never, most times I use a redirect is when I don’t need to see the old page whatsoever. So for the sake of simplicity, and if you don’t want to find out how it all works and the reasons behind my recommendations then you can stop reading here and just remember to use 301(Permanent) redirections whenever you need one.

Whenever you visit a page on a website, the web server generates a HTTP status code. This status code is usually 200 which means that file was found and served to the client without error. The only difference between a temporary and a permanent redirection is the HTTP status code which is generated by the web server, a permanent redirection generates a 301 status code and a temporary redirection generates a 302 status code. A 301 redirect is permanent, they mean that the original page has moved and they inform any search engine or user agent(browser) that comes to the page to update the URL in their database. A 302 redirect on the other hand informs the search engine or user agent(browser) that the requested resource resides temporarily in another location, but it can still be found at the requested URI.

These codes actually affect how search engines treat your site and in turn how they index your site. Incorrect codes can actually cause a search engine such as Google to index your site in a negative way, so it is important that you get it right. Not only do these codes tell search engines how to treat the site, they can also cause your web browser to treat your site in different ways.

Have you ever tried implementing a redirect via the control panel or through .htaccess and even after changing the redirect or removing it, it still doesn’t change in your browser even after multiple refreshes? Usually when this happens you need to close your browser and re-open it again. In most cases the reason this occurs is because you have implemented a permanent redirect and your browser has read the status code(301) and from then on whenever you visit the original location, your browser doesn’t even check the files it just goes straight to the redirects destination. Why does it do this? Because the browser knows that the redirection is permanent hence the redirection shouldn’t ever change, so there is no need for it to even request the original file from the web server, it knows it will only get redirected anyway. Closing and reopening your browser should rectify the problem because the browsers cache is cleared, so the next time you put in the URL of the original resource your browser will actually request the file.

I generally advise that if you are going to setup a redirect, you should use a 301 redirect. This is because of how a 301 redirect affects search engines as opposed to 302 redirects. There are a few reasons that make a 302 redirect a bad choice when it comes to search engines, these are:

-          302 redirects are commonly used by spammers to get more of their domains up in search engine results. Hence Google and other search engines don’t like this redirection technique, and it can really hurt your page rank.

-          A page redirected to by a 302 has to generate it’s own popularity, whereas with 301 redirects your URLs maintain their link popularity. If you set up a 302 redirect, Google and other search engines that determine popularity ratings automatically assume that the link is eventually going to be taken down. Because after all, it’s only a temporary redirect. So the new page doesn’t have any of the link popularity associated with the old page. It has to generate that popularity on its own.

For more reasons you should research the topic more thoroughly – it’s beyond the scope of this tutorial to provide every reason why I don’t recommend 302 redirects. Hope I’ve helped!