Guest Post: ngrep!

Sorry I haven't updated in a while. I've still been trying out stuff - just not blogging about it. Anyway, here's a guest post graciously written by Evan.

. . .

ngrep is a wonderful tool for tracking network activity. While tcpdump may be a better-known command-line tool and Wireshark may be more powerful, ngrep is so much handier in practice that I find I almost never use the other two.

What makes it nice is its simplicity; if I want to look watch for HTTP packets, "ngrep port 80" will do the trick. A local server on port 3000? "ngrep -d lo port 3000". All matching packets are printed to standard out in a reasonably-legible format, so I can get an idea of what's going on:

interface: wlan0 (192.168.1.0/255.255.255.0)
filter: (ip or ip6) and ( port 80 and host slashdot.org )

T 192.168.1.103:60614 -> 216.34.181.45:80 [AP]
GET / HTTP/1.1.
Host: slashdot.org.
User-Agent: Links (2.1pre36; Linux 2.6.36 x86_64; x).
Accept: */*.
Accept-Encoding: gzip, deflate, bzip2.
Accept-Charset: us-ascii, ISO-8859-1, ISO-8859-2, ISO-8859-3, ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8, ISO-8859-9, ISO-8859-10, ISO-8859-13, ISO-8859-14, ISO-8859-15, ISO-8859-16, windows-1250, windows-1251, windows-1252, windows-1256, windows-1257, cp437, cp737, cp850, cp852, cp866, x-cp866-u, x-mac, x-mac-ce, x-kam-cs, koi8-r, koi8-u, koi8-ru, TCVN-5712, VISCII, utf-8.
Accept-Language: en, *;q=0.1.
Connection: Keep-Alive.
Pragma: no-cache.
Cache-Control: no-cache.
.


T 216.34.181.45:80 -> 192.168.1.103:60614 [AP]
HTTP/1.1 200 OK.
Server: Apache/1.3.42 (Unix) mod_perl/1.31.
SLASH_LOG_DATA: shtml.
X-Powered-By: Slash 2.005001305.
X-Fry: That's a chick show. I prefer programs of the genre: World's Blankiest Blank..
X-XRDS-Location: http://slashdot.org/slashdot.xrds.
Cache-Control: no-cache.
Pragma: no-cache.
Content-Type: text/html; charset=iso-8859-1.
Content-Length: 144933.
Date: Fri, 14 Jan 2011 05:14:01 GMT.
X-Varnish: 1334903702 1334903109.
Age: 51.
Connection: keep-alive.
.

But as its name suggests, it can also act as a "network grep": an initial argument that doesn't look like packet descriptors are interpreted as regular expressions to match packets against. So "ngrep GET port 80" will show HTTP GET requests; "ngrep 'Host: slashdot.org' port 80" will show HTTP requests directed to slashdot. Well, technically, any port 80 traffic containing that string would match; including downloading this page...

One area where this can be extremely useful is when you have an opaque program you're trying to work with. For example, iTunes wasn't loading a video from a podcast; it was just showing a blank screen. I ran ngrep on port 80 for a few seconds while loading the video in iTunes, and out of the several dozen packets that came up, one showed a GET request for an expired URL. Problem solved! Sure, I could have found that by checking the web server logs on the server hosting the redirect. But I don't actually have direct access to that server, so I'd have to request them. And without knowing the request iTunes was making, this would have been fairly difficult to debug without learning quite a bit more about iTunes details.

The man page is fairly brief and informative, but here are a few of the options I find most useful:

-W byline: print newlines as newlines instead of as unprintable characters. Extremely useful for looking at HTTP headers

-x: dump packets in hexadecimal. Normally, non-printable characters are printed out as '.', so this lets you debug non-text-oriented protocols.

-i, -v: Like grep, these make the search case insensitive or inverted.

-q: Prevents ngrep from printing out hash marks for non-matched packets

-t: print timestamps for packets

-O/-I: output to a file or read from a file. Useful for grabbing lots of packets and searching through them later with various filters.

And the main filters I find myself using are "src port [port]", "dst host [hostname or ip]", and variations thereof. If neither src nor dst is specified, either is matched. Note, however, that dst or src is often interpreted as the regex to match rather than a packet modifier; adding a blank argument fixes that problem (e.g., "ngrep '' dst port 53" to watch outgoing DNS traffic).

Of course, ngrep is pretty well defeated by SSL/TLS, though you'd be surprised how much is still plaintext. For example, here's the opening of an ssh session:

interface: wlan0 (192.168.1.0/255.255.255.0)
filter: (ip or ip6) and ( port 22 )

T 192.168.1.104:22 -> 192.168.1.103:34445 [AP]
SSH-2.0-OpenSSH_5.1p1 Debian-5..

T 192.168.1.103:34445 -> 192.168.1.104:22 [AP]
SSH-2.0-OpenSSH_5.5..

T 192.168.1.103:34445 -> 192.168.1.104:22 [AP]
...L..sVW.j..2>{...j.....~diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchang
e-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1...Issh-rsa-cert-v00@openssh.c
om,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss....aes128-ctr,aes192-ctr,aes256-ctr,arcfour
256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,r
ijndael-cbc@lysator.liu.se....aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128
-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.l
iu.se...ihmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,h
mac-sha1-96,hmac-md5-96...ihmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripem
d160@openssh.com,hmac-sha1-96,hmac-md5-96....none,zlib@openssh.com,zlib....none,zlib@openss
h.com,zlib...................

T 192.168.1.104:22 -> 192.168.1.103:34445 [AP]
.......bC>.....E.........~diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchang
e-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1....ssh-rsa,ssh-dss....aes128-
cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,r
ijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr....aes128-cbc,3des-cbc,blowfish
-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.l
iu.se,aes128-ctr,aes192-ctr,aes256-ctr...ihmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripem
d160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96...ihmac-md5,hmac-sha1,umac-64@open
ssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96....none,zlib@ope
nssh.com....none,zlib@openssh.com.......................

T 192.168.1.103:34445 -> 192.168.1.104:22 [AP]
.....".......... .......

T 192.168.1.104:22 -> 192.168.1.103:34445 [AP]
............I..i.L7.+ec..~..x^....+...'+"..d...{...w....3.-S.X....]..vj...6&F...b.?J.....`.
..[.G.&Q....sU..ce....L....B.e.....r...A..(.'.;..............

The connection then continues on with similar (mostly non-printable) nonsense, but you can see a bit of the header exchange.

Mac Window Management

fvwm might be basic, but it has a pile of features I wish I could have on a Mac. One of the features I've been missing is window tiling. A disorganized stack of windows just isn't my thing. For a while I was just hiding everything except one window, which worked, but was a bit excessive on a 2560x1600 pixel monitor. Thankfully there is a better way.

There are two external pieces of software that can help with the Tiling

Problem(tm). The first one I tried is SizeUp($13). It divides the screen into four quadrants (they default to 50% of the screen, but this can be customized) and then has key bindings to resize and move a window to fit any half or quarter of the screen. It can also do full screen. It's a very simple interface, and was easy to pick up. I also didn't find any bugs while using it.

The second piece of software I tried was Divvy($15). It divides the screen into cells (up to 10x10). You can then use the mouse to custom select a subset of the cells, which will resize the window to that portion of the screen. You can also define as many custom keyboard shortcuts as you want to correspond to any cell selection. With this you can define as many (possibly overlapping) screen regions as you want to have things snap to. It does have a bit more of a learning curve, and there is a questionable UI choice. Mainly that if you change how many cells the screen is divided into there isn't anyway to update existing shortcuts to use that grid. Instead you have to delete and re-create them. In practice this isn't a big deal however.

After using both pieces of software in demo mode for around a week and a half I broke down and bought Divvy, mostly due to greater flexibility. Now if only I could find something better than command-tab for switching between windows I'd be a very happy person indeed.

Thanks to Caleb for suggesting Divvy.

Reusing screen sessions

Screen is the only way I can keep myself organized, especially given the limited window manager on a mac. In the past I've always only had one terminal running screen and let all other terminals just run straight. This resulted in some awkward workflows however when everything I wanted to get at was in one of the screen tabs. The solution? screen -xRRA

Let's break it down:

x: Attach to a not detached screen session

RR: Attempts to reconnect to the first detached screen session it can find. Otherwise it creates a new screen session.

A: Adjust screen to the size of this terminal instead of resizing the terminal

Ta-da! The same screen session's now in as many terminals as you want.

This tip thanks to Matthew Brewer

Case insensitive vim searching

I don't know how I didn't learn this trick years ago. It's dead simple, but something that comes up fairly often: case insensitive searching.

It's fairly well known that in vim '/' is case sensitive search. That's almost always what I want, but very occasionally I want a case insensitive search. Search can be set to ignore case for the life of the vim session by using :set ignorecase (:set ic) and likewise :set noignorecase turns off the feature. That's really obnoxious if you only want one insensitive search. Luckily there's a better solution! Putting \c at the end of your search query makes only that search case insensitive.