Thursday, April 7, 2016

Driving from Consumption

Driving from consumption at its simplest is just starting from the user's interaction and backtracking from there.  To make this more concrete let's say we're building an extremely simply tiny url server.

Where to start?  There are two places the user interacts with this system.
* When the original url is shortened
* When the shortened url is followed

Which makes more sense to start with?


Shortening a url

If we start with shortening the url we could have a demo that shows whatever the user interaction is, for this example we can just do a POST like so:

curl --data "url=https://i.ytimg.com/vi/oM1EVAYahFE/maxresdefault.jpg" localhost/shorten

That would return some sort of success or failure with a shorter url on success.

If we go this path the first slice will be to return a url that won't work for the success case.... and to return an error for failure cases.  Given what we know now I'm not quite sure what the failure cases might be, and it's hard to think of a demo between it returned a 200 and a url and the full shorten -> fetch pipeline works.  We could demo putting the data in a database, but without knowing the fetching side it's hard to know exactly how we want to organize that.

Let's see if the other starting point is more obvious.


Following a shortened url

A nice starting demo is what happens when we try and follow an invalid short url.  So say curl localhost/s/totally_a_fake_key.  That demo is nice because that's a use case that isn't likely to change too much as we discover more about the system.

We could follow it up with demos of redirecting shortened urls based on a hardcoded hash.

That sounds like an easier place to start and also builds out the most common customer interaction point first, which allows the UX or product team to have something meaningful to demo faster.  It also means that by the time we get to data storage we'll have a good idea what data we need to read and what operations need to be fast.

Okay, lets build it.

<TODO> finish me!


Old post:

Driving from Consumption is simple and logical in theory: start with what the user interacts with and work backwards from there.

Sure, let me just understand the entire problem space and then we can do the DB design and we'll do the UI before the api.  Weird, but I can try that.

Nope.

Actually just focus on the next thing.  Just build the UI.  No actual api calls.  Just hardcoded JSON data.  Is that the right UI?  Do customers like it?  Is it what product wanted?  Does product actually want us to stop here because we came up with different ways of doing it and they want to do usability tests?

Okay, now the API.  Just the API.  No db calls.  The data can be hardcoded inside the tests.  Oh, you want to back it with a model?  Okay, after the hardcoded version works then there can be a model with hardcoded data in its tests.  Then, eventually, finally it can pull from the database.  Though there isn't any data there.  At least not outside of the integration tests.  Now we can put the data in!

You've probably been screaming this whole time.  But wait!  How do you know what the model should look like without considering the database?  You're going to code yourself into a corner!

Sometimes.  Usually not though.  In fact I haven't run into a dead end yet.  Instead the later changes are usually small, simple and obvious.

It's really scary to let go of planning out the entire code arch ahead of time.  So much of what we pride ourselves on as developers is being able to foresee dead ends and avoiding refactors.  Unfortunately all that effort takes time, cognitive load, and requires a certain level of familiarity with the code and the problem space that everyone on your team may not have.  That means they're suddenly in the middle of someone else's idea without understanding where it came from.

If instead of that upfront cost and accidental over designing you accept under designing and small refactors as they become necessary it isn't actually more overall work.  And suddenly everyone can participate in the design because everything is concrete.

Will it work for you?  There's only one way to find out.


Related topics:

Friday, April 8, 2011

i heart tig

I try not to write about something until I feel like I know it fairly well, but tig's causing me to break those rules. At its root it's only a prettier command line interface to git. To use it just run any normal git command and pass the output to tig.

For example: git log --author=jess | tig

That's something I do fairly frequently. This gives you a colored commit history. Not too exciting yet, but wait for it. You can then scroll down through that output and hit enter on any of the commit numbers to see the actual diff. To bring it to the next level add this to your .tigrc:

bind diff r !open http://your_code_review_server/%(commit)

Obviously you'll have to modify this to fit the url type you use, but now typing r in the diff view will jump you straight to your code reviewing site. I know it sounds dry but it's amazing for catching up on code reviews or just exploring the git history.

tig also supports views other than log and diff, I just haven't explored them much yet. Have fun.

Thursday, February 10, 2011

Guest Post: ngrep!

Sorry I haven't updated in a while. I've still been trying out stuff - just not blogging about it. Anyway, here's a guest post graciously written by Evan.

. . .

ngrep is a wonderful tool for tracking network activity. While tcpdump may be a better-known command-line tool and Wireshark may be more powerful, ngrep is so much handier in practice that I find I almost never use the other two.

What makes it nice is its simplicity; if I want to look watch for HTTP packets, "ngrep port 80" will do the trick. A local server on port 3000? "ngrep -d lo port 3000". All matching packets are printed to standard out in a reasonably-legible format, so I can get an idea of what's going on:

interface: wlan0 (192.168.1.0/255.255.255.0)
filter: (ip or ip6) and ( port 80 and host slashdot.org )

T 192.168.1.103:60614 -> 216.34.181.45:80 [AP]
GET / HTTP/1.1.
Host: slashdot.org.
User-Agent: Links (2.1pre36; Linux 2.6.36 x86_64; x).
Accept: */*.
Accept-Encoding: gzip, deflate, bzip2.
Accept-Charset: us-ascii, ISO-8859-1, ISO-8859-2, ISO-8859-3, ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8, ISO-8859-9, ISO-8859-10, ISO-8859-13, ISO-8859-14, ISO-8859-15, ISO-8859-16, windows-1250, windows-1251, windows-1252, windows-1256, windows-1257, cp437, cp737, cp850, cp852, cp866, x-cp866-u, x-mac, x-mac-ce, x-kam-cs, koi8-r, koi8-u, koi8-ru, TCVN-5712, VISCII, utf-8.
Accept-Language: en, *;q=0.1.
Connection: Keep-Alive.
Pragma: no-cache.
Cache-Control: no-cache.
.


T 216.34.181.45:80 -> 192.168.1.103:60614 [AP]
HTTP/1.1 200 OK.
Server: Apache/1.3.42 (Unix) mod_perl/1.31.
SLASH_LOG_DATA: shtml.
X-Powered-By: Slash 2.005001305.
X-Fry: That's a chick show. I prefer programs of the genre: World's Blankiest Blank..
X-XRDS-Location: http://slashdot.org/slashdot.xrds.
Cache-Control: no-cache.
Pragma: no-cache.
Content-Type: text/html; charset=iso-8859-1.
Content-Length: 144933.
Date: Fri, 14 Jan 2011 05:14:01 GMT.
X-Varnish: 1334903702 1334903109.
Age: 51.
Connection: keep-alive.
.

But as its name suggests, it can also act as a "network grep": an initial argument that doesn't look like packet descriptors are interpreted as regular expressions to match packets against. So "ngrep GET port 80" will show HTTP GET requests; "ngrep 'Host: slashdot.org' port 80" will show HTTP requests directed to slashdot. Well, technically, any port 80 traffic containing that string would match; including downloading this page...

One area where this can be extremely useful is when you have an opaque program you're trying to work with. For example, iTunes wasn't loading a video from a podcast; it was just showing a blank screen. I ran ngrep on port 80 for a few seconds while loading the video in iTunes, and out of the several dozen packets that came up, one showed a GET request for an expired URL. Problem solved! Sure, I could have found that by checking the web server logs on the server hosting the redirect. But I don't actually have direct access to that server, so I'd have to request them. And without knowing the request iTunes was making, this would have been fairly difficult to debug without learning quite a bit more about iTunes details.

The man page is fairly brief and informative, but here are a few of the options I find most useful:

-W byline: print newlines as newlines instead of as unprintable characters. Extremely useful for looking at HTTP headers

-x: dump packets in hexadecimal. Normally, non-printable characters are printed out as '.', so this lets you debug non-text-oriented protocols.

-i, -v: Like grep, these make the search case insensitive or inverted.

-q: Prevents ngrep from printing out hash marks for non-matched packets

-t: print timestamps for packets

-O/-I: output to a file or read from a file. Useful for grabbing lots of packets and searching through them later with various filters.


And the main filters I find myself using are "src port [port]", "dst host [hostname or ip]", and variations thereof. If neither src nor dst is specified, either is matched. Note, however, that dst or src is often interpreted as the regex to match rather than a packet modifier; adding a blank argument fixes that problem (e.g., "ngrep '' dst port 53" to watch outgoing DNS traffic).

Of course, ngrep is pretty well defeated by SSL/TLS, though you'd be surprised how much is still plaintext. For example, here's the opening of an ssh session:

interface: wlan0 (192.168.1.0/255.255.255.0)
filter: (ip or ip6) and ( port 22 )

T 192.168.1.104:22 -> 192.168.1.103:34445 [AP]
SSH-2.0-OpenSSH_5.1p1 Debian-5..

T 192.168.1.103:34445 -> 192.168.1.104:22 [AP]
SSH-2.0-OpenSSH_5.5..

T 192.168.1.103:34445 -> 192.168.1.104:22 [AP]
...L..sVW.j..2>{...j.....~diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchang
e-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1...Issh-rsa-cert-v00@openssh.c
om,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss....aes128-ctr,aes192-ctr,aes256-ctr,arcfour
256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,r
ijndael-cbc@lysator.liu.se....aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128
-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.l
iu.se...ihmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,h
mac-sha1-96,hmac-md5-96...ihmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripem
d160@openssh.com,hmac-sha1-96,hmac-md5-96....none,zlib@openssh.com,zlib....none,zlib@openss
h.com,zlib...................

T 192.168.1.104:22 -> 192.168.1.103:34445 [AP]
.......bC>.....E.........~diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchang
e-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1....ssh-rsa,ssh-dss....aes128-
cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,r
ijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr....aes128-cbc,3des-cbc,blowfish
-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.l
iu.se,aes128-ctr,aes192-ctr,aes256-ctr...ihmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripem
d160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96...ihmac-md5,hmac-sha1,umac-64@open
ssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96....none,zlib@ope
nssh.com....none,zlib@openssh.com.......................

T 192.168.1.103:34445 -> 192.168.1.104:22 [AP]
.....".......... .......

T 192.168.1.104:22 -> 192.168.1.103:34445 [AP]
............I..i.L7.+ec..~..x^....+...'+"..d...{...w....3.-S.X....]..vj...6&F...b.?J.....`.
..[.G.&Q....sU..ce....L....B.e.....r...A..(.'.;..............

The connection then continues on with similar (mostly non-printable) nonsense, but you can see a bit of the header exchange.


Friday, November 19, 2010

Mac Window Management

fvwm might be basic, but it has a pile of features I wish I could have on a Mac. One of the features I've been missing is window tiling. A disorganized stack of windows just isn't my thing. For a while I was just hiding everything except one window, which worked, but was a bit excessive on a 2560x1600 pixel monitor. Thankfully there is a better way.

There are two external pieces of software that can help with the Tiling
Problem(tm). The first one I tried is SizeUp($13). It divides the screen into four quadrants (they default to 50% of the screen, but this can be customized) and then has key bindings to resize and move a window to fit any half or quarter of the screen. It can also do full screen. It's a very simple interface, and was easy to pick up. I also didn't find any bugs while using it.


The second piece of software I tried was Divvy($15). It divides the screen into cells (up to 10x10). You can then use the mouse to custom select a subset of the cells, which will resize the window to that portion of the screen. You can also define as many custom keyboard shortcuts as you want to correspond to any cell selection. With this you can define as many (possibly overlapping) screen regions as you want to have things snap to. It does have a bit more of a learning curve, and there is a questionable UI choice. Mainly that if you change how many cells the screen is divided into there isn't anyway to update existing shortcuts to use that grid. Instead you have to delete and re-create them. In practice this isn't a big deal however.

After using both pieces of software in demo mode for around a week and a half I broke down and bought Divvy, mostly due to greater flexibility. Now if only I could find something better than command-tab for switching between windows I'd be a very happy person indeed.

Thanks to Caleb for suggesting Divvy.

Wednesday, November 10, 2010

Reusing screen sessions

Screen is the only way I can keep myself organized, especially given the limited window manager on a mac. In the past I've always only had one terminal running screen and let all other terminals just run straight. This resulted in some awkward workflows however when everything I wanted to get at was in one of the screen tabs. The solution? screen -xRRA

Let's break it down:
x: Attach to a not detached screen session

RR: Attempts to reconnect to the first detached screen session it can find. Otherwise it creates a new screen session.

A: Adjust screen to the size of this terminal instead of resizing the terminal

Ta-da! The same screen session's now in as many terminals as you want.

This tip thanks to Matthew Brewer

Monday, November 8, 2010

Case insensitive vim searching

I don't know how I didn't learn this trick years ago. It's dead simple, but something that comes up fairly often: case insensitive searching.

It's fairly well known that in vim '/' is case sensitive search. That's almost always what I want, but very occasionally I want a case insensitive search. Search can be set to ignore case for the life of the vim session by using :set ignorecase (:set ic) and likewise :set noignorecase turns off the feature. That's really obnoxious if you only want one insensitive search. Luckily there's a better solution! Putting \c at the end of your search query makes only that search case insensitive.