Peter Morville and Jeffery Callender
Data science projects can (very roughly) be divided into two types. The first is a study, aimed at providing quantitative insights to other business units. These typically involve building reports, calculating p-values, and answering product managers’ questions. Fifteen years ago the people doing this were called “statisticians” or “data analysts.” The second type of project feeds directly into customer-facing products. Examples include recommender systems, tagging/classification systems, and search engines.
If you’re a data scientist building customer-facing products, it’s important to recognize that your job is largely subordinate to design. You’re not being paid to optimize objective functions in a vacuum. You’re being paid to make a product that helps users achieve some task. Chipping away at cross-entropy might help with this, but it’s only one piece of a larger puzzle. If you zoom out and look at the entire puzzle – by which I mean learning about product, UX, and design – you’ll be able to add your piece in the right place, rather than jamming it somewhere it doesn’t belong. In 2018 I diversified my blog/article diet to include design, and this year I’m trying to read some classics in the field (and if you have a book recommendation please send me an email).
I began with Search Patterns, an O’Reilly text popular with the designers where I work. It’s now nearly a decade old, but it felt strikingly contemporary. Search is domain-specific, so it’s a different experience every time out. Rather than prescribing a five step program for building search, the authors assemble a toolkit of techniques and heuristics, each of which have their place.
The first chapter, “Pattern Recognition,” is somewhat abstruse. In twenty-four pages the authors cover Klaus Conrad’s concept of apophenia, a critique of categorical thinking inspired by the colour wheel, the philosophy of American architect Christopher Alexander, and introduce a twenty point manifesto.
While all this serves as a basis for an interesting further-reading list, the main points I took away from the chapter are a bit more prosaic. The first is that search is exceedingly difficult, and for this reason, organizations tend to underinvest in it.
Let’s face it: search is a wicked problem with no definitive formulation, considerable uncertainty, and complex interdependencies…Search is both a project and a process. It’s a problem that’s never solved.
The second takeaway is that search is inherently interdisciplinary. It requires designers, engineers, UX researchers, and data scientists to work closely.
The final and perhaps most interesting point (at least to someone coming from a data science background) regards the interactive nature of search.
…search at its best is a conversation. It’s an interactive, iterative process where we find we learn. The answer changes the question. The process moves the goal.
This is in stark contrast to how ML practitioners are trained to think about problems, in which you’re essentially aiming to reduce some error metric of a curve fit. Creating an interactive learning experience is a much more elusive goal.
The second chapter introduces what the authors regard as the five components of the search experience: user, interface, engine, content, and creators. All five components are interesting, but the ones I want to emphasize are the final two. Too often, engineers working on search and discovery assume that their job is to take the data as given and start optimizing. This view is hopelessly narrow. The success of data science projects depends entirely on the data you’re working with. The most important tasks are therefore the creation, cleaning, and maintenance of the content. Morville and Callender suggest five questions to guide this process.
– Who are the current (and potential) creators of content?
– How can we motivate them to improve quality and quantity?
– What tools and processes will make publishing faster and easier?
– How can we enlist users in content creation and organization?
– How can we share analytics to inspire both use and co-creation?
Suppose you’re working on search over a corpus of FAQs (this is a common context in which organizations might invest in search, because it can mitigate customer support costs). You may find that some of your most common queries are bringing up the wrong FAQs. Before you decide that you need an expensive transfer-learning NLP solution, talk to the author of the FAQs. Perhaps the solution is simply to word the FAQs differently. There might be a mismatch between your users’ language and your content strategy. If this is the case, transfer learning won’t even solve the problem: when you serve up the “right” FAQ, the user won’t realize it answers their question, because it isn’t couched in language they recognize.
Chapter 3 is about behaviour. I really enjoyed it – it gives names to a number of behaviours you will recognize in your own Googling, but have likely never articulated. One of my favourites is “thrashing.”
In thrashing, a design flaw resides in users’ heads in the form of the anchoring bias. We set the anchor with our initial query, and then, despite poor results, we insist on small variations of the flawed search phrase rather than trying break this pattern.
(For more on the anchoring bias I recommend this paper.) Thrashing is one of the reasons that autosuggest/autocomplete is a powerful tool. Autosuggest is not simply saving users the trouble of typing out words in full – if this was the only benefit, finite state transducers probably wouldn’t be worth the trouble. By suggesting related words and phrases to the user, you are giving them paths out of local minima. You’re also providing guardrails against unsuccessful keywords, helping users realize that they need to pull anchor and completely change their approach.
The fourth chapter introduces design patterns. A lot of territory is covered here. The content isn’t heavily prescriptive, emphasizing tradeoffs rather than particular approaches. The closest thing to an overarching theme is an undercurrent of conservatism. It’s important to nail the fundamentals of autocomplete, best-first, and pagination before spending time on advanced search, faceted navigation, or personalization. A safe way to cover your bases is to exploit the distribution of queries:
In most cases, the analysis of search query data reveals a power law distribution and invites us to apply the 80/20 rule. A small number of unique search phrases accounts for a large percentage of the total queries. It has become a best practice for managers of large websites to integrate a simple database that matches these common search phrases to good starting points or destinations.
In short, a simple framework for improving your most popular queries will likely give better ROI than a complicated framework (eg. learning to rank) for improving everything simultaneously.
The one section that left something to be desired was the passage on structured results. They discuss some examples but don’t make any real recommendations. This is something I’m really interested in lately, and if you have a good reference please let me know.
The fifth chapter discusses optimizing for your vertical, platform, and audience. This was less interesting to me since I’m working in one particular space right now. That said, the key takeaways were:
- some search conventions are specific to a vertical
- search is only one path to discovery – don’t forget about SEO and Recsys
The final chapter opens with a discussion of methodology. Ethnographic research is mentioned before analytics. (This is an area where I have huge knowledge gaps, and I’m excited to learn more about formal methods in UX.) What follows are three sci-fi allegories that try to shed some light on the current state of search, and suggest directions that may be important in the near future. My favourite is “Semantic Singularity”, a parody of “The Semantic Web” and The Singularity is Near.
I won’t spoil the story – if you’ve made is this far in the review, you ought to buy a copy and read it yourself. It’s a rich, provocative resource and I expect to consult it frequently. I’ll end with a passage from the discussion of “Semantic Singularity”. It articulates why I’ve become convinced that search is a more important problem than more glamorous NLP applications like text summarization, conversational agents and entity extraction:
The idea that software agents will answer our questions (sometimes before we ask) via a conversation interface is seductive… But life and language are messy. Crosswalking for meaning across vocabularies and standards is difficult, if not impossible.
While artificial intelligence lies behind each vision, search is often the killer application.