Archive for the ‘General’ Category

We have been thinking lately about how many dimensions a semantic dictionary should have. Some researchers at Carnegie-Mellon have been approaching the same question from the perspective of neuroscience and real-time imaging of activity in the human brain while understanding language (http://bit.ly/buIZEx).

According to CMU, there are really only THREE basic semantic dimensions: (1) Can I eat it? (2) Can I pick it up? (3) Can I hide in it? Admittedly, this primitive partitioning of the world probably goes back to our primate origins, but does have a certain resonance. Let’s remember it the next time we try to categorize journal articles in nanotechnology or search postings on someone’s Facebook wall.

Even in the world of print, one dictionary is often not enough. Just for English, for example, we can go to standard references like Webster’s Third New International, The American Heritage Dictionary of the English Language, or the Oxford English Dictionary, as well as more specialized lexicons. So how many semantic dictionaries do we really need?

That of course depends on the application. If we are in the situation where our target text data is extremely stable and requires only a general vocabulary, then we might get away with a single semantic dictionary based on a large sample of data processed quite carefully. On the Web, however, we have nothing of the sort, if you haven’t noticed lately.

A sophisticated dictionary that took weeks to build with hairy mathematical algorithms on a reasonable sample of training text may become obsolete overnight. That is not to say that sophisticated dictionaries are unhelpful; but in the merciless competition of the information marketplace, we probably need to be able to pop out a new semantic dictionary based on a gigabyte or more of text in just hours.

Given this kind of turnaround, why would anyone want to rely on a single semantic dictionary with its limited vocabulary and somewhat dated concepts? A new dictionary will of course involve a nontrivial upfront investment, but once a reliable source of tagged data is developed, actual dictionary building can be largely automated. That is the advantage of relying on statistical methods.

One of the challenges with creating and maintaining applications for the web is keeping up with all of today’s different web browsers and their differing under-the-hood technologies and functionality.  New versions of browsers and operating systems are released frequently for a number of reasons, such as feature enhancements to security fixes.  There is a wide variety of web browsers available today, each offering something a bit different from the others.  Operating system vendors have their own, some of them are cross-platform and work on other operating systems, then there are the third-party browsers, and we haven’t even explored the mobile browser realm yet…  Creating and maintaining a set of browser and OS combinations as a company standard toward which applications can be developed and tested has become key for us.

Our standard has been created using statistics on browser and OS usage from W3Schools, broken down by brand and version.  By collecting this data and observing trends over time, we can decide when it’s appropriate to either start or discontinue supporting a browser, OS, or combination of the two.  Our process is to evaluate our browser/OS support matrix each time a new major or minor version of a browser or OS is released, or at most every 6 months (assuming no browser or OS updates have occurred).  Doing an evaluation of the statistics is important even if no updates have occurred, because some browsers may fall below a percentage of use needed for support, or others may have increased enough in usage or popularity to now be supported.

It’s also important to be able to test those combinations to ensure compatibility.  Rather than bearing the expense of having every possible combination in-house, we use a service on the web that specializes in providing those tools to help us test.  The service that we use is called BrowserCam, which gives us the ability to take “snapshots” of our applications in various browser/OS combinations on the web, and remote access on those machines for interactive testing.  And to answer the original question, we have no idea – PlanetWeb2.6 on Dreamcast is not one of our supported combinations.

I’ve worked in Product Management for years, and for years our battles were ones with well-known, battle weary opponents.  The battle lines had been drawn, strategic alliances were made and broken, and weapons were fired (i.e., fingers were pointed) if a product flopped, forecasts weren’t met, or the “biggest prospect ever” walked away.

Sales blamed Marketing, Sales blamed the Product Manager and vice versa, Developers blamed the Requirements Analysts and QA (bad requirements and bad QA people for not catching the bugs prior to launch), and the CEO blamed everybody, but always came back with donuts and cookies for us to enjoy as we licked our collective wounds.

I hate using the whole war/battleground analogy because in the end we’re all on the same team, working toward the same goal.  And if your company is in the business of selling products and services, the goal typically boils down to the bottom line – profitability.

Just over a year ago, I embarked on a new path in my Product Management journey.  I’m now working as a Product Manager for a company whose underlying technology is semantic web technology.  This means we’ve got PhD’s and patent holders with specialties in things like informatics, search engine technology, information retrieval and relevance testing.  So in addition to engineers wanting to continually improve the system and its architecture, I’ve now got scientists who want to (and rightly so) continually improve the core technology, the algorithms.  Tweak, tweak, tweak.  They go to conferences like SIGIR and come back fired up, ready to go. They’re like a new battalion that has shown up on the battlefield and I have to size them up quickly in order to protect my precious project timeline. “Are you friend or foe?”

So I find myself with an interesting dilemma.  How do you tell a scientist to think faster, come to a conclusion faster, improve the technology but get it done NOW so we can get the product built and out the door?  How do I get the engineers to wrap it up? Is it research or is it a project?  I need an answer, and what I keep hearing from all camps is, “It’s both!”

I can hear you Product Managers and Project Managers saying it now.  Just continue with new product development and merge the R&D advancements into the products as they’re ready.  I used to say and do that too, and it worked.  But that was before I landed in a place where what works for typical Web data doesn’t quite work for patent data, and none of it compares to that elusive Holy Grail called the medical domain.  And even tweaks in certain parts of the engineering code can potentially affect relevance and skew the results.

Have I given up on my beloved project schedules, my requirements documents, my product development lifecycle tools?  NEVER. I’m a Product Manager at heart. But just like the ever changing world of the semantic web, and the scientists and engineers I interact with daily, I’m tweaking my process as I go.

A former colleague of mine used to have an entire can of soup for lunch every day. We razzed him about this, but he shook us all off until one day, I looked at the nutrition label on the can. That soup had 1800 mg of sodium altogether! We gravely informed him of this fact, and the soup was soon history.

To understand this story, you would have to know that the recommended daily maximum dietary intake of sodium for an adult is about 900 mg. Without this context, the number 1800 really means nothing. So what do all those numbers in a semantic dictionary mean, if anything?

The key property of semantic dictionary numbers is that they are based on probabilities and so have to fall between 0 and 1. They measure the likelihood that a document containing a given term is related to a given semantic dimension. For example, a dictionary weight of 1.0000 for a term and a dimension would indicate that a document containing the term is absolutely associated with the dimension.

There is a complication here, however. In real life, nothing is ever so certain. If we saw a 1.0000 term weight for a dimension, a more reasonable interpretation is that our sample of training data was too small for estimating the probability of that term accurately. A similar problem arises for a dictionary weight of 0.0000.

In general, a statistician will be highly suspicious of any extreme probabilities like 1.0000 and 0.0000. As a proponent of statistical technology, we have to make a special effort to avoid such probability estimates in our semantic dictionaries. In contrast, certain other mathematic approaches to semantics tend to skate over niceties like this, choosing just to plug in numbers to what is essentially a fixed formula.

If one is careless about the meaning of numbers, though, how can one be careful in capturing the actual meaning of words?

If you have studied the mathematics of linear spaces, then you know that there are infinitely many ways to represent a given point in a space as a set of coordinates. For any particular set of data points, however, one can mechanically derive a particular set of axes that results in the representation requiring the fewest coordinates to capture the most important characteristics of those points.

This is the principle behind Latent Semantic Indexing (LSI), which was in large part why Susan Dumais of Microsoft received the Salton prize at the recently concluded SIGIR Conference. So, why don’t we use LSI?

It all boils down to whether one believes that a linear space is good model for the semantics of natural language; and the main issue here is that of orthogonality. Orthogonality is great in a Pythagorean ideal world, but the real world tends to be quite messy. A rectangular grid can be imposed on places like the U.S. Midwest, but would be quite inappropriate for land management or road building in the Amazon Basin or in Siberia.

An orthogonal system disregards the landscape, which is in fact what we have to live with and in.  Two towns ten miles apart along a navigable river are in a sense closer than two towns five miles apart with a mountain range between them. Our approach to semantics is that of conforming to the landscape of text data, which is probably better described as being fractal than being orthogonal.

Iron Semanticist

12 Aug 2009

Some people have been disparaging a statistical approach to the semantics of natural language. This is essentially a kind of prejudice, as if we came from the wrong side of the technology railroad tracks. It ignores the fact that statistical approaches have performed spectacularly well in some high profile settings.

Have you ever watched the “Iron Chef” on the Food Network? This is where two competing chefs are given an ingredient kept secret until the start of the show, and each contestant then has 60 minutes to create an entire meal around that ingredient. A panel then judges and critiques the two meals and crowns a winner.

In 2003, DARPA ran its own version of “Iron Chef,” though with only a single team of collaborators from eleven academic institutions across the U.S. The team was given a language, with the task of creating a cross-language information retrieval system and a machine translation system within TEN DAYS after learning what the language actually was.

To make challenge harder, the language was not French, Arabic, or Russian, but Cebuano, a dialect spoken in the Philippines. None of the team was familiar with the language, but through the magic of Internet collaboration, they were able in ten days to collect a corpus of resources in Cebuano and English and apply statistical methods to create both a fully workable cross-language retrieval system and a credible start to a translation capability.

The two principal investigators of the Herculean exercise wrote afterward that, given what they learned in those ten days, they would do better next time. They predicted that their team could  build a fully working statistical machine translation facility for a specified language in just a single day given adequate linguistic and computational resources.

In ten days, you could not build even a parser for a language that you have never heard of, much less develop the semantic mapping of that language into some kind of logical model of meaning to support cross-language search and machine translation. Statistical methods do work in semantics.

Our Roots

11 Aug 2009

Semantic Signatures℠ approaches meaning of words from the perspective of their context. In the past couple of months, there has been extensive discussion here and elsewhere about how this differs from RDF, the basis for the Semantic Web. The simplest answer is that we are data-driven where RDF is model-driven.

This dichotomy is nothing new. In fact, if we look at semantics over a hundred years ago, we see the empirical idea of contextual semantics in the structural linguistics of  Ferdinand de Saussure in contrast to the logical formulation of meaning in the predicate calculus of Bertrand Russell and Alfred North Whitehead. The former inferred meaning from the comparative analysis of text; the latter defined a mapping between text and a formal model of possible meanings.

The model-driven approach became less popular after the logician Kurt Gödel proved the incompleteness of all non-trivial logical systems in the 1930′s. Structural linguistics then became the favored approach until Noam Chomsky put the study of language back on a formal basis in the 1950′s, and the semantics of language also tilted to the formal in order to be more consistent with the study of syntax.

This is not to say that one approach is right and the other is wrong. The choice of approach to take should really depend on one’s circumstances. If one has available an appropriate logical model, which today might correspond to a taxonomy and a formal way to relate taxonomic entities, then the model-driven option is compelling. On the other hand, if an appropriate model is lacking or incomplete, but there is plenty of tagged text data to work from, then the data-driven option should be considered.

One can always in fact choose to work with the best of both worlds. We are not the sole providers of data-driven semantic technology, but our statistical characterization of meaning is probably de Saussure himself might have done it if he had access to the Worldwide Web and 21st Century cloud computing.

The current SemanticHacker API offers more than one semantic dictionary. Each one is crafted from a particular collection of categorized documents at a particular time. The choice of a dictionary depends on one’s target application. Ideally, that dictionary will be trained on categorized documents similar to the documents to be analyzed for content.

Currently, the two main types of dictionaries available in English come from the ODP conceptual hierarchy and the USPTO class hierarchy. The dimensions defined for these types have practically no overlap. The differences in language and vocabulary in training data are also huge, and these have major consequences in the dimensional weights computed for terms in the two dictionaries.

In theory, one could employ a USPTO dictionary in a general web application, but one then risks being unable to pick up on popular language and culture. You won’t find “lol” or “Brangelina” in any patent. Similarly, an ODP dictionary may be a bit thin for handling medical journal articles; it would be much better here to have a semantic dictionary trained specifically on medical language and vocabulary.

The cost of building a specialized dictionary varies, mostly due to the complicated legal, technical, and logistical process of collecting the proper training data. Once the data is obtained, however, the actual dictionary process is largely mechanical, although we do carry out extensive quality assessment to determine whether we are running under optimal dictionary building parameters.

With proper training data in hand, we can turn out a semantic dictionary of about 200,000 terms over 2,000 dimensions in only about a day. This turnaround is possible because of our reliance on statistical methods as opposed to more complicated mathematical modeling of other semantic approaches. It means that we could build new dictionaries fast enough to keep up with news cycles as short as one week, given the computational resources needed.

Some linguists believe that early language was always about specific entities–that is, denotational; this kind of reference then evolves into concepts, which are connotational. For example, a baby learns his or her particular meaning of MAMA, which then generalizes into MOTHERHOOD.

We can still see such evolution at work today. About two years ago, the term “Sarah Palin” was only denotational, but after the fall of 2008, it has now become quite connotational. Something similar might be said on the other side of the political spectrum about the term “Barack Obama.”

The whole process of turning denotation into connotation has been extensively studied and is better known as “branding.” Anyone who has ever written a resumé has had experience in doing it.

A semantic dictionary in fact trades on the natural kind of branding. Since we do statistical analyses of context to assign meaning, We may not yet be able to interpret a term for which we have only a small sample of occurrences. Give us a little time, though.