Archive for the ‘semantics’ Category

Aristotle lived about 2,400 years ago, well before the advent of the Worldwide Web. Yet his ideas drive the still emerging Semantic Web. In fact, we could probably do a better job as modern information scientists if we paid a bit more attention to the ancient Greek philosopher.

In his writing called “Categories,” Aristotle addressed the problem of meaning in language and developed a logical framework for semantics. In this work, he invented the theory of subjects and predicates, which modern grammar and formal logic have adopted. This was in effect RDF version 0.0.0.

Aristotle also talked about using taxonomies (from the Greek τάξις + νόμος) to define the meanings of concepts, introducing “genus” and “species” as essential relationships. This approach was adopted by Linnaeus in the 18th Century to catalog the great diversity of life on earth; and more than a hundred years later, formal taxonomies made their way into library science.

Of special interest to us here is Aristotle’s classification of the predicates associated with definitions of meaning. He defined five types: genus, species, difference, property, and accident. The first two are already familiar to information scientists as IS-A relationships. A difference predicate relates to a defining characteristic for a concept. A property is an important characteristic for a concept, but not sufficient to define it. An accident is a true predicate that makes no contribution to meaning.

For example,

(genus/species) Angelina Jolie is an American movie star.
(difference) She is the daughter of American Actor John Voight.
(property) She trained with Lee Stasberg.
(accident) She visited Costa Del Sol.

In automated building of semantic dictionaries, our problem is with accidental predicates. Such predicates have only a weak relationship to a subject and tend to lead to noisy inferred associations. We probably do not want to retrieve a news item about Angelina Jolie given a query about Costa del Sol.

Unfortunately, many and perhaps most predicates in text data are accidental. In current data driven semantic learning systems, we make no distinction here yet, and so there are opportunities here for major improvements. A possible approach here is to employ the techniques of text summarization to identify the most important “predicates” in our data and thus bias our statistics away from accidents toward properties and differences. Aristotle would be amused.

The February issue of Scientific American had an article on the latest thinking about the Whorfian Hypothesis, which states that language strongly influences how humans think. This was a hot idea about sixty years ago, but eventually fell out of academic favor because of the lack of hard empirical evidence. Now that evidence is starting to show up, which has some implications for computational semantics.

The standard view on language and meaning has recently emphasized universality. This is to say that the understanding of language is hardwired in our heads, and so any competent human should qualify as an expert in the algorithmic delineation of meaning. The Whorfian hypothesis throws us a curve here in that we now have to consider language along with culture in our models of thought. A single well-crafted taxonomy or other semantic construct will not fit all.

We see something of this problem on the Worldwide Web. As Jimmy Wales noted this past week, the content of the Web, and Wikipedia in particular, is largely created by twenty- and thirty-something males and so is dominated by their interests. A set of semantic categories derived from the Web in general will certainly be insufficient for understanding text on finance or on medicine and may be challenged even when dealing with the pages frequented by twenty- and thirty-something females.

This does not mean that a given semantic scheme is invalid. Each scheme, however, is limited by the vocabulary it covers and in the kinds of distinctions that that it makes. That should be good news for those of us who make their living in computational semantics.

Watson, IBM’s Jeopardy computer, is showing everyone that its 900-pound gorilla of trivia and is likely to beat its human opponents. Watson could still do something stupid, but its formidable performance says much about the effectiveness of current natural language processing technology and computation resources.

Although Watson has a knowledge base of millions of documents gleaned from the Web, its weakness is that it really does not understand any of this data. It is just an extremely smart entity extraction system; Watson uses the terms of a Jeopardy clue as a selecting a particular entity as an answer, which of course then has to be phrased as a question. It has to figure what kind of entity to look for and what kind of context that entity would be found in.

In a sense, this is a simple kind of semantic search because it involves scanning its entire knowledge base of documents and scoring contexts statistically. The entities of the right kind in the highest-scoring contexts are then the prime candidates for an answer; and Watson can use their statistics to derive a level of confidence that a given candidate is the right answer. This basically relies heavily on brute computational power.

As can be seen in the Jeopardy competition, brute power can be quite effective. In most of the straightforward questions that one might expect that Google would do well on, Watson can simply outsearch its opponents. It can grab enough right answers in this way to make up for its frequent wrong answers on more subtle questions requiring a deeper understanding. This is as much gamesmanship as it is intelligence.

Now imagine how overwhelming Watson could be if it actually developed some understanding and made far fewer wrong answers. The first step in this direction is in fact quite easy: develop a large set of semantic categories corresponding to how humans understand language. Indexing a knowledge base by such predefined categories would have the immediate effect of simplifying the search process so that documents do not always have to be analyzed at the lowest linguistic level. That should allow the searches to be broader, much like allowing a chess computer to analyze more moves ahead.

We of course are in the business of semantic dictionaries, which provide a quick way of assigning semantic categories to text documents. Hey, Watson. If you are listening, give us a call.

Back in the 60′s and 70′s of the last century, the Whorfian hypothesis was a hot subject on college campuses. This was the idea that one’s native language, its syntax and semantics, strongly shaped one’s worldview. For example, Eskimos speaking Inuit supposedly had thirty different words for snow and so had a more complex relationship with their environment than someone speaking English with only one word for snow.

The problem of course is that skiers can make plenty of distinctions about kinds of snow even in English. Despite Whorfian hypothesis being theoretically attractive, it did not square in the end with our actual experience with language. That pretty much took the steam out of the Whorfian hypothesis, but now in the 21st Century, empirical support has been accumulating for a weaker version of it. This was the subject of an article in New York Times Magazine (http://nyti.ms/boqzs5).

The weak Whorfian hypothesis rejects the idea that language establishes an absolute limit on thinking. Thus we can learn about distinctions in types of snow if we really need them. The structure of a language, however, definitely can bias our thinking; and this could have consequences in practical matters like the ranking of retrieved documents. The choice of a particular semantic framework like RDF may therefore affect the performance of an information system in unexpected ways.

So far, experimental results on language and thought have focused on highly specific biases in areas of language like giving spatial directions, assigning gender to nouns, and dividing the spectrum into colors. It seems plausible, though, that this should generalize to the overall semantic problem of dividing up meaning into some kind of compact space. There is more than one way to skin a cat here, and there are probably advantages and disadvantages in each possibility.

A dogmatist might be tempted to argue here that RDF with certain standard taxonomies is the right way and everything else is wrong, but that is probably overreaching. We are not yet savvy enough about semantics to carve tablets in stone about its implementation. At present, one can say only whether a given scheme is optimal in some formal sense; but if it makes no obvious sense to people, then something more comprehendable might be better in the long run even if it is less than optimal.

The weak Whorfian hypothesis forces us to be more honest. If each semantic scheme introduces its own biases, then we need to experiment to see how different approaches work out for a given target application. Given that humans operate with more than one linguistic framework, we should not be so quick to assume than machines can do better at semantics with just a single framework.

Basics

5 Oct 2010

Linguists have long debated whether human language ability is innate or is simply learned by highly plastic neurocircuitry of a general sort. Recent studies with fMRI scans indicate, however, that cognitive skills like language understanding tend to be associated with highly specific brain locations across different individuals, supporting the idea that some kind of language-related structures exists. Studies of people impaired by strokes occurring in language regions also have shown this.

So when a young child learns that Mama is related to a concept of MOTHER, which applies to more than a single individual, this seems to draw upon specialized builtin logic within the human brain. This kind of symbolic capability is not unique to humans, being found to some extent in other large-brained social animals like elephants, whales, dolphins, and chimpanzees; but we certainly have more of it. This can seen in the relative size and organizational complexity of human brains.

The implication here is that concepts like MOTHER, BIRD, HOUSE, or FOOD are real in some sense at the genetic level. We of course do not necessarily all learn the same particular concepts; for example, speakers of different languages in different cultures can be expected to develop divergent concept frameworks. Nevertheless, it is possible to translate between unrelated languages like Inuit and English, meaning that there is still a large overlap in their lingistic repertories of concepts.

Consequently, when we technologists talk about incorporating semantics into search engines and other applications, we need to remember that semantics existed a long time before the first boolean electronic circuit and that what we call “semantics” should be consistent to what goes on in our own heads. This is perhaps only a marketing concern, but the business of selling semantic technology will be that much harder if we cannot agree on what we really mean.

The concept of CONCEPT would seem to be a focus point for semantics that everyone can grasp. Whether we approach language and meaning like Wittgenstein or like Russell or like Korzybski or like Chomsky or like Miller or like Berners-Lee, it helps to get grounded properly.

Because my wife is petite in size, she has difficulty finding clothes and shoes that fit and, being of a certain age, no longer has the option of shopping in the teens section. One understands exactly how this situation came about. Some researchers developed a demographic profile of women in the United States and then applied a multivariate optimization algorithm to calculate the mix of sizes that would generate the most profit for manufacturers. If you happen to be in the tail of the demographic profile, you are just out of luck.

We see something similar happening on the Semantic Web. Developers seem to favor global solutions, which are often highly optimized for capturing content deemed to be the most important somehow. The problem, though, is that this strategy discounts the long tail of distributions, which is unfortunate, since pundits see long-tail processing as the new frontier for Web applications. As with clothing and shoes, an optimal mix of sizes does not fit all.

How we each use our words in language depends on how we learned to speak; and each of us has had a unique history. The differences in our language are most evident when we look at text in specialized areas like medicine, law, government, or technology; but even in “normal” discourse on the Web, we see jargon or unusual usages of common terms that may be missed by some kind of global semantic solution.

This is not to say that global solutions are invalid. They are quite useful in the absence of better information about what someone means; but the Web is moving more towards customization and personalization and localization. Our semantic frameworks should follow that lead. It means that we have to streamline our automated learning of semantic concepts so that we can in fact support individual solutions at least in part.

Hijacked

26 May 2010

Has this ever happened to you? You are Googling for information on the Web, but inadvertently your query happens to share keywords with the latest cultural phenom: the next tweener heart throb, a YouTube video suddenly gone viral, or yet another paranoid political fantasy that refuses to die.

You are a professional, however, and so switch into Advanced Mode to reshape your query, but to no avail. Your information has been buried under pop detritus; it has been hijacked by the maximum likelihood estimate (MLE) on the Web.

At times like this, you want to grab your search engine by the neck and shout, “I am NOT a screaming twelve-year-old girl into dancing cats and fixated on the President’s birth place!” But your search engine continues blithely in the wisdom of the crowd.

It is a reminder that statistically grounded information systems are at the mercy of their training data. If we cede too much control of a system to its finely wrought black box judgment, then we sometimes are going to run off the tracks. This is especially true with web semantics.

If we do in fact want to get under the hood to adjust a semantic system to go against the popular flow, then it helps tremendously if the categories underlying the representation of document content are intelligible to people. Such transparency is a prime motivation for how semantic dictionaries are currently built by TextWise.

Of course, if you care nary a lick about transparency, then may I interest you in this slightly used synthetic collateralized debt obligation….

Going Deep

12 May 2010

When people read text, they may not understand everything in it. For example, a layman might look at an article from a medical journal and see only that it is about some kind of drug. Someone more familiar with medicine would pick up that this is an experimental drug for treating estrogen-sensitive breast cancer. An expert would note that the drug is an aromatase blocker that performs as well as a standard approved drug in a double-blind controlled trials with a large sample of patients.

If an application seeks simply to distinguish documents about pharmaceuticals from documents about toxic financial assets or about the World Cup tournament in South Africa, then it is enough to understand at a superficial level. If a physician is searching for treatment options for a patient with a recurrence of breast cancer, however, a much deeper grasp of content is called for.

A general type of semantic dictionary covering a broad variety of different subjects is more or less forced to opt for broad coverage by default. Collecting enough training data for two thousand dimensions is a major undertaking; having to do it for twenty thousand dimensions will entail a big commitment of resources that one will have to justify. Still, if such a dictionary is critical for a given application, then we need to make the investment.

In many cases the domain of content to be covered can be quite circumscribed. Accordingly, we probably would be better off to add a fairly small number of dimensions to an existing semantic dictionary rather than build a whole new dictionary from scratch. This will require some special statistical balancing of course, but balancing is what dictionary building is all about.

Perfect What?

22 Mar 2010

We have been musing about the true topology of semantic spaces and how this affects our concept of dimensionality. This segués logically into a hot area of contention. In our linear approximation of meaning, how many dimensions do we really need and what should they be?

Some people prefer to approach this problem mathematically. Given a representative sample of documents to describe semantically, we can look at the relationship between terms and documents as a defining a vector space. One can then apply the method of singular vector decomposition (SVD) to find a minimal set of basis vectors to span that space. These singular vectors are like eigenvectors on steroids.

If you have actually read this far into this blog, then you will know that we (TextWise) have a competitor that employs SVD for semantic analysis. We get asked all the time why we have stuck with basic statistical techniques when we could instead be rigorously mathematical. Our usual response is that we have much faster turnaround in building semantic dictionaries, finer-grain descriptions of content, and more intuitive concepts overall.

There are more fundamental concerns, however, both theoretical and practical. On the theoretical side, SVD might be pushing a linear-space semantic model too far if meaning is in fact topological complex. More significantly on the practical side, though, is that one might be getting caught in the common problem of overtraining.

Suppose that we have a hundred thousand blog posting to which we apply SVD to get some optimal set of dimensions for analyzing their content. What then happens next week when we get a million new blogs that we have never seen before? Our perfect basis set is now distinctly handicapped.

Now we could try to reprocess all our data here, but SVD is so computationally intensive as an algorithm that it probably will be too slow to keep up without superextraordinary investments in hardware resources. We also would end up with an unstable system in which it is quite difficult to compare results from one week to the next. Anyway, we made our choice here.

People in the information sciences are fond of high-dimensional vector spaces as models of document content. These are in fact only approximations of reality, however; and in the specific case of semantics, they are probably an oversimplification. We already know something about how the neural circuitry in our brains work when we process the meaning of language; we can find no clean finite-dimensional linear space in the tangle of our synapses.

Neural imaging like PET does support the theory that linguistic concepts correspond to particular clusters of neurons connected in fairly complex feedback loops. Our understanding here is still quite limited, though. We do not know how many such clusters exist or how widely they are distributed. Visual concepts are in a different part of the brain than auditory concepts, for example; and overall, we have not yet found any obvious switchboard, say in the hippocampus, that could somehow tie everything together neatly.

In our computational semantic model, we assume that all concepts are independent and equal. That seems to work in semantic dictionary applications when we have thousands of concepts of concepts as dimensions, but an espistemologist here would have the lurking suspicion that our actual semantic space has to be some kind of complex manifold with all kinds of holes and twisting surfaces like a deranged n-th-order Moebius strip. Meaning is messy.

Our linear Euclidean model may therefore be valid only in a small local region of our actual semantic space, but in practice, that is really where all our apps have to live. One cannot presume to comprehend all possible content in text. We can only slice off a small piece of the pie of meaning, and until world peace and perfect enlightenment break out, that is a good start.