Steps of NLP

Sentence segmentation determines how a text is divided into sentences for further processing. (Palmer, D.D., 2000)

For example, the original text could be a paragraph, "Hello, world. This page is about sentence segmentation. It is not always easy to determine the end of a sentence."

After the implementation of sentence segmentation, the computer will split the paragraph into sentences.

• "Hello, world."

• "This page is about sentence segmentation."

• "It is not always easy to determine the end of a sentence."

Tokenisation is a way of separating text into smaller units called tokens.

It is very similar to sentence segmentation, but this process further breaks down sentences into words and punctuation. (Dr Monica, 2020)

For example, the original text could be a sentence like, "How are you?"

After implementing tokenisation, the sentence splits into individual words and punctuation.

• "How"

• "are"

• "you"

• "?"

POS tagging labels each word in a sentence with its appropriate part of speech.

Parts of speech include nouns, verbs, adverbs, adjectives, pronouns, conjunction and their sub-categories. POS tagging is a useful step as it helps to simplify many different problems in NLP.  (Dr Monica, 2020)

For example, the original text could be a sentence like, "You just gave me a scare."

After implementing POS tagging, each word in the sentence is classified as a noun, verb, adverb, adjective, pronoun, conjunction and their sub-categories.

• "You": personal pronoun

• "just": adverb

• "gave": verb, past tense

• "me": personal pronoun

• "a": determiner

• "scare": noun, singular

These are words that add little value to the meaning of the document.

Stopwords are common words within sentences that don't add value and thus are eliminated before analysis. The removal of stopwords is useful as it means less data for the computer to process, which could translate to faster processing times and improvement in the accuracy of the program. The removal of stopwords must be completed after tokenisation. (Dr Monica, 2020; Roldós, I., 2021)

For example, the original sentence could be, "Hey, Amazon, my package never arrived."

After the implementation of tokenisation and the removal of stopwords, the sentence will look like this, "Amazon package never arrived". You can see that removing the words 'Hey' and 'my' makes the sentence simpler.

Lemmatisation is the process of grouping words based on root definition. It allows us to differentiate between present, past, and indefinite.

For example, under lemmatisation,

• jump = jump

• jumps = jump

• jumping = jumping

• jumped = jumped

(Roldós, I., 2021)

1940s

Machine Translation

NLP originated from the idea of machine translation (MT), which came into existence during the second world war.

This was intended to convert one human language to another human language. For example, translating English to Russian and vice versa, as they were both the primary language used back then. (Kaur, J., 2022)

1960s

The Decline of NLP

In 1957, Noam Chomsky revolutionised previous linguistic concepts, concluding that the sentence structure would have to be changed for a computer to understand a language. In 1958, the programming language, Locator and Identifier Separation Protocol (LISP), a computer language still in use today, was released by John McCarthy.

In 1964, ELIZA, a "typewritten" comment and response process, was designed to imitate a psychiatrist using reflection techniques. It did this by rearranging sentences and following relatively simple grammar rules. However, there needed to be a better understanding of the computer part.

Also, in 1964, the US National Research Council (NRC) created the Automated Language Processing Advisory Committee (ALPAC) to evaluate the progress of NLP research.

In 1966, NRC and ALPAC initiated the first AI and NLP stoppage because, after 12 years of research and $20 million, machine translations were still more expensive than manual human translations. In addition, there were still no computers that came anywhere near being able to carry out basic conversations. (Foote, K. D., 2019)

1980s

Re-emergence with AI

The stoppage of AI research between the 1960s and the 1980s "initiated a new phase of fresh ideas, with earlier concepts of machine translation abandoned and new ideas promoting new research, including expert systems. The 1980s initiated a fundamental reorientation, with simple approximations replacing deep analysis, and the evaluation process became more rigorous." It was a time when computational grammar became a very active field of research linked with the science of reasoning for meaning and considering the user's beliefs and intentions.

LUNAR, developed in 1978 by W.A Woods, could analyse, compare and evaluate the chemical data on a lunar rock and soil composition that was accumulating due to Apollo moon missions. (Kaur, J., 2022; Foote, K. D., 2019)

1990s

Increase in NLP Uptake

In the 1990s, the popularity of statistical models for NLP analyses rose dramatically. The pure statistical NLP methods "have become remarkably valuable in keeping pace with the tremendous flow of online text." In addition, grammar, tools and practical resources related to NLP/MT became available with the parsers.

(Kaur, J., 2022; Foote, K. D., 2019)

Present

Latest NLP Trends

Nowadays, everybody wants a machine to talk. The only way a computer can speak is through NLP. Take the example of Alexa, a conversational product by Amazon.

A query is passed to it by the medium of voice, and it can reply by the same medium. The query can be used to ask anything, search for anything, play songs or even book cabs.

Alexa is only one example. NLP can be applied to the healthcare sector, among many others. Furthermore, it is used for cognitive analytics, sentiment analysis, spam detection, and recruitment.