NLP Development

Natural language processing (NLP) is the forgotten stepsister of the information technology industry. While large-language models (LLM) have captured the world's imagination & the wallets of Silicon Valley (ad)venture capitalists, the field that spawned the models hasn't gotten its due.

For those who are unfamiliar, NLP is a field of Artificial Intelligence (AI) that focuses on development computers that have the ability to understand & use natural language. Its history has been fraught. In the 1950s, one of the first hype cycles took place. Researchers said they could have a computer translate a few Russian sentences. Journalists magnified this into the development of a translator. Of course, the hype went bust but they lost funding in the process.

If you go on Twitter, you'll be able to find tweets for all types of topics, including CSAM! Elon Musk aka Apartheid Clyde (he loves that nickname) has certainly improved the safety along with the search function's effectiveness. However, Social Network Analysis (SNA) Twitter lacks a corpus of tweets. LLMs are superior at SNA so long as you feed it enough .csv files. Thus, my surprise at not finding anything about leveraging LLMs. Academics have been quiet, with few pre-prints to review.

Despite my lack of the required mathematics & little experience conducting formal social scientific investigations—especially with quantitative methods—I decided to try out what a model is capable of.

We are grossly underutilizied models. It is trivial for a model to do the linguistic crunching necessary to spit out all kinds of social network analysis. The applications are numerous: public relations firms determining the extent of public opinion for their clients; government departments trying to figure out what the real impact was of a publicity campaign beyond crude metrics like views; & detecting if someone online is being bullied or close to committing suicide & intervening.