spencermountain/compromise

💫 Industrial-strength Natural Language Processing (NLP) in Python

Simple, Pythonic, text processing--Sentiment analysis, part-of-speech tagging, noun phrase extraction, translation, and more.

CoreNLP: A Java suite of core NLP tools for tokenization, sentence segmentation, NER, parsing, coreference, sentiment analysis, etc.

NLTK Source

Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Unsupervised text tokenizer for Neural Network-based text generation.

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.