Часть 7. Подведение итогов

Codeq's NLP API включает в себя модули суммирования текста, которые могут помочь вам определить наиболее релевантный контент из текстов. В этом уроке мы подробно расскажем, как использовать модули, связанные с обобщением текста, сжатием предложений и извлечением ключевых фраз.

Предыдущие уроки можно найти здесь:

Полный список модулей Codeq NLP API можно найти здесь:



Документация Codeq NLP API
Первое, что вам нужно сделать, прежде чем начать использовать Codeq NLP API, — это зарегистрироваться для создания идентификатора пользователя и ключа пользователя…api .codeq.com



Определите конвейер НЛП и проанализируйте текст

Чтобы вызвать Codeq NLP API, вам необходимо создать экземпляр нашего клиента Python SDK, используя ваши учетные данные API в качестве входных параметров. После этого вам нужно объявить конвейер, содержащий интересующие вас аннотаторы. Клиент и конвейер можно использовать для отправки текста и получения в ответ объекта Document, содержащего выходные данные нужных аннотаторов. Для быстрого обзора вывода вы можете использовать метод document.pretty_print(). Для каждого аннотатора в этом уроке мы подробно расскажем:

  • ключевое слово (KEY), используемое для вызова аннотатора,
  • атрибут (ATTR), в котором хранится вывод.
from codeq_nlp_api import CodeqClient

client = CodeqClient(user_id="USER_ID", user_key="USER_KEY")

pipe = [
    "summarize", "compress", "summarize_compress", "keyphrases"
]

text = "A gunman who shot dead a uniformed officer outside the Biloxi police station remained on the run on Monday, the subject of an intense manhunt along Mississippi’s Gulf coast. It was unclear what prompted the killing of Officer Robert McKeithen, a 23-year veteran who was scheduled to retire this year. Biloxi’s police chief, John Miller, said police did not know if he was targeted, or the victim of a random act. The animal that did this is still on the run, Miller told reporters. We’re going to do everything within our power to bring him to justice for Robert and his family. Authorities say the man approached McKeithen in the station’s parking lot on Sunday night and shot him multiple times, either before or after coming inside the station."

document = client.analyze(text, pipeline=pipe)
print(document.pretty_print())

Подведение итогов

Этот модуль генерирует на выходе извлекающую сводку с наиболее релевантными предложениями входного текста. В случае этого аннотатора вывод сохраняется на уровне объекта Document.

  • KEY: суммировать
  • ATTR: document.summary
pipe = [
    "summarize"
]

text = "A gunman who shot dead a uniformed officer outside the Biloxi police station remained on the run on Monday, the subject of an intense manhunt along Mississippi’s Gulf coast. It was unclear what prompted the killing of Officer Robert McKeithen, a 23-year veteran who was scheduled to retire this year. Biloxi’s police chief, John Miller, said police did not know if he was targeted, or the victim of a random act. The animal that did this is still on the run, Miller told reporters. We’re going to do everything within our power to bring him to justice for Robert and his family. Authorities say the man approached McKeithen in the station’s parking lot on Sunday night and shot him multiple times, either before or after coming inside the station."

document = client.analyze(text, pipeline=pipe)

summary = document.summary

print(summary)
# Output:
# 
# It was unclear what prompted the killing of Officer Robert McKeithen, a 23-year veteran who was scheduled to retire this year. Biloxi's police chief, John Miller, said police did not know if he was targeted, or the victim of a random act.

Сжатие предложения

Этот модуль направлен на создание из заданного предложения нового, более короткого, сохраняющего основную мысль оригинала, но, возможно, опускающего некоторые менее важные детали. Его можно рассматривать как аналог краткого изложения документа, состоящего из одного предложения.

Вывод этого модуля сохраняется на уровне предложения.

  • KEY: сжать
  • ATTR: предложение.compressed_sentence
pipe = [
    "compress"
]

text = "A gunman who shot dead a uniformed officer outside the Biloxi police station remained on the run on Monday, the subject of an intense manhunt along Mississippi’s Gulf coast. It was unclear what prompted the killing of Officer Robert McKeithen, a 23-year veteran who was scheduled to retire this year. Biloxi’s police chief, John Miller, said police did not know if he was targeted, or the victim of a random act. The animal that did this is still on the run, Miller told reporters. We’re going to do everything within our power to bring him to justice for Robert and his family. Authorities say the man approached McKeithen in the station’s parking lot on Sunday night and shot him multiple times, either before or after coming inside the station."

document = client.analyze(text, pipeline=pipe)

for sentence in document.sentences:
    raw_sentence = sentence.raw_sentence
    compressed_sentence = sentence.compressed_sentence
    if raw_sentence != compressed_sentence:
        print("original: %s" % raw_sentence)
        print("compressed: %s\n" % compressed_sentence)

# Output:
# 
# original: A gunman who shot dead a uniformed officer outside the Biloxi police station remained on the run on Monday, the subject of an intense manhunt along Mississippi's Gulf coast.
# compressed: A gunman who shot dead a uniformed officer outside the Biloxi police station remained on the run on Monday.
# 
# original: It was unclear what prompted the killing of Officer Robert McKeithen, a 23-year veteran who was scheduled to retire this year.
# compressed: It was unclear what prompted the killing of Officer Robert McKeithen.
# 
# original: Authorities say the man approached McKeithen in the station's parking lot on Sunday night and shot him multiple times, either before or after coming inside the station.
# compressed: Authorities say the man approached McKeithen in the station's parking lot on Sunday night and shot him multiple times.

Суммирование со сжатием

В этом случае аннотатор генерирует извлекающую сводку с наиболее релевантными предложениями входного текста в их сжатых формах, независимо от того, указан ли в конвейере сжатый аннотатор или нет.

Вывод этого модуля сохраняется на уровне документа.

  • КЛЮЧ:summary_compress
  • ATTR: document.compressed_summary
pipe = [
    "summarize", "summarize_compress"
]

text = "A gunman who shot dead a uniformed officer outside the Biloxi police station remained on the run on Monday, the subject of an intense manhunt along Mississippi’s Gulf coast. It was unclear what prompted the killing of Officer Robert McKeithen, a 23-year veteran who was scheduled to retire this year. Biloxi’s police chief, John Miller, said police did not know if he was targeted, or the victim of a random act. The animal that did this is still on the run, Miller told reporters. We’re going to do everything within our power to bring him to justice for Robert and his family. Authorities say the man approached McKeithen in the station’s parking lot on Sunday night and shot him multiple times, either before or after coming inside the station."

document = client.analyze(text, pipeline=pipe)

print("summary: %s\n" % document.summary)
print("compressed_summary: %s" % document.compressed_summary)
# Output:
#
# summary: It was unclear what prompted the killing of Officer Robert McKeithen, a 23-year veteran who was scheduled to retire this year. Biloxi's police chief, John Miller, said police did not know if he was targeted, or the victim of a random act.
# 
# compressed_summary: It was unclear what prompted the killing of Officer Robert McKeithen. Biloxi's police chief, John Miller, said police did not know if he was targeted, or the victim of a random act.

Извлечение ключевой фразы

Этот модуль отвечает за поиск для заданного документа списка коротких фраз, которые дают пользователю представление о темах, затронутых в документе. Например, для документов более технического характера извлеченные ключевые фразы должны включать технические термины, наиболее соответствующие теме статьи, тогда как для новостной статьи ключевые фразы должны включать имена людей, организаций и т. д., имеющие отношение к статье. .

Вывод этого модуля хранится на уровне документа в двух формах: список ключевых фраз в виде строк и список ключевых фраз в виде кортежей, включая их оценку релевантности.

  • KEY: ключевые фразы.
  • ATTR: document.keyphrases
  • ATTR: document.keyphrases_scored
pipe = [
    "keyphrases"
]
text = "A gunman who shot dead a uniformed officer outside the Biloxi police station remained on the run on Monday, the subject of an intense manhunt along Mississippi’s Gulf coast. It was unclear what prompted the killing of Officer Robert McKeithen, a 23-year veteran who was scheduled to retire this year. Biloxi’s police chief, John Miller, said police did not know if he was targeted, or the victim of a random act. The animal that did this is still on the run, Miller told reporters. We’re going to do everything within our power to bring him to justice for Robert and his family. Authorities say the man approached McKeithen in the station’s parking lot on Sunday night and shot him multiple times, either before or after coming inside the station."

document = client.analyze(text, pipeline=pipe)

print("Keyphrases:\n")
for k in document.keyphrases:
    print(k)

print("Keyphrases Scored:\n")
for k in document.keyphrases_scored:
    print(k)
# Output:
# 
# 
# Keyphrases:
# 
# Biloxi police station
# Biloxi 's police chief
# Officer Robert McKeithen
# the station 's parking lot
# Mississippi 's Gulf coast
# Sunday night and shot
# him multiple times
# Monday
# John Miller
#
# Keyphrases Scored:
#
# ['Biloxi police station', 0.14236053468171103]
# ["Biloxi 's police chief", 0.12844081612661434]
# ['Officer Robert McKeithen', 0.12583178746051182]
# ["the station 's parking lot", 0.11744640267914488]
# ["Mississippi 's Gulf coast", 0.11721120752467706]
# ['Sunday night and shot', 0.11115943448261123]
# ['him multiple times', 0.09358212954494648]
# ['Monday', 0.08635075551783025]
# ['John Miller', 0.0776169319819529]

Заворачивать

В этом руководстве мы описали некоторые модули API Codeq NLP, которые можно использовать для суммирования текстов и получения соответствующих ключевых фраз. В приведенном ниже коде приведены имена конвейеров для вызова каждого аннотатора и переменные, используемые для хранения их вывода: