Python Natural Language Processing (NLP) Exercises : From Basics to BERT by Joshua K. Cage

Author:Joshua K. Cage [Cage, Joshua K.]
Language: eng
Format: azw3
Published: 2020-10-03T00:00:00+00:00

Unlike morphological analysis tools such as Mecab, the Sentence Piece needs to be pre-trained with a text file to perform word segmentation. It is.

Code:

import sentencepiece as spm

spm.SentencePieceTrainer.Train(

'--input=savior.txt, --model_prefix=sentencepiece --character_coverage=1.0 --vocab_size=2000'

)

Output:

The arguments given and their meanings are as follows

Argments

Explanation

--input=savior.txt

Input text file is used as the input text file. The input file is assumed to be a single-column file with no header and separated by a newline code.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.