Machine Learning with TensorFlow, Second Edition MEAP V07 by Chris Mattmann

Author:Chris Mattmann
Language: eng
Format: mobi, epub
Publisher: Manning Publications Co.
Published: 0101-01-01T00:00:00+00:00

if len(tt_idx) == 2: #G

row_idx = tt_idx[0]

col_idx = tt_idx[1]

cnt = t_df.loc[t_df.FirstPoS==row_idx, col_idx]

cnt = cnt + 1 #G

t_df.loc[t_df.FirstPoS==row_idx, col_idx] = cnt

tt_idx.clear()

elif len(tt_idx) == 1 and tags[j][i] == "sent": #H

row_idx = tags[j][i-1]

col_idx = tags[j][i]

cnt = t_df.loc[t_df.FirstPoS==row_idx, col_idx]

cnt = cnt + 1 #H

t_df.loc[t_df.FirstPoS==row_idx, col_idx] = cnt

tt_idx.clear()

#A Iterate each sentence in the PoS tagged corpus dataframe

#B Initialize our row, column index (tt_idx) to a blank list – will only ever have 2 PoS tags

#C Iterate the tags for this sentence from the tagged corpus

#D Add the first element of our row,column index as the current PoS tag for the sentence

#E If it’s any sentence after the first, and it’s the first tag, then the first element is always end of sentence tag from the prior sentence

#F Increment count at row index and column index

#G If we have 2 elements for row,column index, then we are ready to increment the count there

#H If we reached the end of sentence pos tag for the column then we grab the prior tag for our row

index and increment the count

Once you have the counts of the co-occurring PoS tags in the matrix, you need to do a little bit of post-processing to turn them into probabilities. You sum the counts for each row in the PoS bigram count matrix, and then divide each cell count in that row by the sum. This whole process computes the transition probabilities and the initial probabilities and because it involved quite a few steps, I’ve captured the key steps of it for you in figure 10.8, which is handy when taking a look at listing 10.13. Phew! Lots of work, just to do that. Now you will work on the emission probabilities. Aside from a couple of more algorithms, it’s fairly straightforward, so you will get started computing them because it’s the last thing that you will need before running the HMM.

Listing 10.13 Post processing the transition probability matrix

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.