# Machine Learning :: Text feature extraction (tf-idf) – Part II

Read the first part of this tutorial:Text feature extraction (tf-idf) – Part I.

This post is a延续of the first part where we started to learn the theory and practice about text feature extraction and vector space model representation. I really recommend you阅读第一部分of the post series in order to follow this second post.

Since a lot of people liked the first part of this tutorial, this second part is a little longer than the first.

### Introduction

In the first post, we learned how to use theterm-frequency以表示在矢量空间的文本信息。然而，与术语频率方法的主要问题是，它大大加快了频繁的条款和规模下降，这比高频方面经验更丰富罕见的条款。基本的直觉是，在许多文件中经常出现的一个术语不太好鉴别，真正有意义的（至少在许多实验测试）;这里最重要的问题是：你为什么会在例如分类问题，强调术语，是在你的文档的整个语料库几乎礼物？

The tf-idf weight comes to solve this problem. What tf-idf gives is how important is a word to a document in a collection, and that’s why tf-idf incorporates local and global parameters, because it takes in consideration not only the isolated term but also the term within the document collection. What tf-idf then does to solve that problem, is to scale down the frequent terms while scaling up the rare terms; a term that occurs 10 times more than another isn’t 10 times more important than it, that’s why tf-idf uses the logarithmic scale to do that.

### Vector normalization

Suppose we are going to normalize the term-frequency vector$\vec{v_{d_4}}$that we have calculated in the first part of this tutorial. The document$d4$从本教程的第一部分中有这样的文字表示：

d4: We can see the shining sun, the bright sun.

$\vec{v_{d_4}} = (0,2,1,0)$

To normalize the vector, is the same as calculating theUnit Vectorof the vector, and they are denoted using the “hat” notation:$\hat{v}$. The definition of the unit vector$\hat{v}$一个向量的$\vec{v}$is:

$\的DisplayStyle \帽子{V} = \压裂{\ vec的{V}} {\ | \ vec的{V} \ | _p}$

Where the$\hat{v}$is the unit vector, or the normalized vector, the$\vec{v}$is the vector going to be normalized and the$\|\vec{v}\|_p$是矢量的范数（大小，长度）$\vec{v}$in the$L^p$空间(别担心,我将解释它)。

The unit vector is actually nothing more than a normalized version of the vector, is a vector which the length is 1.

But the important question here is how the length of the vector is calculated and to understand this, you must understand the motivation of the$L^p$空间，也被称为Lebesgue spaces.

### Lebesgue spaces

Usually, the length of a vector$\vec{u} = (u_1, u_2, u_3, \ldots, u_n)$is calculated using theEuclidean norma norm is a function that assigns a strictly positive length or size to all vectors in a vector space-, which is defined by:

$\|\vec{u}\| = \sqrt{u^2_1 + u^2_2 + u^2_3 + \ldots + u^2_n}$

$\displaystyle \|\vec{u}\|_p = ( \left|u_1\right|^p + \left|u_2\right|^p + \left|u_3\right|^p + \ldots + \left|u_n\right|^p )^\frac{1}{p}$

and simplified as:

$\的DisplayStyle \ | \ VEC【U} \ | _p =（\总和\ limits_ {I = 1} ^ {N} \左| \ VEC {U】_i \右| ^ P）^ \压裂{1} {P}$

When you read about aL1-norm, you’re reading about the norm with$p=1$, defined as:

$\displaystyle \|\vec{u}\|_1 = ( \left|u_1\right| + \left|u_2\right| + \left|u_3\right| + \ldots + \left|u_n\right|)$

Which is nothing more than a simple sum of the components of the vector, also known as出租汽车距离, also called Manhattan distance.

Taxicab geometry versus Euclidean distance: In taxicab geometry all three pictured lines have the same length (12) for the same route. In Euclidean geometry, the green line has length$6 \times \sqrt{2} \approx 8.48$，并且是唯一的最短路径。
Source:维基百科::出租车通用电气ometry

Note that you can also use any norm to normalize the vector, but we’re going to use the most common norm, the L2-Norm, which is also the default in the 0.9 release of thescikits.learn. You can also find papers comparing the performance of the two approaches among other methods to normalize the document vector, actually you can use any other method, but you have to be concise, once you’ve used a norm, you have to use it for the whole process directly involving the norm (a unit vector that used a L1-norm isn’t going to have the length 1 if you’re going to take its L2-norm later).

### 返回矢量归

Now that you know what the vector normalization process is, we can try a concrete example, the process of using the L2-norm (we’ll use the right terms now) to normalize our vector$\vec{v_{d_4}} = (0,2,1,0)$in order to get its unit vector$\hat{v_{d_4}}$. To do that, we’ll simple plug it into the definition of the unit vector to evaluate it:

$\hat{v} = \frac{\vec{v}}{\|\vec{v}\|_p} \\ \\ \hat{v_{d_4}} = \frac{\vec{v_{d_4}}}{||\vec{v_{d_4}}||_2} \\ \\ \\ \hat{v_{d_4}} = \frac{(0,2,1,0)}{\sqrt{0^2 + 2^2 + 1^2 + 0^2}} \\ \\ \hat{v_{d_4}} = \frac{(0,2,1,0)}{\sqrt{5}} \\ \\ \small \hat{v_{d_4}} = (0.0, 0.89442719, 0.4472136, 0.0)$

And that is it ! Our normalized vector$\hat{v_{d_4}}$has now a L2-norm$\|\hat{v_{d_4}}\|_2 = 1.0$.

### 术语频率 - 逆文档频率（TF-IDF）重量

Now you have understood how the vector normalization works in theory and practice, let’s continue our tutorial. Suppose you have the following documents in your collection (taken from the first part of tutorial):

Train Document Set: d1: The sky is blue. d2: The sun is bright. Test Document Set: d3: The sun in the sky is bright. d4: We can see the shining sun, the bright sun.

Let’s see now, how idf (inverse document frequency) is then defined:

$\的DisplayStyle \ mathrm {IDF}（T）= \日志{\压裂{\左| d \右|} {1+ \左| \ {d：吨\在d \} \右|}}$

where$\left|\{d : t \in d\}\right|$is thenumber of documentswhere the term$t$appears, when the term-frequency function satisfies$\mathrm{tf}(t,d) \neq 0$, we’re only adding 1 into the formula to avoid zero-division.

The formula for the tf-idf is then:

$\mathrm{tf\mbox{-}idf}(t) = \mathrm{tf}(t, d) \times \mathrm{idf}(t)$

and this formula has an important consequence: a high weight of the tf-idf calculation is reached when you have a high term frequency (tf) in the given document (local parameter) and a low document frequency of the term in the whole collection (global parameter).

Now let’s calculate the idf for each feature present in the feature matrix with the term frequency we have calculated in the first tutorial:

$M_{train} = \begin{bmatrix} 0 & 1 & 1 & 1\\ 0 & 2 & 1 & 0 \end{bmatrix}$

Since we have 4 features, we have to calculate$\mathrm{idf}(t_1)$,$\ mathrm {IDF}（T_2）$,$\ mathrm {IDF}（t_3处）$,$\mathrm{idf}(t_4)$:

$\mathrm{idf}(t_1) = \log{\frac{\left|D\right|}{1+\left|\{d : t_1 \in d\}\right|}} = \log{\frac{2}{1}} = 0.69314718$

$\ mathrm {IDF}（T_2）= \log{\frac{\left|D\right|}{1+\left|\{d : t_2 \in d\}\right|}} = \log{\frac{2}{3}} = -0.40546511$

$\ mathrm {IDF}（t_3处）= \日志{\压裂{\左| d \右|} {1+ \左| \ {d：t_3处\在d \} \右|}} = \日志{\压裂{2} {3}} = -0.40546511$

$\mathrm{idf}(t_4) = \log{\frac{\left|D\right|}{1+\left|\{d : t_4 \in d\}\right|}} = \log{\frac{2}{2}} = 0.0$

These idf weights can be represented by a vector as:

$\vec{idf_{train}} = (0.69314718, -0.40546511, -0.40546511, 0.0)$

Now that we have our matrix with the term frequency ($M_{train}$) and the vector representing the idf for each feature of our matrix ($\vec{idf_{train}}$), we can calculate our tf-idf weights. What we have to do is a simple multiplication of each column of the matrix$M_{train}$with the respective$\vec{idf_{train}}$vector dimension. To do that, we can create a squarediagonal matrixcalled$M_{idf}$with both the vertical and horizontal dimensions equal to the vector$\vec{idf_{train}}$尺寸：

$M_{idf} = \begin{bmatrix} 0.69314718 & 0 & 0 & 0\\ 0 & -0.40546511 & 0 & 0\\ 0 & 0 & -0.40546511 & 0\\ 0 & 0 & 0 & 0 \end{bmatrix}$

$M_{tf\mbox{-}idf} = M_{train} \times M_{idf}$

Please note that the matrix multiplication isn’t commutative, the result of$A \times B$will be different than the result of the$B \times A$, and this is why the$M_{idf}$is on the right side of the multiplication, to accomplish the desired effect of multiplying each idf value to its corresponding feature:

$\begin{bmatrix} \mathrm{tf}(t_1, d_1) & \mathrm{tf}(t_2, d_1) & \mathrm{tf}(t_3, d_1) & \mathrm{tf}(t_4, d_1)\\ \mathrm{tf}(t_1, d_2) & \mathrm{tf}(t_2, d_2) & \mathrm{tf}(t_3, d_2) & \mathrm{tf}(t_4, d_2) \end{bmatrix} \times \begin{bmatrix} \mathrm{idf}(t_1) & 0 & 0 & 0\\ 0 & \mathrm{idf}(t_2) & 0 & 0\\ 0 & 0 & \mathrm{idf}(t_3) & 0\\ 0 & 0 & 0 & \mathrm{idf}(t_4) \end{bmatrix} \\ = \begin{bmatrix} \mathrm{tf}(t_1, d_1) \times \mathrm{idf}(t_1) & \mathrm{tf}(t_2, d_1) \times \mathrm{idf}(t_2) & \mathrm{tf}(t_3, d_1) \times \mathrm{idf}(t_3) & \mathrm{tf}(t_4, d_1) \times \mathrm{idf}(t_4)\\ \mathrm{tf}(t_1, d_2) \times \mathrm{idf}(t_1) & \mathrm{tf}(t_2, d_2) \times \mathrm{idf}(t_2) & \mathrm{tf}(t_3, d_2) \times \mathrm{idf}(t_3) & \mathrm{tf}(t_4, d_2) \times \mathrm{idf}(t_4) \end{bmatrix}$

Let’s see now a concrete example of this multiplication:

$M_{tf\mbox{-}idf} = M_{train} \times M_{idf} = \\ \begin{bmatrix} 0 & 1 & 1 & 1\\ 0 & 2 & 1 & 0 \end{bmatrix} \times \begin{bmatrix} 0.69314718 & 0 & 0 & 0\\ 0 & -0.40546511 & 0 & 0\\ 0 & 0 & -0.40546511 & 0\\ 0 & 0 & 0 & 0 \end{bmatrix} \\ = \begin{bmatrix} 0 & -0.40546511 & -0.40546511 & 0\\ 0 & -0.81093022 & -0.40546511 & 0 \end{bmatrix}$

$M_{tf\mbox{-}idf} = \frac{M_{tf\mbox{-}idf}}{\|M_{tf\mbox{-}idf}\|_2}$ $= \begin{bmatrix} 0 & -0.70710678 & -0.70710678 & 0\\ 0 & -0.89442719 & -0.4472136 & 0 \end{bmatrix}$

And that is our pretty normalized tf-idf weight of our testing document set, which is actually a collection of unit vectors. If you take the L2-norm of each row of the matrix, you’ll see that they all have a L2-norm of 1.

### Python practice

Environment Used:Python v.2.7.2,NumPy的1.6.1,SciPy的v.0.9.0,Sklearn (Scikits.learn) v.0.9.

from sklearn.feature_extraction.text import CountVectorizer train_set = ("The sky is blue.", "The sun is bright.") test_set = ("The sun in the sky is bright.", "We can see the shining sun, the bright sun.") count_vectorizer = CountVectorizer() count_vectorizer.fit_transform(train_set) print "Vocabulary:", count_vectorizer.vocabulary # Vocabulary: {'blue': 0, 'sun': 1, 'bright': 2, 'sky': 3} freq_term_matrix = count_vectorizer.transform(test_set) print freq_term_matrix.todense() #[[0 1 1 1] #[0 2 1 0]]

Now that we have the frequency term matrix (calledfreq_term_matrix), we can instantiate theTfidfTransformer, which is going to be responsible to calculate the tf-idf weights for our term frequency matrix:

from sklearn.feature_extraction.text import TfidfTransformer tfidf = TfidfTransformer(norm="l2") tfidf.fit(freq_term_matrix) print "IDF:", tfidf.idf_ # IDF: [ 0.69314718 -0.40546511 -0.40546511 0. ]

tf_idf_matrix = tfidf.transform(freq_term_matrix) print tf_idf_matrix.todense() # [[ 0. -0.70710678 -0.70710678 0. ] # [ 0. -0.89442719 -0.4472136 0. ]]

And that is it, thetf_idf_matrix其实我们以前$M_{tf\mbox{-}idf}$矩阵。您可以通过使用达到相同的效果VectorizerScikit的类。学习是一个vectorizer that automatically combines theCountVectorizerand theTfidfTransformer给你。看到this exampleto know how to use it for the text classification process.

I really hope you liked the post, I tried to make it simple as possible even for people without the required mathematical background of linear algebra, etc. In the next Machine Learning post I’m expecting to show how you can use the tf-idf to calculate the cosine similarity.

Cite this article as: Christian S. Perone, "Machine Learning :: Text feature extraction (tf-idf) – Part II," in亚洲金博宝未知领域，03/10/2011，//www.cpetem.com/2011/10/machine-learning-text-feature-extraction-tf-idf-part-ii/.

### 参考

Wikipedia :: tf-idf

Sklearn text feature extraction code

13 Mar 2015Formating, fixed images issues.
03 Oct 2011Added the info about the environment used for Python examples

## 103 thoughts to “Machine Learning :: Text feature extraction (tf-idf) – Part II”

1. Severtcev says:

Wow!
完美的前奏在TF-IDF，非常感谢你！亚洲金博宝亚洲金博宝很有意思，我想学这个领域很长一段时间，你的职位是一个真正的礼物。这将是非常有趣的阅读更多亚洲金博宝关于该技术的使用情况。而且可能是你有兴趣，请，摆脱对文本语料库表示的其他方法的一些光，如果他们存在？
(sorry for bad English, I’m working to improve it, but there is still a lot of job to do)

2. Excellent work Christian! I am looking forward to reading your next posts on document classification, clustering and topics extraction with Naive Bayes, Stochastic Gradient Descent, Minibatch-k-Means and Non Negative Matrix factorization

Also, the documentation of scikit-learn is really poor on the text feature extraction part (I am the main culprit…). Don’t hesitate to join the mailing list if you want to give a hand and improve upon the current situation.

1. Great thanks Olivier. I really want to help sklearn, I just have to get some more time to do that, you guys have done a great work, I’m really impressed by the amount of algorithms already implemented in the lib, keep the good work !

3. I like this tutorial better for the level of new concepts i am learning here.
这就是说，学习scikits您正在使用哪个版本？
The latest as installed by easy_install seems to have a different module hierarchy (i.e doesn’t find feature_extraction in sklearn). If you could mention the version you used, i will just try out with those examples.

1. 您好阿南德，我很高兴你喜欢它。我已经增加了大约只用一节“的Python惯例”之前，我使用的是scikits.learn 0.9（发布在几个星期前）环境的信息。

4. siamii says:

Where’s part 3? I’ve got to submit an assignment on Vector Space Modelling in 4 days. Any hope of putting it up over the weekend?

1. I’ve no date to publish it since I haven’t got any time to write it =(

5. Niu says:

谢谢again for this complete and explicit tutorial and I am waiting for the coming section.

6. 吴季刚 says:

由于基督教! a very nice work on vector space with sklearn. I just have one question, suppose I have computed the ‘tf_idf_matrix’, and I would like to compute the pair-wise cosine similarity (between each rows). I was having problem with the sparse matrix format, can you please give an example on that? Also my matrix is pretty big, say 25k by 60k. Thanks a lot!

7. Khalid says:

Great post… I understand what tf-idf and how to implement it with a concrete example. But I caught 2 things that I’m not sure about:
1- You called the 2 dimensional matrix M_train, but it has the tf values of the D3 and D4 documents, so you should’ve called that matrix M_test instead of M_train. Because D3 and D4 are our test documents.
2 - 当你计算IDF值的T2（这是“太阳”），它应该是日志（2/4）。因为文件的数目是2 D3有词“太阳” 1次，D4有它的2倍。这使得3，但是我们也加1到值摆脱0分的问题。这使得4 ...我说得对不对还是我失去了一些东西？
Thank you.

1. 胜利者ia says:

你是正确的：这些都是优秀的博客文章，但作者真的有责任/责任回去和纠正错误，这样的（和其他人，例如，第1部分; ...）：缺席训练下划线;设置STOP_WORDS参数;还我的电脑上，词汇索引是不同的。

正如我们赞赏的努力（荣誉的作者！），它也是一个显著伤害那些谁斗争过去在原有材料的（未修正）的错误。

1. 胜利者ia says:

re: my ‘you are correct comment’ (above), I should have added:

“… noting also Frédérique Passot’s comment (below) regarding the denominator:

‘… what we are using is really the number of documents in which a term occurs, regardless of the number of times the term occurs in any given document. In this case, then, the denominator in the idf value for t2 (‘sun’) is indeed 2+1 (2 documents have the term ‘sun’, +1 to avoid a potential zero division error).’ “

2. Yeshwant says:

Khalid,
This is a response to a very old question. However, I still want to respond to communicate what I understand from the article.
Your question 2: “When you calculate the idf value for the t2 (which is ‘sun’) it should be log(2/4)”
我的理解：在数项的分母应该是（一些文件，其中术语出现+ 1），而不是长期的频率。术语“太阳”出现的文件的数目是2（1次在D3和D4中的2倍 - 完全出现3次在两个文件3是频率和2是文件号）。因此，分母为2 + 1 = 3。

8. 阿尔苏 says:

thanks… excellent post…

9. 插口 says:

优秀的帖子！
I have some question. From the last tf-idf weight matrix, how can we get the importance of term respectively(e.g. which is the most important term?). How can we use this matrix to classify documents

10. Thanuj says:

Thank You So Much. You explained it in such a simple way. It was really useful. Once again thanks a lot.

11. Thanuj says:

I have same doubt as Jack(last comment). From the last tf-idf weight matrix, how can we get the importance of term respectively(e.g. which is the most important term?). How can we use this matrix to classify documents.

12. tintin says:

I have a question..
After the tf-idf operation, we get a numpy array with values. Suppose we need to get the highest 50 values from the array. How can we do that?

1. ashwin sudhini says:

high value of f(idf) denotes that the particular vector(or Document) has high local strength and low global strength, in which case you can assume that the terms in it has high significance locally and cant be ignored. Comparing against funtion(tf) where only the term repeats high number of times are the ones given more importance,which most of the times is not a proper modelling technique.

13. Vikram Bakhtiani says:

Hey ,
Thanx fr d code..was very helpful indeed !

1.适用于文档聚类，计算反相的术语频率之后，shud我使用任何关联性系数等Jaccards系数，然后应用聚类算法中像k均值或shud我计算反转术语频率后直接适用d k均值到文档向量？

2. How do u rate inverted term frequency for calcuating document vectors for document clustering ?

谢谢a ton fr the forth coming reply!

14. @Khalid：你在1-指出什么让我困惑过了一分钟（M_train VS M_test）。我想你误会了你的第二点，不过，因为我们用的是什么是真正发生的一个术语，无论任何给定的文档中出现的术语次数的文件数量。在这种情况下，那么，在为T2（“太阳”）的IDF值分母确实2 + 1（2个文件具有的术语“太阳”，1以避免潜在的零分割误差）。

I’d love to read the third installment of this series too! I’d be particularly interested in learning more about feature selection. Is there an idiomatic way to get a sorted list of the terms with the highest tf.idf scores? How would you identify those terms overall? How would you get the terms which are the most responsible for a high or low cosine similarity (row by row)?

谢谢你的帖子_美好的_！

1. Bonnie Varghese says:

如果IDF（T2）进行登录2/4？

15. Matthys Meintjes says:

Excellent article and a great introduction to td-idf normalization.

You have a very clear and structured way of explaining these difficult concepts.

谢谢！

1. 谢谢for the feedback Matthys, I’m glad you liked the tutorial series.

1. param says:

very good & infomative tutorial…. please upload more tutorials related to documents clustering process.

16. Laurent says:

Excellent article ! Thank you Christian. You did a great job.

17. Gavin Igor says:

您可以为使用TFIDF所以我们有TFIDF的矩阵，我们怎么可以用它来计算余弦做余弦相似度任何引用。感谢神奇的物品。

18. 薰衣草 says:

谢谢so much for this and for explaining the whole tf-idf thing thoroughly.

19. Please correct me if i’m worng
与启动后的公式“我们在第一个教程中计算出的频率：”应该不MTEST Mtrain。也开始“这些IDF权重可以由矢量作为表示后：”应该是不idf_test idf_train。

Btw great series, can you give an simple approach for how to implement classification?

20. 迪夫亚 says:

优秀它真的帮助我度过VSM的概念和TF-IDF得到。由于基督教

21. 塞尔吉奥 says:

Very good post. Congrats!!

Showing your results, I have a question:

The tf-idf value increases proportionally to the number of times a word appears in the document, but is offset by the frequency of the word in the corpus, which helps to control for the fact that some words are generally more common than others.

当我看到它，我明白，如果一个字中的所有文档apperars就是一个字只出现在一个文档中不太重要的：

However, in the results, the word “sun” or “bright” are most important than “sky”.

I’m not sure of understand it completly.

22. Awesome! Explains TF-IDF very well. Waiting eagerly for your next post.

23. 有一个明确的解释真棒工作。即使是外行人容易理解的主题..

24. Susan says:

Terrific! I was familiar with tf-idf before but I found your scikits examples helpful as I’m trying to learn that package.

25. Thank you for writing such a detailed post. I learn allot.

26. Eugene Chinveeraphan says:

优秀的帖子！一次偶然的机会找上CountVectorizer更多信息，无意中发现了这一点，但我很高兴我通过两个您的文章（第1部分和第2部分）的读取。

1. Great thanks for the feedback Eugene, I’m really glad you liked the tutorial series.

27. me says:

似乎没有fit_transform（）为你描述..
Any idea why ?
>>> ts
(‘The sky is blue’, ‘The sun is bright’)
>>> V7 = CountVectorizer（）
>>> v7.fit_transform(ts)
<2×2 sparse matrix of type '’
with 4 stored elements in COOrdinate format>
>>> print v7.vocabulary_
{u’is’: 0, u’the’: 1}

1. Ash says:

Actually, there are two small errors in the first Python sample.
1. CountVectorizer应该被实例化，如下所示：
count_vectorizer = CountVectorizer(stop_words='english')
这将确保“是”，“的”等被删除。

2.要打印的词汇，你必须在末尾添加下划线。
print "Vocabulary:", count_vectorizer.vocabulary_

Excellent tutorial, just small things. hoep it helps others.

1. Drogo says:

谢谢ash. although the article was rather self explanatory, your comment made the entire difference.

28. 约翰·凯尔文 says:

29. I’m using scikit learn v .14. Is there any reason my results for running the exact same code would result in different results?

30. Karthik says:

31. Vijay says:

Its useful…..thank you explaining the TD_IDF very elaborately..

32. Mike says:

感谢伟大的解释。

I have a question about calculation of the idf(t#).
在第一种情况下，你写的IDF（T1）=日志（2/1），因为我们没有我们收集此类条款，因此，我们添加1分母。现在，在T2的情况下，你写的日志（2/3），所以分母等于3，而不是4（= 1 + 2 + 1）？万一t3时，你写：日志（2/3），从而分母等于3（= 1 + 1 + 1）。我在这里看到的那种不一致性。你能不能，请解释一下，你是怎么计算的分母值。

谢谢。

1. Hello Mike, thanks for the feedback. You’re right, I just haven’t fixed it yet due to lack of time to review it and recalculate the values.

2. xpsycho says:

You got it wrong, in the denominator you don’t put the sum of the term in each document, you just sum all the documents that have at least one aparition of the term.

3. mik says:

是的，我有同样的问题...

33. huda says:

This is good post

34. huda says:

it is good if you can provide way to know how use ft-idf in classification of document. I see that example (python code) but if there is algorithm that is best because no all people can understand this language.

谢谢

35. Ganesh神 says:

Great post, really helped me understand the tf-idf concept!

36. Samuel Kahn says:

漂亮的文章

37. Nice. An explanation helps put things into perspective. Is tf-idf a good way to do clustering (e.g. use Jaccard analysis or variance against the average set from a known corpus)?

Keep writing:)

38. Neethu Prem says:

Hi Christian,

这让我非常兴奋和幸运，读亚洲金博宝这篇文章。你理解的清晰反映了文件的清晰度。这让我重拾我的信心在机器学习领域。

谢谢a ton for the beautiful explanation.

Would like to read more from you.

谢谢，
Neethu

1. Great thanks for the kind wors Neethu ! I’m very glad you liked the tutorial series.

39. esra'a OK says:

thank you very very much,very wonderful and useful.

40. Arne says:

Thank you for the good wrap up. You mention a number of papers which compare L1 and L2 norm, I plan to study that a bit more in depth. You still know their names?

41. seher says:

how can i calculate tf idf for my own text file which is located some where in my pc?

42. Shubham says:

Brilliant article.

By far the easiest and most sound explanation of tf-tdf I’ve read. I really liked how you explained the mathematics behind it.

43. mehrab says:

superb article for newbies

1. Dayananda says:

Excellent material. Excellent!!!

44. Derrick says:

嗨，伟大的职位！我使用的是TfidVectorizer模块scikit学习产生与规范= L2的TF-IDF矩阵。我把它叫做tfidf_matrix语料的fit_transform后，我一直在检查TfidfVectorizer的输出。我总结了行，但他们并不总和为1的代码是VECT = TfidfVectorizer（use_idf =真，sublunar_tf =真，规范=” L2）。tfidf_matrix = vect.fit_transform（数据）。当我运行tfidf_matrix.sum（轴= 1）的载体是大于1也许我看错矩阵或我误解如何正常化的作品。我希望有人能澄清这一点！谢谢

45. Chris says:

Can I ask when you calculated the IDF, for example, log(2/1), did you use log to base 10 (e) or some other value? I’m getting different calculations!

46. Gonzalo G says:

伟大的教程,刚开始一份新工作在毫升和this explains things very clearly as it should be.

47. Harsimranpal says:

But I need more information, As you show the practical with python, Can you provide it with JAVA language..

48. 塞巴斯蒂安 says:

我有点困惑为什么tf-idf negative numbers in this case? How do we interpret them? Correct me if I am wrong, but when the vector has a positive value, it means that the magnitude of that component determines how important that word is in that document. If the it is negative, I don’t know how to interpret it. If I were to take the dot product of a vector with all positive components and one with negative components, it would mean that some components may contribute negatively to the dot product even though on of the vectors has very high importance for a particular word.

49. 嗨，
thank you so much for this detailed explanation on this topic, really great. Anyway, could you give me a hint what could be the source of my error that I am keep on seeing:

freq_term_matrix = count_vectorizer.transform（TEST_SET）
AttributeError: ‘matrix’ object has no attribute ‘transform’

Am I using a wrong version of sklearn?

50. Mohit Gupta says:

Awesome simple and effective explaination.Please post more topics with such awesome explainations.Looking forward for upcoming articles.
谢谢

51. Alexandro says:

Thank you Chris, you are the only one on the web who was clear about the diagonal matrix.

52. ishpreet says:

Great tutorial for Tf-Idf. Excellent work . Please add for cosine similarity also:)

53. sherlockatsz says:

I understood the tf-idf calculation process. But what does that matrix mean and how can we use the tfidf matrix to calculate the similarity confuse me. can you explain that how can we use the tfidf matrix .thanks

54. lightningstrike says:

55. Anonymous says:

thanks, nice post, I’m trying it out

56. Anonymous says:

Thank you so much for such an amazing detailed explanation!

57. Akanksha Pande says:

best explanation.. Very helpful. Can you please tell me how to plot vectors in text classification in svm.. I am working on tweets classification. I am confused please help me.

58. Koushik says:

I learned so many things. Thanks Christian. Looking forward for your next tutorial.

59. Mhr says:

嗨，I’m sorry if i have mistaken but i could not understand how is ||Vd4||2 = 1.
D4 =的值（0.0，0.89,0.44,0.0），因此归一化将是= SQRT（正方形（0.89）+平方（0.44））= SQRT（0.193）= 0.44

60. 李催情 says:

嗨，it is a great blog!
If I need to do bi-gram cases, how can I use sklearn to finish it?

61. alireza says:

it is very great. i love your teach. very very good

62. Ritesh says:

I am not getting same result, when i am executing the same script.
print (“IDF:”, tfidf.idf_) : IDF: [ 2.09861229 1. 1.40546511 1. ]

My python version is: 3.5
Scikit了解的版本是：o.18.1

什么我需要改变？可能是什么可能的错误？

thanks,

1. It can be many things, since you’re using a different Python interpreter version and also a different Scikit-Learn version, you should expect differences in the results since they may have changed default parameters, algorithms, rounding, etc.

1. Ravithej Chikkala says:

I am also getting: IDF: [2.09861229 1. 1.40546511 1. ]

63. 胜利者 says:

完美的介绍！
No hocus pocus. Clear and simple, as technology should be.
亚洲金博宝非常有帮助
Thank you very much.
Keep posting!

64. 亚太区首席技术官Matt南卡尼 says:

Why is |D| = 2, in the idf equation. Shouldn’t it be 4 since |D| denotes the number of documents considered, and we have 2 from test, 2 from train.

65. LÊ VĂN HẠNH says:

This post is interesting. I like this post…

66. Bren says:

clear cut and to the point explanations….great

67. Shipika Singh says:

hey , hii Christian
您的文章是真正帮助我了解从基础TFD-IDF。我在分类的一个项目，其中我使用向量空间模型，这导致在确定类别在我的测试文档应该存在。机器学习的一部分。如果你认为我有关的东西这将是巨大的。我被困在这一点上。
thank you

68. Eshwar期基于g says:

看到这个例子就知道如何使用它的文本分类过程。“这个”链接不起作用了。能否请您提供相关链接，例如。

谢谢

69. amanda says:

这样一个伟大的解释！谢谢！

70. 另类投资 says:

哇，真棒post.Much再次感谢。将阅读

Say, you got a nice post.Really thank you! Fantastic.

72. togel online says:

Wow, great article post.Much thanks again. Awesome.

73. Mobile Computer says:

74. chocopie says:

1vbXlh You have brought up a very wonderful details , appreciate it for the post.

75. I know this site provides quality based articles or
reviews and additional data, is there any other web page which presents these kinds of
在质量信息？

76. 鲁塞 says:

In the first example. idf(t1), the log (2/1) = 0.3010 by the calculator. Why they obtained 0.69.. Please What is wrong?

This site uses Akismet to reduce spam.Learn how your comment data is processed.