# Machine Learning :: Text feature extraction (tf-idf) – Part II

Read the first part of this tutorial:Text feature extraction (tf-idf) – Part I

This post is acontinuationof the first part where we started to learn the theory and practice about text feature extraction and vector space model representation. I really recommend youto read the first partof the post series in order to follow this second post.

### 介绍

In the first post, we learned how to use theterm-frequencyto represent textual information in the vector space. However, the main problem with the term-frequency approach is that it scales up frequent terms and scales down rare terms which are empirically more informative than the high frequency terms. The basic intuition is that a term that occurs frequently in many documents is not a good discriminator, and really makes sense (at least in many experimental tests); the important question here is: why would you, in a classification problem for instance, emphasize a term which is almost present in the entire corpus of your documents ?

The tf-idf weight comes to solve this problem. What tf-idf gives is how important is a word to a document in a collection, and that’s why tf-idf incorporates local and global parameters, because it takes in consideration not only the isolated term but also the term within the document collection. What tf-idf then does to solve that problem, is to scale down the frequent terms while scaling up the rare terms; a term that occurs 10 times more than another isn’t 10 times more important than it, that’s why tf-idf uses the logarithmic scale to do that.

To overcome this problem, the term frequency$\ mathrm {TF}（T，d）$上的矢量空间中的文件的通常也归一化。让我们来看看我们是如何规范这一载体。

### 矢量归

Suppose we are going to normalize the term-frequency vector$\vec{v_{d_4}}$我们在本教程的第一部分已经计算。该文件$d4$from the first part of this tutorial had this textual representation:

d4: We can see the shining sun, the bright sun.

$\vec{v_{d_4}} = (0,2,1,0)$

$\的DisplayStyle \帽子{V} = \压裂{\ vec的{V}} {\ | \ vec的{V} \ | _p}$

Where the$\hat{v}$是单位矢量，或者归一化矢量，所述$\ VEC {V}$是个vector going to be normalized and the$\ | \ VEC {V} \ | _p$是很正常的(大小、长度)的向量$\ VEC {V}$in the$L ^ p$space (don’t worry, I’m going to explain it all).

The unit vector is actually nothing more than a normalized version of the vector, is a vector which the length is 1.

### 勒贝格空间

$\|\vec{u}\| = \sqrt{u^2_1 + u^2_2 + u^2_3 + \ldots + u^2_n}$

But this isn’t the only way to define length, and that’s why you see (sometimes) a number$p$符合规范的符号，就像在了一起$\|\vec{u}\|_p$。That’s because it could be generalized as:

$\displaystyle \|\vec{u}\|_p = ( \left|u_1\right|^p + \left|u_2\right|^p + \left|u_3\right|^p + \ldots + \left|u_n\right|^p )^\frac{1}{p}$

$\的DisplayStyle \ | \ VEC【U} \ | _p =（\总和\ limits_ {I = 1} ^ {N} \左| \ VEC {U】_i \右| ^ P）^ \压裂{1} {P}$

So when you read about aL2-norm，你正在阅读关于欧几里得范, a norm with$p = 2$, the most common norm used to measure the length of a vector, typically called “magnitude”; actually, when you have an unqualified length measure (without the$p$number), you have theL2-norm(Euclidean norm).

$\displaystyle \|\vec{u}\|_1 = ( \left|u_1\right| + \left|u_2\right| + \left|u_3\right| + \ldots + \left|u_n\right|)$

Taxicab geometry versus Euclidean distance: In taxicab geometry all three pictured lines have the same length (12) for the same route. In Euclidean geometry, the green line has length$6 \倍\ SQRT {2} \约8.48$, and is the unique shortest path.

Note that you can also use any norm to normalize the vector, but we’re going to use the most common norm, the L2-Norm, which is also the default in the 0.9 release of thescikits.learn。You can also find papers comparing the performance of the two approaches among other methods to normalize the document vector, actually you can use any other method, but you have to be concise, once you’ve used a norm, you have to use it for the whole process directly involving the norm (a unit vector that used a L1-norm isn’t going to have the length 1 if you’re going to take its L2-norm later）。

### 返回矢量归

$\hat{v} = \frac{\vec{v}}{\|\vec{v}\|_p} \\ \\ \hat{v_{d_4}} = \frac{\vec{v_{d_4}}}{||\vec{v_{d_4}}||_2} \\ \\ \\ \hat{v_{d_4}} = \frac{(0,2,1,0)}{\sqrt{0^2 + 2^2 + 1^2 + 0^2}} \\ \\ \hat{v_{d_4}} = \frac{(0,2,1,0)}{\sqrt{5}} \\ \\ \small \hat{v_{d_4}} = (0.0, 0.89442719, 0.4472136, 0.0)$

Note that here we have normalized our term frequency document vector, but later we’re going to do that after the calculation of the tf-idf.

### 术语频率 - 逆文档频率（TF-IDF）重量

Train Document Set: d1: The sky is blue. d2: The sun is bright. Test Document Set: d3: The sun in the sky is bright. d4: We can see the shining sun, the bright sun.

$\的DisplayStyle \ mathrm {IDF}（T）= \日志{\压裂{\左| d \右|} {1+ \左| \ {d：吨\在d \} \右|}}$

where$\left|\{d : t \in d\}\right|$是个number of documentswhere the term$t$看来,当term-frequency function satisfies$\ mathrm {TF}（T，d）\neq 0$, we’re only adding 1 into the formula to avoid zero-division.

The formula for the tf-idf is then:

$\mathrm{tf\mbox{-}idf}(t) = \mathrm{tf}(t, d) \times \mathrm{idf}(t)$

Now let’s calculate the idf for each feature present in the feature matrix with the term frequency we have calculated in the first tutorial:

$M_ {}列车= \begin{bmatrix} 0 & 1 & 1 & 1\\ 0 & 2 & 1 & 0 \end{bmatrix}$

Since we have 4 features, we have to calculate$\mathrm{idf}(t_1)$,$\mathrm{idf}(t_2)$,$\mathrm{idf}(t_3)$,$\mathrm{idf}(t_4)$:

$\mathrm{idf}(t_1) = \log{\frac{\left|D\right|}{1+\left|\{d : t_1 \in d\}\right|}} = \log{\frac{2}{1}} = 0.69314718$

$\mathrm{idf}(t_2) = \log{\frac{\left|D\right|}{1+\left|\{d : t_2 \in d\}\right|}} = \log{\frac{2}{3}} = -0.40546511$

$\mathrm{idf}(t_3) = \log{\frac{\left|D\right|}{1+\left|\{d : t_3 \in d\}\right|}} = \log{\frac{2}{3}} = -0.40546511$

$\ mathrm {IDF}（T_4）= \日志{\压裂{\左| d \右|} {1+ \左| \ {d：T_4 \在d \} \右|}} = \日志{\压裂{2} {2}} = 0.0$

$vec {idf_ \{火车}}= (0.69314718, -0.40546511, -0.40546511, 0.0)$

$M_ {} IDF= \begin{bmatrix} 0.69314718 & 0 & 0 & 0\\ 0 & -0.40546511 & 0 & 0\\ 0 & 0 & -0.40546511 & 0\\ 0 & 0 & 0 & 0 \end{bmatrix}$

$M_{tf\mbox{-}idf} = M_{train} \times M_{idf}$

Please note that the matrix multiplication isn’t commutative, the result of$A \times B$will be different than the result of the$B \times A$，这就是为什么$M_ {} IDF$是对乘法的右侧，以完成每个IDF值到其对应的特征相乘的期望的效果：

${bmatrix} \ \开始mathrm {tf} (t_1 d_1) & \ mathrm {tf}(t_2, d_1) & \mathrm{tf}(t_3, d_1) & \mathrm{tf}(t_4, d_1)\\ \mathrm{tf}(t_1, d_2) & \mathrm{tf}(t_2, d_2) & \mathrm{tf}(t_3, d_2) & \mathrm{tf}(t_4, d_2) \end{bmatrix} \times \begin{bmatrix} \mathrm{idf}(t_1) & 0 & 0 & 0\\ 0 & \mathrm{idf}(t_2) & 0 & 0\\ 0 & 0 & \mathrm{idf}(t_3) & 0\\ 0 & 0 & 0 & \mathrm{idf}(t_4) \end{bmatrix} \\ = \begin{bmatrix} \mathrm{tf}(t_1, d_1) \times \mathrm{idf}(t_1) & \mathrm{tf}(t_2, d_1) \times \mathrm{idf}(t_2) & \mathrm{tf}(t_3, d_1) \times \mathrm{idf}(t_3) & \mathrm{tf}(t_4, d_1) \times \mathrm{idf}(t_4)\\ \mathrm{tf}(t_1, d_2) \times \mathrm{idf}(t_1) & \mathrm{tf}(t_2, d_2) \times \mathrm{idf}(t_2) & \mathrm{tf}(t_3, d_2) \times \mathrm{idf}(t_3) & \mathrm{tf}(t_4, d_2) \times \mathrm{idf}(t_4) \end{bmatrix}$

$M_{tf\mbox{-}idf} = M_{train} \times M_{idf} = \\ \begin{bmatrix} 0 & 1 & 1 & 1\\ 0 & 2 & 1 & 0 \end{bmatrix} \times \begin{bmatrix} 0.69314718 & 0 & 0 & 0\\ 0 & -0.40546511 & 0 & 0\\ 0 & 0 & -0.40546511 & 0\\ 0 & 0 & 0 & 0 \end{bmatrix} \\ = \begin{bmatrix} 0 & -0.40546511 & -0.40546511 & 0\\ 0 & -0.81093022 & -0.40546511 & 0 \end{bmatrix}$

And finally, we can apply our L2 normalization process to the$M_{tf\mbox{-}idf}$矩阵。Please note that this normalization is“row-wise”因为我们要处理矩阵的每一行作为一个分离向量进行归一化，而不是矩阵作为一个整体：

$M_{tf\mbox{-}idf} = \frac{M_{tf\mbox{-}idf}}{\|M_{tf\mbox{-}idf}\|_2}$ $= \begin{bmatrix} 0 & -0.70710678 & -0.70710678 & 0\\ 0 & -0.89442719 & -0.4472136 & 0 \end{bmatrix}$

### Python practice

Now the section you were waiting for ! In this section I’ll use Python to show each step of the tf-idf calculation using theScikit.learnfeature extraction module.

The first step is to create our training and testing document set and computing the term frequency matrix:

from sklearn.feature_extraction.text import CountVectorizer train_set = ("The sky is blue.", "The sun is bright.") test_set = ("The sun in the sky is bright.", "We can see the shining sun, the bright sun.") count_vectorizer = CountVectorizer() count_vectorizer.fit_transform(train_set) print "Vocabulary:", count_vectorizer.vocabulary # Vocabulary: {'blue': 0, 'sun': 1, 'bright': 2, 'sky': 3} freq_term_matrix = count_vectorizer.transform(test_set) print freq_term_matrix.todense() #[[0 1 1 1] #[0 2 1 0]]

从进口sklearn.feature_extraction.text TFIDF TfidfTransformer = TfidfTransformer（NORM = “L2”）tfidf.fit（freq_term_matrix）打印 “IDF：”，tfidf.idf_＃IDF：[0.69314718 -0.40546511 -0.40546511 0]

Note that I’ve specified the norm as L2, this is optional (actually the default is L2-norm), but I’ve added the parameter to make it explicit to you that it it’s going to use the L2-norm. Also note that you can see the calculated idf weight by accessing the internal attribute calledidf_。现在适合（）方法计算矩阵中的IDF上，让我们改造freq_term_matrixto the tf-idf weight matrix:

tf_idf_matrix= tfidf.transform(freq_term_matrix) print tf_idf_matrix.todense() # [[ 0. -0.70710678 -0.70710678 0. ] # [ 0. -0.89442719 -0.4472136 0. ]]

And that is it, thetf_idf_matrixis actually our previous$M_{tf\mbox{-}idf}$矩阵。您可以通过使用达到相同的效果VectorizerScikit的类。学习是一个vectorizer that automatically combines theCountVectorizer和theTfidfTransformer给你。看到this exampleto know how to use it for the text classification process.

I really hope you liked the post, I tried to make it simple as possible even for people without the required mathematical background of linear algebra, etc. In the next Machine Learning post I’m expecting to show how you can use the tf-idf to calculate the cosine similarity.

Cite this article as: Christian S. Perone, "Machine Learning :: Text feature extraction (tf-idf) – Part II," in亚洲金博宝未知领域, 03/10/2011,//www.cpetem.com/2011/10/machine-learning-text-feature-extraction-tf-idf-part-ii/

### 参考

Understanding Inverse Document Frequency: on theoretical arguments for IDF

Sklearn text feature extraction code

### 更新

13 Mar 2015Formating, fixed images issues.
03 Oct 2011Added the info about the environment used for Python examples

## 103个想法“机器学习::文本特征提取（TF-IDF） - 第二部分”

1. Severtcev 说：

Wow!
完美的前奏在TF-IDF，非常感谢你！亚洲金博宝亚洲金博宝很有意思，我想学这个领域很长一段时间，你的职位是一个真正的礼物。这将是非常有趣的阅读更多亚洲金博宝关于该技术的使用情况。而且可能是你有兴趣，请，摆脱对文本语料库表示的其他方法的一些光，如果他们存在？
（对不起，糟糕的英语，我正在努力对其进行改进，但仍然有很多工作要做的）

2. 出色的工作基督徒！我期待着阅读的文档分类你的下一个职位，聚类和主题提取朴素贝叶斯，随机梯度下降，Minibatch-K均值和非负矩阵分解

Also, the documentation of scikit-learn is really poor on the text feature extraction part (I am the main culprit…). Don’t hesitate to join the mailing list if you want to give a hand and improve upon the current situation.

1. Great thanks Olivier. I really want to help sklearn, I just have to get some more time to do that, you guys have done a great work, I’m really impressed by the amount of algorithms already implemented in the lib, keep the good work !

3. 我喜欢这个教程的新概念我在这里学习水平较好。
That said, which version of scikits-learn are you using?.
The latest as installed by easy_install seems to have a different module hierarchy (i.e doesn’t find feature_extraction in sklearn). If you could mention the version you used, i will just try out with those examples.

1. 您好阿南德，我很高兴你喜欢它。我已经增加了大约只用一节“的Python惯例”之前，我使用的是scikits.learn 0.9（发布在几个星期前）环境的信息。

4. siamii 说：

Where’s part 3? I’ve got to submit an assignment on Vector Space Modelling in 4 days. Any hope of putting it up over the weekend?

1. I’ve no date to publish it since I haven’t got any time to write it =(

5. 说：

Thanks again for this complete and explicit tutorial and I am waiting for the coming section.

6. Jason Wu 说：

由于基督教! a very nice work on vector space with sklearn. I just have one question, suppose I have computed the ‘tf_idf_matrix’, and I would like to compute the pair-wise cosine similarity (between each rows). I was having problem with the sparse matrix format, can you please give an example on that? Also my matrix is pretty big, say 25k by 60k. Thanks a lot!

7. Khalid 说：

Great post… I understand what tf-idf and how to implement it with a concrete example. But I caught 2 things that I’m not sure about:
1-你调用2维矩阵M_train，但它具有D3和D4文件的TF值，所以你应该已经给那矩阵M_test而不是M_train。由于D3和D4是我们的测试文档。
2- When you calculate the idf value for the t2 (which is ‘sun’) it should be log(2/4). Because number of the documents is 2. D3 has the word ‘sun’ 1 time, D4 has it 2 times. Which makes it 3 but we also add 1 to that value to get rid of divided by 0 problem. And this makes it 4… Am I right or am I missing something?
Thank you.

1. 维多利亚 说：

你是正确的：这些都是优秀的博客文章，但作者真的有责任/责任回去和纠正错误，这样的（和其他人，例如，第1部分; ...）：缺席训练下划线;设置STOP_WORDS参数;还我的电脑上，词汇索引是不同的。

正如我们赞赏的努力（荣誉的作者！），它也是一个显著伤害那些谁斗争过去在原有材料的（未修正）的错误。

1. 维多利亚 说：

re: my ‘you are correct comment’ (above), I should have added:

“… noting also Frédérique Passot’s comment (below) regarding the denominator:

“......我们用的是什么确实是在发生的一个术语，无论任何给定的文档中出现的术语次数的文件数量。在这种情况下，然后，在用于T2（“太阳”）的IDF值分母确实2 + 1（2个文件具有“太阳”术语，1以避免潜在的零分割误差）。“

2. Yeshwant 说：

哈立德，
This is a response to a very old question. However, I still want to respond to communicate what I understand from the article.
Your question 2: “When you calculate the idf value for the t2 (which is ‘sun’) it should be log(2/4)”
我的理解：在数项的分母应该是（一些文件，其中术语出现+ 1），而不是长期的频率。术语“太阳”出现的文件的数目是2（1次在D3和D4中的2倍 - 完全出现3次在两个文件3是频率和2是文件号）。因此，分母为2 + 1 = 3。

8. 阿尔苏 说：

thanks… excellent post…

9. Jack 说：

excellent post!
I have some question. From the last tf-idf weight matrix, how can we get the importance of term respectively(e.g. which is the most important term?). How can we use this matrix to classify documents

10. Thanuj 说：

Thank You So Much. You explained it in such a simple way. It was really useful. Once again thanks a lot.

11. Thanuj 说：

我有同样的疑问，杰克（最后的评论）。从上个TF-IDF权重矩阵，我们怎么能拿到各自任期的重要性（例如，这是最重要的用语？）。我们如何利用这个矩阵来区分文档。

12. tintin 说：

我有个问题..
After the tf-idf operation, we get a numpy array with values. Suppose we need to get the highest 50 values from the array. How can we do that?

1. ashwin sudhini 说：

high value of f(idf) denotes that the particular vector(or Document) has high local strength and low global strength, in which case you can assume that the terms in it has high significance locally and cant be ignored. Comparing against funtion(tf) where only the term repeats high number of times are the ones given more importance,which most of the times is not a proper modelling technique.

13. Vikram Bakhtiani 说：

Hey ,
感谢名单FR d code..was的确非亚洲金博宝常有帮助！

1.For document clustering,after calculating inverted term frequency, shud i use any associativity coefficient like Jaccards coefficient and then apply the clustering algo like k-means or shud i apply d k-means directly to the document vectors after calculating inverted term frequency ?

2.您是如何评价倒词频为calcuating文档向量文本聚类？

由于一吨FR第四到来的答复！

14. @Khalid：你在1-指出什么让我困惑过了一分钟（M_train VS M_test）。我想你误会了你的第二点，不过，因为我们用的是什么是真正发生的一个术语，无论任何给定的文档中出现的术语次数的文件数量。在这种情况下，那么，在为T2（“太阳”）的IDF值分母确实2 + 1（2个文件具有的术语“太阳”，1以避免潜在的零分割误差）。

I’d love to read the third installment of this series too! I’d be particularly interested in learning more about feature selection. Is there an idiomatic way to get a sorted list of the terms with the highest tf.idf scores? How would you identify those terms overall? How would you get the terms which are the most responsible for a high or low cosine similarity (row by row)?

Thank you for the _great_ posts!

1. Bonnie Varghese 说：

如果IDF（T2）进行登录2/4？

15. Matthys Meintjes 说：

Excellent article and a great introduction to td-idf normalization.

你必须解释这些复杂的概亚洲金博宝念非常清晰，结构化的方法。

Thanks!

1. 感谢您的反馈Matthys，我很高兴你喜欢这个系列教程。

1. param 说：

very good & infomative tutorial…. please upload more tutorials related to documents clustering process.

16. Laurent 说：

优秀的文章！谢谢基督徒。你做的非常出色。

17. Gavin Igor 说：

您可以为使用TFIDF所以我们有TFIDF的矩阵，我们怎么可以用它来计算余弦做余弦相似度任何引用。感谢神奇的物品。

18. lavender 说：

Thanks so much for this and for explaining the whole tf-idf thing thoroughly.

1. 感谢您的反馈，我很高兴你喜欢这个系列教程。

19. Please correct me if i’m worng
与启动后的公式“我们在第一个教程中计算出的频率：”应该不MTEST Mtrain。也开始“这些IDF权重可以由矢量作为表示后：”应该是不idf_test idf_train。

顺便说一句伟大的系列赛，你可以给如何实施分类的简单的方法？

20. Divya 说：

优秀它真的帮助我度过VSM的概念和TF-IDF得到。由于基督教

21. Sergio 说：

亚洲金博宝很不错的职位。恭喜！！

Showing your results, I have a question:

我读了维基百科：
The tf-idf value increases proportionally to the number of times a word appears in the document, but is offset by the frequency of the word in the corpus, which helps to control for the fact that some words are generally more common than others.

当我看到它，我明白，如果一个字中的所有文档apperars就是一个字只出现在一个文档中不太重要的：

然而，在结果中，“太阳”或“明亮”是比“天空”最重要的。

I’m not sure of understand it completly.

22. 真棒！解释了TF-IDF非常好。亚洲金博宝热切等待你的下一个职位。

23. 有一个明确的解释真棒工作。即使是外行人容易理解的主题..

1. Great thanks for the feedback Rahul !

24. 雷米 说：

你好，

The explanation is awesome. I haven’t seen a better one yet. I have trouble reproducing the results. It might be because of some update of sklearn.
Would it be possible for you to update the code?

It seem that the formula for computing the tf-idf vector has changed a little bit. Is a typo or another formula. Below is the link to the source code.

https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/feature_extraction/text.py#L954

Many thanks

25. Susan 说：

Terrific! I was familiar with tf-idf before but I found your scikits examples helpful as I’m trying to learn that package.

1. I’m glad you liked Susan, thanks for the feedback !

26. Thank you for writing such a detailed post. I learn allot.

27. Eugene Chinveeraphan 说：

Excellent post! Stumbled on this by chance looking for more information on CountVectorizer, but I’m glad I read through both of your posts (part 1 and part 2).

1. Great thanks for the feedback Eugene, I’m really glad you liked the tutorial series.

28. 说：

似乎没有fit_transform（）为你描述..
Any idea why ?
>>> ts
(‘The sky is blue’, ‘The sun is bright’)
>>> V7 = CountVectorizer（）
>>> v7.fit_transform(ts)
<2×2 sparse matrix of type '’
with 4 stored elements in COOrdinate format>
>>>打印v7.vocabulary_
{u’is’: 0, u’the’: 1}

1. Ash 说：

Actually, there are two small errors in the first Python sample.
1. CountVectorizer应该被实例化，如下所示：
count_vectorizer = CountVectorizer(stop_words='english')
This will make sure the ‘is’, ‘the’ etc are removed.

2. To print the vocabulary, you have to add an underscore at the end.
打印“词汇：” count_vectorizer.vocabulary_

优秀的教程，只是小事情。hoep它可以帮助别人。

1. 德罗戈 说：

Thanks ash. although the article was rather self explanatory, your comment made the entire difference.

29. Kelvin John 说：

30. 我使用scikit学习v 0.14。有什么原因，我的结果运行完全相同的代码会导致不同的结果？

31. KARTHIK 说：

Thanks for taking time to write this article. Found it very useful.

32. Vijay 说：

Its useful…..thank you explaining the TD_IDF very elaborately..

33. Mike 说：

Thanks for the great explanation.

I have a question about calculation of the idf(t#).
在第一种情况下，你写的IDF（T1）=日志（2/1），因为我们没有我们收集此类条款，因此，我们添加1分母。现在，在T2的情况下，你写的日志（2/3），所以分母等于3，而不是4（= 1 + 2 + 1）？万一t3时，你写：日志（2/3），从而分母等于3（= 1 + 1 + 1）。我在这里看到的那种不一致性。你能不能，请解释一下，你是怎么计算的分母值。

Thanks.

1. 您好迈克，感谢您的反馈意见。你说得对，我只是还没有固定它尚未由于缺乏时间来审查它，并重新计算值。

2. xpsycho 说：

你理解错了，分母你不把这个词的总和每个文档中，你只是总结所有具有词的至少一个aparition的文件。

3. mik 说：

是的，我有同样的问题...

34. huda 说：

This is good post

35. huda 说：

这是很好的，如果你能提供的方式来知道如何使用FT-IDF中的文档分类。我看到示例（Python代码），但如果有算法是最好的，因为没有所有的人都能理解这种语言。

Thanks

36. Ganesh 说：

Great post, really helped me understand the tf-idf concept!

37. 塞缪尔·卡恩 说：

漂亮的文章

38. 尼斯。一种解释有助于正确看待这个事情。是TF-IDF的好办法做聚类（例如，从已知的语料用杰卡德分析或方差相对于平均值设定）？

Keep writing:)

39. Neethu Prem 说：

Hi Christian,

It makes me very excited and lucky to have read this article. The clarity of your understanding reflects in the clarity of the document. It makes me regain my confidence in the field of machine learning.

Thanks a ton for the beautiful explanation.

想从你更多。

Thanks,
Neethu

1. Great thanks for the kind wors Neethu ! I’m very glad you liked the tutorial series.

40. esra'a ok 说：

thank you very very much,very wonderful and useful.

1. 感谢您的反馈Esra'a。

41. 阿恩 说：

Thank you for the good wrap up. You mention a number of papers which compare L1 and L2 norm, I plan to study that a bit more in depth. You still know their names?

42. seher 说：

how can i calculate tf idf for my own text file which is located some where in my pc?

43. Shubham 说：

Brilliant article.

By far the easiest and most sound explanation of tf-tdf I’ve read. I really liked how you explained the mathematics behind it.

44. 我hrab 说：

superb article for newbies

1. Dayananda 说：

优良的材质。优秀的！！！

45. Derrick 说：

嗨，great post! I’m using the TfidVectorizer module in scikit learn to produce the tf-idf matrix with norm=l2. I’ve been examining the output of the TfidfVectorizer after fit_transform of the corpora which I called tfidf_matrix. I’ve summed the rows but they do not sum to 1. The code is vect = TfidfVectorizer(use_idf=True, sublunar_tf=True, norm=”l2). tfidf_matrix = vect.fit_transform(data). When I run tfidf_matrix.sum(axis=1) the vectors are larger than 1. Perhaps I’m looking at the wrong matrix or I misunderstand how normalisation works. I hope someone can clarify this point! Thanks

46. 克里斯 说：

Can I ask when you calculated the IDF, for example, log(2/1), did you use log to base 10 (e) or some other value? I’m getting different calculations!

47. Gonzalo G 说：

Great tutorial, just started a new job in ML and this explains things very clearly as it should be.

48. Harsimranpal 说：

但是，我需要更多的信息，当你展示实际使用python，你可以为它提供JAVA语言..

49. 塞巴斯蒂安 说：

我有点困惑，为什么TF-IDF在这种情况下，给出了负数？我们如何解读？纠正我，如果我错了，但是当载体为正值，这意味着该组件的大小确定字是该文件中有多么重要。如果是负数，我不知道如何解释它。如果我是采取向量的点积与所有积极的部件和一个负组件，这将意味着，一些部件可能负点积贡献，即使在载体有一个特定的词非常高的重视。亚洲金博宝

50. 嗨，
非常感谢您对这个主题这个详细的解释，真是太好了。无论如何，你可以给我一个提示，这可能是我的错误，我不断看到的来源：

freq_term_matrix= count_vectorizer.transform(test_set)
AttributeError: ‘matrix’ object has no attribute ‘transform’

我使用sklearn的版本错误？

51. 莫希特古普塔 说：

Awesome simple and effective explaination.Please post more topics with such awesome explainations.Looking forward for upcoming articles.
Thanks

52. Alexandro 说：

Thank you Chris, you are the only one on the web who was clear about the diagonal matrix.

53. ishpreet 说：

Great tutorial for Tf-Idf. Excellent work . Please add for cosine similarity also:)

54. sherlockatsz 说：

I understood the tf-idf calculation process. But what does that matrix mean and how can we use the tfidf matrix to calculate the similarity confuse me. can you explain that how can we use the tfidf matrix .thanks

55. lightningstrike 说：

56. Anonymous 说：

谢谢，nice post, I’m trying it out

57. Anonymous 说：

Thank you so much for such an amazing detailed explanation!

58. Akanksha Pande 说：

best explanation.. Very helpful. Can you please tell me how to plot vectors in text classification in svm.. I am working on tweets classification. I am confused please help me.

59. Koushik 说：

I learned so many things. Thanks Christian. Looking forward for your next tutorial.

60. MHR 说：

嗨，I’m sorry if i have mistaken but i could not understand how is ||Vd4||2 = 1.
the value of d4 = (0.0 ,0.89,0.44,0.0) so the normalization will be = sqrt( square(.89)+square(.44))=sqrt(.193) = .44

61. Cuiqing Li 说：

嗨，这是一个伟大的博客！
如果我需要做双克的情况下，我该如何使用sklearn来完成呢？

62. alireza 说：

it is very great. i love your teach. very very good

63. Ritesh 说：

我没有得到相同的结果，当我执行相同的脚本。
print (“IDF:”, tfidf.idf_) : IDF: [ 2.09861229 1. 1.40546511 1. ]

My python version is: 3.5
Scikit Learn version is: o.18.1

什么我需要改变？可能是什么可能的错误？

谢谢，

1. 它可以是很多东西，因为你使用的是不同的Python解释器的版本也不同Scikit-学习版，你应该会在结果的差异，因为他们可能已经改变了默认参数，算法，圆等

1. Ravithej Chikkala 说：

我也越来越：IDF：2.09861229 1 1.40546511 1]

64. Victor 说：

Perfect introduction!
No hocus pocus. Clear and simple, as technology should be.
非常感谢你。亚洲金博宝
Keep posting!

65. Hitesh Nankani 说：

为什么| d |= 2，在IDF方程。它不应该是4，因为| d |代表的审议的文件数量，我们有2从测试，2个来自火车。

66. LÊ VĂN HẠNH 说：

这篇文章很有意思。我喜欢这个岗位？

67. 布伦 说：

clear cut and to the point explanations….great

68. Shipika Singh 说：

hey , hii Christian
您的文章是真正帮助我了解从基础TFD-IDF。我在分类的一个项目，其中我使用向量空间模型，这导致在确定类别在我的测试文档应该存在。机器学习的一部分。如果你认为我有关的东西这将是巨大的。我被困在这一点上。
thank you

69. Eshwar期基于g 说：

看到this example to know how to use it for the text classification process. “This” link does not work any more. Can you please provide a relevant link for the example.

Thanks

70. amanda 说：

such a great explanation! thankyou!

71. alternative investing 说：

哇，真棒post.Much再次感谢。将阅读

72. Android应用下载PC 说：

Say, you got a nice post.Really thank you! Fantastic.

73. togel online 说：

Wow, great article post.Much thanks again. Awesome.

74. Mobile Computer 说：

75. chocopie 说：

1vbXlh你提出了一个非常美妙的细节，欣赏它的职位。亚洲金博宝

76. 我知道这个网站提供基于高质量的文章或
reviews and additional data, is there any other web page which presents these kinds of
在质量信息？

77. 鲁塞 说：

在第一个例子。IDF（T1），日志（2/1）由计算器= 0.3010。为什么他们获得0.69 ..请有什么不对？

This site uses Akismet to reduce spam.了解您的意见如何处理数据