|
It's easy to make mistakes when testing software or planning a testing effort. Some mistakes are made so often, so repeatedly, by so many different people, that they deserve the label Classic Mistake.
在測試軟件或制訂測試工作計劃時很容易犯一些錯誤。有些錯誤經(jīng)常被許多不同的人一而再、再而三地犯，應該被列為典型錯誤。
Classic mistakes cluster usefully into five groups, which I've called "themes":
典型錯誤可以有效地分為五組，我把這些組稱為“主題”。
· The Role of Testing: who does the testing team serve, and how does it do that?
· 測試的作用：誰承擔測試小組的責任，如何做？
· Planning the Testing Effort: how should the whole team's work be organized?
· 制訂測試工作計劃：應該如何組織整個小組的工作？
· Personnel Issues: who should test?
· 人員問題：誰應該做測試？
· The Tester at Work: designing, writing, and maintaining individual tests.
· 工作中的測試員：設計、編寫和維護各測試。
· Technology Rampant: quick technological fixes for hard problems.
· 過度使用技術：艱難問題的快速技術修復
I have two goals for this paper. First, it should identify the mistakes, put them in context, describe why they're mistakes, and suggest alternatives. Because the context of one mistake is usually prior mistakes, the paper is written in a narrative style rather than as a list that can be read in any order. Second, the paper should be a handy checklist of mistakes. For that reason, the classic mistakes are printed in a larger bold font when they appear in the text, and they're also summarized at the end.
本文有兩個目標。第一，應當識別錯誤，將它們放到具體環(huán)境中，描述它們?yōu)槭裁词清e誤，并給出替代方法的建議。因為一個錯誤的具體環(huán)境通常是先決錯誤，所以本文將以敘事的方式而不是以可以按任意順序閱讀的列表方式來描述。第二，本文應該是一個便于查看的錯誤列表。因為這個原因，文章中出現(xiàn)的典型錯誤都以大號粗體字印刷，并在文章的結尾處匯總。
Although many of these mistakes apply to all types of software projects, my specific focus is the testing of commercial software products, not custom software or software that is safety critical or mission critical.
雖然這些錯誤很多都適用于所有類型的軟件項目，但我的重點將放在商用軟件產(chǎn)品的測試上，而不是定制軟件或者是高度安全或關鍵任務的軟件測試上。
This paper is essentially a series of bug reports for the testing process. You may think some of them are features, not bugs. You may disagree with the severities I assign. You may want more information to help in debugging, or want to volunteer information of your own. Any decent bug reporting system will treat the original bug report as the first part of a conversation. So should it be with this paper. Therefore, follow this link for an ongoing discussion of this topic.
本文主要是測試過程的一系列錯誤報告。你可能認為它們中的部分屬于特性問題而不是 bug。你可能不贊成我設定的嚴重性級別。你可能需要更多的信息以用于幫助排除錯誤，或者希望提供你自己的信息。任何設計良好的錯誤報告系統(tǒng)都將原始的錯誤報告當作是對話的起始部分。本文也是這樣，所以，可以按照鏈接參加這個主題的討論。
Theme One: The Role of Testing
主題一：測試的作用
A first major mistake people make is thinking that the testing team is responsible for assuring quality. This role, often assigned to the first testing team in an organization, makes it the last defense, the barrier between the development team (accused of producing bad quality) and the customer (who must be protected from them). It's characterized by a testing team (often called the "Quality Assurance Group") that has formal authority to prevent shipment of the product. That in itself is a disheartening task: the testing team can't improve quality, only enforce a minimal level. Worse, that authority is usually more apparent than real. Discovering that, together with the perverse incentives of telling developers that quality is someone else's job, leads to testing teams and testers who are disillusioned, cynical, and view themselves as victims. We've learned from Deming and others that products are better and cheaper to produce when everyone, at every stage in development, is responsible for the quality of their work ([Deming86], [Ishikawa85]).
人們犯的第一個主要錯誤是認為測試小組應當負責質(zhì)量保證。這個角色常常分配給組織中的第一測試小組，將它作為最后的防御，成為開發(fā)小組（被指責為產(chǎn)生低劣質(zhì)量）和客戶（必須受到保護以遠離低劣質(zhì)量）的一個屏障。它的特征是測試小組（常稱為“質(zhì)量保證組”）表面上具有阻止產(chǎn)品發(fā)貨的權力。 這本身是一個令人沮喪的任務：測試小組不能提高質(zhì)量，只能強制一個最低水平。更糟糕的是，這種權力常常是看上去比實際的重要。如果發(fā)現(xiàn)這一點，再加上有違常理地暗示開發(fā)人員質(zhì)量是別人的事情，導致測試小組和測試員感到失望、憤事嫉俗、感覺自己是受害者。我們從Deming 和其他人的工作可以得知：如果每個人都在開發(fā)的各個階段對他們的工作質(zhì)量負責，則產(chǎn)品會又好又便宜（[Deming86]，[Ishikawa85]）。
In practice, whatever the formal role, most organizations believe that the purpose of testing is to find bugs. This is a less pernicious definition than the previous one, but it's missing a key word. When I talk to programmers and development managers about testers, one key sentence keeps coming up: "Testers aren't finding the important bugs." Sometimes that's just griping, sometimes it's because the programmers have a skewed sense of what's important, but I regret to say that all too often it's valid criticism. Too many bug reports from testers are minor or irrelevant, and too many important bugs are missed.
實際上，不管表面上的作用是什么，大多數(shù)組織都相信測試的目的是發(fā)現(xiàn) bug。這個定義的危害比前一個定義的危害要小，但是忽略了一個關鍵詞。當我同程序員和開發(fā)經(jīng)理談到測試員的時候，不時聽到一個關鍵的句子：測試員找不到重要的 bug。有時候這種說法只是一種抱怨，有時候是因為程序員對于什么是正確的感覺不對，但我很遺憾地說，它們經(jīng)常是有效的批評。測試員的太多的bug 報告是微小的、不相關的，而有太多重要的錯誤都被遺漏了。
What's an important bug? Important to whom? To a first approximation, the answer must be "to customers". Almost everyone will nod their head upon hearing this definition, but do they mean it? Here's a test of your organization's maturity. Suppose your product is a system that accepts email requests for service. As soon as a request is received, it sends a reply that says "your request of 5/12/97 was accepted and its reference ID is NIC-051297-3". A tester who sends in many requests per day finds she has difficulty keeping track of which request goes with which ID. She wishes that the original request were appended to the acknowledgement. Furthermore, she realizes that some customers will also generate many requests per day, so would also appreciate this feature. Would she:
什么是重要的 bug？對誰而言是重要的？直觀的估計，答案肯定是“對于客戶”。聽到這個定義，幾乎每個人都會點頭稱是，但他們確實這樣認為嗎？這里要測試一下你們組織的成熟度。假設你們的產(chǎn)品是一個接受電子郵件請求服務的系統(tǒng)。當收到請求時，它馬上發(fā)送一個“您在97年5月12日發(fā)送的請求已經(jīng)受理，參考ID是NIC-051297-3”的答復。一個每天發(fā)送很多請求的測試員發(fā)現(xiàn)要分清楚哪個請求與哪個ID對應是非常困難的。她希望最初的請求能夠附加在確認郵件的后面。并且，她意識到某些可戶可能每天也會產(chǎn)生很多請求，所以會高度評價這個功能的。那么她將：
1. file a bug report documenting a usability problem, with the expectation that it will be assigned a reasonably high priority (because the fix is clearly useful to everyone, important to some users, and easy to do)?
寫一個 bug 報告，記錄一個可用性問題，希望能夠分配一個合理的高優(yōu)先級（因為這個修復很明顯對每個人都很用，對有部分用戶來說還非常重要，并且也容易修改）？
2. file a bug report with the expectation that it will be assigned "enhancement request" priority and disappear forever into the bug database?
寫一個 bug 報告，希望它被分配為“功能提升請求”優(yōu)先級并永遠從 bug 數(shù)據(jù)庫中消失？
3. file a bug report that yields a "works as designed" resolution code, perhaps with an email "nastygram" from a programmer or the development manager?
寫一個 bug 報告，產(chǎn)生一個“按設計工作”解決碼，可能還加上一個來自程序員或開發(fā)經(jīng)理的“不同意”電子郵件?
4. not bother with a bug report because it would end up in cases (2) or (3)?
不打算費事去寫 bug 報告，因為它將以情況(2)或(3)結束？
If usability problems are not considered valid bugs, your project defines the testing task too narrowly. Testers are restricted to checking whether the product does what was intended, not whether what was intended is useful. Customers do not care about the distinction, and testers shouldn't either.
如果可用性問題不認為是有效的 bug，那么你們的項目將測試任務定義得太狹窄了。測試員嚴格限制為檢查產(chǎn)品是否按預期工作，而不管這種預期是否有效。客戶不關心這個區(qū)別，測試員也不應該關心。
Testers are often the only people in the organization who use the system as heavily as an expert. They notice usability problems that experts will see. (Formal usability testing almost invariably concentrates on novice users.) Expert customers often don't report usability problems, because they've been trained to know it's not worth their time. Instead, they wait (in vain, perhaps) for a more usable product and switch to it. Testers can prevent that lost revenue.
測試員經(jīng)常是組織中唯一像專家一樣大量使用系統(tǒng)的人。他們會注意到專家會看到的可用性問題。（形式上的可用性測試幾乎不可避免地集中于沒有經(jīng)驗的用戶。）專家客戶常常不會報告可用性問題，因為他們已經(jīng)被訓練的知道不值得花時間去這樣做。相反，他們（也許是徒勞地）等待下一個可用的產(chǎn)品然后切換過去。測試員可以避免這個損失。
While defining the purpose of testing as "finding bugs important to customers" is a step forward, it's more restrictive than I like. It means that there is no focus on an estimate of quality (and on the quality of that estimate). Consider these two situations for a product with five subsystems.
將測試的目的定義為“找到對用戶重要的 bug ”是向前進了一步，但與我所喜歡定義相比仍有限制。這意味著沒有集中于質(zhì)量評估（以及這種評估的質(zhì)量）。考慮一下測試含有五個子系統(tǒng)的產(chǎn)品的兩種情況。
1. 100 bugs are found in subsystem 1 before release. (For simplicity, assume that all bugs are of the highest priority.) No bugs are found in the other subsystems. After release, no bugs are reported in subsystem 1, but 12 bugs are found in each of the other subsystems.
在發(fā)布前，在子系統(tǒng)1中找到了100個bug 。（為了簡單起見，假設所有的 bug 都是最高級別的。）在其他子系統(tǒng)中沒有發(fā)現(xiàn) bug 。在發(fā)布后，在子系統(tǒng)1中沒有報告 bug ，但在其他每個子系統(tǒng)中都報告了12個 bug 。
2. Before release, 50 bugs are found in subsystem 1. 6 bugs are found in each of the other subsystems. After release, 50 bugs are found in subsystem 1 and 6 bugs in each of the other subsystems.
在發(fā)布前，在子系統(tǒng)1中找到了50個 bug 。在其他每個子系統(tǒng)中都找到了6個 bug 。在發(fā)布后，在子系統(tǒng)1中報告了50個 bug ，在其他每個子系統(tǒng)中都報告了6個 bug。
From the "find important bugs" standpoint, the first testing effort was superior. It found 100 bugs before release, whereas the second found only 74. But I think you can make a strong case that the second effort is more useful in practical terms. Let me restate the two situations in terms of what a test manager might say before release:
從“找到重要 bug”的觀點看，第1種測試情況較為理想。在發(fā)布前找到了100個 bug ，但是第2種情況中只找到74個。但我想你們可能會提出一個有力的理由認為第2中測試在實際中更有用。讓我以產(chǎn)品發(fā)版前測試經(jīng)理可能說些什么來重新描述一下兩種測試情況：
1. "We have tested subsystem 1 very thoroughly, and we believe we've found almost all of the priority 1 bugs. Unfortunately, we don't know anything about the bugginess of the remaining five subsystems."
“我們?nèi)鏈y試了子系統(tǒng)1，我們相信已經(jīng)找出了幾乎所有優(yōu)先級為1的 bug。不幸的是，我們對其他五個子系統(tǒng)的的 bug 一無所知。”
2. "We've tested all subsystems moderately thoroughly. Subsystem 1 is still very buggy. The other subsystems are about 1/10th as buggy, though we're sure bugs remain."
“我們比較全面地測試了所有的子系統(tǒng)。子系統(tǒng)1仍舊有不少 bug。其他子系統(tǒng)雖然還有 bug，但只有子系統(tǒng)1的 bug 的十分之一。”
This is, admittedly, an extreme example, but it demonstrates an important point. The project manager has a tough decision: would it be better to hold on to the product for more work, or should it be shipped now? Many factors - all rough estimates of possible futures - have to be weighed: Will a competitor beat us to release and tie up the market? Will dropping an unfinished feature to make it into a particular magazine's special "Java Development Environments" issue cause us to suffer in the review? Will critical customer X be more annoyed by a schedule slip or by a shaky product? Will the product be buggy enough that profits will be eaten up by support costs or, worse, a recall?
必須承認，這是一個極端的例子，但是證明了一個重要的觀點。項目經(jīng)理有一個艱難的決定：是延遲產(chǎn)品交付，再工作一段時間，還是現(xiàn)在就交付使用？許多因素——都是一些大致的評估——都必須予以權衡：競爭對手會搶先發(fā)布產(chǎn)品并占領市場嗎？如果丟掉一個未完工的功能部件會使得某一個雜志的 “Java 開發(fā)環(huán)境” 特別期刊的評論中對我們造成損害嗎？關鍵客戶 X 對產(chǎn)品延期和劣質(zhì)產(chǎn)品哪一個更感到煩惱？產(chǎn)品是否有很多 bug，以至于支持成本會吃掉利潤，或者更糟糕的是將產(chǎn)品召回？
The testing team will serve the project manager better if it concentrates first on providing estimates of product bugginess (reducing uncertainty), then on finding more of the bugs that are estimated to be there. That affects test planning, the topic of the next theme.
如果測試小組首先集中于產(chǎn)品錯誤的估計（減少不確定性），然后再找到更多的錯誤，他們會更好地服務于項目經(jīng)理。這會影響測試計劃。測試計劃將在下個主題中論述。
It also affects status reporting. Test managers often err by reporting bug data without putting it into context. Without context, project management tends to focus on one graph:
這也會影響狀態(tài)報告。測試經(jīng)理常常會被沒有放到具體環(huán)境中的報告 bug數(shù)據(jù)誤導。沒有具體環(huán)境，項目管理傾向于集中于一幅圖：
The flattening in the curve of bugs found will be interpreted in the most optimistic possible way unless you as test manager explain the limitations of the data:
平滑的錯誤曲線很容易以一種樂觀的方式解釋，除非你作為測試經(jīng)理解釋了數(shù)據(jù)的局限：
· "Only half the planned testing tasks have been finished, so little is known about half the areas in the project. There could soon be a big spike in the number of bugs found."
· 只有一半的計劃測試做完了，對于項目的一半所知甚少。很快就有很多錯誤要被發(fā)現(xiàn)了。
· "That's especially likely because the last two weekly builds have been lightly tested. I told the testers to take their vacations now, before the project hits crunch mode."
· 很有可能這樣，因為在過去的兩個周構建只是略微測試了一下。我告訴測試員在項目進入艱難狀態(tài)之前，現(xiàn)在開始休假。
· "Furthermore, based on previous projects with similar amounts and kinds of testing effort, it's reasonable to expect at least 45 priority-1 bugs remain undiscovered. Historically, that's pretty high for a successful product."
· 并且，根據(jù)以前的經(jīng)驗，可以預料到至少還有45個級別為1的 bug還沒有發(fā)現(xiàn)。從歷史看，這對于一個成功產(chǎn)品來說是很高的。
For discussions of using bug data, see [Cusumano95], [Rothman96], and [Marick97].
關于使用 bug 數(shù)據(jù)的討論，請參閱[Cusumano95]、[Rothman96]和[Marick97]。
Earlier I asserted that testers can't directly improve quality; they can only measure it. That's true only if you find yourself starting testing too late. Tests designed before coding begins can improve quality. They inform the developer of the kinds of tests that will be run, including the special cases that will be checked. The developer can use that information while thinking about the design, during design inspections, and in his own developer testing.
我在前面說過，測試員不能直接提高質(zhì)量，他們只能評估它。只有在你發(fā)現(xiàn)測試開始得太晚的時候，這種說法才是正確的。在編碼開始前設計測試將會提高質(zhì)量。他們讓開發(fā)人員知道將進行什么樣的測試，將檢查哪些特殊用例。開發(fā)人員在思考設計、審查設計和自己做測試的時候可以使用這些信息。
Early test design can do more than prevent coding bugs. As will be discussed in the next theme, many tests will represent user tasks. The process of designing them can find user interface and usability problems before expensive rework is required. I've found problems like no user-visible place for error messages to go, pluggable modules that didn't fit together, two screens that had to be used together but could not be displayed simultaneously, and "obvious" functions that couldn't be performed. Test design fits nicely into any usability engineering effort ([Nielsen93]) as a way of finding specification bugs.
盡早測試的作用不僅僅是防止編碼錯誤。像我們將在下一個主題中所討論的那樣，許多測試代表的是用戶任務。設計它們的過程可以在昂貴的重新工作之前發(fā)現(xiàn)用戶界面和可用性問題。我發(fā)現(xiàn)過的問題包括：錯誤消息不能顯示在用戶可以看到的地方，插件不能放到一起，兩個必須同時使用的屏幕不能同時顯示，一個“很明顯”的功能不能執(zhí)行。測試設計作為一個發(fā)現(xiàn)規(guī)格說明書 bug 的方法，很好地與可用性工程工作相適應([Nielsen93])。
I should note that involving testing early feels unnatural to many programmers and development managers. There may be feelings that you are intruding on their turf or not giving them the chance to make the mistakes that are an essential part of design. Take care, especially at first, not to increase their workload or slow them down. It may take one or two entire projects to establish your credibility and usefulness.
我應當說明早期介入測試對于許多程序員和開發(fā)經(jīng)理來說不自然。可能有一種感覺是你干擾了他們，沒有給他們在設計的基礎部分犯錯誤的機會。小心些，尤其是在開始的時候，不要增加他們的工作量或減慢了他們的速度�？赡苄枰恢羶蓚�完整的項目才能建立你們的可信度并顯示出作用。
內(nèi)容太多，共31頁。后面的見附件。
掃碼關注5G通信官方公眾號,免費領取以下5G精品資料
1、回復“YD5GAI”免費領取《中國移動：5G網(wǎng)絡AI應用典型場景技術解決方案白皮書》
2、回復“5G6G”免費領取《5G_6G毫米波測試技術白皮書-2022_03-21》
3、回復“YD6G”免費領取《中國移動：6G至簡無線接入網(wǎng)白皮書》
4、回復“LTBPS”免費領取《《中國聯(lián)通5G終端白皮書》》
5、回復“ZGDX”免費領取《中國電信5G NTN技術白皮書》
6、回復“TXSB”免費領取《通信設備安裝工程施工工藝圖解》
7、回復“YDSL”免費領取《中國移動算力并網(wǎng)白皮書》
8、回復“5GX3”免費領取《 R16 23501-g60 5G的系統(tǒng)架構1》
作者的更多帖子
|
|
|
|
|
| |
|
關注樓主