Expert quality assessment or qualitative assessment by experts. Classification and characteristics of expert methods

The mechanism of consumer choice is quite conservative, since it is based, as a rule, on using only what has already proven itself well. It is more focused on the past than on the possible future.

In market conditions, the basis for assessing the quality of products should be the objective social usefulness of the product, which will reflect its progressiveness and compliance with the features of the new way of life. One of the methods that can provide just such an assessment of products can be the expert method.

Expert method Product quality assessment involves asking experts for their opinions.

Expert is a qualified specialist who meets the requirements of professional and qualimetric competence, efficiency and objectivity, and is also interested in the work of the expert commission.

The use of expert methods is rational if there are one of two reasons:

· the task cannot be completed by any other method;

· other methods of completing a task are either less accurate or more labor-intensive.

Expert methods, together with others or independently, can be used to:

· product classifications;

· determining the range of product quality indicators;

· determination of weight coefficients for individual quality indicators;

· selection of basic product samples and determination of the values ​​of individual quality indicators of these samples;

· determination of organoleptic quality indicators of the products being evaluated;

· determination of complex indicators of product quality.

Involving highly qualified experts in expert commissions makes it possible to obtain an accurate and reproducible assessment of the quality of goods. The experiments carried out confirm that with the correct expert assessment methodology, the error of the results is within 5–10%, and it can be entirely compared with the results of instrumental methods.

To assess the quality of products using the expert method, expert commissions are created. The procedure for creating, goals and tasks of such a commission are determined by order or directive of the head of the ministry, department, association, company. The general algorithm for the creation and operation of an expert commission, which uses group survey methods, is shown in Figure 7.1.

The work of the expert group ends with statistical processing expert assessments, their analysis and making informed decisions.

The main advantages of group expert assessment is the possibility of a comprehensive analysis of problems associated with determining individual characteristics of a product. Interaction between experts increases the amount of total information compared to the information of each member of the group. With a group assessment, there is less chance of error in the final result of the work. In most cases, a group assessment is more reliable than an individual assessment by each expert.

The working group prepares and conducts an expert assessment of product quality. Its task is to prepare experts, assist in quality assessment, processing, analysis, summarizing the results and establishing a collective opinion of experts. It begins its activities with the selection of experts and the formation of an expert group.

Rice. 7.1. General algorithm of operations for the creation and work of an expert commission

The expert commission consists of two groups – working and expert.

The working group includes:

· organizer(he is also the chairman of the expert commission) provides methodological guidance to the work at all stages. He must be a qualified specialist in the field of methodology for assessing product quality, know the basics of its production and the processes that may take place in this product, during its production, transportation and storage. He creates a working group; draws up a work program; takes part in a survey of experts; analyzes the results of each operation, as well as the final results, formulates conclusions and proposals based on the results of the tasting;

· specialist - This is a highly qualified employee of the company where an expert commission is formed. It is desirable that he has universal knowledge about the product, relating to both its production and features of use. The specialist’s task is to analyze the information received from experts in order to adjust the further work of the company. programmer. If the assessment of product quality or processing of results is carried out using computer technology, then the main task of this specialist is to develop a program for carrying out such work and subsequent processing of the results obtained;

· technical workers explain the provisions of the questionnaire if they are not clear enough to the experts, conduct a survey of them, receive questionnaires from the experts, can clarify the results, but should not express their opinion regarding the experts’ assessment.

Formation of an expert group. The organizer and the specialist draw up a list of experts (the official position, time and nature of the work, and profession are taken into account). The group should include specialists from different areas of production and product quality assessment (design, production technology, aesthetics, quality research).

An expert group can be formed from employees of one or more companies that produce (sell) products of this group. In this case, it is necessary to take into account the trends and views on product quality that prevail in a given company, the official position (subordination) of experts, the creation and maintenance of a psychological climate in the group. It is advisable to give preference to creating an expert group based on one company with the invitation to work of several specialists from other companies.

The result of the work of the expert commission largely depends on the number of experts and their competence. Selecting experts is an extremely difficult task. It is due to the fact that:

· it is necessary to take into account the individual abilities of experts, but there is no reliable method for determining these abilities;

· the psychophysiological characteristics of a person, on which the work of experts largely depends, have not been sufficiently studied;

· there is a contradiction between the desire to reduce the cost of examination by using a limited number of experts to assess the quality of a wide range of products and the desire to increase the accuracy and reliability of expert assessment.

According to G.G. Azgaldov and E.P. Reichman, the competence of experts is determined by 4 groups of properties (Fig. 7.2).

Professional training provides knowledge of various aspects of product design and production, consumer requirements, conditions and nature of use (operation).

Qualimetric training ensures that the expert clearly understands the principles and methods of assessing product quality, knowledge of methods for determining individual quality indicators and the ability to use them in the process of practical work (use of various types of rating scales, the ability to distinguish a sufficient number of gradations of the products being assessed).

Rice. 7.2. Expert competence indicators

Interest expert in the work of the expert commission depends on many factors:

· examination purposes;

· the nature of the conclusions;

· individual characteristics of the expert,

· employment at the main job;

· the possibility of using the results of expert assessment in your own practical work.

Efficiency expert includes mobility and elasticity of attention (the ability to quickly switch attention from one indicator to another); composure; efficiency; validity of conclusions; contact (ability to work with people in conflict situations); motivation of the given assessment.

Objectivity expert lies in the fact that he draws conclusions based on the actual level of quality of the products being assessed, and is not guided by other opinions.

Methods for assessing the competence of experts divided into 5 groups: heuristic, statistical, test, documentary and combined.

Heuristic(from the Greek heurisko - I search, I open) the expert’s assessment is based on the assessment of the people around him or self-esteem. There are the following types of heuristic assessments: self-assessment, mutual assessment, work group assessment.

Self-esteem is a method in which the expert himself gives an assessment of his own competence. It has been established that the greater the average value of self-assessment of members of the expert group, the higher the reliability of the average expert assessment. Meanwhile, it must be borne in mind that the results of self-assessment are almost always subjective, which is a disadvantage of this method.

In order to increase the accuracy of self-assessment results, it is carried out in a differentiated manner. The self-assessment indicator is determined as a function of two coefficients: awareness and knowledge of the products being assessed and the reasoning of one’s assessment. T.G. Rodnina and G.O. Vooks propose to determine the expert’s awareness and knowledge of the products being assessed using the “Self-Assessment Questionnaire” (Appendix A). In the questionnaire, the expert notes the regularity of his acquaintance with sources of information, as well as the degree of his familiarity with the products being assessed. Self-esteem is calculated using the formula:

K himself = In esp I ex,

where K itself is the expert’s self-esteem coefficient;

In ex – the weight of indicators of awareness and knowledge;

I ex is an assessment that depends on the degree of awareness.

An example of the calculation of K itself is given in Appendix B.

Mutual assessment– this is a heuristic assessment of an expert, which is given by other experts; it should reduce the subjectivity of each expert’s self-assessment. It is known that there is a relationship between the competence of an expert and the average rating he received from his colleagues. The essence of this type of assessment is that each member of the expert group gives an assessment to all other experts, and then the average assessment of each expert is calculated.

Disadvantages of mutual assessment:

· experts may not know each other well enough;

· the assessment results may be influenced by mutual likes or dislikes;

· different experts may have different perceptions of the concept of “expert competence”;

· use mutual assessment only in those commissions where the experts know each other well;

· Conduct anonymous surveys;

· explain that the results of the survey will only be used to adjust product ratings.

Expert assessment working group is a heuristic assessment that should give a quantitative description of the expert’s interest in the expert assessment and his attentiveness during the survey process. The assessment is given by specialist analysts who conduct a survey of experts after an expert assessment of the product. At the same time, they evaluate the experts’ attitude towards the examination as well as their activity in discussing the results. Most often, it is recommended to use a 10-point score for such an assessment.

Statistical evaluation. – this is an assessment obtained after statistical processing of expert opinions about the object of assessment. These estimates are used to reduce errors that arise during expert assessments. The need for statistical processing of examination results is due to the fact that, unlike measurements that are carried out using technical means and are based on comparison of unknown quantities with known ones, with expert methods there is often no known quantity. In addition, the accuracy of expert assessments is influenced by many objective and subjective factors, resulting in systematic or random errors.

A systematic error is an error that is constantly repeated in the process. this definition. The main reason for its appearance is insufficient or inaccurate information from experts. It can be reduced by familiarizing experts with necessary information before the start of the expert commission's work. Systematic error can be determined by the deviation of each expert’s assessment from the average value obtained in the expert commission.

A random error depends on the psychophysiological characteristics of the expert (composure, attentiveness, confidence in one’s own rightness). It decreases if the expert often takes part in the work of expert commissions and has practical experience.

Assessing the objectivity of the expert- this is an assessment of the expert’s ability to unintentionally evaluate a specific sample of a product. This is a very important characteristic of the expert’s competence, which significantly affects the accuracy of the expert assessment results.

Test scores allow, through testing, to determine the psychophysiological characteristics of experts. They are used to assess the objectivity, qualimetric and professional preparedness of the expert. Testing allows you to assess personality traits that cannot be done by any other methods. Disadvantage this method it may be considered impossible to compare data obtained by the testing method with data obtained using other objective methods.

When compiling tests, you must adhere to the following requirements:

· the expert must understand the statement test task and the conditions to which his decision meets;

· the probability of randomly guessing the solution to a task should be close to zero;

· the test task must have an exact answer;

· it is necessary to justify the approximation of the test task and real problems that the expert solves when assessing product quality.

Assessment of qualimetric training expert is an assessment theoretical knowledge methods for assessing product quality indicators and the ability to use them in practical work. This assessment can be carried out through oral or written knowledge assessment using tests.

It is somewhat more difficult to test such knowledge as the ability to use Various types rating scales, determine subjective probability, recognize a sufficient number of gradations of quality indicators.

Documentary assessment– this is an expert’s assessment based on documents that confirm his compliance with established requirements. For example, the expert's competence can be confirmed general information which are given in the application form: experience – copy work book; professional training– diploma vocational education; special training - a certificate or a standard certificate of special training; advanced training – with a corresponding certificate.

Documentary assessment can only be an addition to other methods of assessing the expert’s competence and has no independent significance. This is due to the fact that some of the information in the documents duplicates other assessments.

Combined assessment involves the use of various methods for assessing the competence of experts. Since different methods of this assessment have their own advantages and disadvantages, their combined use can enhance the advantages and mitigate the disadvantages of individual methods.

Control questions and tasks

1. What is the essence of consumer sensory analysis methods?

2. What conditions must be created when creating tasting commissions for consumer assessment?

3. What rules must be followed when compiling a consumer assessment questionnaire?

4. What is the essence of paired and triangular research methods?

5. What is the essence of analytical methods of sensory analysis?

6. For what purpose is the dilution method most often used?

7. What is the essence of scoring the quality of goods?

8. What requirements must be observed when developing a point system for assessing the quality of a product?

9. What is the purpose of establishing significance coefficients for quality indicators?

10. What is the essence of the expert method for assessing the quality of goods?

11. Name the main stages and operations that take place during an expert assessment.

12. What methods exist for assessing the competence of experts?

13. Describe the main methods for assessing the competence of experts (heuristic, test, documentary).

Expert assessments often arise in practice, for example, when tasting food products. In general, they are typical for sociological surveys, for example, a quality control manager conducts a survey of customers in a supermarket. When using the expert method, an order scale is often used to assess quality. The issue of comparison is resolved according to the principle “better is worse”, “more is less”. More detailed information how many times better or worse is often not required.

Pairwise comparison (objects are compared to each other in pairs). When constructing an order scale or so-called ranked series, experts use the method pairwise comparison. In tab. Figure 1 shows an example of ranking six objects using pairwise comparison. This is the result of the work of one expert who assessed objects in a certain way. Preference for one object over another is indicated by 1, the opposite situation is indicated by 0.

Ranked series(order scale) for objects, the comparative assessment of which is given in table. 1 will look like:

Q4

If you use several experts, you can get a more accurate result.

You can use more advanced criteria, for example, advantage is determined by a score of 1, worse quality is determined by a score of -1, and equivalent quality is determined by a score of 0. The mechanism for compiling a ranked series remains the same.

Psychologists have proven that pairwise comparison underlies any choice (i.e. you choose products by comparing them in pairs), however, the order scale is often drawn up in advance (not a ranked series) and reference points are fixed in it, which are called points.

This is how the twelve-point scale of earthquake intensity MSK - 64, the Mohs mineralogical scale, the five-point scale for assessing knowledge, points in figure skating, etc. appeared. Table 2 shows the Mohs mineral hardness scale as an example.

Table 1. Ranking of six objects by pairwise comparison

Table 2. Mohs scale

Each subsequent mineral leaves a scratch on the previous one, i.e. is more solid. The measurement results obtained by pairwise comparison can be refined by the method of successive approximation.

The influence of the composition of experts on the results of the examination

When forming an expert group, it is advisable to conduct testing, mutual assessment of experts and check the consistency of opinions.

Testing consists of experts solving problems with results known to the testing organizers, but unknown to the experts, and testing, using the Fisher criterion, the hypothesis that the assessments of different experts belong to the same general population of assessments.

Self-assessment consists of each expert answering questions in a specially designed questionnaire within a limited time. This test is carried out on a computer and then scored. Experts can evaluate each other, but this requires a trusting environment and experience of working together. The consistency of expert opinion can be assessed by the value of the concordance coefficient:

Where S- the sum of squared deviations of all rank estimates for each object of examination from the average value;
n- number of experts;
m- number of objects of examination.

The concordance coefficient varies in the range 0<W<1, причем 0 - полная несогласованность, 1 - полное единодушие.

Example

It is necessary to determine the degree of consistency in the opinions of five experts, the results of which ranked seven objects are shown in Table 3. To determine the degree of consistency, a special measure is used - the Kendall concordance coefficient (from the Latin. concordare- bring into line, organize).

Table 3. Data for assessing the consistency of opinions of five experts

We estimate the arithmetic mean number of ranks:

Q avg = (21 + 15 + 9 + 28 + 7 + 25 + 35)/7 = 20.

Then we estimate the sum of squared deviations from the mean: S = 630. We determine the value of the concordance coefficient:

W = 12 * 630 / 25 * (343 - 7) = 0,9.

Is it a lot or a little? If we carry out the corresponding calculations in STATISTICA, then you can get the following table of results:

Rice. 1. Results of analysis in STATISTICA

From this table it follows that different expert opinions in this example are insignificant: p<0.00014.

Influence on the results of the examination of the quantitative composition of experts. As the number of experts in the group increases, the accuracy of the measurement increases, which is typical for repeated measurements.

Number of experts n, ensuring a given measurement accuracy, can be established by knowing the law of distribution of expert opinions and the maximum permissible standard error of estimation Sx

The mechanism of consumer choice is quite conservative, since it is based, as a rule, on using only what has already proven itself well. It is more focused on the past than on the possible future.

In market conditions, the basis for assessing the quality of products should be the objective social usefulness of the product, which will reflect its progressiveness and compliance with the features of the new way of life. One of the methods that can provide just such an assessment of products can be the expert method.

Expert method Product quality assessment involves asking experts for their opinions.

Expert is a qualified specialist who meets the requirements of professional and qualimetric competence, efficiency and objectivity, and is also interested in the work of the expert commission.

The use of expert methods is rational if there are one of two reasons:

the task cannot be completed by any other method;

other methods of completing a task are either less accurate or more labor-intensive.

Expert methods, together with others or independently, can be used to:

product classifications;

determining the range of product quality indicators;

determining the weight coefficients of individual quality indicators;

selection of basic product samples and determination of the values ​​of individual quality indicators of these samples;

determination of organoleptic quality indicators of the products being evaluated;

determination of complex indicators of product quality.

Involving highly qualified experts in expert commissions makes it possible to obtain an accurate and reproducible assessment of the quality of goods. The experiments carried out confirm that with the correct expert assessment methodology, the error of the results is within 5–10%, and it can be entirely compared with the results of instrumental methods.

To assess the quality of products using the expert method, expert commissions are created. The procedure for creating, goals and tasks of such a commission are determined by order or directive of the head of the ministry, department, association, company. The general algorithm for the creation and operation of an expert commission, which uses group survey methods, is shown in Figure 7.1.

The work of the expert group ends with statistical processing of expert assessments, their analysis and making informed decisions.

The main advantages of group expert assessment is the possibility of a comprehensive analysis of problems associated with determining individual characteristics of a product. Interaction between experts increases the amount of total information compared to the information of each member of the group. With a group assessment, there is less chance of error in the final result of the work. In most cases, a group assessment is more reliable than an individual assessment by each expert.

The working group prepares and conducts an expert assessment of product quality. Its task is to prepare experts, assist in quality assessment, processing, analysis, summarizing the results and establishing a collective opinion of experts. It begins its activities with the selection of experts and the formation of an expert group.

An expert is a specialist (professional), whose assessments and judgments the decision maker considers useful to take into account when making decisions.

Let's give some explanations. Naturally, the opinions of a novice therapist and a world-famous cardiologist are not comparable in terms of the level of assessment of the patient’s condition and recommendations for his treatment in a rather complex situation. However, if a patient needs emergency help, and a specialist of a high professional level cannot be invited for one reason or another, then the decision will be made by whoever is nearby, even if he is less qualified. In this situation, it is he who turns out to be the expert.

Thus, the formation of the composition of the expert commission depends on:

· specific decision-making situation;

· the ability of the organizers of the examination to attract highly qualified specialists for the work;

· opportunities for specialists to take part in the work of the expert commission.

It should also be noted that at present there is no generally accepted unified methodology for assessing the qualities of an expert, therefore opinions about the professional level of a specialist often differ significantly. In real practice, the decision maker either strives to form an opinion about the professional level of the candidate expert, or entrusts this to those who are entrusted with organizing the examination. Therefore, in the absence of the capabilities and experience in organizing and conducting examinations, it makes sense to turn to the services of independent examination centers, information and analytical centers, etc., whose main tasks are to analyze the situation, assess the objects of examination, prepare and evaluate alternative solutions.

When assessing the qualities of an expert, it is necessary to take into account his professional knowledge, experience and effectiveness of work as part of expert commissions. There are many ways to assess the qualities of an expert, each of which can be successfully used in a particular case. They are divided into three main groups:

· a priori;

· a posteriori;

· test.

A priori methods are called methods for assessing the qualities of an expert that do not use information about the results of his participation in previous examinations.

A posteriori methods are called methods for assessing the qualities of an expert based on information about the results of his participation in previous examinations.

Test methods are methods for assessing the qualities of an expert that involve conducting a special test.

A priori methods for assessing an expert. This group includes, first of all, the most common methods of self-assessment, the essence of which is that the expert himself evaluates his qualities. The following methods can be used:

· self-esteem on one of the point scales (3, 5, 10 or 100 points);

· self-assessment using verbal-numeric scales, which, along with the numerical values ​​of the gradations, contain their qualitative description;

· self-assessment on a verbal scale, in which the expert gives a verbal assessment of his knowledge and experience, using qualitative gradations of a pre-developed scale;

· self-assessment using the differential method, in which the expert evaluates his qualities using two main private criteria: a criterion characterizing his familiarity with the main sources of information in this field (for example, special domestic and foreign periodicals, patent information, in-house information, etc.); etc.) and a criterion characterizing familiarity with the object of examination (for example, knowledge of industry specifics, a specific enterprise, familiarity with product samples, etc.).

In this case, the expert’s comprehensive self-assessment is calculated using the formula:

TO k = ( TO and + ß TO h) / 2,

Where TO k - comprehensive self-assessment of the expert, TO and - awareness coefficient (in fractions of unity), TO z is the coefficient of familiarity with the problem, ß is the weighting coefficient.

The issues of measuring indicators and types of scales will be discussed in more detail in the next topic.

The accuracy of self-assessment methods is somewhat controversial. On the one hand, no one better than the expert himself represents the totality of knowledge and experience that he possesses. On the other hand, a person makes the biggest mistake when assessing his own capabilities.

Another group of a priori methods for assessing the qualities of an expert includes mutual assessment methods, the use of which assumes that experts evaluate each other. The most common of these is the expert list method. The procedure for its use is as follows. Each expert compiles a list of specialists whom he considers competent in this field. Then, based on these lists, the expert’s competence coefficient is calculated - the ratio of the number of lists in which a given specialist is present to the total number of compiled lists.

The a priori methods for assessing the qualities of an expert also include the most common documentary or questionnaire method. In it, to assess certain qualities of a specialist, it is proposed to use objective characteristics that have documentary evidence, for example, such as: work experience, academic degree, academic title, position held, number of publications, citation index, etc. It should be borne in mind that, based only on these objective factors, it is impossible to adequately assess the suitability and usefulness of an expert’s participation in the work of a particular expert group.

A posteriori estimation methods. These methods involve the use of the results of his participation in previous surveys when assessing the qualities of an expert. With their help, to a certain extent, such qualities as conformism, opportunism, and expert competence can be identified. In particular, to assess the level of competence of a specialist, the method of paired comparisons can be used. Its essence lies in the fact that the expert is sequentially presented with pairs of compared evaluation objects and he selects from each pair the most preferable object from his point of view. After all pairs are presented, analysts have information about the expert’s comparative preferences regarding the objects being assessed. In this case, a situation may arise when, in a direct comparison, the first object is preferable to the second, the second is preferable to the third, but, at the same time, the third object turns out to be preferable to the first, i.e. an obvious contradiction arises. In real practice, such contradictions (of course, in longer chains of comparison) do not happen so rarely. Naturally, the fewer contradictions in an expert’s assessments, the more justified his competence is.

The a posteriori method also involves assessing the reliability of the expert’s judgments. As a criterion for assessing such reliability, the reliability coefficient is used - the relative frequency of cases in which the expert attributed the greatest probability to subsequently confirmed events. When using this method, the average value of the reliability coefficient of the expert commission is also calculated and the individual coefficients of the experts are compared with it.

Another method of a posteriori assessment is the method of deviation from the resulting group estimate. This method is based on calculating the deviation coefficient - Co.

TOoi = D oi / D max,

Where To oi- coefficient of deviation of judgments of the i-th expert, D oi- deviation of the individual assessment of the i-th expert from the resulting assessment, D max - the maximum possible deviation of the expert's assessment from the resulting assessment.

Test methods for assessing the qualities of an expert. Their essence is the execution by the subject expert of some previously prepared task. These methods are quite well known, because are widely used to determine the professional suitability of specialists in various fields of activity. The advantage of these methods is that they allow not only to establish whether an expert has a certain professional level, but also to identify the skills and experience necessary for productive participation in the work of the expert commission. It is necessary, however, to remember that conducting test experiments requires compliance with the following important rules:

· the test must be developed specifically for specific objects of expert assessment;

· the true values ​​of the parameters being assessed (correct answers) must be known to the analytical group conducting the test, but unknown to the expert being tested;

· a scale should be developed to determine the accuracy of assessments given by an expert;

· the probability of accidentally guessing the true estimate should be very small.

The use of test methods makes it possible to evaluate such an important professional quality of an expert as the reproducibility of expert assessments. To do this, several test experiments are carried out, close to the conditions of real examination. In this case, the time interval between them should be sufficient for the subject to have time to forget the results of the previous one. Then the obtained estimates are compared with each other. More stable (reproducible) expert assessments indicate both his greater professional competence and his greater suitability for participation in the expert commission.

OBJECTIVES OF THE LESSON: 1) study of methods for quantitative assessment of the quality of experts when forming an expert group;

2) the use of methods for statistical assessment of the quality of experts when creating an expert group designed to assess the quality of a particular product.

1. GENERAL PROVISIONS

One of the most important issues that have to be resolved in the process of expert assessment of product quality is the selection of experts. Just as in metrology the accuracy of a measurement result depends on the accuracy of the instrument used to make this measurement, so the objectivity of an expert assessment and its accuracy depend mainly on the qualifications of the expert, his ability for analytical thinking, synthetic generalization, on his outlook, erudition, psychophysiological abilities etc.

Currently, in practice, the following methods are used to quantitatively assess the quality of experts:

1. Heuristic, in which the meaning of assessments is determined by a person; heuristic assessment methods are based on the fact that the idea that others (or himself) have about a given expert fairly accurately reflects his true quality; heuristic assessments include: self-assessment, mutual assessment, expert assessment by members of the working group.
2. Static, in which the value of the ratings is obtained as a result of processing expert judgments about the products being evaluated.
3. Test tests, in which assessment values ​​are obtained as a result of special tests based on solving specially selected test problems.
4. Documentary, in which assessment values ​​are obtained based on the analysis of documentary data about experts.
5. Combined, in which the assessment values ​​are obtained using any combination of the methods listed above.

The most justified at present are statistical methods for assessing the quality of experts, so we will consider them in more detail.

Methods of static assessment include assessment of the deviation from the average opinion of the expert group Kos and assessment of the reproducibility of the CVR result, which can be obtained after processing the results of special surveys preceding the operation of forming an expert group.

The first method is based on the premise that the true value of the value (property) determined by experts is the value of the average assessment of the expert group. The smaller the deviation of the value of an individual assessment assigned by an expert from the group average assessment, the higher the quality of this expert, which can be taken into account by assigning each expert a corresponding “weight” or weighting coefficient.

In the case when an expert determines the numerical values ​​of the properties being assessed (in points, fractions of a unit or percent), it is proposed to use the distance between the “average” series of assessment values ​​and the assessment values ​​assigned by this expert as the expert’s assessment. For example:

in this case n

j=1/2A Ki - Kij

where Ki is the average value of the i-valued value (property);

Kij is the value of the i-th value (property) assigned by the j-th expert; n

The second method is based on the premise that the quality of an expert who is characterized by reproducibility of assigned assessment values ​​can be considered high, i.e. the assessment values ​​of the same object in several rounds should be quite close. In this case, we can talk about the stability of his opinion. The methods for assessing the quality of an expert based on the reproducibility of the result are similar to the methods for assessing the deviation from the average opinion.

In the case when experts rank the assessed quantities (properties), to quantify their quality or consistency, most often in practice the concordance coefficient is used, which is calculated as follows:

m2 (n3 - n) - m Tj

where Si is the sum of ranks assigned by m experts to the i-th factor;

S - average value of rank sums;

The quantities Si and S are determined by the relations:

Si=Kij; S=Si/n

where j is the expert number;

i is the number of the factor being assessed.

The value of Tj is determined from the expression:

Tj= (t3jl – tjl),

where tjl is the number of identical ranks of the 1st type in the assessments of the j -th expert;

Lj is the number of groups of factors with the same ranks in the assessments of the j expert.

The concordance coefficient varies from zero to one, and its value equal to one corresponds to complete agreement of expert opinions. Typically, the consistency of expert opinions is considered sufficient when W >0.5.

The consistency of expert opinions may or may not be random. The significance level of the concordance coefficient is used to quantify the degree of this randomness. This level is found from the table based on data on the number of degrees of freedom n=1 and value 2 (Pearson criterion).

Value 2 is determined by the formula:

mn (n +1) - 1 Tj

The significance level of the concordance coefficient determined in this way gives the probability of a random coincidence of expert opinions, i.e. reliability of assessments. If the consistency of expert opinions is high and non-random, then this expert group can be taken as the basis for an expert assessment of product quality. If the consistency of opinions is insufficient (W<0,5), то анализируя оценки отдельных экспертов, выделяют эксперта (экспертов), имеющего наиболее отличающиеся оценки от общей совокупности оценок других экспертов и подсчитывают значение коэффициента конкордации опять. Эту операцию повторяют до тех пор, пока не будет достигнута приемлемая согласованность экспертных оценок. При этом рекомендуется, чтобы количество оставшихся экспертов было не менее 2/3 от первоначального состава экспертной группы. В противном случае требуется заново сформировать экспертную группу. Для решения большинства задач по экспертной оценке качества продукции, оптимальными могут считаться экспертные группы, включающие от 7 до 20 экспертов.

II Independent work.

Task 1. For a given analogue of the product being evaluated, make an expert assessment of the ergonomic properties in accordance with the developed hierarchical structure of the properties of this product (see practical lesson No. 2). Determine the numerical values ​​of the corresponding ergonomic indicators, guided by the following rating scale:

Carry out the necessary statistical processing of the obtained data in order to assess the quality of each expert participating in the experiment. It is recommended to present the results of the task in the form of the following table:

Task 2. Using the express method of expert quality assessment, rank a number of proposed products (3...5 pieces), similar to the product being assessed, depending on their level of quality, carry out the necessary statistical processing of the data obtained in order to assess the quality of each expert participating in the experiment . It is recommended to present the results of the task in the form of the following table: