Methodology

Measurements—by allowing us to study, compare, assess, and draw conclusions about growth—assist us in making policy decisions. The assessment of attributes through assignment of numbers is at the core of all scientific inferences. With the 2022 edition of India Justice Report, we continue to map the scope of improvements as well as stagnancies in justice capacity across states through four core pillars of the justice system, namely the police, judiciary, prisons and legal aid and a standalone pillar state human rights commission. As with the previous two editions, new indicators have been added with an aim to deepen and broaden our assessments. Seventeen new indicators across the four core pillars have been added, bringing the total number of indicators to 102.

This IJR assesses the capacity and performance of State Human Rights Commissions (SHRCs) separately and not as part of the overall ranking of a state.

All pillars are measured on the basis of six themes: budget, infrastructure, human resource, workload, diversity and trends (or intention to improve).

Step 1: Outline

Data indicators of four pillars:

The indicators across the pillars cover the following themes:

1. Infrastructure
2. Human Resources
3. Diversity (Gender, SC/ST/OBC)
4. Budgets
5. Workload
6. Trends (Change over last five years)

Each theme represents a precondition necessary for the functioning of a pillar. Budgets measure the funds received, utilised, and spent per functionary or per capita; infrastructure, the basic physical resources available; human resources looks into personnel sanctioned and available on the ground; workload is the weight of service delivery upon a functionary within a particular subsystem; and diversity assesses how representative these systems are of the populations they are set up to serve. A sixth theme, ‘trends’, is used where possible to assess whether there has been improvement or deterioration over five years in a particular theme. This too is taken account of when arriving at the overall ranking.

Step 2: Clustering

The vast variations across India in terms of both demography and geography make comparisons difficult. For example, the police capacity in a state like Rajasthan or Madhya Pradesh is incomparable to that of Goa or Sikkim. To undertake fair comparisons between states, the report divides states and UTs into four clusters:

Cluster I (ranked): Eighteen large and mid-sized states or states with a population of 10 million and above.

Cluster II (ranked): Seven small-sized states with a population of up to 10 million, namely Arunachal Pradesh, Goa, Himachal Pradesh, Meghalaya, Mizoram, Sikkim, and Tripura.

Cluster III (not ranked):

Data for 8 UTs is provided but they are not ranked. These include Andaman & Nicobar Islands, Chandigarh, Dadra & Nagar Haveli and Daman & Diu (DNH & DD), Jammu & Kashmir¹, Ladakh, Lakshadweep, and Puducherry.

Cluster IV (not ranked):

Three states where the Armed Forces Special Powers Act, 1958 (AFSPA) is in force, namely Manipur, Assam and Nagaland. The report provides data on these but does not rank them.

Step 3: Filtering

All indicators are chosen based on government data availability and comparability across states. Benchmarks are taken from hard laws, policy pronouncements and Supreme Court judgements, wherever available. Government recommendations are also used. There are seventeen new indicators across pillars, some of which are indicators measuring diversity in subordinate courts in case of judiciary, presence of CCTV cameras in police stations, women help desks in police stations and share of overcrowded jails in a state.

Baseline

The IJR 2022 uses the latest official data available at the time of going to press. These are:

Comptroller and Auditor General of Accounts (CAG) documents were preferred over state budget documents due to the uneven availability of budget documents and variations in the way each records budget heads. However, for legal aid, state budget documents were used because the budgetary data was not available in CAG documents.

E-prisons portal

For the two indicators on overcrowded prisons—the share of overcrowded jails in a state and the share of jails with 150 per cent occupancy—the report used data available on the e-prisons portal. The portal lists 1,367 prisons, of which 53 prisons were not considered in the total number, as they are not functional or there was no information regarding the prisons available with the respective prison departments, or are covered under the Revenue Department. This report analysed e-prisons’ data over a three-month period—July to October 2022.

Cases pending at the end of the year

For the ‘cases pending at the end of the year’ indicator under the* *judiciary pillar, data from the National Judicial Data Grid (NJDG) has been used. The data for cases pending at the beginning/end of the year is not available with NJDG, hence, the older data from court news has been carried forward for certain calculations. This might result in slight variations if compared with figures from various high court websites. For an illustration see example of Madhya Pradesh High Court below:

State Citizen Portals

These state-wise portals are expected to offer nine basic services. They were assessed for accessibility, language and completeness of services. The compliance of the state citizen portal² was assessed by checking these nine services twice from September 2022 to November 2022 to evaluate improvements in the working of the portals. One mark was given for the portal being available in more than one language; and one mark was given per service for completeness of content. Where a service was disaggregated into various sub-parts that mark was also subdivided. Illustratively, if the service sought to provide details on stolen/ recovered vehicles, arms and other properties, each of the three sub-categories was allocated a maximum score of 0.3. Partial marks were therefore still accorded to ‘incomplete’ services.

State Human Rights Commissions

In order to collect data related to the existing 25 SHRCs, 136 RTI applications were filed, as the complete data for SHRCs has neither been collected nor published, or proactively disclosed to the public. The performance on each of the seven indicators was calculated using the same methods as in Step 4 and 5. Assam, Nagaland and Manipur are excluded from ranking due to the presence of Armed Forces Special Powers Act (AFSPA). Jammu & Kashmir is excluded since after the enactment of the Jammu and Kashmir Reorganisation Act, 2019 the SHRC was disbanded. The Protection of Human Rights Act, 1993 provides for the establishment, powers and functions of the SHRCs, and has been used as the benchmark to assess their performance. International standards set up under the Global Alliance of National Human Rights Institutions (GANHRI) and the Paris Principles have also been referred to.

Step 4: Scoring Method

As with IJR 2019 and 2020, raw data was rebased on a common scale so that every indicator could be scored on a scale of 1 to 10, with 1 being the lowest or least desirable status, and 10 indicating the highest or best score. The scores in-between were calibrated to show where a state stood in relation to the best and the lowest. Where a state met or exceeded the benchmark it had set for itself it received a score of 10. In cases where there were no benchmarks available, a state received a ‘top’ score of 10. This does not mean that the state has reached an ideal capacity, merely that it is best in class. The scores of every indicator were aggregated and averaged to arrive at a pillar score, also scored on a scale of 1 to 10. Averages were arrived at using geometric mean because the method is less prone to distortion by extreme outlying figures. Thus, for each pillar every state got a score out of 10, and a rank in its cluster. The pillar scores were then averaged to arrive at the overall score, also out of 10.

Step 5: Scoring and Ranking

For each cluster, the report applied the methodology outlined in Step 4 to every indicator in the pillar. For states whose values were missing for certain indicators due to an unavoidable reason — for example, in Haryana where there is no reservation for Scheduled Tribes—the number of indicators was reduced. Certain states have not received the grant for modernisation funds, the number of indicators in such states has also been accordingly reduced while working out their scores. In assessing judiciary for Andhra Pradesh and Telangana only three-year trends have been used due to unavailability of separate data prior to their bifurcation.

Step 6: Uniformity in Indicator Counts Across Themes and Weights

Each indicator, theme, and pillar has been assigned equal weightage so as not to privilege any one aspect over another. The study avoids subjectivity by giving any one element higher or lower weightage, since every data point influences the whole outcome.

Step 7: Data Checks

The data was checked down to source data at two points in time: after the preliminary set of rankings was generated, and after the final set of rankings was generated (in other words, before web and print outputs). A third round of checking was carried out on the final outputs.

Other Points

Rounding off decimals

The report looked at decimals through the ease of reading the data. Where the numbers were large, it did not include decimals and where they were small and the variance was in fractions, decimals were included—one or two places as needed.

Use of percentage points

The report uses percentage points as a unit of measurement for the trend or change indicators. This is calculated as the difference between two percentages to highlight an increase or decrease.

Union Territories and states

UTs and AFSPA states are not ranked as already mentioned in Step 2. As of August 2019, the state of Jammu & Kashmir was reorganized into two Union Territories: Jammu & Kashmir, and Ladakh. Separate data for these two UTs is not available for trend indicators, hence these two are not included in trend indicators. Similarly, the UTs of Dadra & Nagar Haveli and Daman & Diu were merged on 26 January 2020. Separate data for these two is combined.

Geometric mean over arithmetic mean

In a scenario where a state scores high or low in a pillar because it is doing extremely well or extremely poorly in a handful of variables, the geometric mean tends to normalize outliers i.e. extreme variables, better.

Shared court jurisdictions

For states that share court jurisdictions, the report used the same data where justifiable. For example, population per high court judge was combined for Punjab, Haryana, and Chandigarh since the two states and the UT are serviced by the same high court.

Strengths and limitations

Ranking of states on the basis of justice capacity is an unprecedented exercise in the context of our country with an aim to bring together disparate and hitherto siloed information. The capacity of 7 small states and 18 large states to deliver justice is once again ranked in this year’s India Justice Report. We examine the systems more thoroughly with each report and, as usual, only use the most accurate official data. The processing of so much data enables the precise location of potential intervention and remediation sites. Not only that, but even internal gaps caused by unequal data availability indicate how urgent it is to establish reliable, consistent, timely, and publicly accessible data systems across the country that facilitate collaborative internal planning for success in the future. The report benefits from ongoing assessments and recommendations from government organisations, judges, retired DGPs, police, and other experts in various sub-systems because it is a partnership between numerous specialised civil society groups. The inclusion of thus many different viewpoints confirms the selection of indicators and rating. The report is a purely quantitative exercise on selected aspects of the justice system. Its assessment is often limited by the unavailability and paucity of data and its inconsistencies. It does not aspire to capture the views of the duty holder or functionary and stakeholder that relate to the qualitative performance and functioning of each sub-system as perception studies and surveys do. Nevertheless, the assessment of the structures involved in the administration of justice point to levels of service and response. The data delineation here is also a necessary supplement to other qualitative studies and helps indicate possible solutions to many entrenched problems. We hope that the report will encourage others to strive to go deeper in evaluating the structure of the justice delivery system holistically and in ever more detail.

Notes

1

In the earlier report, Jammu & Kashmir was included in Cluster IV. Since August 2019, it has become a Union Territory, hence shifted to Cluster III. Either way, as a UT or AFSPA state it is not ranked. Dadra & Nagar Haveli and Daman & Diu were two different UTs. The merger of these two UTs took place in 2019. The data for these two UTs is merged wherever used.

2

The SMART Policing initiative of the Ministry of Home Affairs advises states to provide services to citizens online through the state citizen portal. https://digitalpolice.gov.in/