This research is based on the “Banks and Finance” database provided by the informational agency “Mobile”. It is a verified source of diverse information about international financial companies that is used extensively in the academic literature. The database provides monthly financial data that allowed us to obtain a panel dataset of Russian banks. Financial data includes balance sheet, income statement, calculated ratios and other information. There are 2071 banks in the “Mobile” database and the data was initially extracted from 2007 to 2016.

In order to generate a representative sample from the database, some data filtration methods were applied. First the bank’s distribution by the ownership type was considered. The focus of this paper is the individual profit maximizing banks, so all state-owned banks were omitted. The definition of state-owned bank was taken from the paper by Vernikov and Bokov (2008), where the direct government ownership is assigned to banks with more than 50% of shares that belong to the government (including different territorial entities of Russia or municipal corporations). According to this definition, 36 government banks were excluded from the dataset.

The main reduction of the sample size appeared due to the fact that only a small share of banks (395 banks) was assigned a rating grade. The data about history of rating changes was taken from Cbonds.ru and Bankodrom.ru that are the main on-line aggregators of banking statistics. The extracted data contained assessments of national RAs (RAEX, Rus-Rating, AK&M, NRA, Ria-Rating) and international agencies (Moody’s, Standard & Poor’s or Fitch). Then the data on banks’ defaults were collected from Cbr.ru and Banki.ru. 86 Russian banks (which received a rating assessment at least once) had default in the concerned time period. Both ratings and defaults were added to the financial data with a two-quarter lag between them. This time lag was chosen due to the fact that the process of assigning a rating by the RA takes some time to complete all the necessary procedures.

The historical distribution of all Russian banks (before any filtrations) for the period from 2007 to 2017 is demonstrated on the Fig. 1. From Fig. 1, we can conclude that there was an increase in the number of defaults after the 2008 crisis. Moreover, the increase in sharpness of banking regulation proposed by the policy of Elvira Nabiullina in the Central Bank of Russia since 2014 is also highly noticeable on the diagram. The growth rate of defaults in 2014 achieved 98%. The default policy of the Central Bank had no correlation with the fact whether the bank had a rating assessment or not. It can be seen from the consistency of proportion of default banks amongst overall number of banks (447 defaults/2071 total = 21.6% of defaults) and banks that were assigned a rating at least once (86 defaults/395 total = 21.8% of defaults). That shows us the fact that the policy of the Central Bank was aimed at all banks with financial distress and influenced the banks with top popularity at the same rate as unfamiliar banks without rating. This fact means that there is no sample bias of modeling a behavior of banks with a rating grade and extrapolating the results to a total population of Russian banks. Moreover, it means that banks that have not yet been assigned the rating grade are not worse than those with rating, so that again emphasizes the need for obtaining a reliable model that can assess the credit risk grade of a bank based on the publicly available information.

The initial database was imbalanced (223 of defaults compared to 11,404 of non-defaults). The nature of imbalanced data is intrinsic (corresponds to the nature of a data set). Furthermore, there is no data available for the “default” class after the bank had experienced the default. It leads to embedded rarity and within-class imbalances as well as the failure of generalizing inductive rules by learning algorithms (He and Garcia, 2009). Therefore, the combination of random undersampling and oversampling methods was used to obtain the final data provided for the regression analysis. Thus, we randomly selected a set of majority class examples and removed these samples from data and then the random set of minority class with new banks’ names was added.

RAs assign their grades in a symbolic form. However, in order to obtain coefficient estimates in an econometric model, these symbols should be transformed into numerical values. Moreover, symbolic ratings of different rating agencies should be unified to the base scale. The process of comparison of rating scales was taken from the paper of Karminsky et al. (2011).

As a result of comparisons of multiple mapping, it was found by Karminsky et al. (2011) that the best transformations of scales are obtained by using the class of linear-logarithmic transformations. In this case, the parametrization of mappings implies finding a pair of coefficients for mapping each of the scales into a basic one (free term and coefficient in front of the logarithm of the described rating scale). Moody’s was chosen as a “dependent” agency and, therefore, the base scale is associated with the international scale of this agency. The reason for this choice was the fact that Moody’s international scale is the closest one to the base scale. Therefore, the mapping was done by the following regression:

LN(M)=αiLN(Ri)+bi (1)

where M is a Moody’s international scale, taken as a base scale and R_i is the scale of the rating that should be transformed to a base scale. To build the models, the agencies’ statistics from the first quarter of 2006 to the fourth quarter of 2010 for Russian banks were used. Therefore, it allows a solution using the least-squares regression. The calculated coefficients for international agencies like Moody’s, Standard & Poor’s, Fitch (both international and national scales) and national agencies RAEX, NRA, Rus-Rating and AK&M were taken from the paper of Karminky et al. (2011).

In this research, one more Russian rating agency was added to the comparison list: this agency is Ria-Rating with the estimated regression coefficients α_i = 0.278 and b_i = 2.392 (all coefficients are statistically significant at 1% level). The comparison of rating scales is summarized in Appendix Table. The symbolic rating was transformed into the numeric scale beginning from 1 that is given to banks with the best rating and ending with the last largest number for the worst rating.

In order to avoid the loss in consistency for the model, the ratings were assumed to be unchanged until the moment of the new rating assignment. The final version of the dependent variable was obtained by averaging all single scale numeric grades of a bank in a particular quarter for all rating agencies. However, the averaging procedure brought us to the numerous non-integer rating groups (e.g. rating = 17.43) and the difference between this groups was too small to be properly modeled. For this reason, the numeric rating was rounded to the closest integer. Therefore, in this paper, 30 different groups of ratings were considered.

The models introduced in this paper allow interested agents to determine the probability of default and credit ratings for Russian banks, having at their disposal only public information. As for the modeling methods of this research, binary logit/probit regressions were chosen for PD estimation and multinomial ordered logit/probit for credit ratings modeling. It was shown (Jiao et al., 2007; Karminsky and Kostrov, 2017; Zan et al., 2004) that the predictions of more complex modeling methods like artificial intelligence models do not outperform the standard binary and ordered multinomial models. These methods were described and applied in the paper of Karminsky and Khromova (2016), and in Karminsky and Kostrov (2017).

The optimal set of indicators was selected on the basis of the most significant parameters that were chosen by a stepwise procedure (Hajek and Michalak, 2013), overall significance of the model (likelihood ratio test, pseudo-R², in-sample fit of the model) and the smallest Akaike and Schwartz information criteria (AIC & BIC). Also, the expected signs of the coefficients were considered. During the comparative analysis of logit and probit regressions, the decision, based on the minimization of the AIC & BIC and the greatest number of significant coefficients, was made in favor of the probit model. Moreover, concerning the specification of unobserved heterogeneity term, random effects models were chosen to be the best fit for both PD and ratings models. This conclusion was based on the Durbin-Wu-Hausmann test and RHO statistic.

The variable specification of the models was continuously challenged by the choice of financial variables, their cross terms and macroeconomic variables used as principal components (PCs) (explained in the section 4.3). The final models were checked for multicollinearity and all explanatory variables had correlations less than 35% and reasonable descriptive statistics. The results obtained by the panel probit regressions of PD modeling are shown in Table 1, while credit ratings models are provided in Table 2. All regressions were conducted for two different samples: a sample without imbalanced data reduction (N = 11,627) and a sample with omission of non-defaults combined with additional random inclusion of defaults (N = 3,289). Two different types of variable specifications were tested: a model with financial variables only (2) and a model with financial variables combined with Principal Components (3).

, (2)

, (3)

In order to interpret the signs of the estimated coefficients correctly, one should remember that the higher dependent variable is, the higher is the probability of default and the higher is the numeric value of a rating, which corresponds to banks with low financial stability. Keeping this in mind, we can conclude that all signs of coefficients coincide with their expected impact on PD and credit ratings for all regressions.

The first model specification (models 1.1 and 2.1) included only financial variables that were based on the BFSR methodology described in previous studies of authors (Karminsky and Khromova, 2016). The ratio of equity to assets, which shows the structure of a bank’s capital, appeared highly significant. This ratio shows the capitalization of a bank and is included in the model in order to capture the capital adequacy of a bank. It captures the ability of a bank to cover risks with its own resources and is inversely proportional to the financial leverage. Another parameter, the ratio of operating expenses to revenues, that shows the inefficiency of a bank, adversely affects banks credit risks and is significant in all models. The net interest margin shows the profitability of a bank and becomes highly significant in the models with PCs. The interbank ratio shows the share of issued loans in overall received funds on the interbank market. With the increase in this coefficient, a bank becomes less dependent on interbank loans and therefore its rating is raised. This parameter is highly significant in the regressions constructed without multicollinearity. It was repeatedly proved that market share is a significant factor for credit risks of a bank. In this research, the market share was estimated as the ratio of a bank’s equity to overall equity of all banks. The increase in market share significantly improves the financial performance of a bank in all models. The logarithm of total assets shows the size of a bank and has a positive relationship with a bank’s financial stability. Note that the second power of the bank’s size appeared also to be significant and shows a positive sign that tells us about the parabolic relationship between bank size and PD. It means that the largest and the smallest banks are the most unstable banks, so the idea of “too big to fail” was not supported by our research. The current liquidity of a bank is also a very important factor in evaluating its rating. A higher level of current assets compared to current liabilities decreases financial instability in each model. The impact of the ratio of loan loss reserves to gross loans appeared to be significant and almost the same in all models: the high level of reserves indicates the presence of “bad” loans issued by a bank and leads to a downgrade in its rating and increase in PD.

Macroeconomic variables and cross terms of financial variables are heavily correlated with each other, which inevitably leads to multicollinearity problems if no measures are taken. Therefore, due to this, the model is constructed primarily to be applied in forecasting and principal component analysis (PCA) is used to eliminate potential problems. PCA (Hotelling, 1933; Pearson, 1901) is exploited to reveal the intrinsic structure of the relations between the involved individuals and to reduce the number of dimensions needed to capture the dispersion. Implementation of this method is done in several steps. First, the means and standard deviations of each group of variables under consideration of PC are constructed. These groups are:

Let us assume that initial variables from liquidity group are called LV1, LV2, LV3, LV4, LV5, from market share group — SV1, SV2, SV3 and from macroeconomic group — MV1, MV2, MV3, MV4. Then the data are transformed so that the stochastic process yields zero mean and unit variance, i.e.

(4)

Secondly, the normalized data are given a new orthogonal basis via constructing linear combinations of LV1^norm ... LV5^norm; SV1^norm ... SV3^norm; MV1^norm ... MV4^norm. New groups of variables are uncorrelated by construction inside the group. However, external multicollinearity may still be present, but it is below 35%. Note that only two principal components from each group were tested in the model as the first two components cumulatively explain more than 80% of the initial variables.

As soon as PCA transformation is completed, the coefficients are no longer interpretable in an economic sense. In order to calculate marginal effects of variables that are inside PCs, we need to make the return procedure from principal components coefficients to initial coefficients.

The process will be shown on the example of macroeconomic variables. As each of PC1, PC2 is a linear combination of MV1 ... MV4, one can plug into the estimation equation coefficients for each of the principal components and restructure the equation. The transformation will construct a way from principal components back to the original variables and will provide an opportunity to interpret the model:

Zit=β0+∑j=12βjPCjit+uit (5)

Zit=β0+∑j=12βj(∑k=14γjkMVk)it+uit (6)

Zit=β0+(∑k=12∑j=14βjγjkMVk)it+uit (7)

Marginal effect at any point is calculated as

(8)

If ∑j=18βjγjk>0, one can consider MV has a positive effect on probability of default of a bank.

The reverse procedure of marginal effects provided us with the expected sign interpretation. In the second specification (models 1.2 and 2.2), various cross products and macro variables were tested. An asset-liquidity group principal component shows the interdependence of banks’ asset quality and liquidity, observing a tendency that banks with better loan portfolio tend to have a stable liquidity. Moreover, it shows a correlation between a GDP growth rate with liquidity of a bank. An increase in GDP growth rate leads to an increase in investments and savings of firms and households and they, in its turn, pay off their debts to banks more easily and banks’ liquid funds increase. A market share group principal component was also highly significant in all models and shows the importance to include different methods of estimation of a market share.

The comparison of predictive power of different variable specifications models gives us an expected result: the model with principal components (PC) that includes financial variables, their cross terms and macro variables gives us the highest level of forecasts in any sample of data. As the result of this step of the research, the predicted values for the best model specification for both PD default model (2.2) and credit ratings model (1.2) were computed.

In order to compare the forecasting power of PD and credit ratings models, they should be presented in the same scale. There are various papers that study calibration of ratings and defaults (Karminsky et al., 2015; Pomasanov and Vlasov, 2008) and some rating agencies publish their ratings scales correspondence to PD officially (Moody’s, 2011). In this research, the calibration scale of Standard & Poor’s national rating provided by Pomasanov and Vlasov (2008) was taken as a basis scale of calibration.

This scale clearly shows the non-linear pattern of PD and rating grade. In order to correspond the scale provided in Table 3 to the base rating’s scale of this research (see Appendix Table), one exponential and two polynomial transformations were applied. The results of extrapolation of PD to all numeric rating grades of the base scale are provided in Fig. 2.

Fig. 2 shows that the rating scale corresponds to PD non-linearly. The correspondence of PD to the highest ratings from ruAAA to ruA+ was estimated by the exponential function and, for the middle ratings (from ruA+ to ruCCC+), a convex polynomial quadratic function was used. That proves that PD increases with accelerating pace for these ratings’ grades. For the bottom ratings (from ruCCC+ to ruD), a concave polynomial quadratic function proved to be the most appropriate approximation and shows the decelerating rate of change in PD to rating scale.

Calibrated models of PD and credit ratings now can be compared by their in-sample predictive power. For the precise visualization, PD model’s forecasts were converted into the base rating scale and the difference between the actual rating grade and the rating grade predicted by PD model was calculated. Fig. 3 shows the distributions of forecasted errors of rating model (1.2) and PD model (2.2) calibrated in the same scale of rating grades. Moreover, the percentages of precise predictions and deviations of less than one rating grade were calculated for each model.

From Fig. 3, we can see the disproportional distributions of forecasted errors for both models. The share of correct forecasts (Δ = 0) in the rating model was 18%, while the default model appeared to be even more skewed than ratings model and forecasted correctly only in 6% of cases. In addition, the percentage of forecasts with a deviation of not more than one rating class from the actual rating (| Δ | < 1) for the rating model and PD model was 34.8% and 12.3% respectively. The rating model had the property of forecasting a grade lower than actual rating, while the PD model, on the contrary, underestimated the financial problem of banks and predicted a grade higher than actual rating. Fig. 3 shows that, in the PD model, the positive prediction error dominates the negative one, which means that the actual numeric ratings exceed their forecasts in this model. However, the decreasing numerical values assigned to ratings relative to their symbolic grades means this tendency implies the reverse: the ratings forecasted by the PD model are overstated. This happened because the defaults model initially had imbalanced data and even formation of a representative sample did not solve the problem of over-education of the PD model towards the non-defaulted banks. The distribution of ratings’ model forecasted errors is much more symmetric with respect to zero, however, it also shows the skewed pattern that was discussed before.

To sum up the comparison, we should conclude that the first hypothesis of this paper was not rejected after empirical modelling. Indeed, ratings models tend to overestimate the financial instability of a bank, whereas PD models underestimate it.

In order to construct a reliable synergic model, the ratings’ grade forecasts by PD and rating model should be computed for the same observations. Note that the rating model was estimated for 11627 observations, while PD model has 3489 estimates. Each observation has its own ID and time correspondence, so we find all id_time estimates that are present in both of these data sets. The overlapping of these datasets included 3011 estimates as the PD model had some artificially generated defaults. Then the regressions in which the dependent variable was the actual rating and explanatory variables were the fitted values of rating and PD models, were run on the 3011 observations.

The first synergic model was obtained as a linear combination of PD default and rating model:

Yit=α+β1PDit+β2Ratit+uit (9)

where Y_it is the actual rating, PD_it and Rat_it are the predicted ratings by PD model and ratings model respectively. The regressions output is provided below in Table 4 and the obtained distribution of the forecasted errors of this synergic model is illustrated on Fig. 4.

The linear synergic model shows much higher predictive power than PD or rating model on its own. It can predict 32% of precise rating grades and up to 58% of ratings with an error less than one grade. However, it still contains heavy tails. In order to solve this problem, we use the logarithmic model specification of synergic model:

Yit=α+β1Log20(Ratit–PDit)+uit (10)

The regressions output is provided below in Table 5 and the obtained distribution of the forecasted errors of this synergic model is illustrated in Fig. 5.

The synergic model that was obtained by the logarithm of difference of rating and PD forecasts was found have the highest predicted power with the smallest deviations. Note that this distribution has very small tails and so such of the model does not have any prediction errors higher than three rating grades. Therefore, this optimal combination can bring us to the most consistent estimates with forecasting power of more than 44% of precise ratings and 83% of deviation less than one rating grade.

The second part of this section is devoted to the analysis of the out-of-sample predictive power of the logarithmic synergic model. In order to accomplish this task, the data were limited to the observations from 2007 to 2015. Based on the new coefficients of the PD and ratings models, the forecast for the year 2016 was made. In order to calculate the predicted ratings, the predicted probabilities of each rating grade were calculated as the difference between the values of the standard normal distribution (F) at two points, that were calculated using the estimated boundary values (cut_j) and the product of vectors of estimated coefficients (β) and the values of explanatory variables for the year 2016 (x'_k) according to the formula:

Pr(outcomek=j)=F(cutj–x'kβ)–F(cutj–1–x'kβ) (11)

The rating grade with the highest predicted probability was selected as the rating model’s forecast. Concerning the PD model, the probability forecasts were estimated and then calibrated to the rating scale. Then the predicted rating grades of the PD model and the rating model were taken with the functional form and estimated coefficients on 2007–2015 data for synergic models. The following coefficients and forecast error’s distributions were obtained under the out-of-sample fit check of the logarithmic synergic model (Table 6).

Then the actual financial data for the year 2016 was used as explanatory variables in both models and two separate forecasts of PD model (calibrated into ratings) and rating model were obtained. The following forecasts were placed into two different specifications of synergic models and the final synergic forecasts for the year 2016 were obtained. These forecasts were compared with the actual one assigned to a bank in the year 2016 and the distributions of forecast errors, illustrated on Fig. 6 were constructed.

The results show a slight expected deterioration in the predictive power of the synergic models under the out-of-sample fit check. Nevertheless, the logarithmic model can accurately predict the grade of the expected rating with a probability of 31.4%. In addition, the analysis of the out-of-sample power of the model shows that, in 72.3% of the cases, the prediction error of the expected rating of a bank will not exceed one rating grade. Based on this analysis, we can conclude that the logarithmic synergic model can have a practical use for predicting the credit risks. Therefore, the aim of this research was achieved by constructing a synergic model with higher forecasting power by using the set of alternative models: the model of probability of default and the model of credit ratings for Russian banks (hypothesis 2 was not rejected). Moreover, it should be noted that the horizontal scale in all forecast errors distributions shows the deviation of actual rating from the forecast based on the 30-grades rating scale. Such scale is much more detailed than the usual 22-grade scale of any international rating agency. That means that all results in this research are even more precise if we transform our forecasts to the 22-grade scale.