Received: 18 May 2017 Revised: 25 September 2017 Accepted: 30 September 2017

DOI: 10.1002/JPER.17-0314

ORIGINAL ARTICLE

The role of menopause on the relationship between metabolic risk

factors and periodontal disease via salivary oxidative parameters
Esra Sinem Kemer Doğan1 Fatma Yeşim Kırzıoğlu2 Burak Doğan1 Özlem Fentoğlu2
Banu Kale3 Süleyman Akif Çarsancaklı3 Hikmet Orhan4

1 Department of Periodontology, Faculty of

Dentistry, Mustafa Kemal University, Hatay,
Turkey
2 Department of Periodontology, Faculty of
Dentistry, Süleyman Demirel University,
Isparta, Turkey
3 Private practice, Isparta, Turkey
4 Department of Biostatistics and Medical
Informatics, Faculty of Medicine, Süleyman
Demirel University, Isparta, Turkey
Correspondence
Dr. Esra Sinem Kemer Doğan, Mustafa Kemal
University, Faculty of Dentistry, Department
of Periodontology, 31001, Antakya, Hatay,
Turkey.
Email: esraa_kemer@hotmail.com
Abstract
Background: Periodontal disease is shown to be aggravated by an increase in the
count of metabolic risk factors. This study aims to evaluate the effects of metabolic
risk factors on periodontal parameters and salivary oxidative stress markers related to
menopausal status.
Methods: One hundred and seventy-six women were categorized according to
menopausal status, either premenopause (Pre/M) (n = 86) or postmenopause (Post/M)
(n = 90). The count of metabolic risk factors was evaluated. Sociodemographics and
systemic status were determined via questionnaire and medical records. After clinical
periodontal measurements and saliva collection, myeloperoxidase (MPO), total oxi-
dant status (TOS) and total antioxidant capacity (TAOC) were determined by enzyme-
linked immunosorbent assay and automatic colorimetric method. Oxidative stress
index (OSI) was also calculated.

Results: The count of metabolic risk factors was higher in the Post/M group than the
Pre/M group. Periodontal parameters and TOS levels were elevated by an increase
in the count of metabolic risk factors. Multivariate regression analyses revealed that
periodontal (clinical attachment level and missed teeth) and oxidative (MPO and OSI)
parameters increased and TAOC levels decreased due to menopause. Additionally,
positive relationships between periodontal and oxidative parameters were determined.

Conclusion: These findings suggest that salivary oxidative stress level may be an indi-
cator of worsened periodontal status related to menopause and the count of metabolic
risk factors.

KEYWORDS
Menopause, metabolic syndrome X, oxidative stress, periodontal diseases, risk factors, saliva

1 I N T RO D U C T I O N periodontal disease and systemic diseases can be the for-

Periodontitis is a chronic inflammatory disease characterized
by the breakdown of periodontal tissues as a result of an
immune response against pathogens in the microbial dental
plaque.1 Various systemic risk factors that can change the host
response can contribute to this periodontal breakdown.2,3
One of the possible mechanisms of the relationship between
mation of oxidative stress, which can be induced by the
imbalance between the excess reactive oxygen species
production and antioxidant mechanisms.4
Saliva is composed of biomarkers such as immunoglobulin,
enzyme, gingival crevicular fluid and microbial components
and can be used as a marker for the evaluation of both peri-
odontal and systemic diseases.5 Specific biomarkers can be

J Periodontol. 2018;89:331–340. wileyonlinelibrary.com/journal/jper © 2018 American Academy of Periodontology 331

332
used to determine oxidative stress. Total antioxidant capacity
(TAOC) and total oxidant status (TOS) levels are more benefi-
cial in terms of acquiring information about the total oxidative
status.6,7 Oxidative stress index (OSI) also was reported to be
more advantageous in the determination of oxidative status.7
It has been shown that antioxidant levels decrease and oxi-
dant status increases as a result of periodontal6,8 and systemic
diseases.9,10
Myeloperoxidase (MPO) plays an important role in
killing phagocytized bacteria by means of oxygen-dependent
mechanisms.11 It has been found that periodontal destruction
increases MPO activity and that MPO can be used as a marker
for the evaluation of periodontal diseases.12,13 Furthermore,
various chronic diseases such as diabetes,14 hyperlipidemia,15
and hypertension16 can affect MPO levels.
Menopause is a natural and physiologic process, but
elevated serum inflammatory17 and pro-oxidant biomarker
levels18 and increased prevalence of metabolic risk factors19
in postmenopausal women were reported. Estrogen deficiency
can lead to alterations in oral soft and hard tissues, and
postmenopausal women who are susceptible to osteoporosis
could also be more susceptible to periodontal destruction.20
Menopause is also related to changes in lipid metabolism,
and it has been suggested that alterations in lipid metabolism
can be a risk factor for cardiovascular disease (CVD) in post-
menopausal women.19
Studies have revealed that having more than one risk fac-
tor can increase the risk of the systemic disease21,22 and be
more dangerous for multifactorial diseases such as CVD21 and
metabolic syndrome.22 Although various reports have indi-
cated the role of several metabolic risk factors on periodon-
tal disease severity,2,22,23 to the best of our knowledge, none
of them considered the impact of the menopausal status on
this association. Therefore, we hypothesized that menopause
could aggravate the effects of metabolic risk factors on peri-
odontal disease. In line with this hypothesis, the aim of
our study was to evaluate how periodontal parameters and
oxidative markers in saliva were affected by the increase in
metabolic risk factors related to menopausal status.
2 M AT E R I A L S A N D M E T H O D S
2.1 Study population
This study was approved by Süleyman Demirel University
Faculty of Medicine Clinical Researches Ethics Committee,
Isparta, Turkey (11.02.2015/33) in accordance with the Dec-
laration of Helsinki, which was revised in 2013.
From September 2013 to March 2015, 340 individuals who
applied to Süleyman Demirel University, Faculty of Dentistry
for routine controls were invited to participate in the study.
They were referred to Süleyman Demirel University Hospital
KEMER DOĞAN ET AL.
for the medical control. Among them, 222 women were eval-
uated, and 176 women aged 30 to 70 years, who were eligi-
ble for inclusion, were recruited. Written consent forms were
obtained.
Patients with pregnancy or lactating at the time of the study;
history of chemotherapy, radiotherapy or renal diseases; his-
tory of systemic antibiotic administration within the previ-
ous 3 months, surgical menopause and/or history of hormone
replacement therapy; osteopenia or osteoporosis, type 1 dia-
betes mellitus, history of bisphosphonates, periodontal treat-
ment within the last 6 months, drug-induced gingival enlarge-
ment, current or former smokers, and < 8 teeth (< 2 teeth in
each quadrant) were excluded.
Women were categorized according to menopausal status,
either premenopause (Pre/M) (n = 86, aged 30 to 55 years)
or postmenopause (Post/M) (n = 90, aged 45 to 65 years).
Pre/M is defined as regular menstrual cycles in the last year,
and Post/M is defined as not having monthly menstrual cycles
in the last ≥1 year.24 Information regarding sociodemograph-
ics (systemic health habits, oral hygiene agents, etc.) was col-
lected via questionnaire. Body mass index (BMI) and waist
circumferences (WC) were measured. BMI was calculated as
the body weight (kilograms) divided by the height squared
(square meters).25 Systolic and diastolic blood pressure (BP)
measurements were taken in a sitting position from the par-
ticipant's right arm.26
Metabolic risk factors were determined using National
Cholesterol Education Program Adult Treatment Panel III
(NCEP ATP III) criteria which includes five risk factors as
WC over 88 cm, BP over 130/85 mmHg, fasting triglyceride
(TG) level over 150 mg/dl, fasting high-density lipoprotein
(HDL) cholesterol level less than 50 mg/dl and fasting blood
glucose over 110 mg/dl.27 Participants were categorized into
five groups as “no, 1, 2, 3, and ≥4 risk factors” based on the
NCEP ATP III criteria.
2.2 Periodontal examination
Periodontal examination of all teeth including the probing
pocket depth (PD), clinical attachment level (CAL), plaque
index (PI),28 gingival index (GI),29 sulcus bleeding index
(SBI),30 and number of missed teeth (MT) were measured
by one calibrated dentist (E.D.) PD and CAL were measured
at six sites (buccal and lingual aspects, each with mesial,
median and distal points), and PI, GI and SBI were evalu-
ated at four sites (mesio-buccal, mid-buccal, disto-buccal, and
mid-lingual) around each tooth using a periodontal probe.∗
Intra-examiner correlation coefficient was shown as 0.88
for PD and 0.85 for CAL. Weighted k values (–1 mm)
ranged from 0.82 to 0.90 for PD and 0.82 to 0.87 for CAL,
respectively.
∗ Williams periodontal probe, Hu-Friedy, Chicago, IL.
KEMER DOĞAN ET AL.
2.3 Metabolic parameters
Metabolic parameters including fasting blood glucose (FBG),
glycated hemoglobin (HbA1c), triglyceride (TG), total
cholesterol (TC), low-density lipoprotein cholesterol (LDL)
and high-density lipoprotein cholesterol (HDL) were deter-
mined by an endocrinologist (B.K.) who evaluated detailed
medical history and biochemical tests to specify any previ-
ously mentioned diseases.
2.4 Saliva samples and laboratory analysis
Unstimulated whole saliva samples were collected prior to
periodontal examination in the morning after an overnight
fast. Patients were seated and were instructed to expectorate
into the cup one time per 60 seconds throughout 10 minutes.31
Obtained saliva samples were measured, and saliva flow rate
(SV) was calculated (mL/min). Samples were centrifuged at
4,000 g for 10 minutes. At 4◦ C, and supernatants were stored
in Eppendorf tubes at −80◦ C until analysis.
MPO levels were determined using a commercially avail-
able ELISA kit,∗ whose sensitivity and assay range is
0.116 ng/mL and 0.3 to 30 ng/mL respectively, according
to the manufacturer's instructions based on the principle of
double-antibody sandwich technique. Color changes were
measured spectrophotometrically at 450 nm. The level of
MPO was expressed as ng/mL.
Saliva TOS† and TAOC‡ levels were determined by com-
mercially available kits using the colorimetric method.7,32
TAOC levels were measured based on the decolorization
of the 2,2′ -azino-bis (3-ethylbenzthiazoline-6-sulphonic acid)
radical cation by antioxidants. The change in absorbance at
660 nm indicated the level of TAOC of the sample. TOS levels
were determined using spectrophotometric analysis measur-
ing the color intensity of the oxidation reaction of the ferrous
ion-chelator complex to ferric ion by the oxidants in the saliva.
Hydrogen peroxide was used as a standard for preparation
of calibration curve. The results were expressed as millimo-
lar trolox equivalent per liter (mmol trolox Eq/l) for TAOC
and micromolar hydrogen peroxide equivalent per liter (𝜇mol
H2 O2 Eq/l) for TOS. In order to calculate the OSI, which was
found to be a potential indicator of the degree of oxidative
stress, the percentage ratio of TOS:TAOC was used.7 Before
the calculation, the TAOC, given in millimolar trolox equiv-
alent per liter, was converted to micromoles equivalent per
liter, and the following formula was used to calculate the OSI
value:
∗ Human myeloperoxidase [MPO] ELISA Kit, Sunred Biotechnology, Shang-
hai, China.
† Total
oxidant status [TOS] Assay kit, REL Assay Diagnostics, Gaziantep,
Turkey.
‡ Total
antioxidant status [TAS] Assay kit, REL Assay Diagnostics,
Gaziantep, Turkey.
333
OSI = [(TOS, 𝜇mol H2 O2 Eq∕l)∕ (TAOC, 𝜇mol trolox Eq∕l)]
×100
2.5 Statistical analyses
A statistical package program§ was used for statistical anal-
yses. A sample size software¶ was used to determine the
achieved power for studied parameters with entering required
prior values. Twenty-five subjects were needed per count of
metabolic risk factors group to get achieved power of > 99%
for all periodontal parameters. A power of > 95% at the
𝛼 = 0.05 level for periodontal (PI, GI, SBI, PD, CAL)
and oxidative (TAOC, MPO) parameters with a power
of > 85% for TOS between the Pre/M and Post/M groups were
observed.
Controlling for the following normal distribution of the
data and homogeneity of variances were analyzed using the
Kolmogorov-Smirnov test and Levene test, respectively. From
these tests, the data were found not to be normally distributed.
A Mann-Whitney U test for non-normal distributed contin-
uous variables and chi-square test for categorical variables
were used to compare the parameters between the Pre/M and
Post/M groups, and the Kruskal-Wallis H test was applied to
compare the parameters among the risk factor groups. Sig-
nificant outcomes by the Kruskal-Wallis analyses were also
analyzed using the Mann-Whitney U test to determine the sig-
nificance level between bivariate variables.
Multivariate regression analyses were applied to access the
associations among periodontal parameters, oxidative mark-
ers, and menopausal status. Specifically, the following four
models were created: model 1, adjusted for age (30 to 40, 40
to 50, 50 to 60, and 60 to 70 years old); model 2, additionally
adjusted with BMI (< 25, 25 to 30, and > 30 kg/m2 ), tooth-
brushing (two to three times/day, once a day, or < 1 time/day)
and flossing (yes or no); model 3, additionally adjusted with
education (primary school, high school or university) and the
count of metabolic risk factors (no, 1, 2, 3, and ≥4 risk fac-
tors); and model 4, additionally adjusted with menopausal sta-
tus (Pre/M or Post/M). P < 0.05 was considered statistically
significant.
3 RESULTS
3.1 Characteristics of subjects
When the Pre/M group was compared with the Post/M group,
age (38.37 ± 7.19 versus 54.31 ± 6.07), BMI (26.92 ± 6.46
versus 31.72 ± 6.5), WC (86.21 ± 17.5 versus 97.16 ± 15.92)
and BP (117.21 ± 8.18 versus 124.89 ± 13.09; 74.59 ± 6.15
§ SPSS, v.21.0 for Windows, IBM, Chicago, IL.
¶ G*power, v.3.1.9.2 for Windows, University of Kiel, Kiel, Germany.
334 KEMER DOĞAN ET AL.

versus 78.89 ± 7.70 for systolic and diastolic BP, respec- TABLE 1 Sociodemographics and metabolic parameters accord-
tively) were found to be statistically significant (P < 0.05). ing to menopausal status
The count of metabolic risk factors was also higher in the Pre/M Post/M
Post/M group than in the Pre/M group (P < 0.05). There were Variables (n = 86) (n = 90) P valuea
no significant differences regarding toothbrushing and floss- Age (years)
ing frequencies (P > 0.05). All evaluated metabolic parame- 30 to 40 56 (100%) 0 (0%) 0.000
ters except HbA1c, HDL and TC/HDL were also observed to
40 to 50 23 (54.80%) 19 (45.20%)
be significantly higher in the Post/M group compared with the
50 to 60 7 (12.10%) 51 (87.90%)
Pre/M group (P < 0.05) (Table 1).
60 to 70 0 (0%) 20 (100%)
BMI (kg/m2 )
3.2 Periodontal parameters and oxidative < 25 38 (77.60%) 11 (22.40%) 0.000
stress markers 25 to 30 23 (46%) 27 (54%)
The more risk factors, the higher periodontal parameters were > 30 25 (32.50%) 52 (67.50%)
seen (P < 0.05) (Table 2). Multivariate regression analyses of Toothbrushing
periodontal parameters related to menopausal status revealed 2 to 3 times/day 18 (50%) 18 (50%)
that, with menopause, MT increased in all adjusted mod- 1 time/day 33 (55.90%) 26 (44.10%) 0.327
els (P < 0.05), whereas CAL was shown to increase inde- < 1 time/day 35 (43.20%) 46 (56.80%)
pendently of the count of metabolic risk factors in model 3
Flossing
(Table 3).
No 58 (50%) 58 (50%) 0.675
TOS elevated with an increase in the count of metabolic
Yes 28 (46.70%) 32 (53.30%)
risk factors (P < 0.05) (Table 2); however, no significance
Diabetes Mellitus
was determined in adjusted models (P > 0.05) (Table 3). With
menopause, MPO and OSI levels increased in all adjusted No 61 (52.6%) 55 (47.4%)
models, and TAOC decreased in models 1 and 3 (P < 0.05) Yes 25 (41.7%) 35 (58.3%) 0.204
(Table 4). Metabolic Risk Factors
Multivariate regression analyses of periodontal parame- WC ≥88 cm 39 (45.30%) 58 (64.40%) 0.011
ters and oxidative markers indicated that, despite an increase BP ≥130/85 mmHg 8 (9.30%) 35 (38.90%) 0.000
in CAL elevated MPO levels in models 1 and 3, PD ele- TG ≥150 mg/dl 19 (22.10%) 30 (33.30%) 0.096
vated TOS levels in models 1 and 2 (P < 0.05). SV nega- HDL < 50 mg/dl 29 (33.70%) 24 (26.70%) 0.308
tively affected TOS levels in all adjusted models (P < 0.05) FBG ≥110 mg/dl 26 (30.20%) 37 (41.10%) 0.132
(Table 5). Count of Metabolic Risk Factors
No 28 (70%) 12 (30%) 0.006
1 25 (48.10%) 27 (51.90%)
4 DISCUSSI O N 2 17 (48.60%) 18 (51.40%)
3 10 (43.50%) 13 (56.50%)
In the present study, it was found that the increase in metabolic
≥4 6 (23.10%) 20 (76.90%)
risk factors related to menopausal status worsened periodontal
Metabolic Parameters
health and elevated the oxidative stress level in saliva. To our
FBG (mg/dl) 125.44 ± 73.01 127.44 ± 63.68 0.035
knowledge, this is the first study that evaluated the multiple
effects of metabolic risk factors on periodontal and salivary HbA1c (%) 8.19 ± 2.85 7.46 ± 1.97 0.464
oxidative parameters related to menopausal status. TC (mg/dl) 156.19 ± 43.07 186.7 ± 58.79 0.003
Postmenopausal women could be more susceptible to peri- TG (mg/dl) 131.86 ± 72.09 162.49 ± 105.71 0.040
odontal destruction,20 and lower bone mineral density seems HDL (mg/dl) 52.86 ± 8.59 55.98 ± 11.66 0.132
to be the prominent factor.33,34 The risk of various systemic LDL (mg/dl) 106.57 ± 37.27 122.92 ± 41.92 0.022
diseases, such as hypertension, proatherogenic lipid changes, TC/HDL 3.92 ± 0.92 4.21 ± 1.25 0.447
diabetes and severe CVD, is higher in postmenopausal Continuous variables are shown as unadjusted mean ± SD; categorical variables
women compared with their premenopausal counterparts;35 are shown as n (%).
thus, it can be considered that increases in metabolic risk Bold denotes statistical significance at P < 0.05.
a P values were computed with chi-square test for categorical variables or Mann-
factors related to menopause could affect periodontal disease.
Whitney U tests for continuous variables.
As a matter of fact, each risk factor affects periodontal status
through various mechanisms, and a number of studies exists
in which the effects of multiple risk factors on periodontal
KEMER DOĞAN ET AL. 335

TABLE 2 Comparisons of the periodontal and oxidative parameters according to the count of metabolic risk factors
Count of metabolic risk factors
No 1 2 3 ≥4
Variables (n = 40) (n = 52) (n = 35) (n = 23) (n = 26) P valuea
Periodontal Parameters
PI 0.93±0.44 1.44±0.56b 1.7±0.59b,c 1.62±0.62b 1.77±0.60b,c 0.000
GI 0.90±0.30 1.29±0.39b 1.44±0.42b 1.49±0.48b 1.54±0.39b,c 0.000
SBI 1.42±0.57 2.22±0.78b 2.44±0.83b 2.64±0.89b,c 2.56±0.60b,c 0.000
PD (mm) 2.73±0.44 3.09±0.56b 3.32±0.63b 3.2±0.54b 3.8±0.66b,c,d,e 0.000
CAL (mm) 2.84±0.55 3.29±0.76b 3.66±0.84b,c 3.66±0.90b 4.25±0.75b,c,d,e 0.000
MT 5.55±5.80 8.44±5.85b 7.69±6.32 8.3±7.75 9.69±6.93b 0.031
SV 0.47±0.20 0.48±0.22 0.41±0.19 0.46±0.29 0.34±0.14b 0.032
Oxidative Markers
MPO 8.13±2.37 7.84±2.09 8.59±1.55 8.02±2.18b 8.12±2.16 0.376
TAOC 5.31±6.18 5.58±8.02 5.5±5.60 3.69±2.23 3.02±1.94 0.790
b b,c,e
TOS 0.99±0.48 1.14±0.47 1.27±0.57 1.08±0.50 1.47±0.61 0.007
OSI 0.05±0.09 0.05±0.04 0.05±0.04 0.04±0.02 0.07±0.04b,e 0.079
Values are shown as unadjusted mean ± SD.
Bold denotes statistical significance at P < 0.05.
a
P values were computed with Kruskal-Wallis H tests.
b
Significantly different from group No (P < 0.05, Mann-Whitney U test).
c Significantly different from group 1 (P < 0.05, Mann-Whitney U test).
d
Significantly different from group 2 (P < 0.05, Mann-Whitney U test).
e
Significantly different from group 3 (P < 0.05, Mann-Whitney U test).

disease are evaluated.2,22,23,36 Primary aim of this study was
to investigate the multiple effects of these risk factors on
periodontal health related to menopausal status and in order
to eliminate the complicated effects of multiple risk factors
and confounders which may change the course of the results,
adjusted models were created and analyzed carefully.
In the present study, metabolic risk factors were deter-
mined using NCEP ATP III criteria.27 In line with the liter-
ature, we found that metabolic risk factors increased in the
Post/M group, and this increase was mainly effected by WC
and BP.19 After menopause, various alterations such as an
increase in body weight and abdominal adiposity; body lipid
distribution changes35 and increases in TC, LDL and TG lev-
els were reported.37 Our results are in accordance with the
literature.35,37 Elevated TG levels with menopause are also
associated with an increase in abdominal obesity and insulin
resistance.38 Increased FBG levels in the Post/M group com-
pared with the Pre/M group supports this relationship.
It has been shown that risk factors are likely to be found
together and not singularly,39 and having more than one risk
factor can increase the risk of disease.21,22 Similar to our
results, studies indicated that an increase in the number of
metabolic syndrome components was shown to elevate peri-
odontal disease risk.2,22 Recently, Kaye et al.23 exhibited sim-
ilar results in men. Parallel to Furuta et al.,36 we revealed
that as the number of risk factors increased, MT raised nearly
50% (Table 2). Additionally, similar to von Bultzingslowen
et al.,40 decreased SV by metabolic risk factors was observed.
Depending on these results, we also corroborate the wors-
ening of periodontal status with an increase in the count of
metabolic risk factors by menopause.
Estrogen deficiency contributes to hard and soft tissue alter-
ations, causing periodontal destruction.20 The incidence of
tooth loss increases after menopause,41 and our results con-
firm the report. Furthermore, the relation between menopause
and CAL, independently of the count of metabolic risk fac-
tors, emphasized the pure effect of menopause on periodontal
disease.
It is believed that impaired oxidant-antioxidant capacity is
responsible for periodontal disease pathogenesis,4 thus our
secondary aim was to evaluate the effects of the salivary
oxidative status on the relation among menopause, the count
of metabolic risk factors and periodontal disease. Increased
saliva TOS levels by periodontal disease have previously been
reported.8,42 MPO levels, which play a major role in periodon-
tal disease pathogenesis,13 were reported to be elevated by
periodontal destruction.12 In this study, not only the positive
relations between CAL and MPO, PD and TOS, and SV and
TOS supported the literature but also the pure effect of peri-
odontal disease on oxidative stress markers in saliva as MPO
and TOS levels was determined in models 3 and 4.
It has been stated that reduced estrogen levels after
menopause are associated with decreased antioxidant defence
and increased oxidative stress.43 In the present study, the
Post/M group had higher salivary MPO and OSI but lower
TAOC levels than the Pre/M group (Table 4). Although
 336

TABLE 3 Multivariate regression models (𝛽 a [95% CI]) of periodontal parameters related to menopausal status
Dependent variables
Independent
variables PI GI SBI PD CAL MT SV
Model 1b
Menopause −0.14 (−0.43 to 0.09) −0.16 (−0.32 to 0.03) −0.18 (−0.65 to 0.03) 0.06 (−0.18 to 0.32) 0.15 (−0.04 to 0.57) 0.22 (0.33 to 5.29) 0.12 (−0.05 to 0.15)
Age 0.51 (0.14 to 0.34) 0.63 (0.15 to 0.28) 0.63 (0.27 to 0.53) 0.51 (0.16 to 0.34) 0.52 (0.22 to 0.45) 0.35 (0.75 to 2.61) −0.26 (−0.08 to −0.01)
Model 2b
Menopause −0.10 (−0.36 to 0.12) −0.14 (−0.28 to 0.04) −0.16 (−0.58 to 0.04) 0.06 (−0.16 to 0.31) 0.16 (−0.03 to 0.56) 0.23 (0.45 to 5.46) 0.13 (−0.04 to 0.15)
Age 0.31 (0.05 to 0.24) 0.45 (0.09 to 0.21) 0.44 (0.16 to 0.40) 0.35 (0.08 to 0.26) 0.39 (0.14 to 0.37) 0.35 (0.68 to 2.65) −0.29 (−0.08 to −0.01)
BMI 0.19 (0.03 to 0.25) 0.22 (0.04 to 0.19) 0.24 (0.10 to 0.39) 0.28 (0.11 to 0.33) 0.22 (0.10 to 0.37) −0.02 (−1.28 to 1.05) −0.03 (−0.05 to 0.04)
Toothbrushing 0.31 (0.12 to 0.30) 0.23 (0.05 to 0.17) 0.21 (0.07 to 0.32) 0.10 (−0.02 to 0.17) 0.07 (−0.05 to 0.18) 0.00 (−0.98 to 1.00) 0.11 (−0.01 to 0.06)
Flossing −0.09 (−0.28 to 0.05) −0.11 (−0.22 to 0.01) −0.09 (−0.38 to 0.06) −0.07 (−0.26 to 0.07) −0.07 (−0.34 to 0.07) −0.10 (−3.05 to 0.46) 0.04 (−0.05 to 0.09)
Model 3b
Menopause −0.08 (−0.33 to 0.14) −0.11 (−0.26 to 0.06) −0.14 (−0.54 to 0.07) 0.08 (−0.13 to 0.33) 0.18 (0.02 to 0.59) 0.24 (0.57 to 5.52) 0.11 (−0.05 to 0.14)
Age 0.22 (0.00 to 0.20) 0.35 (0.05 to 0.18) 0.34 (0.09 to 0.35) 0.29 (0.04 to 0.23) 0.31 (0.09 to 0.32) 0.26 (0.24 to 2.30) −0.24 (−0.08 to 0.00)
BMI 0.10 (−0.04 to 0.19) 0.12 (−0.02 to 0.15) 0.15 (0.00 to 0.31) 0.19 (0.03 to 0.26) 0.11 (−0.03 to 0.26) −0.03 (−1.47 to 1.04) 0.05 (−0.04 to 0.06)
Toothbrushing 0.25 (0.07 to 0.27) 0.16 (0.01 to 0.14) 0.14 (0.00 to 0.26) 0.06 (−0.05 to 0.14) 0.02 (−0.10 to 0.14) −0.06 (−1.48 to 0.61) 0.14 (−0.01 to 0.07)
Flossing −0.08 (−0.27 to 0.06) −0.10 (−0.21 to 0.01) −0.08 (−0.36 to 0.07) −0.07 (−0.26 to 0.06) −0.07 (−0.33 to 0.07) −0.08 (−2.80 to 0.67) 0.05 (−0.05 to 0.09)
Education −0.14 (−0.22 to 0.02) −0.15 (−0.16 to 0.00) −0.16 (−0.33 to 0.00) −0.05 (−0.16 to 0.08) −0.06 (−0.21 to 0.08) −0.23 (−3.06 to −0.48) 0.01 (−0.05 to 0.05)
Count of 0.16 (0.00 to 0.14) 0.17 (0.01 to 0.10) 0.15 (0.00 to 0.19) 0.20 (0.03 to 0.16) 0.24 (0.07 to 0.24) −0.06 (−1.00 to 0.47) −0.18 (−0.06 to 0.00)
metabolic
risk factors
CI, confidence interval.
Bold denotes statistical significance at P < 0.05.
a Standardized 𝛽 coefficient represents the change in periodontal parameters (dependent variables) for each unit increase in the predictor variable as menopausal status.
b
Model 1 is adjusted for age; model 2 is adjusted for model 1 + BMI, toothbrushing and flossing; model 3 is adjusted for model 2 + education, and count of metabolic risk factors.
KEMER DOĞAN ET AL.
KEMER DOĞAN ET AL. 337

TABLE 4 Multivariate regression models (𝛽 a [95% CI]) of oxidative parameters related to menopausal status
Dependent variables
Independent variables MPO TAOC TOS OSI
Model 1b
Menopause 0.46 (0.90 to 2.89) −0.23 (−5.41 to −0.04) −0.02 (−0.26 to 0.21) 0.30 (0.01 to 0.06)
Age −0.21 (−0.70 to 0.05) 0.17 (−0.24 to 1.76) 0.33 (0.05 to 0.22) −0.19 (−0.02 to 0.00)
Model 2b
Menopause 0.46 (0.91 to 2.90) −0.22 (−5.35 to 0.09) −0.03 (−0.26 to 0.21) 0.30 (0.01 to 0.06)
Age −0.27 (−0.81 to −0.03) 0.21 (−0.13 to 2.01) 0.28 (0.02 to 0.20) −0.22 (−0.02 to 0.00)
BMI 0.05 (−0.37 to 0.61) −0.10 (−1.96 to 0.57) 0.12 (−0.03 to 0.18) 0.08 (−0.01 to 0.02)
Toothbrushing 0.17 (−0.05 to 0.78) −0.01 (−1.17 to 0.98) 0.01 (−0.09 to 0.10) 0.00 (−0.01 to 0.10)
Flossing 0.06 (−0.44 to 0.99) −0.04 (−2.39 to 1.42) −0.03 (−0.19 to 0.14) 0.00 (−0.02 to 0.02)
Model 3b
Menopause 0.45 (0.87 to 2.88) −0.24 (−5.56 to −0.18) −0.01 (−0.25 to 0.22) 0.30 (0.01 to 0.06)
Age −0.27 (−0.84 to −0.01) 0.32 (0.30 to 2.54) 0.21 (−0.02 to 0.18) −0.24 (−0.02 to 0.00)
BMI 0.07 (−0.36 to 0.71) −0.11 (−1.45 to 1.29) 0.06 (−0.08 to 0.16) 0.09 (−0.01 to 0.02)
Toothbrushing 0.17 (−0.06 to 0.81) 0.06 (−0.74 to 1.53) −0.05 (−0.13 to 0.07) −0.01 (−0.01 to 0.01)
Flossing 0.07 (−0.42 to 1.03) −0.05 (−2.51 to 1.26) −0.02 (−0.18 to 0.15) 0.01 (−0.02 to 0.02)
Education −0.05 (−0.66 to 0.42) 0.19 (−0.06 to 2.76) −0.14 (−0.21 to 0.03) −0.07 (−0.02 to 0.01)
Count of metabolic risk factors −0.09 (−0.44 to 0.17) −0.14 (−1.39 to 0.21) 0.09 (−0.04 to 0.10) −0.04 (−0.01 to 0.01)
CI, confidence interval.
Bold denotes statistical significance at P < 0.05.
a
Standardized 𝛽 coefficient represents the change in oxidative parameters (dependent variables) for each unit increase in the predictor variable as menopausal status.
b Model 1 is adjusted for age; model 2 is adjusted for model 1 + BMI, toothbrushing and flossing; model 3 is adjusted for model 2 + education, and count of metabolic

risk factors.

estrogen increases the MPO release from neutrophils,44 ele-
vated systemic and periodontal parameters in postmenopausal
individuals in models 1 and 2 could be suggested to contribute
to an increase in salivary oxidative stress markers. Further-
more, we can conclude that menopause itself (without any risk
factors) may increase salivary oxidative stress parameters in
model 3.
The association between periodontal disease and various
systemic diseases such as diabetes,45 hyperlipidemia46 or
obesity47 via mitochondrial dysfunction and oxidative stress
have already been reported. It has been stated that oxidative
stress may act as a potential common link in the relation-
ship between periodontitis and each component of metabolic
syndrome.48 TAOC and TOS levels were shown to be more
beneficial in terms of acquiring information about the total
oxidative status.6,7 Demirbag et al.9 showed that as the num-
ber of metabolic syndrome components increased, TAOC lev-
els decreased, and oxidative stress, OSI and total peroxidase
levels increased in the blood. In accordance with the literature,
elevated TOS levels were reported as the count of metabolic
risk factors increased. On the other hand, specific biomarkers
can be used to determine oxidative status.4,8,43,46 The ratio of
reduced glutathione (GSH) / GSSG (oxidized form of GSH),
a critical factor in the activation of redox-sensitive transcrip-
tion factors49 may clarify oxidative status in an extracellular
environment.46
The present study has some limitations. The cross-sectional
study design helps us to evaluate the current status of
menopause, which means that causal inferences cannot be
made. The effects of the confounders such as age and BMI,
which were found to be different between the Pre/M and
Post/M groups and could affect the study results, have been
tried to be limited in adjusted regression models. However,
there is a possibility that residual confounding variables also
including individual features (diet habits, physical activity,
drugs etc.), which can cause systemic effects, make it difficult
to comment on the results.
5 CONC LU SI ON S
The present study not only confirms the effect of menopause
on periodontal disease but also supports the notion that the
increased count of metabolic risk factors could play an impor-
tant role in the relationship between periodontal disease and
menopause. Salivary oxidative stress level may be used as an
indicator of the increased systemic and periodontal inflam-
matory response due to menopausal status. In order to reveal
more specific interactions among periodontal, menopausal
and systemic diseases, inflammatory markers and oxidants /
antioxidants in saliva and also estrogen levels, which may be
used as an indicator of menopause, can be evaluated.
 338

TABLE 5 Multivariate regression models (𝛽 a [95% CI]) of periodontal parameters and oxidative markers
Dependent
variables PI GI SBI PD CAL MT SV
MPO
Model 1b −0.02 (−0.69 to 0.55) −0.02 (−1.05 to 0.87) −0.06 (−0.64 to 0.34) 0.09 (−0.34 to 0.91) 0.22 (0.02 to 1.01) 0.03 (−0.05 to 0.07) 0.20 (0.27 to 3.84)
Model 2b −0.11 (−1.03 to 0.33) −0.10 (−1.54 to 0.56) −0.15 (−0.90 to 0.16) 0.04 (−0.55 to 0.81) 0.21 (−0.06 to 1.02) 0.05 (−0.05 to 0.08) 0.18 (0.11 to 3.70)
Model 3b −0.10 (−1.01 to 0.38) −0.09 (−1.55 to 0.65) −0.15 (−0.92 to 0.19) 0.07 (−0.50 to 0.91) 0.26 (0.04 to 1.16) 0.05 (−0.05 to 0.08) 0.18 (−0.01 to 3.66)
Model 4b −0.07 (−0.91 to 0.42) −0.05 (−1.31 to 0.80) −0.10 (−0.78 to 0.29) 0.03 (−0.58 to 0.77) 0.20 (−0.07 to 1.01) −0.03 (−0.07 to 0.05) 0.14 (−0.34 to 3.20)
TAOC
Model 1b 0.06 (−1.03 to 2.10) −0.01 (−2.39 to 2.23) 0.03 (0.00 to 1.40) −0.10 (−2.56 to 0.74) −0.17 (−2.48 to 0.15) −0.04 (−0.20 to 0.12) −0.11 (−7.20 to 1.27)
b
Model 2 0.10 (−0.82 to 2.68) 0.02 (−2.26 to 2.85) 0.07 (−0.85 to 1.78) 0.08 (−2.48 to 1.09) −0.16 (−2.45 to 0.33) −0.05 (−0.21 to 0.12) −0.11 (−7.19 to 1.35)
Model 3b 0.15 (−0.32 to 3.21) 0.08 (−1.50 to 3.71) 0.13 (−0.45 to 2.22) 0.04 (−2.14 to 1.48) −0.11 (−2.19 to 0.69) −0.03 (−0.19 to 0.14) −0.13 (−7.84 to 0.68)
Model 4b 0.14 (−0.42 to 3.08) 0.06 (−1.75 to 3.44) 0.11 (−0.60 to 2.07) −0.02 (−2.00 to 1.59) −0.08 (−1.96 to 0.93) 0.01 (−0.16 to 0.18) −0.12 (−7.50 to 0.98)
TOS
Model 1b 0.04 (−0.10 to 0.17) 0.05 (−0.14 to 0.25) 0.01 (−0.10 to 0.11) 0.21 (0.03 to 0.31) 0.15 (−0.02 to 0.21) 0.03 (−0.01 to 0.02) −0.22 (−0.90 to −0.19)
Model 2b 0.00 (−0.15 to 0.15) 0.00 (−0.22 to 0.22) −0.04 (−0.14 to 0.09) 0.19 (0.01 to 0.31) 0.13 (−0.04 to 0.20) 0.03 (−0.01 to 0.02) −0.22 (−0.91 to −0.20)
Model 3b −0.04 (−0.18 to 0.12) −0.04 (−0.27 to 0.18) −0.08 (−0.17 to 0.06) 0.17 (−0.02 to 0.29) 0.10 (−0.07 to 0.18) 0.01 (−0.01 to 0.02) −0.21 (−0.89 to −0.17)
Model 4b −0.04 (−0.19 to 0.12) −0.04 (−0.28 to 0.18) −0.09 (−0.17 to 0.06) 0.17 (−0.02 to 0.29) 0.10 (−0.06 to 0.19) 0.02 (−0.01 to 0.02) −0.21 (−0.89 to −0.17)
OSI
Model 1b −0.09 (−0.02 to 0.01) −0.08 (−0.03 to 0.01) −0.13 (−0.02 to 0.00) 0.16 (0.00 to 0.03) 0.16 (0.00 to 0.02) −0.04 (0.00 to 0.00) 0.10 (−0.01 to 0.06)
Model 2b −0.13 (−0.03 to 0.01) −0.13 (−0.04 to 0.01) −0.18 (−0.02 to 0.00) 0.15 (0.00 to 0.03) 0.14 (0.00 to 0.02) −0.04 (0.00 to 0.00) 0.10 (−0.01 to 0.06)
Model 3b −0.14 (−0.03 to 0.00) −0.13 (−0.04 to 0.01) −0.20 (−0.02 to 0.00) 0.16 (0.00 to 0.03) 0.16 (0.00 to 0.02) −0.05 (0.00 to 0.00) 0.09 (−0.02 to 0.06)
Model 4b −0.12 (−0.03 to 0.01) −0.11 (−0.04 to 0.01) −0.17 (−0.02 to 0.00) 0.15 (0.00 to 0.03) 0.12 (0.00 to 0.02) −0.10 (0.00 to 0.00) 0.08 (−0.02 to 0.06)
CI, confidence interval.
Bold denotes statistical significance at P < 0.05.
a
Standardized 𝛽 coefficient represents the change in dependent variables for each unit increase in the predictor variable as periodontal parameters.
b Model 1 is adjusted for age; model 2 is adjusted for model 1 + BMI, toothbrushing and flossing; model 3 is adjusted for model 2 + education, and count of metabolic risk factors; model 4 is adjusted for model 3 + menopausal

status.
KEMER DOĞAN ET AL.
KEMER DOĞAN ET AL.
ACK NOW L E D G M E N T
The authors report no conflicts of interest related to this study.
REFERENCES
1. Socransky SS, Haffajee AD. The bacterial etiology of destructive
periodontal disease: current concepts. J Periodontol. 1992;63:322–
331.
2. Morita T, Ogawa Y, Takada K, et al. Association between peri-
odontal disease and metabolic syndrome. J Public Health Dent.
2009;69:248–253.
3. Knight ET, Liu J, Seymour GJ, Faggion CM, Jr, Cullinan MP. Risk
factors that may modify the innate and adaptive immune responses
in periodontal diseases. Periodontol 2000. 2016;71:22–51.
4. D'Aiuto F, Nibali L, Parkar M, Patel K, Suvan J, Donos N. Oxidative
stress, systemic inflammation, and severe periodontitis. J Dent Res.
2010;89:1241–1246.
5. Kaufman E, Lamster IB. Analysis of saliva for periodontal
diagnosis–a review. J Clin Periodontol. 2000;27:453–465.
6. Baltacioglu E, Kehribar MA, Yuva P, et al. Total oxidant status
and bone resorption biomarkers in serum and gingival crevicular
fluid of patients with periodontitis. J Periodontol. 2014;85:317–
326.
7. Erel O. A new automated colorimetric method for measuring total
oxidant status. Clin Biochem. 2005;38:1103–1111.
8. Akalin FA, Baltacioglu E, Alver A, Karabulut E. Lipid peroxidation
levels and total oxidant status in serum, saliva and gingival crevic-
ular fluid in patients with chronic periodontitis. J Clin Periodontol.
2007;34:558–565.
9. Demirbag R, Yilmaz R, Gur M, et al. DNA damage in metabolic
syndrome and its association with antioxidative and oxidative mea-
surements. Int J Clin Pract. 2006;60:1187–1193.
10. Roberts CK, Barnard RJ, Sindhu RK, Jurczak M, Ehdaie A,
Vaziri ND. Oxidative stress and dysregulation of NAD(P)H oxi-
dase and antioxidant enzymes in diet-induced metabolic syndrome.
Metabolism. 2006;55:928–934.
11. Klebanoff SJ. Myeloperoxidase-halide-hydrogen peroxide antibac-
terial system. J Bacteriol. 1968;95:2131–2138.
12. Yamalik N, Caglayan F, Kilinc K, Kilinc A, Tumer C. The
importance of data presentation regarding gingival crevicular fluid
myeloperoxidase and elastase-like activity in periodontal disease
and health status. J Periodontol. 2000;71:460–467.
13. Behle JH, Sedaghatfar MH, Demmer RT, et al. Heterogeneity of
systemic inflammatory responses to periodontal therapy. J Clin
Periodontol. 2009;36:287–294.
14. Dodds MW, Yeh CK, Johnson DA. Salivary alterations in
type 2 (non-insulin-dependent) diabetes mellitus and hyper-
tension. Community Dent Oral Epidemiol. 2000;28:373–
381.
15. Puntoni M, Sbrana F, Bigazzi F, Minichilli F, Ferdeghini E,
Sampietro T. Myeloperoxidase modulation by LDL apheresis
in familial hypercholesterolemia. Lipids Health Dis. 2011;10:
185.
16. Shah TJ, Leik CE, Walsh SW. Neutrophil infiltration and systemic
vascular inflammation in obese women. Reprod Sci. 2010;17:116–
124.
339
17. Ridker PM, Hennekens CH, Buring JE, Rifai N. C-reactive pro-
tein and other markers of inflammation in the prediction of car-
diovascular disease in women. N Engl J Med. 2000;342:836–
843.
18. Sanchez-Rodriguez MA, Zacarias-Flores M, Arronte-Rosales A,
Correa-Munoz E, Mendoza-Nunez VM. Menopause as risk factor
for oxidative stress. Menopause. 2012;19:361–367.
19. Gohlke-Barwolf C. Coronary artery disease–is menopause a risk
factor?. Basic research in cardiology. 2000;95(Suppl 1):I77–83.
20. Otomo-Corgel J. Dental management of the female patient. Peri-
odontol 2000. 2013;61:219–231.
21. Daviglus ML, Talavera GA, Aviles-Santa ML, et al. Prevalence
of major cardiovascular risk factors and cardiovascular diseases
among Hispanic/Latino individuals of diverse backgrounds in the
United States. JAMA. 2012;308:1775–1784.
22. Han DH, Lim SY, Sun BC, Paek D, Kim HD. The association of
metabolic syndrome with periodontal disease is confounded by age
and smoking in a Korean population: the Shiwha-Banwol Environ-
mental Health Study. J Clin Periodontol. 2010;37:609–616.
23. Kaye EK, Chen N, Cabral HJ, Vokonas P, Garcia RI. Metabolic
syndrome and periodontal disease progression in men. J Dent Res.
2016;95:822–828.
24. World Health Organization. Research on the menopause in the
1990s: report of a WHO scientific group. World Health Organ Tech
Rep Ser 1996;866:1–107.
25. WHO. Obesity: preventing and managing the global epidemic.
Report of a WHO Consultation. WHO Technical Report Series 894.
Geneva: World Health Organization; 2000.
26. Ahn YB, Shin MS, Byun JS, Kim HD. The association of hyperten-
sion with periodontitis is highlighted in female adults: results from
the Fourth Korea National Health and Nutrition Examination Sur-
vey. J Clin Periodontol. 2015;42:998–1005.
27. Third report of the national cholesterol education program (NCEP)
expert panel on detection, evaluation, and treatment of high blood
cholesterol in adults (Adult Treatment Panel III) final report. Circu-
lation 2002;106:3143.
28. Silness J, Loe H. Periodontal disease in pregnancy. II. Correla-
tion between oral hygiene and periodontal condtion. Acta Odontol
Scand. 1964;22:121–135.
29. Loe H, Silness J. Periodontal disease in pregnancy. I. Prevalence
and severity. Acta Odontol Scand. 1963;21:533–551.
30. Muhlemann HR, Son S. Gingival sulcus bleeding–a leading symp-
tom in initial gingivitis. Helv Odontol Acta. 1971;15:107–113.
31. Navazesh M. Methods for collecting saliva. Ann N Y Acad Sci.
1993;694:72–77.
32. Erel O. A novel automated method to measure total antioxi-
dant response against potent free radical reactions. Clin Biochem.
2004;37:112–119.
33. Penoni DC, Fidalgo TK, Torres SR, et al. Bone density and clini-
cal periodontal attachment in postmenopausal women. J Dent Res.
2017;96:261–269.
34. Tezal M, Wactawski-Wende J, Grossi SG, Ho AW, Dunford R,
Genco RJ. The relationship between bone mineral density and peri-
odontitis in postmenopausal women. J Periodontol. 2000;71:1492–
1498.
340
35. Polotsky HN, Polotsky AJ. Metabolic implications of menopause.
Semin Reprod Med. 2010;28:426–434.
36. Furuta M, Liu A, Shinagawa T, et al. Tooth loss and metabolic
syndrome in middle-aged Japanese adults. J Clin Periodontol.
2016;43:482–491.
37. Li Z, McNamara JR, Fruchart JC, et al. Effects of gender and
menopausal status on plasma lipoprotein subspecies and particle
sizes. J Lipid Res. 1996;37:1886–1896.
38. Carr MC. The emergence of the metabolic syndrome with
menopause. J Clin Endocrinol Metab. 2003;88:2404–2411.
39. Meigs JB. Invited commentary: insulin resistance syndrome? Syn-
drome X? Multiple metabolic syndrome? A syndrome at all? Fac-
tor analysis reveals patterns in the fabric of correlated metabolic
risk factors. Am J Epidemiol. 2000;152:908–911. discussion
912.
40. von Bultzingslowen I, Sollecito TP, Fox PC, et al. Salivary dys-
function associated with systemic diseases: systematic review and
clinical management recommendations. Oral Surg Oral Med Oral
Pathol Oral Radiol Endod. 2007;103(Suppl):S57. e51-15.
41. Tezal M, Wactawski-Wende J, Grossi SG, Dmochowski J, Genco
RJ. Periodontal disease and the incidence of tooth loss in post-
menopausal women. J Periodontol. 2005;76:1123–1128.
42. Baltacioglu E, Yuva P, Aydin G, et al. Lipid peroxidation lev-
els and total oxidant/antioxidant status in serum and saliva from
patients with chronic and aggressive periodontitis. Oxidative stress
index: a new biomarker for periodontal disease. J Periodontol.
2014;85:1432–1441.
43. Baltacioglu E, Akalin FA, Alver A, Balaban F, Unsal M, Karab-
ulut E. Total antioxidant capacity and superoxide dismutase
activity levels in serum and gingival crevicular fluid in post-
KEMER DOĞAN ET AL.
menopausal women with chronic periodontitis. J Clin Periodontol.
2006;33:385–392.
44. Bekesi G, Kakucs R, Varbiro S, et al. Induced myeloperoxidase
activity and related superoxide inhibition during hormone replace-
ment therapy. BJOG. 2001;108:474–481.
45. Sun X, Mao Y, Dai P, et al. Mitochondrial dysfunction is involved
in the aggravation of periodontitis by diabetes. J Clin Periodontol.
2017;44:463–471.
46. Kirzioglu FY, Fentoglu O, Bulut MT, et al. Is a cholestrol-enriched
diet a risk factor for alveolar bone loss. J Periodontol. 2016;87:529–
538.
47. Atabay VE, Lutfioglu M, Avci B, Sakallioglu EE, Aydogdu
A. Obesity and oxidative stress in patients with different peri-
odontal status: a case-control study. J Periodontal Res. 2017;52:
51–60.
48. Bullon P, Morillo JM, Ramirez-Tortosa MC, Quiles JL, Newman
HN, Battino M. Metabolic syndrome and periodontitis: is oxidative
stress a common link. J Dent Res. 2009;88:503–518.
49. Chapple IL, Matthews JB. The role of reactive oxygen and antiox-
idant species in periodontal tissue destruction. Periodontol 2000.
2007;43:160–232.
How to cite this article: Kemer Doğan ES, Kırzıoğlu
FY, Doğan B, et al. The role of menopause on
the relationship between metabolic risk factors and
periodontal disease via salivary oxidative parame-
ters. J Periodontol. 2018;89:331–340. https://doi.org/
10.1002/JPER.17-0314