Deep Learning for ECG Analysis: Benchmarksand Insights from PTB-XL

Deep Learning for ECG Analysis: Benchmarksand Insights from PTB-XL

심전도 분석을 위한 딥러닝: 벤치마크 및 PTB-XL의 인사이트

원문 : https://arxiv.org/pdf/2004.13701.pdf

 

Abstract— Electrocardiography is a very common, noninvasive diagnostic procedure and its interpretation is increasingly supported by automatic interpretation algorithms. The progress in the field of automatic ECG interpretation has up to now been hampered by a lack of appropriate datasets for training as well as a lack of well-defined evaluation procedures to ensure comparability of different algorithms. To alleviate these issues, we put forward first benchmarking results for the recently published, freely accessible PTB-XL dataset, covering a variety of tasks from different ECG statement prediction tasks over age and gender prediction to signal quality assessment. We find that convolutional neural networks, in particular resnetand inception-based architectures, show the strongest performance across all tasks outperforming feature-based algorithms by a large margin. These results are complemented by deeper insights into the classification algorithm in terms of hidden stratification, model uncertainty and an exploratory interpretability analysis. We also put forward benchmarking results for the ICBEB2018 challenge ECG dataset and discuss prospects of transfer learning using classifiers pretrained on PTB-XL. With this resource, we aim to establish the PTB-XL dataset as a resource for structured benchmarking of ECG analysis algorithms and encourage other researchers in the field to join these efforts. 

요약- 심전도는 매우 일반적인 비침습적 진단 절차이며 자동 해석 알고리즘에 의한 해석이 점점 더 많이 지원되고 있습니다. 지금까지 자동 심전도 해석 분야의 발전은 학습을 위한 적절한 데이터 세트의 부족과 서로 다른 알고리즘의 비교 가능성을 보장하기 위한 잘 정의된 평가 절차의 부족으로 인해 방해를 받아 왔습니다. 이러한 문제를 완화하기 위해 최근 무료로 공개된 PTB-XL 데이터 세트에 대한 벤치마킹 결과를 제시했으며, 연령 및 성별 예측부터 신호 품질 평가에 이르기까지 다양한 작업을 포괄합니다. 컨볼루션 신경망, 특히 리셋 및 인셉션 기반 아키텍처가 모든 작업에서 특징 기반 알고리즘을 큰 차이로 능가하는 가장 강력한 성능을 보인다는 사실을 발견했습니다. 이러한 결과는 숨겨진 계층화, 모델 불확실성 및 탐색적 해석 가능성 분석 측면에서 분류 알고리즘에 대한 심층적인 인사이트를 통해 보완됩니다. 또한 ICBEB2018 챌린지 ECG 데이터 세트에 대한 벤치마킹 결과를 제시하고 PTB-XL에서 사전 훈련된 분류기를 사용한 전이 학습의 전망에 대해 논의합니다. 이 리소스를 통해 PTB-XL 데이터셋이 ECG 분석 알고리즘의 구조화된 벤치마킹을 위한 리소스로 자리매김하고, 이 분야의 다른 연구자들이 이러한 노력에 동참할 수 있도록 장려하고자 합니다.

 

Index Terms— Decision support systems, Electrocardiography, Machine learning algorithms

색인 용어- 의사 결정 지원 시스템, 심전도, 머신 러닝 알고리즘

 

I. INTRODUCTION

 

 

CARDIOVASCULAR diseases (CVDs) rank among diseases of highest mortality [1] and were in this respect only recently surpassed by cancer in high-income countries [2]. Electrocardiography (ECG) is a non-invasive tool to assess the general cardiac condition of a patient and is therefore as first-in-line examination for diagnosis of CVD. In the US, during about 5% of the office visits an ECG was ordered or provided [3]. In spite of these numbers, ECG interpretation remains a difficult task even for cardiologists [4] but even more so for residents, general practioners [4], [5] or doctors in the emergency room who have to interprete ECGs urgently. A second major application area that will even grow in importance in the future is the telemedicine, in particular the monitoring of Holter ECGs. In both of these exemplary cases medical personnel could profit from significant reliefs if they were supported by advanced decision support systems relying on automatic ECG interpretation algorithms.

심혈관 질환(CVD)은 사망률이 가장 높은 질병 중 하나이며[1], 최근 고소득 국가에서는 암을 추월한 바 있습니다[2]. 심전도(ECG)는 환자의 전반적인 심장 상태를 평가할 수 있는 비침습적 도구로, CVD 진단을 위한 1차 검사로 사용됩니다. 미국에서는 진료실 방문의 약 5%에서 심전도 검사를 주문하거나 제공했습니다[3]. 이러한 수치에도 불구하고 심전도 해석은 심장 전문의[4]에게도 여전히 어려운 작업이지만 레지던트, 일반의[4], [5] 또는 긴급하게 심전도를 해석해야 하는 응급실 의사에게는 더더욱 어려운 작업입니다. 앞으로 그 중요성이 더욱 커질 두 번째 주요 응용 분야는 원격 의료, 특히 홀터 심전도 모니터링입니다. 이 두 가지 예시적인 사례에서 의료진은 자동 심전도 해석 알고리즘에 기반한 고급 의사 결정 지원 시스템의 지원을 받으면 상당한 도움을 받을 수 있습니다.

 

During recent years, we have witnessed remarkable advances in automatic ECG interpretation algorithms. In particular, deep-learning-based approaches have reached or even surpassed cardiologist-level performance for selected subtasks [6]–[10] or enabled statements that were very difficult to make for cardiologists e.g. to accurately infer age and gender from the ECG [11]. Due to the apparent simplicity and reduced dimensionality compared to imaging data, also the broader machine learning community has gained a lot of interest in ECG classification as documented by numerous research papers each year, see [12] for a recent review.

최근 몇 년 동안 자동 심전도 해석 알고리즘이 괄목할 만한 발전을 이루었습니다. 특히 딥러닝 기반 접근 방식은 일부 하위 작업에서 심장 전문의 수준의 성능에 도달하거나 이를 뛰어넘었으며[6]-[10], 심전도로부터 나이와 성별을 정확하게 추론하는 등 심장 전문의가 수행하기 매우 어려웠던 작업도 가능하게 했습니다[11]. 영상 데이터에 비해 단순하고 차원이 줄어들었기 때문에 광범위한 머신 러닝 커뮤니티에서도 매년 수많은 연구 논문에서 문서화되는 것처럼 심전도 분류에 많은 관심을 보이고 있으며, 최근 리뷰는 [12]를 참조하세요.

 

We see deep learning algorithms in the domain of computer vision as a role model for the deep learning algorithms in the field of ECG analysis. The tremendous advances for example in the field of image recognition relied crucially on the availability of large datasets and the competitive environment of classification challenges with clear evaluation procedures. In reverse, we see these two aspects as two major issues that hamper the progress in algorithmic ECG analysis: First, open ECG datasets are typically very small [13] and existing large datasets remain inaccessible for the general public. This issue has been at least partially resolved by the publication of the PTB-XL dataset [14], [15] hosted by PhysioNet [16], which provides a freely accessible ECG dataset of unprecedented size with predefined train-test splits based on stratified sampling. Second, the existing datasets typically provide only the raw data, but there exist no clearly defined benchmarking tasks with corresponding evaluation procedures. This severely restricts the comparability of different algorithms, as experimental details such as sample selection, train-test splits, evaluation metrics and score estimation can largely impact the final result. To address this second issue, we propose a range of different tasks showcasing the variability of the dataset ranging from the prediction of ECG statements over age and gender prediction to the assessment of signal quality. For these tasks, we present first benchmarking results for deep-learning-based time series classification algorithms. We use the ICBEB2018 dataset to illustrate the promising prospects of transfer learning especially in the small dataset regime establishing PTB-XL as a pretraining resource for generic ECG classifiers, very much like ImageNet [17] in the computer vision domain.

우리는 컴퓨터 비전 분야의 딥러닝 알고리즘을 심전도 분석 분야의 딥러닝 알고리즘의 롤모델로 보고 있습니다. 예를 들어 이미지 인식 분야의 엄청난 발전은 대규모 데이터 세트의 가용성과 명확한 평가 절차가 있는 분류 과제의 경쟁 환경에 결정적으로 의존했습니다. 역으로, 우리는 이 두 가지 측면이 알고리즘 심전도 분석의 발전을 저해하는 두 가지 주요 문제로 보고 있습니다: 첫째, 개방형 ECG 데이터 세트는 일반적으로 매우 작으며[13] 기존의 대규모 데이터 세트는 일반 대중이 접근할 수 없습니다. 이 문제는 PhysioNet [16]에서 호스팅하는 PTB-XL 데이터 세트 [14], [15]의 공개로 적어도 부분적으로 해결되었으며, 이 데이터 세트는 계층화된 샘플링을 기반으로 사전 정의된 훈련-테스트 분할을 통해 전례 없는 크기의 ECG 데이터 세트에 자유롭게 액세스할 수 있습니다. 둘째, 기존 데이터 세트는 일반적으로 원시 데이터만 제공하지만 해당 평가 절차에 따라 명확하게 정의된 벤치마킹 작업이 존재하지 않습니다. 이는 샘플 선택, 훈련-테스트 분할, 평가 지표 및 점수 추정과 같은 실험 세부 사항이 최종 결과에 큰 영향을 미칠 수 있기 때문에 서로 다른 알고리즘의 비교 가능성을 심각하게 제한합니다. 이 두 번째 문제를 해결하기 위해 연령 및 성별에 따른 심전도 문장 예측부터 신호 품질 평가에 이르기까지 데이터 세트의 가변성을 보여주는 다양한 작업을 제안합니다. 이러한 과제에 대해 먼저 딥러닝 기반 시계열 분류 알고리즘에 대한 벤치마킹 결과를 제시합니다. 특히 컴퓨터 비전 영역의 ImageNet [17]과 같이 일반 심전도 분류기를 위한 사전 학습 리소스로서 PTB-XL을 구축하는 소규모 데이터 세트 체제에서 전이 학습의 유망한 전망을 설명하기 위해 ICBEB2018 데이터 세트를 사용합니다.

 

II. MATERIALS & METHODS

 

A. PTB-XL dataset

In this section, we briefly introduce the PTB-XL dataset [15] that underlies most experiments presented below. The PTBXL dataset comprises 21837 clinical 12-lead ECG records of 10 seconds length from 18885 patients, where 52 % were male and 48 % were female. The ECG statements used for annotation are conform to the SCP-ECG standard [18] and were assigned to three non-mutually exclusive categories diag. (short for diagnostic), form and rhythm. In total, there are 71 different statements, which decompose into 44 diagnostic, 12 rhythm and 19 form statements. Note that there are 4 form statements that are also assigned to the set of diagnostic ECG statements. For diagnostic statements also a hierarchical organization into five coarse superclasses (NORM: normal ECG, CD: conduction disturbance, MI: myocardial infarction, HYP: hypertrophy and STTC: ST/T changes) and 24 sub-classes is provided, see Figure 1. For further details on the dataset and the annotation scheme, we refer the reader to the original publication [15]. To illustrate the versatility of tasks that can be addressed within the dataset, we also incorporate the further metadata provided, namely demographic information such as age and gender or signal quality as assessed by a technical expert.

A. PTB-XL 데이터 세트 

이 섹션에서는 아래에 제시된 대부분의 실험의 기반이 되는 PTB-XL 데이터 세트[15]에 대해 간략히 소개합니다. PTBXL 데이터 세트는 1만 8885명의 환자로부터 얻은 10초 길이의 임상 12-리드 ECG 기록 21837개로 구성되어 있으며, 52%는 남성, 48%는 여성입니다. 주석에 사용된 ECG 문장은 SCP-ECG 표준[18]을 준수하며 상호 배타적이지 않은 세 가지 범주인 진단(진단의 줄임말), 형태 및 리듬에 할당되었습니다. 총 71개의 다른 진술이 있으며, 이는 44개의 진단, 12개의 리듬, 19개의 형태 진술로 분해된다. 진단 ECG 문 집합에도 4개의 형식 문이 할당되어 있습니다. 진단 문에 대해서도 5개의 거친 슈퍼 클래스(NORM: 정상 ECG, CD: 전도 장애, MI: 심근 경색, HYP: 비대, STTC: ST/T 변화)와 24개의 하위 클래스로 계층적 조직이 제공됩니다(그림 1 참조). 데이터 세트와 주석 체계에 대한 자세한 내용은 원본 출판물 [15]을 참조하시기 바랍니다. 데이터 세트 내에서 처리할 수 있는 작업의 다양성을 설명하기 위해 제공된 추가 메타데이터, 즉 기술 전문가가 평가한 연령 및 성별과 같은 인구통계학적 정보 또는 신호 품질도 통합했습니다.

 

B. Time series classification algorithms

For benchmarking different classification algorithms, we focus on algorithms that operate on raw multivariate time series data. An alternative class of algorithms operates on derived or transformed features such as Fourier or Wavelet coefficients or handcrafted features extract from single beats after beat segmentation, see [19], [20] for review in the context of ECG classification and [21] for (mostly univariate) time series classification in general. Deep learning approaches for time series classification are covered in a variety of recent, excellent reviews [22]–[24].

B. 시계열 분류 알고리즘
다양한 분류 알고리즘을 벤치마킹하기 위해 원시 다변량 시계열 데이터에서 작동하는 알고리즘에 중점을 둡니다. 다른 종류의 알고리즘은 푸리에 또는 웨이블릿 계수 또는 비트 분할 후 단일 비트에서 추출한 수작업 특징과 같이 파생되거나 변형된 특징에 대해 작동하며, ECG 분류의 맥락에서 검토하려면 [19], [20]을, 일반적으로 (대부분 단변량) 시계열 분류에 대해서는 [21]을 참조하세요. 시계열 분류를 위한 딥러닝 접근 방식은 최근의 다양한 우수 리뷰 [22]-[24]에서 다룹니다. 

 

We evaluate adaptations of a range of different algorithms from the literature that can be broadly categorized as follows, see Appendix I for experimental details:

저희는 다양한 알고리즘의 적응을 평가합니다. 다음과 같이 크게 분류할 수 있는 문헌에서 다양한 알고리즘을 평가했습니다, 실험에 대한 자세한 내용은 부록 I을 참조하세요:

 

Fig. 1: Summary of the PTB-XL dataset in terms of diagnostic super and subclasses where the size of area represents the fraction of samples (figure adapted from [15]).

그림 1: 면적의 크기가 샘플의 비율을 나타내는 진단 슈퍼 및 하위 클래스의 관점에서의 PTB-XL 데이터 세트 요약(그림은 [15]에서 각색).

 

• convolutional neural networks:

– standard: fully convolutional [25], Deep4Net [26]

– resnet-based: one-dimensional adaptations of standard resnets [25], [27], wide resnets [28] and xresnets [29]

– inception-based: InceptionTime [30]

 

• recurrent neural networks: LSTM [31], GRU [32]

• baseline classifiers:

– feature-based: Wavelet + shallow NN inspired by [33]

– naive: predicting the frequency of each term in the training set

- 컨볼루션 신경망:
- 표준: 완전 컨볼루션 [25], Deep4Net [26]
- 레스넷 기반: 표준 레스넷[25], [27], 와이드 레스넷[28], 엑스 레스넷[29]의 1차원적 변형
- 인셉션 기반: 인셉션타임[30]

- 순환 신경망: LSTM [31], GRU [32]
- 기준선 분류기:
- 특징 기반: 웨이블릿 + [33]에서 영감을 얻은 얕은 NN
- 나이브: 훈련 세트에서 각 용어의 빈도 예측

 

For reasons of clarity, we only report the performance for selected representatives including the best-performing method for each group. Typically the differences within the different groups are rather small. For completeness, the full results including all architectures are available in the accompanying code repository [34]. To encourage future benchmarking on this dataset, we release our repository [34] used to produce the results presented below along with instructions on how to evaluate the performance of custom classifiers in this framework. Finally, we would like to stress that the deep learning models were trained on the original time series data without any further preprocessing such as removing baseline wander and/or filtering, which are commonly used in literature approaches but introduce further hyperparameters into the approach.

명확성을 위해 각 그룹에서 가장 실적이 좋은 방법을 포함하여 선정된 대표자의 성과만 보고합니다. 일반적으로 각 그룹 내에서의 차이는 다소 작습니다. 완전성을 위해 모든 아키텍처를 포함한 전체 결과는 함께 제공되는 코드 리포지토리에서 확인할 수 있습니다[34]. 이 데이터 세트에 대한 향후 벤치마킹을 장려하기 위해, 아래에 제시된 결과를 생성하는 데 사용된 저장소[34]를 이 프레임워크에서 사용자 정의 분류기의 성능을 평가하는 방법에 대한 지침과 함께 공개합니다. 마지막으로, 딥러닝 모델은 문헌 접근 방식에서 일반적으로 사용되지만 접근 방식에 추가적인 하이퍼파라미터를 도입하는 기준선 방황 및/또는 필터링 제거와 같은 추가 전처리 없이 원본 시계열 데이터에 대해 학습되었다는 점을 강조하고 싶습니다.

 

C. Multi-label classification metrics

C. 다중 레이블 분류 메트릭

 

In this subsection, we review metrics for multi-label classification problems, see [35] for a review on multi-label classification metrics and algorithms. Multi-label classification metrics can be categorized broadly as sample-centric and label-centric metrics. The main difference between metrics from both categories is the question whether to first aggregate the scores across labels and then across samples or vice versa. To obtain a comprehensive view of the classification performance, we pick one exemplary metric from each category as proposed on theoretical grounds by [36]. Here, we focus on metrics that can be evaluated based on soft classifier outputs, where no thresholding has been applied yet, as this allows to get a more complete picture of the discriminative power of a given classification algorithm. In addition, it disentangles the selection of an appropriate classifier from the issue of threshold optimization, that will anyway have to be adjusted to match the clinical requirements rather than to optimize a certain global target metric. a) Term-centric metrics: In general label-centric metrics are based on averages across class-specific metrics, which can further subdivided into micro- and macro-averages. In our setting, macro-averaging is preferred, since we expect class imbalance and do not want the score to be dominated by a few large classes. In addition, the distribution of pathologies in the dataset does not follow the natural distribution in the population but rather reflects the data collection process. Averaging class-wise AUCs over all classes yields the termcentric macro AUC (henceforth abbreviated as AUC), which we will use as primary evaluation metric. b) Sample-centric metrics: Sample-centric evaluation metrics measure how accurately classification algorithms assign labels to a given sample, which is an information-retrieval point of view. For the selection of sample-centric metrics, we follow the evaluation procedures over the course of todate three CAFA classification challenges [37]. The CAFA challenges address protein function prediction, which is also an inherent multi-label problem and shows strong structurally similarities to the task of ECG classification. For a given prediction Pi(τ ) for given threshold τ ∈ [0, 1] and corresponding ground-truth annotations Ti , we can define sample-centric precision pr(τ ), recall/sensitivity rc(τ )

이 하위 섹션에서는 다중 라벨 분류 문제에 대한 메트릭을 검토하며, 다중 라벨 분류 메트릭 및 알고리즘에 대한 검토는 [35]를 참조하세요. 다중 레이블 분류 메트릭은 크게 샘플 중심 메트릭과 레이블 중심 메트릭으로 분류할 수 있습니다. 두 카테고리의 메트릭 간의 주요 차이점은 먼저 라벨별로 점수를 집계한 다음 샘플별로 집계할지, 아니면 그 반대의 경우인지에 대한 질문입니다. 분류 성능에 대한 포괄적인 시각을 얻기 위해, [36]의 이론적 근거에 따라 각 범주에서 하나의 모범적인 메트릭을 선택합니다. 여기서는 임계값이 아직 적용되지 않은 소프트 분류기 출력을 기반으로 평가할 수 있는 메트릭에 중점을 두는데, 이는 주어진 분류 알고리즘의 변별력을 보다 완벽하게 파악할 수 있기 때문입니다. 또한, 특정 글로벌 목표 메트릭을 최적화하기보다는 임상 요구 사항에 맞게 조정해야 하는 임계값 최적화 문제에서 적절한 분류기 선택을 분리할 수 있습니다. a) 용어 중심 메트릭: 일반적으로 레이블 중심 지표는 클래스별 지표의 평균을 기반으로 하며, 이는 마이크로 평균과 매크로 평균으로 더 세분화될 수 있습니다. 저희 설정에서는 클래스 불균형이 예상되고 점수가 몇몇 큰 클래스에 의해 좌우되는 것을 원하지 않기 때문에 매크로 평균을 선호합니다. 또한 데이터 세트의 병리 분포는 모집단의 자연 분포를 따르지 않고 데이터 수집 과정을 반영합니다. 모든 클래스에 대한 클래스별 AUC의 평균을 구하면 용어 중심 매크로 AUC(이하 AUC로 약칭)가 산출되며, 이를 기본 평가 지표로 사용할 것입니다. b) 샘플 중심 지표: 샘플 중심 평가 지표는 분류 알고리즘이 주어진 샘플에 라벨을 얼마나 정확하게 할당하는지를 측정하는 것으로, 정보 검색 관점입니다. 샘플 중심 메트릭을 선택하기 위해, 우리는 현재까지 세 가지 CAFA 분류 과제[37]에 대한 평가 절차를 따릅니다. CAFA 과제는 단백질 기능 예측을 다루며, 이는 또한 고유한 다중 레이블 문제이며 ECG 분류 작업과 구조적으로 매우 유사합니다. 주어진 임계값 τ ∈ [0, 1]에 대한 주어진 예측 Pi(τ )와 해당 지상 진실 주석 Ti 에 대해 샘플 중심 정밀도 pr(τ ), 회수/감도 rc(τ ) 를 정의할 수 있습니다.

 

 

where Nτ = {i ∈ 1, . . . , N| P f 1(f ∈ Pi(τ )) > 0} and Nτ = |Nτ |. Here, we handle a possibly vanishing denominator when calculating the average precision in the same way as it is done in the CAFA challenges [38] by restricting the mean to the subset of samples with at least one prediction at the given threshold. Note that this procedure assumes a single threshold rather than class-dependent thresholds. We focus on Fmax as secondary performance metric, which was considered as main metric in the CAFA challenge. To this end, one defines a threshold-dependent F1-score as the harmonic mean of precision and recall, i.e.

여기서 Nτ = {i ∈ 1, . . . , N| P f 1(f ∈ Pi(τ )) > 0} 및 Nτ = |Nτ |. 여기서는 평균 정밀도를 계산할 때 소실될 가능성이 있는 분모( 와 같은 방식으로 평균 정밀도를 계산할 때 소실될 가능성이 있는 분모 CAFA 챌린지[38]에서 수행되는 것과 동일한 방식으로 소실 분모를 처리합니다. 주어진 임계값에서 예측이 하나 이상 있는 샘플의 하위 집합에 대한 평균 샘플의 하위 집합으로 제한합니다. 이 절차는 클래스별 임계값이 아닌 클래스 종속 임계값이 아닌 단일 임계값을 가정합니다. 우리는 CA에서 주요 메트릭으로 간주되었던 주요 지표로 간주되었습니다. 이를 위해 임계값에 종속적인 F1 점수를 고조파로 정의합니다. 평균으로 정의합니다

 

To summarize F1(τ ) by a single number, the threshold is varied and the maximum score, from now on referred to as Fmax, is reported. As in the CAFA challenge, the threshold is optimized on the respective test set for each classification task and classifier under consideration. This procedure allow for a black-box evaluation just based on soft classifier outputs.

F1(τ )을 하나의 숫자로 요약하면 임계값은 다음과 같습니다. 가 달라지며 이제부터는 최대 점수인 Fmax가 보고됩니다. CAFA 챌린지에서와 마찬가지로 임계값은 각 분류 작업에 대한 각 테스트 세트에서 최적화됩니다. 및 고려 중인 분류기에 대한 각 테스트 세트에 대해 임계값이 최적화됩니다. 이 절차를 통해 소프트 분류기 출력만을 기반으로 한 블랙박스 평가가 가능합니다.

 

TABLE I: Number of ECG statments per sample for a given level.

표 I: 특정 레벨에 대한 샘플당 ECG 상태 수입니다.

III. BENCHMARKING RESULTS ON PTB-XL AND ICBEB2018

 

PTB-XL comes with a variety of labels and further metadata. The presented experiments in this section serve two purposes: On the one hand, we provide first benchmarking results for future reference and, on the other hand, they illustrate the versatility of analyses that can be carried out based on the PTB-XL dataset. In Section III-A, we evaluate classifiers for different selections and granularities of ECG statements, which represents the core of analysis. It is complemented by Section III-B, where we validate our findings on the ICBEB2018 dataset and investigate aspects of transfer learning using PTB-XL for pretraining. Finally, we illustrate ways of leveraging further metadata within PTB-XL to construct age and gender prediction models, see Section III-C, and to build signal quality assessment models based on the provided signal quality annotations, see Section III-D.

PTB-XL에는 다양한 레이블과 추가 메타데이터가 제공됩니다. 이 섹션에 제시된 실험은 두 가지 목적을 가지고 있습니다: 한편으로는 향후 참조를 위한 첫 번째 벤치마킹 결과를 제공하고, 다른 한편으로는 PTB-XL 데이터 세트를 기반으로 수행할 수 있는 분석의 다양성을 보여줍니다. 섹션 III-A에서는 분석의 핵심인 심전도 문장의 다양한 선택과 세분화에 대한 분류기를 평가합니다. 섹션 III-B에서는 ICBEB2018 데이터 세트에 대한 결과를 검증하고 사전 학습을 위해 PTB-XL을 사용한 전이 학습의 측면을 조사합니다. 마지막으로, PTB-XL 내에서 추가 메타데이터를 활용하여 연령 및 성별 예측 모델을 구축하는 방법(섹션 III-C 참조)과 제공된 신호 품질 주석을 기반으로 신호 품질 평가 모델을 구축하는 방법(섹션 III-D 참조)을 설명합니다.

 

A. ECG statement prediction on PTB-XL

A. PTB-XL의 ECG 문 예측

We start by introducing, performing and evaluating all experiments that are directly related to ECG-statements, where we cover the three different major categories diagnostic diag., form and rhythm and level (sub-diag. and super-diag. as proposed in [15]) resulting in different number of labels per experiment and per sample as can seen in Table I. In the next step, we select only samples with at least one label in the given label selection. Our proposed evaluation as described in Section II-C is applied the same way for each experiment, where we report the term-centric macro-averaged AUC and the sample-centric Fmax-score.

먼저 심전도 상태와 직접적으로 관련된 모든 실험을 도입, 수행 및 평가하는데, 여기서는 진단 진단, 형태 및 리듬, 수준(하위 진단 및 슈퍼 진단, [15]에서 제안한 바와 같이)의 세 가지 주요 범주를 다루기 때문에 표 I에서 볼 수 있듯이 실험당 및 샘플당 라벨 수가 달라지게 됩니다. 다음 단계에서는 주어진 라벨 선택에서 적어도 하나의 라벨을 가진 샘플만 선택합니다. 섹션 II-C에서 설명한 제안된 평가는 각 실험에 대해 동일한 방식으로 적용되며, 용어 중심의 매크로 평균 AUC와 샘플 중심의 Fmax-score를 보고합니다.

 

In Table II, we report the results for all six experiments each applied to all models (as introduced in Section II-B), Figure 2 shows the result for all six experiments using barplots with associated bootstrap confidence intervals, see Appendix I for details. In all six experiments, deep-learning-based methods show a high predictive performance. Interestingly, even though all models are optimized based on binary cross-entropy loss rather than on the target metrics directly, the ranking according to both sample-based and term-based metrics largely coincides across all algorithms, which is why we focus on macro AUC in the following. The best-performing resnet or inceptionbased models reach macro AUCs ranging from 0.89 in the form category, over around 0.93 in the diagnostic categories to 0.96 in the rhythm category. These performance metrics can in principle used for a rudimentary assessment of the difficulty of the different prediction tasks. However, one has to keep in mind that for example the form prediction task has a considerably smaller training set compared to the other experiments due to approximately 12k ECGs without any form annotations

표 II에서는 모든 모델에 적용된 6개의 실험 결과를 각각 보고하고 있으며(섹션 II-B에서 소개한 대로), 그림 2는 부트스트랩 신뢰 구간이 연결된 막대그래프를 사용하여 6개의 실험 결과를 모두 보여줍니다(자세한 내용은 부록 I 참조). 6가지 실험 모두에서 딥러닝 기반 방법이 높은 예측 성능을 보였습니다. 흥미롭게도 모든 모델이 목표 지표가 아닌 이진 교차 엔트로피 손실을 기반으로 최적화되었음에도 불구하고, 샘플 기반 지표와 용어 기반 지표에 따른 순위는 모든 알고리즘에서 거의 일치하므로 다음에서는 매크로 AUC에 초점을 맞춥니다. 가장 성능이 좋은 리셋 또는 인셉션 기반 모델은 폼 카테고리에서 0.89, 진단 카테고리에서 약 0.93, 리듬 카테고리에서 0.96에 이르는 매크로 AUC에 도달합니다. 이러한 성능 지표는 원칙적으로 다양한 예측 작업의 난이도를 기초적으로 평가하는 데 사용할 수 있습니다. 그러나 예를 들어 형태 예측 작업의 경우 형태 주석이 없는 약 12,000개의 ECG로 인해 다른 실험에 비해 훈련 세트가 상당히 작다는 점을 염두에 두어야 합니다.

 

TABLE II: Overall discriminative performance of ECG classification algorithms on PTB-XL. For each experiment and each metric the best mean performing model is highlighted in bold font. For all experiments, 95% confidence intervals were calculating via bootstrapping on the test set, see Appendix I for notation and further details.

표 II: PTB-XL에서 ECG 분류 알고리즘의 전반적인 판별 성능. 각 실험과 각 메트릭에 대해 가장 우수한 평균 성능을 보인 모델은 굵은 글꼴로 강조 표시되어 있습니다. 모든 실험에서 95% 신뢰 구간은 테스트 세트에서 부트스트래핑을 통해 계산되었으며, 표기법 및 자세한 내용은 부록 I을 참조하십시오.

Fig. 2: Graphical summary of experiments described in Section III-A. For comparability, the algorithms are ranked according to prediction performance in each category.

그림 2: 섹션 III-A에 설명된 실험의 그래픽 요약. 비교 가능성을 위해 알고리즘은 각 카테고리의 예측 성능에 따라 순위를 매겼습니다.

 

As first general observation upon investigating the different model performances in more detail, we find that resnetarchitectures and inception-based architectures perform best across all experiments, but all convolutional architectures show a comparable performance level. In fact, the results of all convolutional models, up to very few exceptions, remain compatible within error bars. Recurrent architectures are consistently slightly less performant than their convolutional counterparts but, at least for diagnostic and rhythm statements, still competitive. The second general observation is that the performances of both convolutional as well as recurrent deep learning models turn out to be considerably stronger than the performance of the baseline algorithm operating on wavelet features. However, this statement has to be taken with caution, as the performance of feature-based classifiers is typically rather sensitive to details of feature selection choice of derived and details of the proprocessing procedure.

다양한 모델 성능을 자세히 조사했을 때 가장 먼저 일반적으로 관찰한 것은 모델 성능을 자세히 살펴본 결과, 모든 실험에서 리셋 아키텍처와 인셉션 기반 아키텍처의 성능이 가장 우수하지만 모든 실험에서 가장 우수한 성능을 보였지만, 모든 컨볼루션 아키텍처는 도 비슷한 수준의 성능을 보였습니다. 실제로 모든 컨볼루션 모델의 결과 는 극소수의 예외를 제외하고는 모든 컨볼루션 모델의 오차 범위 내에서 호환성을 유지합니다. 리커런트 아키텍처는 컨볼루션 아키텍처에 비해 지속적으로 약간 낮은 성능을 보이지만 하지만 적어도 진단 및 리듬 문에 대해서는 여전히 경쟁력이 있습니다. 두 번째 일반적인 관찰은 컨볼루션과 반복적 딥러닝 모두의 성능 학습 모델의 성능이 기준 알고리즘의 성능보다 훨씬 강력하다는 것입니다. 웨이블릿에서 작동하는 기준 알고리즘의 성능 기능에서 작동하는 기준 알고리즘의 성능보다 훨씬 강력하다는 것입니다. 그러나 이 결과는 신중하게 받아들여야 합니다, 특징 기반 분류기의 성능은 일반적으로 특징 선택의 세부 사항에 파생된 특징 선택의 세부 사항에 다소 민감합니다. 및 처리 절차의 세부 사항에 다소 민감하기 때문입니다.

 

In addition to single-model-performance, we also report the performance of an ensemble formed by averaging the predictions of all considered models (except the naive model). As can be seen in Table II, ensembling leads in many case to slight performance increases, but the best-performing single resnet or inception models always remain compatible with the ensemble result within error bars. The largest performance improvement of the ensemble model compared to single model performance is observed in the rhythm category, where the ensemble model outperforms all convolutional models except for xresnet1d101 and inception1d (as can be seen in Figure 2f). The ensemble results are only supposed to serve as rough orientation as the focus of this work is on single-model performance.

단일 모델 성능 외에도 고려된 모든 모델(나이브 모델 제외)의 예측을 평균하여 형성된 앙상블의 성능도 보고합니다. 표 II에서 볼 수 있듯이 앙상블을 사용하면 많은 경우 성능이 약간 향상되지만, 가장 성능이 좋은 단일 리셋 또는 초기 모델은 항상 앙상블 결과와 오차 범위 내에서 호환성을 유지합니다. 단일 모델 성능에 비해 앙상블 모델의 성능이 가장 크게 개선된 것은 리듬 범주에서 관찰되며, 앙상블 모델은 그림 2f에서 볼 수 있듯이 xresnet1d101 및 inception1d를 제외한 모든 컨볼루션 모델보다 성능이 뛰어납니다. 이 작업의 초점은 단일 모델 성능에 맞춰져 있기 때문에 앙상블 결과는 대략적인 방향성을 제시하는 역할만 합니다.

 

As a final remark, throughout this paper we use the recommended train-test splits provided by PTB-XL [15], which consider patient assignments and use input data at a sampling frequency of 100 Hz. Deviations from this setup are investigated in Appendix II.

마지막으로, 이 백서에서는 환자 할당을 고려하고 100Hz의 샘플링 주파수에서 입력 데이터를 사용하는 PTB-XL[15]에서 제공하는 권장 훈련-테스트 분할을 사용합니다. 이 설정의 편차는 부록 II에서 조사됩니다.

 

TABLE III: Classification performance on the ICBEB2018 dataset. In addition to sample-centric Fmax and term-centric macro-AUC, we also report the term-centric Fβ=2 and Gβ=2 to be used in the PhysioNet/CinC challenge 2020.

표 III: ICBEB2018 데이터 세트의 분류 성능. 샘플 중심 Fmax 및 용어 중심 매크로-AUC 외에도, PhysioNet/CinC 챌린지 2020에서 사용될 용어 중심 Fβ=2 및 Gβ=2도 보고합니다.

 

B. ECG statement prediction on ICBEB2018 and transfer learning

B. ICBEB2018 및 전이 학습을 통한 ECG 문장 예측

 

Beyond analyses on the PTB-XL dataset itself, we see further application of it as generic pretraining resource for ECG classification task, in a similar way as ImageNet [17] is commonly used for pretraining image classification algorithms. One freely accessible dataset from the literature that is large enough to reliably quantify the effects of transfer learning is the ICBEB2018 dataset, which is based on data released for the 1st China Physiological Signal Challenge 2018 held during the 7th International Conference on Biomedical Engineering and Biotechnology (ICBEB 2018) [39]. It comprises 6877 12- lead ECGs lasting between 6 s and 60 s. Each ECG record is annotated by up to three statements by up to three reviewers taken from a set of nine classes (one normal and eight abnormal classes, see Figure 3). We use the union of labels turning the dataset into a multi-label dataset. As the original test set is not available, we define 10 cross-validation folds by stratified sampling preserving the overall label distribution in each fold following [15].

PTB-XL 데이터 세트 자체에 대한 분석 외에도 이미지 분류 알고리즘의 사전 훈련에 ImageNet [17]이 일반적으로 사용되는 것과 유사한 방식으로 심전도 분류 작업을 위한 일반적인 사전 훈련 리소스로서 이 데이터 세트를 추가로 적용할 수 있습니다. 전이 학습의 효과를 안정적으로 정량화할 수 있을 만큼 충분히 큰 문헌에서 무료로 액세스할 수 있는 데이터 세트 중 하나는 제7회 국제 생의학 공학 및 생명공학 컨퍼런스(ICBEB 2018)[39]에서 개최된 제1회 중국 생리 신호 챌린지 2018에 공개된 데이터를 기반으로 한 ICBEB2018 데이터 세트입니다. 이 데이터는 6초에서 60초 사이에 지속되는 6877개의 12-리드 ECG로 구성되어 있으며, 각 ECG 기록에는 9개의 클래스(정상 클래스 1개, 비정상 클래스 8개, 그림 3 참조)에서 최대 3명의 검토자가 최대 3개의 진술로 주석을 달았습니다. 우리는 레이블의 조합을 사용하여 데이터 세트를 다중 레이블 데이터 세트로 전환합니다. 원본 테스트 세트를 사용할 수 없으므로 [15]에 따라 각 폴드의 전체 라벨 분포를 보존하는 계층화된 샘플링을 통해 10개의 교차 검증 폴드를 정의합니다.

 

Fig. 3: Summary of the ICBEB2018 dataset [39] in terms of ECG statements.

그림 3: ICBEB2018 데이터 세트 [39]의 요약은 다음과 같습니다. ECG 진술.

 

We start by analyzing the classification performance of classifiers trained on ICBEB2018 from scratch as an independent validation of the results obtained on PTB-XL. Table III shows the performance of classifiers that were trained using the the same experimental setup as in Section III-A. In all cases, we train a classifier from scratch by training on the first eight folds using the ninth and tenth fold as validation and test sets, respectively. Interestingly, the ICBEB2018 dataset was recently selected as training dataset for the PhysioNet/CinC challenge 2020 1 . For this reason we also report two further label-based performance metrics that will supposedly serve as evaluation metrics in the challenge, namely a macro-averaged Fβ-score (β = 2) and a macro-averaged Gβ-score with β = 2, where Gβ = T P/(T P + F P + β · F N), in both cases with sample weights chosen inversely proportional to the number of labels. Values of β > 1 allow to assign more weight to recall than precision, which might be a desirable property. However, applying this equally to the NORM-class seems questionable since high precision is required in this case. In addition, the corresponding scores are sensitive to the chosen classification threshold, which we determine by maximizing the Fβ/Gβscore on the training set, which is an undesirable aspect as it entangles the discriminitive performance of the classification algorithm with the process of threshold determination. Nevertheless, both Fβ and Gβ show a quantitative similarity in terms of ranking between our threshold-free metrics. Comparing to the quantitative classification performance on PTB-XL as presented in Section III-A, we see a largely consistent picture on ICBEB2018 in the sense of a rather uniform performance level among the convolutional architectures, all of which remain consistent within error bars, a slightly weaker performance of the recurrent architectures and a considerable performance gap to the feature-based baseline classifier.

먼저 PTB-XL에서 얻은 결과를 독립적으로 검증하기 위해 ICBEB2018에서 훈련된 분류기의 분류 성능을 처음부터 다시 분석합니다. 표 III은 섹션 III-A에서와 동일한 실험 설정을 사용하여 훈련된 분류기의 성능을 보여줍니다. 모든 경우에서 9번째와 10번째 폴드를 각각 검증 및 테스트 세트로 사용하여 처음 8개의 폴드에 대해 훈련함으로써 분류기를 처음부터 훈련합니다. 흥미롭게도, ICBEB2018 데이터 세트는 최근 PhysioNet/CinC 챌린지 2020의 훈련 데이터 세트로 선정되었습니다1. 이러한 이유로 챌린지에서 평가 지표로 사용될 것으로 예상되는 두 가지 추가 라벨 기반 성능 지표, 즉 매크로 평균 Fβ 점수(β = 2)와 매크로 평균 Gβ 점수(β = 2, Gβ = T P/(T P + F P + β - F N), 두 경우 모두 샘플 가중치가 라벨 수에 반비례하여 선택됨)도 보고하고 있습니다. β 값이 1보다 크면 정확도보다 리콜에 더 많은 가중치를 부여할 수 있으며, 이는 바람직한 특성일 수 있습니다. 그러나 이 경우 높은 정밀도가 필요하기 때문에 이를 NORM 클래스에 동일하게 적용하는 것은 의문입니다. 또한 해당 점수는 선택한 분류 임계값에 민감하며, 이는 훈련 세트에서 Fβ/Gβs 점수를 최대화하여 결정하는데, 이는 분류 알고리즘의 판별 성능과 임계값 결정 과정이 얽혀 있기 때문에 바람직하지 않은 측면이 있습니다. 그럼에도 불구하고 Fβ와 Gβ는 임계값이 없는 메트릭 간의 순위 측면에서 정량적 유사성을 보여줍니다. 섹션 III-A에 제시된 PTB-XL의 정량적 분류 성능과 비교하면, ICBEB2018에서는 컨볼루션 아키텍처 간에 다소 균일한 성능 수준이 오차 범위 내에서 일관되게 유지되고, 리커런트 아키텍처의 성능이 약간 약하며, 특징 기반 기준 분류기와의 성능 격차가 상당히 크다는 점에서 대체로 일관된 모습을 볼 수 있습니다.

 

In the next experiment, we leverage PTB-XL by finetuning a classifer trained on PTB-XL on ICBEB2018 data. To this end, we take a classifier trained on PTB (using all ECG statements) and replace the top layer of the fully connected classification head to account for the different number ECG statements in ICBEB2018. This classifier is then finetuned on ICBEB2018 data. To systematically investigate the transition into the small dataset regime, we do not only present results for finetuning on the full dataset (8 training folds) but for the full range of one eighth to eight training folds i.e. from 85 to 5500 training samples. For each training size and fixed model architecture (xresnet1d101), we compare models trained from scratch to models that pretrained on PTB-XL and then finetuned on ICBEB2018. Figure 4 summarizes the results of this experiment, and illustrates the fact that for large dataset sizes pretraining on PTB-XL does not improve the performance compared to training from scratch but even potentially slightly deteriorates it, even though the two results remain compatible within error bars. However, for smaller dataset sizes of a single training fold or fractions of it, we see a clear advantage from pretraining. Most notably, the performance of the finetuned model remains much more stable upon decreasing the size of the training set and consequently outperforms the model trained from scratch by a large margin in the the case of small training sizes. In the most extreme case of one eighth of the original training fold corresponds to 85 samples, where the performance of the finetuned classifier only drops by about 10% in terms of AUC compared to a classifier trained on a training set that is 64 times larger. Since the small dataset regime is the most natural application domain for pretraining on a generic ECG dataset, we see this as a very encouraging sign for future applications of PTB-XL as a pretraining resource for relatively small datasets.

다음 실험에서는 PTB-XL로 훈련된 분류기를 ICBEB2018 데이터에 대해 미세 조정하여 PTB-XL을 활용합니다. 이를 위해 PTB에서 훈련된 분류기(모든 ECG 문장을 사용)를 가져와 완전히 연결된 분류 헤드의 최상위 레이어를 ICBEB2018에서 다른 수의 ECG 문장을 고려하도록 교체합니다. 그런 다음 이 분류기는 ICBEB2018 데이터에 대해 미세 조정됩니다. 소규모 데이터 세트 체제로의 전환을 체계적으로 조사하기 위해 전체 데이터 세트(8개 훈련 폴드)에 대한 미세 조정 결과뿐만 아니라 8분의 1에서 8개의 훈련 폴드, 즉 85개에서 5500개의 훈련 샘플 전체 범위에 대한 결과를 제시합니다. 각 훈련 크기와 고정 모델 아키텍처(xresnet1d101)에 대해 처음부터 훈련한 모델과 PTB-XL에서 사전 훈련한 후 ICBEB2018에서 미세 조정한 모델을 비교합니다. 그림 4는 이 실험의 결과를 요약한 것으로, 데이터 세트 크기가 큰 경우 PTB-XL로 사전 학습하는 것이 처음부터 학습하는 것보다 성능이 향상되지 않고 심지어 두 결과가 오차 범위 내에서 호환성을 유지하더라도 성능이 약간 저하될 수 있다는 사실을 보여줍니다. 그러나 단일 훈련 폴드 또는 그보다 작은 데이터 세트 크기의 경우 사전 훈련이 분명한 이점을 제공합니다. 특히, 훈련 세트의 크기를 줄여도 미세 조정된 모델의 성능이 훨씬 더 안정적으로 유지되며, 결과적으로 훈련 크기가 작은 경우 처음부터 훈련한 모델보다 큰 폭으로 성능이 향상됩니다. 가장 극단적인 경우, 원래 훈련 배의 8분의 1은 85개의 샘플에 해당하며, 이 경우 미세 조정된 분류기의 성능은 64배 더 큰 훈련 세트에서 훈련된 분류기에 비해 AUC 측면에서 약 10% 정도만 떨어집니다. 작은 데이터 세트 체제는 일반적인 ECG 데이터 세트에 대한 사전 학습에 가장 자연스러운 적용 영역이므로, 향후 상대적으로 작은 데이터 세트에 대한 사전 학습 리소스로서 PTB-XL을 적용하는 데 매우 고무적인 신호로 보입니다.

 

 

Fig. 4: Effect of transfer learning from PTB-XL to ICBEB2018 upon varying the size of the ICBEB2018 training set.

그림 4: PTB-XL에서 ICBEB2018로의 전이 학습 효과 ICBEB2018로의 전이 학습 효과 세트.

 

TABLE IV: Age regression performance for models trained on all patients and evaluated on all/healthy/non-healthy subpopulation in terms of mean absolute error (MAE) and R-squared (R2).

표 IV: 모든 환자를 대상으로 학습하고 전체/건강한/건강하지 않은 하위 집단에 대해 평가한 모델의 연령 회귀 성능(평균 절대 오차(MAE) 및 R-제곱(R2) 기준).

C. Age regression and gender classification

C. 연령 회귀 및 성별 분류

The following experiment is inspired by the recent work from [11] that demonstrated that deep neural networks are capable of accurately inferring age and gender from standard 12-lead ECGs. Here, we look into both tasks again based on PTB-XL. The experiment is supposed to illustrate the possibility of leveraging demographic metadata in the PTBXL dataset. We applied the same model architectures from Section III-A but with adjusted final layers, where for gender prediction a binary and for age prediction a linear output neuron was trained and optimized such that the binary crossentropy or mean squared error is minimized respectively. Both networks were trained separately but with the same train-test-splits and identical hyperparameters as in previous experiments, except that for final output prediction where we computed the mean of all windows instead of the maximum (as used above). In order to study the effect of pathologies on performance for this task, in addition to all subjects we also evaluated the models only for healthy subjects and for nonhealthy subjects. Here, we define the set of healthy records as the set of records with NORM as the only diagnostic label and non-healthy as its complement.

다음 실험은 심층 신경망이 표준 12-리드 심전도로부터 나이와 성별을 정확하게 추론할 수 있음을 입증한 [11]의 최근 연구에서 영감을 얻었습니다. 여기서는 PTB-XL을 기반으로 두 가지 작업을 다시 살펴봅니다. 이 실험은 PTBXL 데이터 세트에서 인구통계학적 메타데이터를 활용할 수 있는 가능성을 보여주기 위한 것입니다. 섹션 III-A의 동일한 모델 아키텍처를 적용했지만 최종 레이어를 조정하여 성별 예측의 경우 이진, 연령 예측의 경우 선형 출력 뉴런을 각각 이진 교차 엔트로피 또는 평균 제곱 오차가 최소화되도록 훈련하고 최적화했습니다. 두 네트워크 모두 개별적으로 훈련되었지만, 최종 출력 예측의 경우 위에서 사용한 것처럼 최대값 대신 모든 창의 평균을 계산한 것을 제외하고는 이전 실험과 동일한 훈련-테스트-분할 및 동일한 하이퍼파라미터를 사용하여 훈련했습니다. 이 과제에서 병리가 성능에 미치는 영향을 연구하기 위해 모든 피험자 외에도 건강한 피험자와 건강하지 않은 피험자에 대해서만 모델을 평가했습니다. 여기서는 건강한 기록 집합을 유일한 진단 레이블로 NORM을 사용하고 그 보완으로 비건강을 사용하는 기록 집합으로 정의합니다.

 

TABLE V: Gender prediction performance for models trained on all patients and evaluated on all/healthy/non-healthy subpopulations in terms of accuracy (acc) and area under the receiver operating curve (AUC).

표 V: 모든 환자를 대상으로 학습하고 모든/건강한/비건강한 하위 집단에 대해 평가한 모델의 성별 예측 성능(정확도(acc) 및 수신기 작동 곡선 아래 면적(AUC) 기준).

The results for the age regression experiment are shown in Table IV. Overall, testing only on healthy subjects yielded better results in each category as compared to testing only on nonhealthy or all subjects (MAE=6.86 compared to MAE=7.38 and MAE=7.16 respectively). These observations are in line with [11], [40]. Furthermore, these results are competitive to [11], who reported a value of MAE=6.9 years (R-squared = 0.7) but with thirty times more data (≈20k versus ≈750k samples [11]). Table V shows the corresponding results for gender prediction. As already suggested in [41], [42] the differences between male and female are also present in ECG, which is also confirmed by our model yielding a accuracy of 84.9%(89.8%) and an AUC of 0.92(0.96) on all(healthy) patients. This performance level, in particular on the healthy subpopulation, is competitive with results from the literature [11] (90.4% accuracy and an AUC of 0.97). As a final word of caution, we want to stress that the results for age and gender prediction algorithms are not directly comparable across different datasets due to different dataset distributions not only in terms of the labels themselves but also in terms of co-occurring diseases. This is apparent from the performance differences of our classifier for both subtasks when evaluated on the full dataset and on the two different subpopulations.

연령 회귀 실험의 결과는 표 IV에 나와 있습니다. 전반적으로 건강한 피험자만을 대상으로 테스트했을 때 건강하지 않은 피험자 또는 모든 피험자를 대상으로 테스트했을 때보다 각 범주에서 더 나은 결과를 얻었습니다(각각 MAE=7.38 및 MAE=7.16에 비해 MAE=6.86). 이러한 관찰 결과는 [11], [40]과 일치합니다. 또한 이러한 결과는 MAE=6.9년(R-제곱 = 0.7)의 값을 보고했지만 30배 더 많은 데이터(≈20만 개 대 ≈75만 개 샘플 [11])를 사용한 [11]과 경쟁할 수 있습니다. 표 V는 성별 예측에 대한 해당 결과를 보여줍니다. 이미 [41], [42]에서 제안한 바와 같이, 심전도에서도 남성과 여성의 차이가 존재하며, 이는 모든(건강한) 환자에 대해 84.9%(89.8%)의 정확도와 0.92(0.96)의 AUC를 산출하는 우리 모델에서도 확인할 수 있습니다. 특히 건강한 하위집단에 대한 이러한 성능 수준은 문헌[11]의 결과(정확도 90.4%, AUC 0.97)와 비교할 때 경쟁력이 있습니다. 마지막으로, 연령 및 성별 예측 알고리즘의 결과는 레이블 자체뿐만 아니라 함께 발생하는 질병의 측면에서 데이터 세트 분포가 다르기 때문에 서로 다른 데이터 세트 간에 직접 비교할 수 없다는 점을 강조하고 싶습니다. 이는 전체 데이터 세트와 두 개의 서로 다른 하위 집단에 대해 평가했을 때 두 하위 작업에 대한 분류기의 성능 차이를 보면 알 수 있습니다.

 

D. Signal quality assessment

D. 신호 품질 평가

 As part of a technical validation of the database each sample underwent a second iteration by a technical expert to annotate the data with respect to signal artifacts, see [15] for a detailed description. The annotations were given without any regular syntax, for this reason the annotations were coded as a binary targets, where targets are set to one if any annotation is given for NOISE (either globally present static noise (static noise) or local bursts of high voltage induced by external sources (burst noise)), DRIFT (baseline wandering). In total this binary target is set for ≈ 22% (i.e. ≈ 78% of signals contain no artifacts). Using these annotations coded as binary targets might help to develop a signal quality classifier for creating validation data to test for robustness with respect to artifacts. For this purpose we conducted experiments along the lines of Section III-A i.e. again using the same models, hyperparameters and train-test-splits as above. Overall, our models reach AUC scores around 0.81, which seems to indicate a slightly weaker predictive performance compared to ECG statement prediction models discussed in Section III-A, even though performance measures for different tasks are obviously not directly comparable. According to a first analysis, a significant portion of this performance deficiency can be attributed label noise (i.e. missing annotation in case of artifacts or misleading annotation in case of normal signals). However, a more thorough analysis should attempt to incorporate the full report strings instead of just binary labels. In any case, models trained on this task can still be used as a prescreening procedure for ECG quality assessment.

데이터베이스의 기술적 검증의 일환으로 각 샘플은 기술 전문가가 신호 아티팩트와 관련하여 데이터에 주석을 달기 위해 두 번째 반복 작업을 거쳤습니다(자세한 설명은 [15]를 참조하십시오). 주석은 규칙적인 구문 없이 주어졌기 때문에 주석은 이진 타깃으로 코딩되었으며, NOISE(전역적으로 존재하는 정적 잡음(정적 잡음) 또는 외부 소스에 의해 유도된 고전압의 국부적 버스트(버스트 잡음)), DRIFT(기준선 방황)에 대한 주석이 주어지면 타깃이 1로 설정됩니다. 이 바이너리 목표는 총 ≈ 22%(즉, 신호의 ≈ 78%에 아티팩트가 없음)로 설정됩니다. 이진 타깃으로 코딩된 이러한 주석을 사용하면 아티팩트에 대한 견고성을 테스트하기 위한 검증 데이터를 생성하기 위한 신호 품질 분류기를 개발하는 데 도움이 될 수 있습니다. 이를 위해 위와 동일한 모델, 하이퍼파라미터, 훈련-테스트 분할을 사용하여 섹션 III-A에 따라 다시 실험을 수행했습니다. 전반적으로, 우리 모델은 약 0.81의 AUC 점수에 도달했는데, 이는 서로 다른 작업에 대한 성능 측정값을 직접 비교할 수는 없지만 섹션 III-A에서 설명한 ECG 문장 예측 모델에 비해 예측 성능이 약간 약한 것으로 보입니다. 1차 분석에 따르면 이러한 성능 부족의 상당 부분은 라벨 노이즈(아티팩트가 있는 경우 주석이 누락되거나 정상 신호인 경우 잘못된 주석)에 기인할 수 있습니다. 그러나 보다 철저한 분석을 위해서는 이진 레이블이 아닌 전체 보고서 문자열을 통합해야 합니다. 어쨌든 이 작업에 대해 훈련된 모델은 ECG 품질 평가를 위한 사전 선별 절차로 여전히 사용될 수 있습니다.

 

Fig. 5: Hierarchical decomposition of class-specific AUCs onto subclasses and individual diagnostic statements exhibiting hidden stratification, i.e. inferior algorithmic performance on certain diagnostic subpopulations that remains hidden when considering only the superior superclass performance, see the description in Section IV-B for details. AUC is given in square brackets and the number of label occurrences in the test set in parentheses. The transparency of each colored node is relative to the minimum and maximum AUC in the last layer.

그림 5: 클래스별 AUC의 계층적 분해 하위 클래스 및 개별 진단 문에 대한 계층적 분해 숨겨진 계층화, 즉 특정 진단 하위 집단에 대한 알고리즘 성능 저하 숨겨져 있는 특정 진단 하위집단, 즉 우수한 슈퍼클래스 성능만 고려할 때 숨겨져 있는 특정 진단 하위집단에 대한 열등한 알고리즘 성능, 자세한 내용은 섹션 IV-B의 설명을 참조하십시오. AUC는 대괄호 괄호 안에 표시되며, 테스트 세트의 레이블 발생 횟수는 괄호 안에 표시됩니다. 각 색상 노드의 투명도는 상대적입니다. 마지막 레이어의 최소 및 최대 AUC에 상대적입니다.

 

IV. DEEPER INSIGHTS FROM CLASSIFICATION MODELS

IV. 분류 모델에서 얻은 더 깊은 인사이트

 

Until now we investigated our experiments quantitatively in order to compare different model architectures. However, a quantitatively evaluation focusing on overall predictive performance, as presented in the previous section, might not take important qualitative aspects into account, such as the predictive performance for single, potentially sparsely populated ECG statements. Here, we focus our analysis on a single xresnet1d101 model, but we verified that the results presented below are largely consistent across different model architectures.

지금까지는 다양한 모델 아키텍처를 비교하기 위해 실험을 정량적으로 조사했습니다. 그러나 이전 섹션에서 제시된 것처럼 전반적인 예측 성능에 초점을 맞춘 정량적 평가는 잠재적으로 드물게 채워질 수 있는 단일 ECG 문에 대한 예측 성능과 같은 중요한 정성적 측면을 고려하지 않을 수 있습니다. 여기서는 단일 xresnet1d101 모델에 초점을 맞춰 분석했지만, 아래에 제시된 결과는 여러 모델 아키텍처에 걸쳐 대체로 일관성이 있음을 확인했습니다.

Fig. 6: AUC curves for two subset of samples revealing hidden stratification within the IVCD class.

그림 6: 숨겨진 계층화를 보여주는 두 개의 샘플 하위 집합에 대한 AUC 곡선 숨겨진 계층화를 보여주는 두 개의 샘플 하위 집합

 

A. Hierarchical organization of diagnostic labels

A. 진단 라벨의 계층적 구성

As first analysis, we cover the hierarchical organization of diagnostic labels and its impact on predictive performance. The PTB-XL dataset provides proposed assignements to one of five superclasses and one of 23 subclasses for each diagnostic ECG statement, which represents one possible ontology that can be used to organize ECG statements. In Figure 5, we show the hierarchical decomposition for the diagnostic labels in sub- and superclasses, where we propagated predictions from experiment diag. upwards the hierarchy over sub-diag. to super-diag. by summing up prediction probabilities of the corresponding child nodes and limiting the output probabilities to one. We experimented with other aggregation strategies such as using the maximum or the mean of the predictions of the child nodes but observed only minor impact on the results. The same holds for models trained on the specific level, where no propagation is needed. The training of hierarchical classifiers is a topic with a rich history in the machine learning literature, see for example [43] for a dedicated review and [44] for a recent deep learning approach to the topic. Extensive experiments on this topic are beyond the scope of this manuscript, but our first experiments on this topic indicate that the performance of a model trained on a coarser granularity is largely compatible or in some cases even slightly inferior to a model trained on the finest label granularity and propagating prediction scores upwards the label hierarchy

첫 번째 분석으로 진단 레이블의 계층적 구성과 예측 성능에 미치는 영향을 다룹니다. PTB-XL 데이터 세트는 각 진단 ECG 문에 대해 5개의 수퍼클래스 중 하나와 23개의 서브클래스 중 하나에 제안된 할당을 제공하며, 이는 ECG 문 구성에 사용할 수 있는 하나의 가능한 온톨로지를 나타냅니다. 그림 5에서는 하위 클래스 및 슈퍼클래스의 진단 레이블에 대한 계층적 분해를 보여 주며, 실험 진단에서 하위 진단을 거쳐 슈퍼 진단으로 계층을 상향하여 해당 하위 노드의 예측 확률을 합산하고 출력 확률을 1로 제한하는 방식으로 예측을 전파했습니다. 자식 노드 예측의 최대값 또는 평균을 사용하는 등 다른 집계 전략도 실험해 보았지만 결과에 미치는 영향은 미미했습니다. 전파가 필요하지 않은 특정 수준에서 훈련된 모델도 마찬가지입니다. 계층적 분류기의 훈련은 머신 러닝 문헌에서 풍부한 역사를 가진 주제이며, 관련 리뷰는 [43], 최근의 딥 러닝 접근 방식은 [44]를 참조하세요. 이 주제에 대한 광범위한 실험은 이 원고의 범위를 벗어나지만, 이 주제에 대한 첫 번째 실험에 따르면 더 거친 세분성으로 훈련된 모델의 성능이 가장 세밀한 라벨 세분성으로 훈련되고 예측 점수를 라벨 계층 위로 전파하는 모델과 대체로 호환되거나 경우에 따라 약간 열등한 것으로 나타났습니다.

 

B. Hidden stratification and co-occurring pathologies

B. 숨겨진 층화 및 동시 발생 병리

The hierarchical organization of the diagnostic labels allows for deeper insights and potential pitfalls of model evaluation that are crucial for clinical applications. In particular, we focus on the issue of hidden stratification that was put forward in [45] and describes potential inferior algorithmic performance on certain diagnostic subpopulations that remains hidden from the outside if only the superclass performance is reported. We analyze this effect in a top-down fashion using the results obtained by propagating the finest granularity scores upwards the label hierarchy as described above. In Figure 5, we illustrate how the label AUC of a particular superclass or subclass decomposes into the label AUCs of the corresponding subclasses. One reason for weak classifier performance are ECG statements classes that are too scarcely populated to allow training a discriminative classifier on them and for which also the score estimate on the test set is unreliable due to the small sample size. However, there are further ECG statements that stand out from other members of the same subclass, where the performance deficiency cannot only be attributed to effects of small sample sizes. For example, consider the classes NST (non-specific ST changes), LAO/LAE (left atrial overload/enlargement) and IVCD (non-specific intraventricular conduction disturbance (block)) in the bottom layer of the hierarchy, where the classifier shows a weak performance, which is in fact hidden when reporting only the corresponding superclass or subclass performance measures. At least for NST and IVCD, these findings can be explained by the fact that both statements are by definition non-specific ECG statements and potentially subsum rather heterogenous groups of findings. 

진단 레이블의 계층적 구성을 통해 임상 적용에 중요한 모델 평가의 잠재적 함정과 심층적인 통찰력을 얻을 수 있습니다. 특히, 우리는 [45]에서 제시된 숨겨진 계층화 문제에 초점을 맞추고, 슈퍼클래스 성능만 보고되는 경우 외부에서 숨겨져 있는 특정 진단 하위집단에 대한 잠재적인 알고리즘 성능 저하를 설명합니다. 위에서 설명한 대로 가장 세분화된 점수를 라벨 계층 구조 위로 전파하여 얻은 결과를 사용하여 하향식 방식으로 이 효과를 분석합니다. 그림 5에서는 특정 슈퍼클래스 또는 하위클래스의 레이블 AUC가 해당 하위클래스의 레이블 AUC로 어떻게 분해되는지 보여줍니다. 분류기 성능이 약한 이유 중 하나는 표본 수가 너무 적어 판별 분류기를 훈련할 수 없고 표본 크기가 작아 테스트 세트의 점수 추정치도 신뢰할 수 없는 ECG 문장 클래스입니다. 그러나 동일한 하위 클래스의 다른 구성원들과 두드러지는 ECG 진술이 더 있으며, 이러한 성능 부족은 작은 표본 크기의 영향 때문이라고만 볼 수 없습니다. 예를 들어, 계층 구조의 최하위 계층에 있는 NST(비특이적 ST 변화), LAO/LAE(좌심방 과부하/확대) 및 IVCD(비특이적 심실 내 전도 장애(차단)) 클래스를 생각해 보면, 분류기가 약한 성능을 보이는데, 이는 실제로 해당 슈퍼클래스 또는 하위 클래스 성능 측정값만 보고할 때 숨겨지는 것입니다. 적어도 NST 및 IVCD의 경우, 이러한 결과는 두 진술이 정의상 비특이적인 ECG 진술이며 잠재적으로 다소 이질적인 결과 그룹을 포함할 수 있다는 사실로 설명할 수 있습니다.

 

Although identifying hidden stratification is straightforward to identify in hindsight given the hierarchical organization of the diagnostic labels, [45] also demonstrated how to identify groups of samples exhibiting hidden stratification for a given class label under consideration using an unsupervised clustering approach. For demonstration, we carried out such a comparable analysis for IVCD in order to understand the comparably weak classification performance on the particular statement compared to other conduction disturbances. Indeed, clustering the model’s output probabilities revealed two clusters, where one subset performed much better than the other as can be seen in Figure 6. Interestingly, it turned out that the two clusters largely align with the presence/absence of NORM as additional ECG statement. The blue line (all) represents the performance as is (AUC 0.74), the green line is the performance for samples out of one cluster (AUC 0.59, for which most of the sample were also associated with NORM), the orange line for the second cluster (AUC 0.91, predominantly samples without NORM). As can be seen clearly, samples with IVCD in combination with NORM are much harder to classify

숨겨진 계층화를 식별하는 것은 간단하지만 진단 레이블의 계층적 조직을 고려할 때 뒤늦게 식별하는 것은 간단합니다. 진단 레이블, [45] 또한 식별하는 방법을 보여주었습니다. 숨겨진 계층화를 나타내는 샘플 그룹을 식별하는 방법도 보여주었습니다. 비지도 클러스터링 접근법을 사용하여 고려 중인 클래스 라벨에 대해 숨겨진 계층화를 나타내는 샘플 그룹을 식별하는 방법을 시연했습니다. 데모를 위해, 저희는 특정 문장에 대한 분류 성능이 상대적으로 특정 문장에 대한 약한 분류 성능 다른 전도 장애에 비해 상대적으로 약한 분류 성능을 이해하기 위해 이러한 분석을 수행했습니다. 실제로 클러스터링 모델의 출력 확률에서 두 개의 클러스터가 나타났습니다. 그림 6에서 볼 수 있듯이 한 하위 집합이 다른 하위 집합보다 훨씬 우수한 성능을 보였습니다. 에서 볼 수 있습니다. 흥미롭게도 이 두 클러스터는 추가적으로 NORM의 유무에 따라 크게 일치하는 것으로 나타났습니다. ECG 문과 거의 일치하는 것으로 나타났습니다. 파란색 선(전체)은 성능을 나타냅니다. 성능을 나타내며(AUC 0.74), 녹색 선은 한 클러스터의 샘플 중 한 클러스터의 샘플(AUC 0.59, 샘플의 대부분이 샘플의 대부분이 NORM과 연관됨), 주황색 선은 두 번째 클러스터의 성능입니다. 선은 두 번째 클러스터에 대한 성능(AUC 0.91, 주로 NORM이 없는 샘플 없음). 명확하게 알 수 있듯이, IVCD가 있는 샘플은 가 있는 샘플은 분류하기가 훨씬 더 어렵습니다.

 

These kinds of investigations are very important for the identification of hidden stratification in the model which are induced by data and their respective labels [45]. Models trained on coarse labels might hide this kind of clinically relevant stratification, because of both subtle discriminative features and low prevalence. Further studies might investigate hidden stratification below our deepest level of labels. At this point, it remains to stress that the PTB-XL dataset does not provide any clinical ground truth on the considered samples but only provides cardiologists’ annotations based on the ECG signal itself, which could compromise the analysis. However, we still see an in-depth study towards the identification subgroups with certain combinations of co-occurring ECG statements/pathologies, along the lines of the example of IVCD presented above, as a promising direction for future research in the sense that it can potentially provide pointers for future clinical investigations.

이러한 종류의 조사는 데이터와 해당 레이블에 의해 유도되는 모델에서 숨겨진 계층화를 식별하는 데 매우 중요합니다[45]. 거친 레이블로 훈련된 모델은 미묘한 변별 특징과 낮은 유병률로 인해 임상적으로 관련된 이러한 종류의 계층화를 숨길 수 있습니다. 추가 연구에서는 가장 깊은 수준의 레이블 아래에 숨겨진 계층화를 조사할 수 있습니다. 현재로서는 PTB-XL 데이터 세트가 고려된 샘플에 대한 임상적 근거 자료를 제공하지 않고 ECG 신호 자체에 기반한 심장 전문의의 주석만 제공하므로 분석에 영향을 미칠 수 있다는 점을 강조하고 싶습니다. 그러나 위에 제시된 IVCD의 예와 같이 특정 조합의 동시 발생 심전도 진술/병리를 가진 식별 하위 그룹에 대한 심층적인 연구는 향후 임상 조사에 대한 지침을 제공할 수 있다는 점에서 향후 연구의 유망한 방향이라고 생각합니다.

 

Fig. 7: Relation between model uncertainty (standard deviation of ensemble predictions as in [46]) and diagnosis likelihood as quantified by the annotating cardiologist, see Section IV-C for details.

그림 7: 모델 불확실성(앙상블 예측의 표준편차, [46]에서와 같이)과 진단 사이의 관계 주석 달기 심장 전문의가 정량화한 가능성, 자세한 내용은 섹션 IV-C를 참조하십시오.

 

subgroups with certain combinations of co-occurring ECG statements/pathologies, along the lines of the example of IVCD presented above, as a promising direction for future research in the sense that it can potentially provide pointers for future clinical investigations.

함께 발생하는 특정 조합의 ECG 진술/병리의 특정 조합을 가진 하위 그룹, 위에 제시된 IVCD의 예에 따라 의 예에 따라, 향후 연구의 유망한 방향으로서 잠재적으로 향후 임상 조사에 대한 임상 조사.

 

C. Model uncertainty and diagnosis likelihoods

C. 모델 불확실성 및 진단 가능성

 Besides this hierarchical organization of diagnostic labels, PTB-XL comes along with associated likelihoods for each diagnostic label ranging from 15 to 100, where 15 indicates less and 100 strong confidence for one label. These likelihoods were extracted from the original ECG report string for all diagnostic statements based on certain keywords [15]. As an initial experiment to assess the quality of this likelihood information, we compare the likelihoods to model uncertainty estimates for a model trained on diagnostic statements. To quantify the model uncertainty, we follow the simple yet very powerful approach put forward in [46] that defines model uncertainty via the variance of an ensemble of identical models for different random initializations. Here, we use an ensemble of 10 models and for simplicity even omit the optional stabilizing adversarial training step, which was reported to lead to slightly improved uncertainty estimates [46], in this first exploratory analysis. In Figure 7, we plot model uncertainty versus diagnosis likelihood and observe the expected monotonic behavior. Only the likelihood 100 stands out from this trend and shows a large number of outliers. One possible explanation for this observation is an overconfidence of human annotators when it comes to seemingly very obvious statements that goes in with the human inability to precisely quantify uncertainties, which is a well-known phenomenon in cognitive psychology, see e.g. [47]. However, we perceive the overall alignment of diagnosis likelihood with model uncertainty as an interesting observation as it correlates perceived human uncertainty with algorithmic uncertainty, a statement that is normally impossible for clinical datasets due to the unavailability of appropriate labels.

이러한 진단 레이블의 계층적 구성 외에도 PTB-XL은 각 진단 레이블에 대해 15에서 100까지의 관련 가능성을 제공하며, 여기서 15는 한 레이블에 대한 신뢰도가 낮음을 나타내고 100은 강한 신뢰도를 나타냅니다. 이러한 가능성은 특정 키워드를 기반으로 한 모든 진단 문장에 대한 원본 ECG 보고서 문자열에서 추출되었습니다[15]. 이 가능성 정보의 품질을 평가하기 위한 초기 실험으로, 진단 문장에 대해 학습된 모델에 대한 모델 불확실성 추정치와 가능성을 비교합니다. 모델 불확실성을 정량화하기 위해, 우리는 [46]에서 제시된 간단하지만 매우 강력한 접근 방식을 따르는데, 이는 서로 다른 무작위 초기화에 대한 동일한 모델 앙상블의 분산을 통해 모델 불확실성을 정의합니다. 여기서는 10개의 모델로 구성된 앙상블을 사용하며, 이 첫 번째 탐색적 분석에서 불확실성 추정치를 약간 개선하는 것으로 보고된 [46]의 선택적 안정화 적대 훈련 단계도 생략하여 단순화했습니다. 그림 7에서는 모델 불확실성과 진단 가능성을 플롯하고 예상되는 단조로운 동작을 관찰합니다. 확률 100만이 이 추세에서 눈에 띄며 많은 수의 이상값을 보여줍니다. 이러한 관찰에 대한 한 가지 가능한 설명은 인지 심리학에서 잘 알려진 현상인 불확실성을 정확하게 정량화하지 못하는 인간의 무능력과 함께 매우 명백해 보이는 진술에 대한 인간 주석자의 과신입니다(예: [47] 참조). 그러나 우리는 진단 가능성과 모델 불확실성이 전반적으로 일치하는 것을 인지된 인간의 불확실성과 알고리즘 불확실성의 상관관계, 즉 적절한 라벨을 사용할 수 없기 때문에 일반적으로 임상 데이터 세트에서는 불가능한 진술과 연관시키는 흥미로운 관찰로 인식합니다.

 

Fig. 8: Two exemplary attribution maps for a resnet model for the classes PVC (left) and PACE (right).

그림 8: PVC(왼쪽) 및 PACE(오른쪽) 클래스에 대한 리셋 모델에 대한 두 가지 예시 어트리뷰션 맵.

 

D. Prospects of interpretability methods

The acceptance of machine learning and in particular deep learning algorithms in the clinical context is often limited by the fact that data-driven algorithms are perceived as black boxes by doctors. In this direction, the recent advances in the field of explainable AI has the prospect to at least partially alleviate this issue. In particular, we consider post-hoc interpretability that can be applied for a trained model, see e.g. [48]. The general applicability of interpretability methods to multivariate timeseries and in particular ECG data was demonstrated in [49], see also [50], [51] for futher accounts on interpretability methods for ECG data. Here, we focus on exemplary for the form statement “premature ventricular complex” (PVC) and the rhythm statement PACE indicating an active pacemaker. The main reason for choosing these particular classes is the easy verifiable also for non-cardiologists. In Figure 8, we show two exemplary but representative attribution maps obtained via the -rule with  = 0.1 within the framework of layer-wise relevance propagation [52]. For PVC the relevance is located at the extra systole across all leads. For PACE, the relevance is scattered across the whole signal aligning nicely with the characteristic pacemaker spikes (just before each QRS complex) in each beat. It is a non-trivial finding that the relevance patterns for the two ECG statements from above align with medical knowledge. A more extensive, statistical analysis of the attribution maps both within patients across different beats and across different ECGs with common pathologies is a promising direction for future work.

 

V. SUMMARY AND CONCLUSIONS

V. 요약 및 결론

Electrocardiography is among the most common diagnostic procedures carried out in hospitals and doctor’s offices. We envision a lot potential for automatic ECG interpretation algorithms in different medical application domains, but we see the current progress in the field hampered by the lack of appropriate benchmarking datasets and well-defined evaluation procedures. We propose a variety of benchmarking tasks based on the PTB-XL dataset [15] and put forward first baseline results for deep-learning-based time classification algorithms that are supposed to guide future reasearchers working on this dataset. We find that convolutional, in particular resnetand inception-based, architectures show the best performance but recurrent architectures are also competitive for most prediction tasks. Furthermore, we demonstrate the prospects of transfer learning by finetuning a classifier pretrained on PTB-XL on a different target dataset, which turns out to be particularly effective in the small dataset regime. Finally, we provide different directions for further in-depth studies on the dataset ranging from the analysis of co-occurring pathologies, over the correlation of human-provided diagnosis likelihoods with model uncertainties to the application of interpretability methods. We release the training and evaluation code for all ECG statement prediction tasks, trained models as well as the complete model predictions in an accompanying code repository [34].

심전도 검사는 병원과 의사 사무실에서 가장 흔히 시행되는 진단 절차 중 하나입니다. 다양한 의료 애플리케이션 영역에서 자동 심전도 해석 알고리즘의 잠재력은 무궁무진하지만, 적절한 벤치마킹 데이터 세트와 잘 정의된 평가 절차의 부재로 인해 현재 이 분야의 발전이 저해되고 있는 것으로 보입니다. 우리는 PTB-XL 데이터셋[15]을 기반으로 다양한 벤치마킹 작업을 제안하고, 향후 이 데이터셋을 연구하는 연구자들에게 지침이 될 수 있는 딥러닝 기반 시간 분류 알고리즘에 대한 첫 번째 기준선 결과를 제시합니다. 컨볼루션, 특히 리셋 및 인셉션 기반 아키텍처가 가장 우수한 성능을 보이지만 대부분의 예측 작업에서는 리커런트 아키텍처도 경쟁력이 있다는 것을 발견했습니다. 또한, PTB-XL에서 사전 학습된 분류기를 다른 대상 데이터 세트에서 미세 조정하여 전이 학습의 가능성을 입증했으며, 이는 소규모 데이터 세트 체제에서 특히 효과적인 것으로 나타났습니다. 마지막으로, 동시 발생 병리 분석, 사람이 제공한 진단 가능성과 모델 불확실성의 상관관계, 해석 가능성 방법의 적용에 이르기까지 데이터 세트에 대한 심층 연구를 위한 다양한 방향을 제시합니다. 모든 ECG 문장 예측 작업에 대한 훈련 및 평가 코드, 훈련된 모델 및 전체 모델 예측을 함께 제공되는 코드 저장소에 공개합니다[34].