현업에서는 어떻게 서비스 모니터링 메일링할까? (ElasticSearch 수집정보, 여러서비스 사용률)

안녕하세요

최근 회사에 여러 서비스가 모니터링, 사용률 제공이 필요 했습니다. 

 오류 - 즉각적인 이메일 알람 (ElasticSearch 데이터 필요)

 사용률 - 일단위 이메일 알람  (여러 DB Connection 필요)

* 사작

spring boot를 사용하였습니다.

- build.gradle

dependencies {
   implementation 'org.springframework.boot:spring-boot-starter-data-elasticsearch'
   implementation 'org.springframework.boot:spring-boot-starter-mail'
   implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
   implementation 'org.springframework.boot:spring-boot-starter-thymeleaf'
   implementation 'org.springframework.boot:spring-boot-starter-web'
   implementation 'org.springframework.boot:spring-boot-starter-validation'
   compile 'org.springframework.boot:spring-boot-devtools'
   compile group: 'org.thymeleaf.extras', name: 'thymeleaf-extras-java8time', version: '3.0.1.RELEASE'
   compile group: 'com.sun.mail', name: 'javax.mail', version: '1.6.2'
   compile('org.modelmapper:modelmapper:1.1.0')
   compile group: 'org.apache.commons', name: 'commons-lang3', version: '3.9'
   compile group: 'org.apache.commons', name: 'commons-collections4', version: '4.1'
   compile group: 'com.google.guava', name: 'guava', version: '28.0-jre'
   compile group: 'com.omnicns', name: 'common-java', version: '0.0.22'
   compile group: 'com.omnicns', name: 'common-jsp', version: '0.0.2'
   compile group: 'com.omnicns', name: 'common-spring', version: '0.0.15'
   compileOnly 'org.projectlombok:lombok'
   implementation 'com.h2database:h2'
   annotationProcessor 'org.projectlombok:lombok'

   compile group: 'mysql', name: 'mysql-connector-java', version: '8.0.13'
   compile group: 'org.mariadb.jdbc', name: 'mariadb-java-client', version: '2.2.1'
   compile group: 'com.oracle', name: 'ojdbc6', version: '11.2.0.3'
   compile fileTree(dir: "$project.rootDir/libs", includes: ['*.jar'])
   testImplementation('org.springframework.boot:spring-boot-starter-test') {
      exclude group: 'org.junit.vintage', module: 'junit-vintage-engine'
   }
}

- application.yaml

spring:
  application:
    name: service-monitor
  pid:
    file: ${spring.application.name}.pid
  h2:
    console:
      enabled: true
    path: /h2-console

server:
  port: 8090

logging:
  config: classpath:logback-spring.xml

projects:
  properties:
    email-from: serviceteam@company.com
    email-serviceteam: serviceteam@company.com
    email-business: business@company.com
    email-company: company@company.com
    monitor-day-count-size: 20
  monitor:
    hibernate:
      property:
        "[hibernate.dialect]": org.hibernate.dialect.H2Dialect
        "[hibernate.show_sql]": false
        "[hibernate.hbm2ddl.auto]": create-drop
        "[hibernate.cache.use_second_level_cache]": false
        "[hibernate.cache.use_query_cache]": false
    datasource:
      hikari:
        jdbc-url: jdbc:h2:mem:monitor
        driver-class-name: org.h2.Driver
        username: sa
        password: companypwd

  brain:
    hibernate:
      property:
        "[hibernate.dialect]": org.hibernate.dialect.MySQLDialect
        "[hibernate.show_sql]": true
        "[hibernate.hbm2ddl.auto]": none
        "[hibernate.cache.use_second_level_cache]": false
        "[hibernate.cache.use_query_cache]": false
    datasource:
      hikari:
        jdbc-url: jdbc:mysql://1.1.1.1:3306/company?serverTimezone=Asia/Seoul
        driver-class-name: com.mysql.jdbc.Driver
        username: company
        password: companypwd
  mindcares:
    kr:
      hibernate:
        property:
          "[hibernate.dialect]": org.hibernate.dialect.Oracle10gDialect
          "[hibernate.show_sql]": true
          "[hibernate.hbm2ddl.auto]": none
          "[hibernate.cache.use_second_level_cache]": false
          "[hibernate.cache.use_query_cache]": false
      datasource:
        hikari:
          jdbc-url: jdbc:oracle:thin:@1.1.1.170:1521:omnidb
          driver-class-name: oracle.jdbc.driver.OracleDriver
          username: company
          password: companypwd
    seniorcare:
      hibernate:
        property:
          "[hibernate.dialect]": org.hibernate.dialect.MySQL5InnoDBDialect
          "[hibernate.show_sql]": true
          "[hibernate.hbm2ddl.auto]": none
          "[hibernate.cache.use_second_level_cache]": false
          "[hibernate.cache.use_query_cache]": false
      datasource:
        hikari:
          jdbc-url: jdbc:mariadb://1.1.1.171:3306/company_hasri
          driver-class-name: org.mariadb.jdbc.Driver
          username: hasri
          password: companypwd

    medicare:
      hibernate:
        property:
          "[hibernate.dialect]": org.hibernate.dialect.MySQL5InnoDBDialect
          "[hibernate.show_sql]": true
          "[hibernate.hbm2ddl.auto]": none
          "[hibernate.cache.use_second_level_cache]": false
          "[hibernate.cache.use_query_cache]": false
      datasource:
        hikari:
          jdbc-url: jdbc:mariadb://1.1.1.170:3306/medicare
          driver-class-name: org.mariadb.jdbc.Driver
          username: medicare
          password: companypwd

    ceracheck:
      hibernate:
        property:
          "[hibernate.dialect]": org.hibernate.dialect.MySQL5InnoDBDialect
          "[hibernate.show_sql]": true
          "[hibernate.hbm2ddl.auto]": none
          "[hibernate.cache.use_second_level_cache]": false
          "[hibernate.cache.use_query_cache]": false
      datasource:
        hikari:
          jdbc-url: jdbc:mariadb://1.1.1.171:3306/company_busines
          driver-class-name: org.mariadb.jdbc.Driver
          username: company_dev
          password: companypwd

- config

 

-- elasticsearch, email config

@Slf4j
@EnableConfigurationProperties(ProjectProperties.class)
@Configuration
public class CoreConfigration {

    @Autowired
    ProjectProperties projectProperties;


    @Bean
    public JavaMailSenderImpl mailSender() {
        JavaMailSenderImpl javaMailSender = new JavaMailSenderImpl();
        javaMailSender.setProtocol("smtp");
        javaMailSender.setHost("127.0.0.1");
        javaMailSender.setPort(25);
        return javaMailSender;
    }
    @Bean
    public Java8TimeDialect java8TimeDialect() {
        return new Java8TimeDialect();
    }

}

@Configuration
public class ElasticSearchClientConfig extends AbstractElasticsearchConfiguration {

    @Override
    @Bean
    public RestHighLevelClient elasticsearchClient() {
        final ClientConfiguration clientConfiguration = ClientConfiguration.builder()
                .connectedTo("1.1.1.181:9200")
                .build();
        return RestClients.create(clientConfiguration).rest();
    }
}

- multiple database config

 서비스별 config naming

   -- BrainDatabaseConfiguration

   -- MindcareCeraCheckDatabaseConfiguration

   -- MindcareKRDatabaseConfiguration

   -- MindcareMedicareDatabaseConfiguration

   -- MindcareSeniorCareDatabaseConfiguration

   -- MonitorDatabaseConfiguration  <- primary

@Configuration("monitorDatabaseConfiguration")
@EnableJpaRepositories(basePackages = "com.company.service.monitor.repository.monitor", entityManagerFactoryRef = "monitorEntityManager", transactionManagerRef = "monitorTransactionManager")
public class MonitorDatabaseConfiguration {

     @Bean(value = "monitorHibernateProperties")
    @ConfigurationProperties("projects.monitor.hibernate.property")
    public HashMap<String, Object> monitorHibernateProperties() {
        return new HashMap<String, Object>();
    }

    @Primary
    @Bean
    public LocalContainerEntityManagerFactoryBean monitorEntityManager() throws SQLException {
        final LocalContainerEntityManagerFactoryBean em = new LocalContainerEntityManagerFactoryBean();
        em.setDataSource(dataSource());
        em.setPackagesToScan("com.company.service.monitor.domain.monitor");
        final HibernateJpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
        em.setJpaVendorAdapter(vendorAdapter);
        final HashMap<String, Object> properties = monitorHibernateProperties();
        em.setJpaPropertyMap(properties);
        return em;
    }

    @Primary
    @Bean
    public PlatformTransactionManager monitorTransactionManager() throws SQLException {
        final JpaTransactionManager transactionManager = new JpaTransactionManager();
        transactionManager.setEntityManagerFactory(monitorEntityManager().getObject());
        return transactionManager;
    }

    @Bean(value = "monitorHikariConfig")
    @ConfigurationProperties(prefix="projects.monitor.datasource.hikari")
    public HikariConfig hikariConfig() {
        return new HikariConfig();
    }

    @Primary
    @Bean(value = "monitorDataSource")
    public DataSource dataSource() throws SQLException {
        return new HikariDataSource(hikariConfig());
    }
}

...

- 서비스별 package 구분 

 

- schedule 구분

  

-- elasticsearch 

@Component
@Slf4j
public class ElasticSearchScheduler {

    @Autowired
    MailService javaMailSender;

    @Qualifier("elasticsearchClient")
    @Autowired
    RestHighLevelClient restHighLevelClient;

    @Autowired
    SpringTemplateEngine springTemplateEngine;

    @Value("${projects.properties.email-serviceteam}")
    String emailServiceteam;

    Date omnifit2Last = new Date();
    @Scheduled(cron = "*/5 * * * * *")
    public void omnifit2Monitor() throws Throwable {
       ...
        if (datas.size() > 0) {
            String title = String.format("%s %s", new SimpleDateFormat("yyyy-MM-dd hh:mm:ss").format(new Date()), "omnifit2 5초간격 ERROR 발생 (" + datas.size() + "건)");
            Context context = new Context();
            context.setVariable("logs", datas);
            String html = springTemplateEngine.process("emails/log-error", context);
            javaMailSender.sendHtml(emailServiceteam, title, html);
        }
    }


    Date mindcareLast = new Date();
    @Scheduled(cron = "*/5 * * * * *")
    public void mindcareMonitor() throws Throwable {
       ...
        if (datas.size() > 0) {
            String title = String.format("%s %s", new SimpleDateFormat("yyyy-MM-dd hh:mm:ss").format(new Date()), "mindcare 5초간격 ERROR 발생 (" + datas.size() + "건)");
            Context context = new Context();
            context.setVariable("logs", datas);
            String html = springTemplateEngine.process("emails/log-error", context);
            javaMailSender.sendHtml(emailServiceteam, title, html);
        }
    }} 

-- serviceScheduler

@Component
@Slf4j
public class ServiceScheduler {


   ...
    @Scheduled(cron = "1 1 8 * * *")
    public void mindcareMonitor() throws Throwable {
        String title = String.format("[모니터 서비스통계] %s %s", new SimpleDateFormat("yyyy-MM-dd").format(new Date()), "현황 공유 건");
        Context context = new Context();
        context.setVariable("dayCountSize", monitorDayCountSize);
        mindcareService.getTotal().entrySet().stream().forEach(it -> context.setVariable(it.getKey(), it.getValue()));
        String html = springTemplateEngine.process("emails/mindcares/total", context);
        javaMailSender.sendHtml(emailOmnifit, title, html);
    }
}

- thymeleaf view

-- total.html (이메일 발송에 style을 inline으로 해야 필터링에 걸리지 않는다 - 디자인깨짐 방지) 

<!DOCTYPE html>
<html lang="en" xmlns:th="http://www.thymeleaf.org">
<head>
    <meta charset="UTF-8">
    <title>business 기준 데이터</title>
</head>
<body>
<h1>business 국문향</h1>
<h3>회원통계</h3>
<div>
    <table style="border: 1px solid #444444; border-collapse: collapse; text-align:center;" cellpadding="6px;" width="740px">
        <th style="border: 1px solid #444444; background-color:#323a46; color:#98a6ad;" width="14%;">구분</th>
        <th style="border: 1px solid #444444; background-color:#323a46; color:#98a6ad;" width="18%">시작일</th>
        <th style="border: 1px solid #444444; background-color:#323a46; color:#98a6ad;" width="18%">종료일</th>
        <th style="border: 1px solid #444444; background-color:#323a46; color:#98a6ad;" width="25%">전체 회원수</th>
        <th style="border: 1px solid #444444; background-color:#323a46; color:#98a6ad;" width="25%">정회원수 (%)</th>
        <tr>
            <td style="border: 1px solid #444444;">누적전체</td>
            <td style="border: 1px solid #444444;" th:text="${kr_totalSignUpUser.minRegDt}"></td>
            <td style="border: 1px solid #444444;" th:text="${kr_totalSignUpUser.maxRegDt}"></td>
            <td style="border: 1px solid #444444; text-align:right;font-weight: bold;" th:text="${#numbers.formatInteger(kr_totalSignUpUser.totCnt, 0, 'COMMA')}"></td>
            <td style="border: 1px solid #444444; text-align:right;"> <span th:text="${#numbers.formatInteger(kr_totalSignUpUser.mem01, 0, 'COMMA')}"></span> (<span style="background-color:rgb(243, 209, 170)" ><span th:text="${kr_totalSignUpUser.memRat01}"></span> <span>%</span></span>)</td>
        </tr>
        <tr>
            <td style="border: 1px solid #444444;">이전 달</td>
            <td style="border: 1px solid #444444;" th:text="${kr_afterMonthSignUpUser.minRegDt}"></td>
            <td style="border: 1px solid #444444;" th:text="${kr_afterMonthSignUpUser.maxRegDt}"></td>
            <td style="border: 1px solid #444444; text-align:right;" th:text="${#numbers.formatInteger(kr_afterMonthSignUpUser.totCnt, 0, 'COMMA')}"></td>
            <td style="border: 1px solid #444444; text-align:right;"> <span th:text="${#numbers.formatInteger(kr_afterMonthSignUpUser.mem01, 0, 'COMMA')}"></span> (<span style="background-color:rgb(211, 245, 161)" ><span th:text="${kr_afterMonthSignUpUser.memRat01}"></span> <span>%</span></span>)</td>
        </tr>
        <tr>
            <td style="border: 1px solid #444444;">이전 일</td>
            <td style="border: 1px solid #444444;" th:text="${kr_afterDaySignUpUser.minRegDt}"></td>
            <td style="border: 1px solid #444444;" th:text="${kr_afterDaySignUpUser.maxRegDt}"></td>
            <td style="border: 1px solid #444444; text-align:right;" th:text="${#numbers.formatInteger(kr_afterDaySignUpUser.totCnt, 0, 'COMMA')}">34</td>
            <td style="border: 1px solid #444444; text-align:right;"> <span th:text="${#numbers.formatInteger(kr_afterDaySignUpUser.mem01, 0, 'COMMA')}"></span> (<span style="background-color:rgb(211, 245, 161)" ><span th:text="${kr_afterDaySignUpUser.memRat01}"></span> <span>%</span></span>)</td>
        </tr>
    </table>
</div>

<h3>측정통계</h3>
<div>
    <table style="border: 1px solid #444444; border-collapse: collapse; text-align:center;" cellpadding="6px;" width="740px">
        <th style="border: 1px solid #444444; background-color:#323a46; color:#98a6ad;" width="14%">구분</th>
        <th style="border: 1px solid #444444; background-color:#323a46; color:#98a6ad;" width="18%">시작일</th>
        <th style="border: 1px solid #444444; background-color:#323a46; color:#98a6ad;" width="18%">종료일</th>
        <th style="border: 1px solid #444444; background-color:#323a46; color:#98a6ad;" width="25%">전체 측정수</th>
        <th style="border: 1px solid #444444; background-color:#323a46; color:#98a6ad;" width="25%">측정 정회원수 (%)</th>
        <tr>
            <td style="border: 1px solid #444444;">누적전체</td>
            <td style="border: 1px solid #444444;" th:text="${kr_totalMeasUser.minRegDt}"></td>
            <td style="border: 1px solid #444444;" th:text="${kr_totalMeasUser.maxRegDt}"></td>
            <td style="border: 1px solid #444444; text-align:right;" th:text="${#numbers.formatInteger(kr_totalMeasUser.totCnt, 0, 'COMMA')}"></td>
            <td style="border: 1px solid #444444; text-align:right;"> <span th:text="${#numbers.formatInteger(kr_totalMeasUser.memUz001, 0, 'COMMA')}"></span> (<span style="background-color:rgb(243, 209, 170)" ><span th:text="${kr_totalMeasUser.uz001Rat}"></span> <span>%</span></span>)</td>
        </tr>
        <tr>
            <td style="border: 1px solid #444444;">이전 달</td>
            <td style="border: 1px solid #444444;" th:text="${kr_afterMonthMeasUser.minRegDt}"></td>
            <td style="border: 1px solid #444444;" th:text="${kr_afterMonthMeasUser.maxRegDt}"></td>
            <td style="border: 1px solid #444444; text-align:right;" th:text="${#numbers.formatInteger(kr_afterMonthMeasUser.totCnt, 0, 'COMMA')}"></td>
            <td style="border: 1px solid #444444; text-align:right;"> <span th:text="${#numbers.formatInteger(kr_afterMonthMeasUser.memUz001, 0, 'COMMA')}"></span> (<span style="background-color:rgb(211, 245, 161)" ><span th:text="${kr_afterMonthMeasUser.uz001Rat}"></span> <span>%</span></span>)</td>
        </tr>
        <tr>
            <td style="border: 1px solid #444444;">이전 일</td>
            <td style="border: 1px solid #444444;" th:text="${kr_afterDayMeasUser.minRegDt}"></td>
            <td style="border: 1px solid #444444;" th:text="${kr_afterDayMeasUser.maxRegDt}"></td>
            <td style="border: 1px solid #444444; text-align:right;" th:text="${#numbers.formatInteger(kr_afterDayMeasUser.totCnt, 0, 'COMMA')}"></td>
            <td style="border: 1px solid #444444; text-align:right;"> <span th:text="${#numbers.formatInteger(kr_afterDayMeasUser.memUz001, 0, 'COMMA')}"></span> (<span style="background-color:rgb(211, 245, 161)" ><span th:text="${kr_afterDayMeasUser.uz001Rat}"></span> <span>%</span></span>)</td>
        </tr>
    </table>
</div>

<h4>전체 정회원 중 3회이상 측정회원</h4>
<div>
    <table style="border: 1px solid #444444; border-collapse: collapse; text-align:center;" cellpadding="6px;" width="555px">
        <th style="border: 1px solid #444444; background-color:#323a46; color:#98a6ad;" width="66%">정회원수</th>
        <th style="border: 1px solid #444444; background-color:#323a46; color:#98a6ad;" width="34%">3회이상 측정회원수</th>
        <tr>
            <td style="border: 1px solid #444444; text-align:right;" th:text="${#numbers.formatInteger(kr_manyMeasUser.totCnt, 0, 'COMMA')}"></td>
            <td style="border: 1px solid #444444; text-align:right;"><span th:text="${#numbers.formatInteger(kr_manyMeasUser.cnt, 0, 'COMMA')}"></span> (<span style="background-color:rgb(243, 209, 170)"><span th:text="${kr_manyMeasUser.meaRat}"></span> <span>%</span></span>)</td>
        </tr>
    </table>
</div>

<h4>최근 일주일 간 Top 10 측정업체</h4>
<div>
    <table style="border: 1px solid #444444; border-collapse: collapse; text-align:center;" cellpadding="6px;" width="555px">
        <th style="border: 1px solid #444444; background-color:#323a46; color:#98a6ad;" width="66%">업체정보</th>
        <th style="border: 1px solid #444444; background-color:#323a46; color:#98a6ad;" width="34%">측정건수</th>

        <tr
                th:each="kr_measCrop, iterStat : ${kr_measCrops}"
                th:style="${iterStat.even ? 'background-color: #f1f3fa;' : ''}"
        >
            <td style="border: 1px solid #444444;"><span th:text="${kr_measCrop.corpNm}"></span> (<span th:text="${kr_measCrop.corpId}"></span>)</td>
            <td style="border: 1px solid #444444; text-align:right;" th:text="${#numbers.formatInteger(kr_measCrop.cnt, 0, 'COMMA')}"></td>
        </tr>
    </table>
</div>
</body>
</html>

메일링 (디자인은 기억하지말아주세요.ㅎ)

1.

2.

3.

https://github.com/visualkhh/lib-spring/tree/master/boot/service-monitor

visualkhh@gmail.com

감사합니다.