Enterprise Java Application Architecture and Implementation

Wednesday, August 7, 2019

A Spring Boot based Log4j2 starter project

Spring Boot uses Logback as a default logging solution. In order to switch to Log4j2, there are some configurations that has to be made.

The following customized configurations provide the following features:

A Maven POM for a Spring Boot starter project
Logstash layout dependency for JSon log files to feed into ELK
Log4j configuration with MDC support
Log4j configuration produces multiple log files

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.urcorp</groupId>
<artifactId>app-springboot-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
</parent>
<artifactId>app-log4j2-starter</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>app-log4j2-starter</name>
<description>Log4j2 starter</description>

<properties>
<log4j2-logstash-layout.version>0.19</log4j2-logstash-layout.version>
</properties>

<dependencies>

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-log4j2</artifactId>
</dependency>


<dependency>
<groupId>com.vlkan.log4j2</groupId>
<artifactId>log4j2-logstash-layout</artifactId>
<version>${log4j2-logstash-layout.version}</version>
</dependency>
</dependencies>

<profiles>
<profile>
<id>local</id>
<activation>
<os>
<family>windows</family>
</os>
</activation>
<properties>
<log.file.location>C:/app/logs</log.file.location>
</properties>
</profile>
<profile>
<id>server</id>
<activation>
<os>
<family>!windows</family>
</os>
</activation>
<properties>
<log.file.location>/app/logs</log.file.location>
</properties>
</profile>
</profiles>
</project>

log4j.xml

<?xml version="1.0" encoding="UTF-8"?>

<Configuration xmlns:xi="http://www.w3.org/2001/XInclude" status="DEBUG" name="log4j2-ur-app" monitorInterval="30">

<Properties>

<Property name="logPath">${log.file.location}</Property>

<Property name="text.log.pattern">%d{yyyy-MM-HH:mm:ss.SSS} [%t - %X{HOSTNAME,APPLICATION,CORRELATION_ID,MESSAGE_ID}] %highlight{%level}{FATAL=bg_red, ERROR=red, WARN=yellow, INFO=green, DEBUG=blue} %c{3} - %m%n</Property>

</Properties>

<Appenders>

<Console name="console" target="SYSTEM_OUT">

<PatternLayout pattern="${text.log.pattern}" />

</Console>

<RollingFile name="serverJson" fileName="${logPath}/server.json"

filePattern="${logPath}/server_%d{yyyy-MM-dd}.%i.json">

<LogstashLayout dateTimeFormatPattern="yyyy-MM-dd'T'HH:mm:ss.SSSZZZ"

templateUri="classpath:LogstashJsonEventLayoutV1.json" prettyPrintEnabled="false" stackTraceEnabled="true" />

<xi:include href="log4j2-policies.xml" />

<xi:include href="log4j2-rolloverStrategies.xml" />

</RollingFile>

<RollingFile name="serverText" fileName="${logPath}/server.log"

filePattern="${logPath}/server_%d{yyyy-MM-dd}.log">

<PatternLayout pattern="${text.log.pattern}" />

<xi:include href="log4j2-policies.xml" />

<xi:include href="log4j2-rolloverStrategies.xml" />

</RollingFile>

<RollingFile name="exceptionText" fileName="${logPath}/exception.log"

filePattern="${logPath}/exception_%d{yyyy-MM-dd}.log">

<Filters>

<ThresholdFilter level="WARN" onMatch="ACCEPT" />

</Filters>

<PatternLayout pattern="${text.log.pattern}" />

<xi:include href="log4j2-policies.xml" />

<xi:include href="log4j2-rolloverStrategies.xml" />

</RollingFile>

<RollingFile name="eventText" fileName="${logPath}/event.log"

filePattern="${logPath}/event_%d{yyyy-MM-dd}.log">

<PatternLayout pattern="${text.log.pattern}" />

<xi:include href="log4j2-policies.xml" />

<xi:include href="log4j2-rolloverStrategies.xml" />

</RollingFile>

<RollingFile name="statsText" fileName="${logPath}/stats.log"

filePattern="${logPath}/stats_%d{yyyy-MM-dd}.log">

<PatternLayout pattern="${text.log.pattern}" />

<xi:include href="log4j2-policies.xml" />

<xi:include href="log4j2-rolloverStrategies.xml" />

</RollingFile>

<RollingFile name="perfText" fileName="${logPath}/perf.log"

filePattern="${logPath}/perf_%d{yyyy-MM-dd}.log">

<PatternLayout pattern="${text.log.pattern}" />

<xi:include href="log4j2-policies.xml" />

<xi:include href="log4j2-rolloverStrategies.xml" />

</RollingFile>

<RollingFile name="sqlText" fileName="${logPath}/sql.log"

filePattern="${logPath}/sql_%d{yyyy-MM-dd}.log">

<PatternLayout pattern="${text.log.pattern}" />

<xi:include href="log4j2-policies.xml" />

<xi:include href="log4j2-rolloverStrategies.xml" />

</RollingFile>

</Appenders>

<Loggers>

<Root level="info" additivity="false">

<AppenderRef ref="console" />

<AppenderRef ref="serverText" />

<AppenderRef ref="serverJson" />

<AppenderRef ref="exceptionText" />

</Root>

<logger name="EVENT" level="info" additivity="false">

<AppenderRef ref="eventText" />

<AppenderRef ref="console" />

</logger>

<logger name="STATISTICS" level="info" additivity="false">

<AppenderRef ref="statsText" />

</logger>

<logger name="PERFORMANCE" level="info" additivity="false">

<AppenderRef ref="perfText" />

</logger>

<logger name="SQL" level="info" additivity="false">

<AppenderRef ref="sqlText" />

<AppenderRef ref="console" />

</logger>

<logger name="com.zaxxer.hikari" level="debug" additivity="false">

<AppenderRef ref="sqlText" />

</logger>

<logger name="java.sql" level="debug" additivity="false">

<AppenderRef ref="sqlText" />

</logger>

<logger name="org.apache.ibatis" level="debug" additivity="false">

<AppenderRef ref="sqlText" />

</logger>

<logger name="org.mybatis" level="debug" additivity="false">

<AppenderRef ref="sqlText" />

</logger>

<logger name="com.urcorp.dao.generated.mapper" level="trace" additivity="false">

<AppenderRef ref="sqlText" />

</logger>

<logger name="org.springframework.jdbc.core" level="trace" additivity="true">

<AppenderRef ref="sqlText" />

</logger>

</Loggers>

</Configuration>

log4j2-policies.xml

<?xml version="1.0" encoding="UTF-8"?>

<Policies>

<OnStartupTriggeringPolicy />

<SizeBasedTriggeringPolicy size="300 MB" />

<TimeBasedTriggeringPolicy interval="1"

modulate="true" />

</Policies>

log4j2-rolloverStrategies.xml

<?xml version="1.0" encoding="UTF-8"?>

<DefaultRolloverStrategy>

<Delete basePath="${logPath}" maxDepth="2">

<IfFileName glob="*.(json|log)" />

<IfLastModified age="7d" />

</Delete>

</DefaultRolloverStrategy>

package com.urcorp.common.logging;

public enum LogFileTypes {
EXCEPTION("EXCEPTION."),
EVENT("EVENT."),
PERFORMANCE("PERFORMANCE."),
SERVER("SERVER."),
SQL("SQL."),
STATISTICS("STATISTICS.");

private final String group;

private LogFileTypes(final String group) {
this.code = group;
}

public String group() {
return group;
}
}

A Maven Parent POM for a Spring Boot based application

Spring Boot based Java application is a good choice for new Java applications. The Spring Boot eco-system provides numerous libraries and tools.

In order to better manage the versions of huge amount of libraries, it is a good approach to upgrade the libraries along with Spring Boot upgrades. It will save huge maintenance and integration effort by taking advantage of the work Spring Boot team has already done.

The following sample parent POM provides the following customized features:

removed the Logback and Appache Commons Logging related dependencies
added unit test dependencies
enabled resource filtering
configured Google stype check

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>


<parent>
<groupId>com.urcorp</groupId>
<artifactId>urcorp</artifactId>
<version>1.0</version>
</parent>

<groupId>com.urcorp.urdepartment</groupId>
<artifactId>app-springboot-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>pom</packaging>
<name>app-springboot-parent</name>
<description>Parent POM for Spring Boot based Application</description>

<properties>
<maven.build.timestamp.format>yyyy-MM-dd'T'HH:mm:ss'Z'</maven.build.timestamp.format>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>





<spring-boot.version>2.1.5.RELEASE</spring-boot.version>


<awaitility.version>3.1.6</awaitility.version>


<google.format.version>1.5</google.format.version>
<junit-vintage-engine.version>5.5.1</junit-vintage-engine.version>
<maven-compiler-plugin.version>3.8.1</maven-compiler-plugin.version>
<maven-surefire-plugin.version>2.22.2</maven-surefire-plugin.version>
<spotless-maven-plugin.version>1.23.0</spotless-maven-plugin.version>
</properties>

<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>${spring-boot.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>


<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
<version>${spring-boot.version}</version>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</exclusion>
<exclusion>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-access</artifactId>
</exclusion>
<exclusion>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</exclusion>
<exclusion>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</exclusion>
<exclusion>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>log4j-over-slf4j</artifactId>
</exclusion>
</exclusions>
</dependency>


<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
<exclusions>
<exclusion>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
</dependencyManagement>

<dependencies>


<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>


<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>


<dependency>
<groupId>org.junit.platform</groupId>
<artifactId>junit-platform-launcher</artifactId>
<scope>test</scope>
</dependency>


<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-params</artifactId>
<scope>test</scope>
</dependency>


<dependency>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
<scope>test</scope>
</dependency>


<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-inline</artifactId>
<scope>test</scope>
</dependency>


<dependency>
<groupId>org.awaitility</groupId>
<artifactId>awaitility</artifactId>
<version>${awaitility.version}</version>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<resources>
<resource>
<directory>src/main/resources</directory>
<filtering>true</filtering>
</resource>
<resource>
<directory>src/test/resources</directory>
<filtering>true</filtering>
</resource>
</resources>
<plugins>
<plugin>
<groupId>com.diffplug.spotless</groupId>
<artifactId>spotless-maven-plugin</artifactId>
<version>${spotless-maven-plugin.version}</version>
<configuration>
<java>
<includes>

<include>src/**/*.java</include>
</includes>
<googleJavaFormat>

<version>${google.format.version}</version>

<style>GOOGLE</style>
</googleJavaFormat>
<removeUnusedImports />
</java>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>${maven-compiler-plugin.version}</version>
<configuration>
<source>${maven.compiler.source}</source>
<target>${maven.compiler.source}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<excludes>
</excludes>
</configuration>
</plugin>
</plugins>
</build>

</project>

Monday, January 29, 2018

Cassandra vs Oracle

Source:
https://www.datastax.com/wp-content/uploads/2013/11/WP-DataStax-Oracle.pdf

Oracle is a solid RDBMS that performs well for the use cases for which it was designed (e.g. ERP and accounting applications). It is not architected to tackle the new wave of big data, online applications developed today. The scale- up, master-slave, non-distributed architecture of Oracle falls short of what modern online applications need.

Scalability and Performance Limitations

Oracle’s scale-up, master-slave design limits both its scalability and performance for servicing the online elasticity and performance SLA needs of many online applications.

The failure of Oracle to add capacity online in an elastic, scale-out vs. scale-up manner to service increasing user workloads, keep performance high, and easily consume fast incoming data from countless geographical locations is widely recognized.

Benefits of Cassandra

Massively scalable architecture – a masterless design where all nodes are the same.
Linear scale performance – online node additions produce predictable increases in performance.
Continuous availability – redundancy of both data and function mean no single point of failure.
Transparent fault detection and recovery – easy failed node recovery.
Flexible, dynamic schema data modeling – easily supports structured, semi-structured, and unstructured data.
Guaranteed data safety – commit log design ensures no data loss.
Active everywhere design – all nodes may be written to and read from.
Tunable data consistency – support for strong or eventual data consistency.
Multi-data center replication – cross data center and multi-cloud availability zone support for writes/reads built in.
Data compression – data compressed up to 80% without performance overhead.
CQL (Cassandra Query Language) – an SQL – like language that makes moving from an RDBMS very easy.

DataStax Cassandra

Built-in analytics functionality for Cassandra data via integration with a number of Hadoop components (e.g. MapReduce, Hive, Pig, Mahout, etc.)
Enterprise search capability on Cassandra data via Solr integration.
Enterprise security including external/internal authentication and object permission management, transparent data encryption, client-tonode and node-to-node encryption, and data auditing.
Visual cluster management for all administration tasks including backup/restore operations, performance monitoring, alerting, and more.

Cassandra Use Cases

Time-series data management (e.g. financial, sensor data, web click stream, etc.)
Online web retail
Web buyer behavior and personalization management
Recommendation engines
Social media input and analysis
Online gaming
Fraud detection and analysis
Risk analysis and management
Supply chain analytics
Web product searches
Write intensive transactional systems

Data Modeling Differences

In traditional databases such as Oracle, data is modeled in a standard “third normal form” design without the need to know what questions will be asked of the data.

By contrast, in NoSQL, the questions asked of the data are what drive the data model design and the data is highly de-normalized.

Data Processing Concerns of Modern Applicaiton

Legacy Application	Modern Application
Slow/medium velocity data	High velocity data
Data coming in from one/few locations	Data coming in from many locations
Rigid, static structured data	Flexible, fluid, multi-type data
Low/medium data volumes; purge often	High data volumes; retain forever
Deploy app central location/one server	Deploy app everywhere/many servers
Write data in one location	Write data everywhere/anywhere
*Primary concern: scale reads*	*Scale writes and reads*
Scale up for more users/data	Scale out for more users/data

DataStax Cassandra vs. Oracle at Functional Level

Feature/Function	DataStax/Cassandra	Oracle RDBMS
High Availability	Continuous availability with built in redundancy and hardware rack awareness in both single and multiple data centers	General replication; Oracle Dataguard (for failover) and Oracle RAC (single point of failure with storage) bout of which are expensive add-ons. GoldenGate also offered for certain use cases.
Scalability Model	Linear performance gains via node additions	Scale up via adding CPU’s RAM or Oracle RAC or Exadata
Replication Model	Peer-to-peer; number of copies configurable across cluster and each datacenter.	Peer-to-peer; number of copies configurable across cluster and each datacenter
Multi-data center/geography/cloud capabilities	Multi-directional, 1-many data center support built in, with true read/write anywhere capability	Nothing specific for multi-data center
Data partitioning/sharding model	Automatic; done via primary key; random or ordered	Table partitioning option to enterprise edition; manual server sharding
Data volume support	TB-PB capable	TB capable; PB with Exadata
Analytic support	Analytics on Caddadra data via Hadoop integration( MapReduce, Hive, Pig, Mahout )	Analytic functions in Oracle RDBMS via SQL MapReduce. Haddop support done in NoSQL appliance
Enterprise search support	Built into dataStax Enterprise via Solr integration	Handled via Oracle search (cost add-on)
Mixed workload support	All handled in one cluster with built-in workload isolation; no workload competes for resources with another	Handled via Oracle Exadata
Data Model	Google Bigtable like; a wide column store	Relational/tabular
Flexibility of data model	Flexible. Designed for structured, semi-structured, and unstructured data	Rigid; primarily structured data
Data consistency mode	Tunable consistency (CAP theorem consistency per operation (e.g. per insert, delete, etc.) across cluster	Traditional ACID
Transaction Support	Provides full Atomic, Isolated, and Durable (AID) transactions including batch transactions and “lightweight” transactions with Cassandra 2.0 and higher	Traditional ACID
Security	Support for all key security needs: Login ID/passwords, external security support; object permission management; transparent data encryption; client to mode, node to node encryption; data auditing	Full security support
Storage model	Targeted directories with separation (e.g. put some column families on SSD’s, some on spinning disk)	Tablespaces
Data compression	Built in	Various methods
Memory usage model	Distributed object/row caches across all nodes in a cluster	Standard data/metadata caches with query cache
Logical database container	Keyspace	Database
Primary data object	Column family/table	Table
Data variety support	Structured, semi-structured, unstructured	Primarily structured
Indexes	Primary, secondary. Extensible via Solr indexes	B-Tree, bitmap, clustered, others
Core language	CQL (Cassadra Query Language; resembles SQL)	SQL
Primary query utilities	CQL shell	SQL*Plus
Visual query tools	DataStax DevCenter and 3^rd party support (Aqua data Studio)	SQL Developer from Oracle etc.
Development Language support	Many (Java, C#, Python)	Many
Geospatial support	Done via Solr ingtegration	Oracle Geospatial option (cost add-on)
Logging (e.g., web, application) data support	Handled via log4j	Nothing built in
Backup/recovery	Online, point-in-time restore	Online, point-in-time restore
Enterprise management/monitoring	DataStax OpsCenter	Oracle Enterprise Manager

Subscribe to: Posts (Atom)