Platform & L3 Development Engineering
An enterprise database activity monitoring and security platform in production since 2008 — working as a Platform/L3 Development Engineer on a legacy Java-based system serving enterprise clients across banking, healthcare, and financial services sectors.
Overview
Industry: Enterprise Cybersecurity / Data Protection
Product Type: Database Activity Monitoring (DAM) Platform
Architecture: Multi-tier appliance stack — DB Agent → Gateway → Management Server → Manager of Managements, Java 8 backend on custom CentOS 7/OEL 9, Oracle DB 19c, Tomcat 9x, managed via Python-based CLI orchestrator.
Period: August 2021 — present
SecureSphere DAM monitors, audits, and controls database access across enterprise infrastructure. The platform operates as a network appliance stack: lightweight agents on monitored database hosts forward audit traffic through Gateways to a central Management Server, which handles policy enforcement, reporting, alert delivery, and administrative control. The system runs in high-availability configurations and trust configurations. Work covered versions 14.x through 15.x with direct involvement in production upgrade validation and failure remediation.
The nature of the role was L3 troubleshooting-development: reproducing customer-reported failures in lab, identifying root causes through cross-layer log analysis and code inspection, and delivering fixes as Java patches, shell script modifications, or platform-level configuration changes.
Architecture Context
Environment
- Custom CentOS 7 / OEL 9
- Java 8, Apache Tomcat 9.x
- Oracle DB 11c/19c — co-located with App, cross-access via shared OS groups
- HA infrastructure: corosync / keepalived, Oracle Data Guard
- CLI Python-based platform orchestration
- Active platform services: server, db, watchdog, portguard, platform network, export/import
Stack
- Backend: Java 8, Spring Framework, Ant/Maven multi-module monorepo
- ORM / DB migration: Hibernate, Flyway
- Template engine: Apache Velocity
- Database: Oracle DB 11c/19c
- Platform: CentOS 7/OEL9, bash, Python 2.7/3.x
- Security layer: TLS/SSL, JCEKS/PKCS12 keystores, keytool
- CLI tooling: custom python CLI
- HA: corosync, keepalived, Oracle Data Guard
- Cloud: AWS, Azure
Features
- Database activity monitoring via agent → gateway → APP audit pipeline
- Policy-based access control with login&password/LDAP-integrated authentication/SSO
- TLS certificate lifecycle management with HSM support and Client-Server synchronization
- Platform HA with corosync/keepalived and Oracle Data Guard replication
- In-place upgrade pipeline (IPU) with Oracle DB export/import
- URM (User Rights Management) scanning across Oracle, SQL Server, and LDAP/AD sources
- Automated audit archiving with configurable job scheduling
- Learning engine with profile-based behavioral enforcement
My Contribution
- Performed root-cause analysis and code-level fixes across the full platform stack: Java backend, Oracle DB, Linux OS, network layer, and TLS/certificate subsystem
- Owned L3 investigation and resolution of production customer cases: audit pipeline failures, MX startup issues, upgrade regressions, HA configuration failures, and certificate lifecycle breakdowns
- Delivered Java-level fixes for data integrity issues: sequence counter desync, query construction defects, ORM-layer constraint violations, and application logic bypasses
- Diagnosed and resolved OS-level platform regressions: filesystem permission and UMask issues, JVM entropy configuration, and post-upgrade environment breakdowns
- Contributed fixes to the IPU (in-place upgrade) pipeline: export/import log separation, TAR existence validation, and Flyway migration mismatch resolution
- Rewrote MX-HA cluster provisioning from
crmtopcsfor OEL9 compatibility — shipped as a platform-wide change - Improved Client-Server certificate renewal reliability and developed keystore management tooling
- Developed and shipped a suite of platform diagnostic scripts integrated into the impctl toolchain: Oracle JVM health checks, keystore inspection and merge utilities, certificate management automation, and log parsing tools
- Added diagnostic mode to the audit index pipeline: extended logging to capture CSV file real size, permissions, owner, and path at download time
Engineering Highlights
Oracle JVM corruption — platform non-starter. Platform updates can corrupt internal DB engine components in ways that don’t surface as configuration errors. Effective diagnosis requires understanding the full startup dependency chain — not just the application layer. Recovery from this class of failure demands working knowledge of both the DB internals and the platform service initialization order simultaneously.
Oracle sequence counter desync — silent data loss at the DB layer. Unique constraint violations don’t always mean duplicate input data — they can indicate state drift between the DB engine’s internal counters and the actual data. Application-layer resilience requires anticipating this class of DB state inconsistency and handling it without escalating to manual DBA intervention on customer systems.
Post-upgrade permission regression — invisible failures at the OS boundary. Security updates can silently break cross-user file access by modifying ownership and UMask values. This class of failure presents as missing data at the application layer with no corresponding error — it is undiagnosable without OS-level awareness of how two co-operating service accounts interact on shared filesystem paths.
rngd CPU saturation — library upgrade as a hidden infrastructure dependency. Upgrading a security library can introduce unexpected OS-level resource contention with no obvious connection to the change. Entropy pool exhaustion caused by SecureRandom blocking is a well-known JVM deployment issue — but it only becomes visible under the specific call patterns introduced by newer library versions. Cross-referencing customer incident timelines with GTI snapshots was the key diagnostic step.
Upgrade data loss — missing safeguards in destructive pipeline operations. A multi-step upgrade pipeline that includes destructive operations must treat each preceding step as a hard precondition — not an assumption. Log file reuse across phases destroys post-mortem visibility exactly when it’s needed most. The lesson: any script that drops data must explicitly verify that a recoverable backup exists before proceeding.
Client-Server certificate renewal desync — protocol assumptions vs. deployment reality. Certificate renewal protocols designed for always-available topology fail non-obviously in environments where components are intermittently offline. Retaining the previous certificate until the peer confirms receipt is a basic availability contract — but it requires explicit implementation, not implicit assumption.
CRM → PCS cluster provisioning — CLI migration is an architectural decision. Replacing a cluster management CLI is not a command substitution exercise. Different tools expose different assumptions about idempotency, state management, and error handling. A provisioning script that works on a clean environment must also work on a partially configured one — idempotent resource cleanup is not optional.
ORA-01795 under high agent count — unbounded inputs as a latent failure mode. SQL IN clause limits are a known Oracle constraint, but they only become failures when input size reaches a threshold that may not appear in standard testing. Any code path that constructs queries from external input lists must treat list size as a variable — not a constant.
URM scan unique constraint — identity lifecycle as a data integrity edge case. Directory service identity management introduces data lifecycle scenarios that relational merge logic doesn’t anticipate by default. Delete-and-recreate of a domain user produces structurally identical but semantically distinct records. Correct merge semantics require matching on the full identity key — not just the human-readable name.
Network security layer as a diagnostic blind spot. A distributed appliance architecture spanning DB agents, gateways, and management servers crosses multiple network boundaries in enterprise environments — each of which may have firewalls, deep packet inspection, or network-level antivirus in place. These security controls can silently drop traffic, partially block communication channels, or flag legitimate platform components as potentially malicious — without producing any error visible to the platform itself. Diagnosing this class of failure is further complicated by customer security policies that restrict access to firewall logs, IDS/IPS rule sets, or network capture points. In practice, this means the platform cannot rely on the network layer being transparent or cooperative. The only reliable mitigation is exhaustive structured logging at every communication stage — agent-to-gateway, gateway-to-MX, and MX-to-SOM — so that when a network control silently interferes, the platform’s own logs can at minimum identify where in the communication chain the failure occurred, even when the cause itself remains outside diagnostic reach.
Technologies
Java 8, Spring Framework, Spring Security, Maven, Hibernate, Flyway, Apache Velocity, Oracle DB 11с/19c, Apache Tomcat 9x, CentOS 7, OEL 9, bash, Python 2.7/3.x, keytool, corosync, keepalived, Oracle Data Guard, TLS/SSL
