Speaker
Description
The massive data throughput at the High Energy Photon Source (HEPS) exposes critical I/O bottlenecks inherent in the conventional "write-then-read" data handling paradigm, severely limiting analytical throughput. To address this challenge, we have designed and developed LightVortex, an end-to-end real-time data feeding platform for HEPS. LightVortex implements a coherent processing pipeline that ingests raw DAQ binary streams, automatically parses them based on their beamline origin, and restructures them for computational efficiency. Processed data is staged in a massive distributed cache pool; within this cache, a lightweight indexing mechanism that encodes metadata directly into file paths and names enables rapid data retrieval. A unified I/O interface abstracts away the differences between accessing live data streams and persisted files, presenting a consistent data view to applications. This design decouples scientific workflows from data locality and format complexities, thereby accelerating scientific discovery. The platform is currently undergoing integration, deployment, and testing with key HEPS scientific computing applications.
