publications

2023

1. 

   Can My WiFi Handle the Metaverse? A Performance Evaluation Of Meta’s Flagship Virtual Reality Hardware

   Matthijs Jansen, Jesse Donkervliet, Animesh Trivedi, and Alexandru Iosup

   In Companion of the 2023 ACM/SPEC International Conference on Performance Engineering, ICPE 2023, Coimbra, Portugal, April 15-19, 2023, 2023

   Abs Bib PDF Code Slides

   Extending human societies into virtual space through the construction of a metaverse has been a long-term challenge in both industry and academia. Achieving this challenge is now closer than ever due to advances in computer systems, facilitating large-scale online platforms such as Minecraft and Roblox that fulfill an increasing number of societal needs, and extended reality (XR) hardware, which provides users with state-of-the-art immersive experiences. For a metaverse to succeed, we argue that all involved systems must provide consistently good performance. However, there is a lack of knowledge on the performance characteristics of extended reality devices. In this paper, we address this gap and focus on extended- and virtual-reality hardware. We synthesize a user-centered system model that models common deployments of XR hardware and their trade-offs. Based on this model, we design and conduct real-world experiments with Meta’s flagship virtual reality device, the Quest Pro. We highlight two surprising results from our findings which show that (i) under our workload, the battery drains 15% faster when using wireless offloading compared to local execution, and (ii) the outdated 2.4 GHz WiFi4 gives surprisingly good performance, with 99% of samples achieving a frame rate of at least 65 Hz, compared to the 72 Hz performance target. Our experimental setup and data are available at https://github.com/atlarge-research/measuring-the-metaverse.

   @inproceedings{DBLP:conf/wosp/JansenDTI23,
     author = {Jansen, Matthijs and Donkervliet, Jesse and Trivedi, Animesh and Iosup, Alexandru},
     editor = {Vieira, Marco and Cardellini, Valeria and Marco, Antinisca Di and Tuma, Petr},
     title = {Can My WiFi Handle the Metaverse? {A} Performance Evaluation Of Meta's
                      Flagship Virtual Reality Hardware},
     booktitle = {Companion of the 2023 {ACM/SPEC} International Conference on Performance
                      Engineering, {ICPE} 2023, Coimbra, Portugal, April 15-19, 2023},
     pages = {297--303},
     publisher = {{ACM}},
     year = {2023},
     url = {https://doi.org/10.1145/3578245.3585022},
     doi = {10.1145/3578245.3585022},
     timestamp = {Sat, 29 Apr 2023 19:25:35 +0200},
     biburl = {https://dblp.org/rec/conf/wosp/JansenDTI23.bib},
     bibsource = {dblp computer science bibliography, https://dblp.org}
   }

2. 

   Continuum: Automate Infrastructure Deployment and Benchmarking in the Compute Continuum

   Matthijs Jansen, Linus Wagner, Animesh Trivedi, and Alexandru Iosup

   In Companion of the 2023 ACM/SPEC International Conference on Performance Engineering, ICPE 2023, Coimbra, Portugal, April 15-19, 2023, 2023

   Abs Bib PDF Code Slides

   As the next generation of diverse workloads like autonomous driving and augmented/virtual reality evolves, computation is shifting from cloud-based services to the edge, leading to the emergence of a cloud-edge compute continuum. This continuum promises a wide spectrum of deployment opportunities for workloads that can leverage the strengths of cloud (scalable infrastructure, high reliability), edge (energy efficient, low latencies), and endpoints (sensing, user-owned). Designing and deploying software in the continuum is complex because of the variety of available hardware, each with unique properties and trade-offs. In practice, developers have limited access to these resources, limiting their ability to create software deployments. To simplify research and development in the compute continuum, in this paper, we propose Continuum, a framework for automated infrastructure deployment and benchmarking that helps researchers and engineers to deploy and test their use cases in a few lines of code. Continuum can automatically deploy a wide variety of emulated infrastructures and networks locally and in the cloud, install software for operating services and resource managers, and deploy and benchmark applications for users with diverse configuration options. In our evaluation, we show how our design covers these requirements, allowing Continuum to be (i) highly flexible, supporting any computing model, (ii) highly configurable, allowing users to alter framework components using an intuitive API, and (iii) highly extendable, allowing users to add support for more infrastructure, applications, and more. Continuum is available at https://github.com/atlarge-research/continuum.

   @inproceedings{DBLP:conf/wosp/JansenWTI23,
     author = {Jansen, Matthijs and Wagner, Linus and Trivedi, Animesh and Iosup, Alexandru},
     editor = {Vieira, Marco and Cardellini, Valeria and Marco, Antinisca Di and Tuma, Petr},
     title = {Continuum: Automate Infrastructure Deployment and Benchmarking in
                      the Compute Continuum},
     booktitle = {Companion of the 2023 {ACM/SPEC} International Conference on Performance
                      Engineering, {ICPE} 2023, Coimbra, Portugal, April 15-19, 2023},
     pages = {181--188},
     publisher = {{ACM}},
     year = {2023},
     url = {https://doi.org/10.1145/3578245.3584936},
     doi = {10.1145/3578245.3584936},
     timestamp = {Sat, 29 Apr 2023 19:25:34 +0200},
     biburl = {https://dblp.org/rec/conf/wosp/JansenWTI23.bib},
     bibsource = {dblp computer science bibliography, https://dblp.org}
   }

3. 

   The SPEC-RG Reference Architecture for The Compute Continuum

   Matthijs Jansen, Auday Al-Dulaimy, Alessandro V. Papadopoulos, Animesh Trivedi, and Alexandru Iosup

   In 23rd IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing, CCGrid 2023, Bangalore, India, May 1-4, 2023, 2023

   Abs Bib PDF Code Slides

   As the next generation of diverse workloads like autonomous driving and augmented/virtual reality evolves, computation is shifting from cloud-based services to the edge, leading to the emergence of a cloud-edge compute continuum. This continuum promises a wide spectrum of deployment opportunities for workloads that can leverage the strengths of cloud (scalable infrastructure, high reliability) and edge (energy efficient, low latencies). Despite its promises, the continuum has only been studied in silos of various computing models, thus lacking strong end-to-end theoretical and engineering foundations for computing and resource management across the continuum. Consequently, devel-opers resort to ad hoc approaches to reason about performance and resource utilization of workloads in the continuum. In this work, we conduct a first-of-its-kind systematic study of various computing models, identify salient properties, and make a case to unify them under a compute continuum reference architecture. This architecture provides an end-to-end analysis framework for developers to reason about resource management, workload distribution, and performance analysis. We demonstrate the utility of the reference architecture by analyzing two popular continuum workloads, deep learning and industrial IoT. We have developed an accompanying deployment and benchmarking framework and first-order analytical model for quantitative reasoning of continuum workloads. The framework is open-sourced and available at https://github.com/atlarge-research/continuum.

   @inproceedings{DBLP:conf/ccgrid/JansenAPTI23,
     author = {Jansen, Matthijs and Al{-}Dulaimy, Auday and Papadopoulos, Alessandro V. and Trivedi, Animesh and Iosup, Alexandru},
     editor = {Simmhan, Yogesh and Altintas, Ilkay and Varbanescu, Ana Lucia and Balaji, Pavan and Prasad, Abhinandan S. and Carnevale, Lorenzo},
     title = {The {SPEC-RG} Reference Architecture for The Compute Continuum},
     booktitle = {23rd {IEEE/ACM} International Symposium on Cluster, Cloud and Internet
                      Computing, CCGrid 2023, Bangalore, India, May 1-4, 2023},
     pages = {469--484},
     publisher = {{IEEE}},
     year = {2023},
     url = {https://doi.org/10.1109/CCGrid57682.2023.00051},
     doi = {10.1109/CCGrid57682.2023.00051},
     timestamp = {Fri, 21 Jul 2023 22:25:52 +0200},
     biburl = {https://dblp.org/rec/conf/ccgrid/JansenAPTI23.bib},
     bibsource = {dblp computer science bibliography, https://dblp.org}
   }

4. 

   Beyond von Neumann in the Computing Continuum: Architectures, Applications, and Future Directions

   Dragi Kimovski, Nishant Saurabh, Matthijs Jansen, Atakan Aral, Auday Al-Dulaimy, André B. Bondi, Antonino Galletta, Alessandro V. Papadopoulos, Alexandru Iosup, and Radu Prodan

   IEEE Internet Computing, 2023

   Abs Bib PDF

   The article discusses the emerging non-von Neumann computer architectures and their integration in the computing continuum for supporting modern distributed applications, including artificial intelligence, big data, and scientific computing. It provides a detailed summary of the available and emerging non-von Neumann architectures, which range from power-efficient single-board accelerators to quantum and neuromorphic computers. Furthermore, it explores their potential benefits for revolutionizing data processing and analysis in various societal, science, and industry fields. The paper provides a detailed analysis of the most widely used class of distributed applications and discusses the difficulties in their execution over the computing continuum, including communication, interoperability, orchestration, and sustainability issues.

   @article{10207712,
     author = {Kimovski, Dragi and Saurabh, Nishant and Jansen, Matthijs and Aral, Atakan and Al-Dulaimy, Auday and Bondi, André B. and Galletta, Antonino and Papadopoulos, Alessandro V. and Iosup, Alexandru and Prodan, Radu},
     journal = {IEEE Internet Computing},
     title = {Beyond von Neumann in the Computing Continuum: Architectures, Applications, and Future Directions},
     year = {2023},
     volume = {},
     number = {},
     pages = {1-11},
     doi = {10.1109/MIC.2023.3301010}
   }

2021

1. 

   GradeML: Towards Holistic Performance Analysis for Machine Learning Workflows

   Tim Hegeman, Matthijs Jansen, Alexandru Iosup, and Animesh Trivedi

   In ICPE ’21: ACM/SPEC International Conference on Performance Engineering, Virtual Event, France, April 19-21, 2021, Companion Volume, 2021

   Abs Bib PDF Code

   Today, machine learning (ML) workloads are nearly ubiquitous. Over the past decade, much effort has been put into making ML model-training fast and efficient, e.g., by proposing new ML frameworks (such as TensorFlow, PyTorch), leveraging hardware support (TPUs, GPUs, FPGAs), and implementing new execution models (pipelines, distributed training). Matching this trend, considerable effort has also been put into performance analysis tools focusing on ML model-training. However, as we identify in this work, ML model training rarely happens in isolation and is instead one step in a larger ML workflow. Therefore, it is surprising that there exists no performance analysis tool that covers the entire life-cycle of ML workflows. Addressing this large conceptual gap, we envision in this work a holistic performance analysis tool for ML workflows. We analyze the state-of-practice and the state-of-the-art, presenting quantitative evidence about the performance of existing performance tools. We formulate our vision for holistic performance analysis of ML workflows along four design pillars: a unified execution model, lightweight collection of performance data, efficient data aggregation and presentation, and close integration in ML systems. Finally, we propose first steps towards implementing our vision as GradeML, a holistic performance analysis tool for ML workflows. Our preliminary work and experiments are open source at https://github.com/atlarge-research/grademl.

   @inproceedings{DBLP:conf/wosp/HegemanJIT21,
     author = {Hegeman, Tim and Jansen, Matthijs and Iosup, Alexandru and Trivedi, Animesh},
     editor = {Bourcier, Johann and Jiang, Zhen Ming (Jack) and Bezemer, Cor{-}Paul and Cortellessa, Vittorio and Pompeo, Daniele Di and Varbanescu, Ana Lucia},
     title = {GradeML: Towards Holistic Performance Analysis for Machine Learning
                      Workflows},
     booktitle = {{ICPE} '21: {ACM/SPEC} International Conference on Performance Engineering,
                      Virtual Event, France, April 19-21, 2021, Companion Volume},
     pages = {57--63},
     publisher = {{ACM}},
     year = {2021},
     url = {https://doi.org/10.1145/3447545.3451185},
     doi = {10.1145/3447545.3451185},
     timestamp = {Sun, 02 Oct 2022 16:17:46 +0200},
     biburl = {https://dblp.org/rec/conf/wosp/HegemanJIT21.bib},
     bibsource = {dblp computer science bibliography, https://dblp.org}
   }

2020

1. 

   DDLBench: Towards a Scalable Benchmarking Infrastructure for Distributed Deep Learning

   Matthijs Jansen, Valeriu Codreanu, and Ana Lucia Varbanescu

   In Deep Learning on Supercomputers, 2020

   Abs Bib PDF Code Slides

   Due to its many applications across various fields of research, engineering, and daily life, deep learning has seen a surge in popularity. Therefore, larger and more expressive models have been proposed, with examples like Turing-NLG using as many as 17 billion parameters. Training these very large models becomes increasingly difficult due to the high computational costs and large memory footprint. Therefore, several approaches for distributed training based on data parallelism (e.g., Horovod) and model/pipeline parallelism (e.g., GPipe, PipeDream) have emerged. In this work, we focus on an in-depth comparison of three different parallelism models that address these needs: data, model and pipeline parallelism. To this end, we provide an analytical comparison of the three, both in terms of computation time and memory usage, and introduce DDLBench, a comprehensive (open-source, ready-to-use) benchmark suite to quantify these differences in practice. Through in-depth performance analysis and experimentation with various models, datasets, distribution models and hardware systems, we demonstrate that DDLBench can accurately quantify the capability of a given system to perform distributed deep learning (DDL). By comparing our analytical models with the benchmarking results, we show how the performance of real-life implementations diverges from these analytical models, thus requiring benchmarking to capture the in-depth complexity of the frameworks themselves.

   @inproceedings{DBLP:conf/sc/JansenCV20,
     author = {Jansen, Matthijs and Codreanu, Valeriu and Varbanescu, Ana Lucia},
     title = {DDLBench: Towards a Scalable Benchmarking Infrastructure for Distributed Deep Learning},
     booktitle = {Deep Learning on Supercomputers},
     pages = {31--39},
     publisher = {{IEEE}},
     year = {2020},
     url = {https://doi.org/10.1109/DLS51937.2020.00009},
     doi = {10.1109/DLS51937.2020.00009},
     biburl = {https://dblp.org/rec/conf/sc/JansenCV20.bib},
     bibsource = {dblp computer science bibliography, https://dblp.org}
   }