\section{Introduction}
\label{sec:introduction}

Emerging applications such as \acp{llm} revolutionize modern computing and fundamentally change how we interact with computing systems.
A key component of these models is the use of \acp{dnn}, which are a type of machine learning model inspired by the structure of the human brain - composed of multiple layers of interconnected nodes that mimic a network of neurons, \acp{dnn} are utilized to perform various tasks such as image recognition or natural language and speech processing.
Consequently, \acp{dnn} make it possible to tackle many new classes of problems that were previously beyond the reach of conventional algorithms.

However, the ever-increasing use of these technologies poses new challenges for hardware architectures, as the energy required to train and run these models reaches unprecedented levels.
Recently published numbers approximate that the development and training of Meta's LLaMA model over a period of about five months consumed around $\qty{2638}{\mega\watt\hour}$ of electrical energy and caused a total emission of $\qty{1015}{tCO_2eq}$ \cite{touvron2023}.
As these numbers are expected to increase in the future, it is clear that the energy footprint of current deployment of \ac{ai} applications is not sustainable \cite{blott2023}.


In a more general view, the energy demand of computing for new applications continues to grow exponentially, doubling about every two years, while the world's energy production only grows linearly, at about $\qty{2}{\percent}$ per year \cite{src2021}.
This dramatic increase in energy consumption is due to the fact that while the energy efficiency of compute processor units has continued to improve, the ever-increasing demand for computing however is outpacing this progress.
In addition, Moore's Law is slowing down as further device scaling approaches physical limits.

\begin{figure}[!ht]
	\centering
	\input{plots/energy_chart}
	\caption[Total energy of computing.]{Total energy of computing \cite{src2021}.}
	\label{plt:enery_chart}
\end{figure}

The exponential grow in compute energy will eventually be constrained by market dynamics, flattening the energy curve and making it impossible to meet future computing demands.
It is therefore required to achieve radical improvements in energy efficiency in order to avoid such a scenario.

In recent years, domain-specific accelerators, such as \acp{gpu} or \acp{tpu} have become very popular, as they provide orders of magnitude higher performance and energy efficiency for \ac{ai} applications  than general-purpose processors \cite{kwon2021}.
However, research must also take into account off-chip memory - moving data between the computation unit and the \ac{dram} is very costly, as fetching operands consumes more power than performing the computation on them itself.
While performing a double precision floating point operation on a $\qty{28}{\nano\meter}$ technology might consume an energy of about $\qty{20}{\pico\joule}$, fetching the operands from \ac{dram} consumes almost 3 orders of magnitude more energy at about $\qty{16}{\nano\joule}$ \cite{dally2010}.

Furthermore, many types of \acp{dnn} used for language and speech processing, such as \acp{rnn}, \acp{mlp} and some layers of \acp{cnn}, are severely limited by the memory bandwidth that the \ac{dram} can provide, making them \textit{memory-bound} \cite{he2020}.
In contrast, compute-intensive workloads, such as visual processing, are referred to as \textit{compute-bound}.

\begin{figure}[!ht]
	\centering
	\input{plots/roofline}
	\caption[Roofline model of GPT revisions.]{Roofline model of GPT revisions \cite{ivobolsens2023}.}
	\label{plt:roofline}
\end{figure}

In the past, specialized types of \ac{dram} such as \ac{hbm} have been able to meet the high bandwidth requirements.
However, recent \ac{ai} technologies require even greater bandwidth than \ac{hbm} can provide \cite{kwon2021}.

All things considered, to meet the need for more energy-efficient computing systems, which are increasingly becoming memory-bound, new approaches to computing are required.
This has led researchers to reconsider past \ac{pim} architectures and advance them further \cite{lee2021}.
\Ac{pim} integrates computational logic into the \ac{dram} itself, to exploit minimal data movement cost and extensive internal data parallelism \cite{sudarshan2022}, making it a good fit for memory-bound problems.

This work analyzes various \ac{pim} architectures, identifies the challenges of integrating them into state-of-the-art \acp{dram}, examines the changes required in the way applications lay out their data in memory and explores a \ac{pim} implementation from one of the leading \ac{dram} vendors.
The remainder of this work is structured as follows:
\cref{sec:dram} gives a brief overview of the architecture of \acp{dram}, in detail that of \ac{hbm}.
In \cref{sec:pim} various types of \ac{pim} architectures are presented, with some concrete examples discussed in detail.
\cref{sec:vp} is an introduction to virtual prototyping and system-level hardware simulation.
After explaining the necessary prerequisites, \cref{sec:implementation} implements a concrete \ac{pim} architecture in software and provides a development library that applications can use to take advantage of in-memory processing.
The \cref{sec:results} demonstrates the possible performance enhancement of \ac{pim} by simulating a typical neural network inference.
Finally, \cref{sec:conclusion} concludes the findings and identifies future improvements in \ac{pim} architectures. 
