de.uni-kl.ems.dram.vp.system
============================

**DRAMSys** [1] is a flexible DRAM subsystem design space exploration
framework that consists of models reflecting the DRAM functionality, power
consumption, temperature behaviour and retention time errors.

## Basic Setup

Open a terminal window, go to your home directory, create a directory for your
projects and change to it.

``` bash
$ cd
$ mkdir projects
$ cd projects
```

Configure git on your machine. Some basic configurations follow. Replace
**rhrkuser** with your own RHRK user when configuring your email.

``` bash
$ git config --global user.name "FirstName OtherNames LastName"
$ git config --global user.email rhrkuser@rhrk.uni-kl.de
$ git config --global credential.helper ’cache --timeout=3600’
$ git config --global color.ui auto 
```

Login using your **git.rhrk** account. Fork the repository by clicking in
**Fork** in the upper-right corner of this page.

Now clone **your fork** of the official repository into a local folder on your
computer. Replace the occurrences of the word **user** with your own RHRK user
name. **Note that it appears twice in the line below**.

``` bash
$ git clone --recursive https://user@git.rhrk.uni-kl.de/user/dram.vp.system.git
```

The *--recursive* flag tells git to initialize all submodules within the
repository. **DRAMPower** [2] and **tinyxml** are examples third party
repositories that were embedded within the source tree as submodules.

Now you can implement, test, commit and push features into your **fork** of
the official repository.

When you consider your work stable enough to be merged into the official
repository it is time to open a **pull request** using the web interface of
git.rhrk.uni-kl.de.

Your changes will be reviewed and finally integrated to the official
repository.

After cloning go to the project directory.

``` bash
$ cd dram.vp.system
```

When working with a fork, the official repository must be added as a remote for
your fork. Replace the word **user** with your own RHRK user name.

``` bash
$ git remote add upstream https://user@git.rhrk.uni-kl.de/EIT-Wehn/dram.vp.system.git
```

In general, from time to time you should update your fork in order to keep it
synchronized with the official repository. It is important to keep track of
the evolution of the official repository and avoid a huge divergence.

To get the latest changes from the official repository merged into your fork
you can use the commands that follow:

``` bash
$ git fetch upstream
$ git checkout master
$ git merge upstream/master
$ git push origin HEAD
$ git submodule update --init --recursive
```

### Dependencies

Make sure you have properly installed in your system the required libraries
and programs.

- **General dependencies**

You can use [utils/install.sh](./utils/install.sh) in order to install dependencies. First
read and understand the script then execute it. Type your password if
required.

```bash
$ bash install.sh
```

- **SystemC 2.3.1 and TLM 2.0**

You can use [utils/getsysc.sh](./utils/getsysc.sh) to download and install SystemC 2.3.1
and TLM 2.0. First read and understand the script then execute it.

``` bash
$ bash getsysc.sh
```

Alternatively, the sources can be downloaded from
[here](http://accellera.org/downloads/standards/systemc). For installation
instructions see the installation notes file contained in the release package.

- **qwt-6.1**

You can use [utils/getqwt.sh](./utils/getqwt.sh) in order to install qwt-6.1. First read
and understand the script then execute it.

```bash
$ bash getqwt.sh
$ cd ~/qwt-6.1
$ sudo make install
```

For further information refer to [this](http://qwt.sourceforge.net/)

To grant flexibility to the user the paths where to find some essential
libraries and headers can be specified with environment variables. Make sure
you have the environment variables below in your ~/.bashrc file.

Note that some of the variables are automatically generated by the scripts. If
you install the libraries in custom paths in your system you have to adapt the
environment variables accordingly.

```bash
# SystemC home and target architecture
export SYSTEMC_HOME=$HOME/systemc-2.3.1a
export SYSTEMC_TARGET_ARCH=linux64

# DRAMSys libraries and headers
export PYTHON_HOME=/usr/lib/python3.5
export PYTHON_HEADERS=/usr/include/python3.5m

export LIBQWT_HOME=/usr/local/qwt-6.1.4-svn/lib
export LIBQWT_HEADERS=/usr/local/qwt-6.1.4-svn/include

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH:+${LD_LIBRARY_PATH}:}$LIBQWT_HOME
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${SYSTEMC_HOME}/lib-$SYSTEMC_TARGET_ARCH
```

### Buiding with QTCreator
Execute the *QTCreator*.

``` bash
$ qtcreator &
```

Use the menu bar and open the DRAMSys project.

**File -> Open Project -> dram.vp.system/DRAMSys/DRAMSys.pro**

When you open the project for the first time a configuration window pops-up.
Then click in **Configure Project** and after that **Build** the project.

To speedup the building process one can use the additional **make** option
**-j[jobs]**. The command line below returns a good number to be passed to
make as the number of jobs that can run simultaneously to improve the building
time.

``` bash
$ cat /proc/cpuinfo | grep processor | wc -l
```

In the left bar go to **Projects -> Build & Run -> Build Steps -> Make**.
Click in **Details** then **Make arguments** and add **-j** followed by the
number you got.

In case you face a problem related to the **Qt version** double check the **Qt
version** configuration shown in the image below.

![Qt Creator Configuration](DRAMSys/docs/images/QtCreatorConfig.png)

### Building without QTCreator

In case you prefer a command line interface to the QTCreator GUI you can also
use **qmake** to generate a Makefile and then compile the project.

``` bash
$ mkdir build
$ cd build
$ qmake ../DRAMSys/DRAMSys.pro
$ make -j4
```

The compilation generates executable binary files **DRAMSys** and
**traceAnalyzer** that can be found inside sub-directories.

From the build directory use the commands below to execute DRAMSys.

``` bash
$ cd simulator
$ ./DRAMSys
```

From the build directory use the commands below to execute the traceAnalyzer.

``` bash
$ cd traceAnalyzer
$ export QT_QPA_PLATFORMTHEME=qgnomeplatform
$ ./traceAnalyzer
```

### Building on OSX
- Install XCode

- Install SystemC manually in /opt:

``` bash
$ ./configure --prefix=/opt/systemc
$ make
$ sudo make install
```

Or you can install via homebrew:

``` bash
$ brew install systemc
```

in this case, systemc will be installed, e.g. in:
```
/usr/local/Cellar/systemc/2.3.1
```
and make a simlink of the lib directory:
```
ln -s lib/ lib-macosx64
```


- Install the required python 3 over homebrew:

``` bash
$ brew install python3
```

Python3 (via homebrew) will be installed in 
```
/usr/local/Cellar/python3/3.5.2_2/Frameworks/Python.framework
```

or you can install manually using official package provided in [link](https://www.python.org/downloads/)

**Note:** Official Python Package will be installed in 
```
/Library/Frameworks/Python.framework
``` 

- Install the QtCreator using offical setup file from [link](https://www.qt.io/download-open-source/#section-2)

**Note:** You have later setup PATH for Qt5 and its tool if you install QtCreator manually, e.g:

``` bash
# Setting PATH for Qt5 and its tools
PATH="/Users/<username>/Qt5.7.0/5.7/clang_64/bin:${PATH}"
export PATH
```

- Install the QWT manually to /opt/qwt, then do:

``` bash
$ cd /Library/Frameworks
$ sudo ln -s /opt/qwt-6.1.2/lib/qwt.framework/ .
```
- Export correct Environment Variables in your terminal's profile, e.g:

``` bash
# Setting for DRAMSys
# SystemC via official source
export SYSTEMC_HOME=/opt/systemc
export SYSTEMC_TARGET_ARCH=macosx64
# SystemC via Homebrew
#export SYSTEMC_HOME=/usr/local/Cellar/systemc/2.3.1
#export SYSTEMC_TARGET_ARCH=macosx64

# Python via official pkg
export PYTHON_HOME=/Library/Frameworks/Python.framework/Versions/3.5/lib
export PYTHON_HEADERS=/Library/Frameworks/Python.framework/Versions/3.5/Headers

# Python3 via Homebrew
#export PYTHON_HOME=/usr/local/Cellar/python3/3.5.2_2/Frameworks/Python.framework/Versions/3.5/lib
#export PYTHON_HEADERS=/usr/local/Cellar/python3/3.5.2_2/Frameworks/Python.framework/Versions/3.5/Headers

export LIBQWT_HOME=/opt/qwt-6.1.4/lib
export LIBQWT_HEADERS=/opt/qwt-6.1.4/lib/qwt.framework/Headers
```

-  For the trace analyzer the file:
```
/opt/local/Library/Frameworks/Python.framework/Versions/3.5/include/python3.4m/pyport.h 
```
has to be changed like [this](https://trac.macports.org/attachment/ticket/44288/issue10910-workaround.txt)

- Install package [xerces](http://xerces.apache.org/mirrors.cgi) if your system does not have.

- Type following command inside your dram.vp.system folder:

``` bash
$ mkdir build
$ qmake ../DRAMSys/DRAMSys.pro
$ make -j<number_jobs>
```

Now you can try to run DRAMSys and traceAnalyzer App inside folder simulator and traceAnalyzer, respectively

### DRAMSys Configuration

The **DRAMSys** executable supports one argument which is a XML file that
contains configurable aspects of the desired simulation. If no argument is
passed through the command line a default configuration file will be loaded.

The XML code below shows a typic configuration:

``` xml
<simulation>
    <!-- Configuration for the DRAMSys Simulator -->
    <simconfig src="ddr3.xml" />
    <!-- Temperature Simulator Configuration -->
    <thermalconfig src="config.xml" />
    <!-- Memory Device Specification: Which Device is on the DDR3 DIMM -->
    <memspec src="MICRON_1Gb_DDR3-1600_8bit_G.xml"></memspec>
    <!-- Addressmapping Configuration of the Memory Controller -->
    <addressmapping src="am_ddr3_8x1Gbx8_dimm_p1KB_brc.xml"></addressmapping>
    <!-- Memory Controller Configuration: -->
    <mcconfig src="fifoStrict.xml"/>
    <!--
    The following trace setup is only used in standalone mode.
    In library mode e.g. in Platform Architect the trace setup is ignored.
    -->
    <tracesetup>
        <!--
        Specify here a trace file for each of the trace players. Trace
        players without a file will not generate transactions.
        -->
        <device clkMhz="200">ddr3_example.stl</device>
        <device clkMhz="100">ddr3_SAMSUNG_M471B5674QH0_DIMM_example.stl</device>
    </tracesetup>
</simulation>
```
Some configuration fields reference other XML files which contain more
specialized chunks of the configuration like memory specification, address
mapping and memory configurations.

The XML configuration files are parsed by the program and the configuration
details extracted are assigned to the correspondent attributes of the internal
configuration structure.

The **device** configuration consists of two parameters - clkMhz
(operation frequency for this device) - and a **trace file**.

A **trace file** is a pre-recorded file containing memory transactions. Each
memory transaction has a timestamp that tells the simulator when it shall
happen, a transaction type (read or write) and a memory address given in
hexadecimal.

Here is an example syntax:

```
# Comment lines begin with #
# [clock-cyle]: [write|read] [hex-address]
31:	read	0x400140
33:	read	0x400160
56:	write	0x7fff8000
81:	read	0x400180
```

The timestamp corresponds to the time the request is to be issued and it is
given in cycles of the bus master device. Example: the device is a FPGA with
frequency 200 MHz (clock period of 5 ns). If the timestamp is 10 it means that
the request is to be issued when time is 50 ns.

A **trace player** is **equivalent** to a bus master **device**
(processor, FPGA, etc.). It reads an input trace file and translates each line into
a new memory request. By adding a new device element into the trace setup section
one can specify a new trace player, its operating frequency and the trace file
for that trace player.

#### Configuration File Sections

The main configuration file is divided into self-contained sections. Each of
these sections refers to sub-configuration files.

Below, the sub-configurations are listed and explained.

- **Simulator Configuration**

  The content of
  [ddr3.xml](DRAMSys/library/resources/configs/simulator/ddr3.xml) is
  presented below as an example.

  ```xml
  <simconfig>
      <SimulationName value="ddr3" />
      <Debug value="0" />
      <DatabaseRecording value="1" />
      <PowerAnalysis value="1" />
      <EnableWindowing value = "1" />
      <WindowSize value="100" />
      <ThermalSimulation value="0"/>
      <SimulationProgressBar value="1"/>
      <NumberOfMemChannels value="1"/>
      <NumberOfDevicesOnDIMM value = "8" />
      <ECCControllerMode value = "Disabled" />
  </simconfig>
  ```

  - *SimulationName* (boolean)
    - Give the name of the simulation for distingushing from other simulations.
  - *Debug* (boolean)
    - "1": enables debug output on console
    - "0": disables debug output
  - *DatabaseRecording* (boolean)
    - "1": enables trace file recording for the trace analyser tool
    - "0": disables trace file recording
  - *PowerAnalysis* (boolean)
    - "1": enables live power analysis with the DRAMPower tool
    - "0": disables power analysis
  - *EnableWindowing* (boolean)
    - "1": enables temporal windowing
    - "0": disables temporal windowing
  - *WindowSize* (unisgned int)
    - Size of the window in clock cycles used to evaluate average bandwidth and average power consumption
  - *NumberOfMemChannels* (unsigned int)
    - Number of memory channels
  - *ThermalSimulation* (boolean)
    - "1": enables thermal simulation
    - "0": static temperature during simulation
  - *SimulationProgressBar* (boolean)
    - "1": enables the simulation progress bar 
    - "0": disables the simulation progress bar
  - *NumberOfDevicesOnDIMM* (unsigned int)
    - Number of devices on dual inline memory module
  - *CheckTLM2Protocol* (boolean)
    - "1": enables the TLM 2.0 Protocol Checking 
    - "0": disables the TLM 2.0 Protocol Checking 
  - *ECCControllerMode* (string)
    - "Disabled": No ECC Controller is used
    - "Hamming": Enables an ECC Controller with classic SECDED implementation using Hamming Code

- **Temperature Simulator Configuration**

  The content of
  [config.xml](DRAMSys/library/resources/configs/thermalsim/config.xml) is
  presented below as an example.

  ```xml
  <thermalsimconfig>
      <TemperatureScale value="Celsius" />
      <StaticTemperatureDefaultValue value="89" />
      <ThermalSimPeriod value="100" />
      <ThermalSimUnit value="us" />
      <PowerInfoFile value="../../DRAMSys/library/resources/configs/thermalsim/powerInfo.xml"/>
      <IceServerIp value="127.0.0.1" />
      <IceServerPort value="11880" />
      <SimPeriodAdjustFactor value="10" />
      <NPowStableCyclesToIncreasePeriod value="5" />
      <GenerateTemperatureMap value="1" />
      <GeneratePowerMap value="1" />
  </thermalsimconfig>
  ```

  - *TemperatureScale* (string)
    - "Celsius"
    - "Fahrenheit"
    - "Kelvin"
  - *StaticTemperatureDefaultValue* (int)
    - Temperature value for simulations with static temperature
  - *ThermalSimPeriod* (double)
    - Period of the thermal simulation
  - *ThermalSimUnit* (string)
    - "s": seconds
    - "ms": millisecond
    - "us": microseconds
    - "ns": nanoseconds
    - "ps": picoseconds
    - "fs": femtoseconds
  - *PowerInfoFile* (string)
    - File containing power related information: devices identifiers, initial power values and power thresholds.
  - *IceServerIp* (string)
    - 3D-Ice server IP address
  - *IceServerPort* (unsigned int)
    - 3D-Ice server port
  - *SimPeriodAdjustFactor* (unsigned int)
    - When substantial changes in power occur (i.e., changes that exceed the thresholds), then the simulation period will be divided by this number causing the thermal simulation to be executed more often.
  - *NPowStableCyclesToIncreasePeriod* (unsigned int)
    - Wait this number of thermal simulation cycles with power stability (i.e., changes that do not exceed the thresholds) to start increasing the simulation period back to its configured value.
  - *GenerateTemperatureMap* (boolean)
    - "1": generate temperature map files during thermal simulation
    - "0": do not generate temperature map files during thermal simulation
  - *GeneratePowerMap* (boolean)
    - "1": generate power map files during thermal simulation
    - "0": do not generate power map files during thermal simulation

- **Memory Specification**

  A file with memory specifications. This information comes from datasheets and
  measurements, and usually does not change.

- **Address Mapping**

  XML files describe the address mapping to be used in the simulation.

  Example for 1GB x64 DIMM with: 8 x 1 Gbit x8 Devices (Micron MT41J128M8) with Page Size: 1KB

  [am_ddr3_8x1Gbx8_dimm_p1KB_brc.xml](DRAMSys/library/resources/configs/amconfigs/am_ddr3_8x1Gbx8_dimm_p1KB_brc.xml)

  ``` xml
  <!-- 
  DDR3 Example:
  1GB x64 DIMM with: 8 * 1 Gb x8 Devices (e.g. Micron MT41J128M8) with Page Size: 1KB
  
  Device Characteristics:
  
  Rows: 16 K [13:0]  -> 14 bit
  Bank: 8    [2:0]   -> 3  bit
  Cols: 1 K  [9:0]   -> 10 bit
  
  Due to the DIMM we have a Byte Offset Y
         
  2 2 2 | 2 2 2 2 2 2 2 1 1 1 1 1 1 1 | 1 1 1 
  9 8 7 | 6 5 4 3 2 1 0 9 8 7 6 5 4 3 | 2 1 0 9 8 7 6 5 4 3 | 2 1 0
  B B B | R R R R R R R R R R R R R R | C C C C C C C C C C | Y Y Y
  -->
  
  <addressmapping>
      <channel from="128"  to="128"  /> <!-- only one channel -->
      <bank    from="27"   to="29"   />
      <row     from="13"   to="26"   />
      <column  from="3"    to="12"   />
      <bytes   from="0"    to="2"    />
  </addressmapping>
  ```

  Some more examples with graphical representation follow:
  
  [am_wideio.xml](DRAMSys/library/resources/configs/amconfigs/am_wideio.xml)

  ``` xml
  <!-- Row Bank Column -->
  
  <addressmapping>
      <channel from="27" to="28" />
      <row   from="14" to="26" />		
      <bank  from="11" to="13" />
      <column from="4" to="10" />
      <bytes from="0" to="3" /> 
  </addressmapping>
  ```

  ![Address Mapping Sample 1](DRAMSys/docs/images/am_wideio_rbc.png)

  ``` xml
  <!-- Bank Row Column -->

  <addressmapping>
      <channel from="27" to="28" />
      <bank  from="24" to="26" />
      <row   from="11" to="23" />
      <column from="4" to="10" />
      <bytes from="0" to="3" />
  </addressmapping>
  ```

  ![Address Mapping Sample 2](DRAMSys/docs/images/am_wideio_brc.png)

- **Memory Configuration**

  The content of [fifo.xml](DRAMSys/library/resources/configs/memconfigs/fifo.xml) is
  presented below as an example.

  ``` xml
  <mcconfig>
      <BankwiseLogic value="0"/>
      <OpenPagePolicy value="1" />
      <MaxNrOfTransactions value="8" />
      <Scheduler value="FIFO" />
      <Capsize value="5" />
      <!-- 4 Modes: NoPowerDown, Staggered, TimeoutPDN, TimeoutSREF -->
      <PowerDownMode value="NoPowerDown" />
      <PowerDownTimeout value="100" />
      <!-- Error Modelling -->
      <ErrorChipSeed value="42" />
      <ErrorCSVFile value="../../DRAMSys/library/src/error/error.csv" />
      <!-- Modes:
          - NoStorage,
          - Store (store data without errormodel),
          - ErrorModel (store data with errormodel)
      -->
      <StoreMode value="NoStorage" />
      <ControllerCoreDisableRefresh value="0"/>
  </mcconfig>
  ```

  - *BankwiseLogic* (boolean)
    - "1": perform bankwise-refresh [3] and bankwise-powerdown [4]
    - "0": do not perform bankwise operations
  - *OpenPagePolicy* (boolean)
    - "1": use open page precharge policy
    - "0": do not use open page precharge policy
  - *MaxNrOfTransactions* (unsigned int)
    - Maximum number of transactions.
  - *Scheduler* (string)
    - "FIFO": first in, first out
    - "FIFO_STRICT": out-of-order treatment of queue elements not allowed
    - "FR_FCFS": first-come, first-served
  - *Capsize* (unsigned int)
    - Capacitor cell size.
  - *PowerDownMode* (enum EPowerDownMode)
    - "NoPowerDown": no power down mode (active idle)
    - "Staggered": staggered power down policy [5]
    - "TimeoutPDN": precharge idle
    - "TimeoutSREF": self refresh
  - *ReadWriteGrouping* (boolean)
    - "1": enable read writing grouping
    - "0": disable read writing grouping
  - *ReorderBuffer* (boolean)
    - "1": use reordering buffer
    - "0": do not use reordering buffer
  - *ErrorChipSeed* (unsigned int)
    - Seed to initialize the random error generator.
  - *ErrorCSVFile* (string)
    - CSV file with error injection information.
  - *StoreMode* (enum StorageMode)
    - "NoStorage": no storage
    - "Store": store data without error model
    - "ErrorModel": store data with error model [6]
  - *ControllerCoreDisableRefresh* (boolean)
    - "1": disables refreshes
    - "0": normal operation (refreshes enabled)
  - *ControllerCoreForceMaxRefBurst* (boolean)
    - "1": always postpone, resulting in a ControllerCoreMaxPostponedARCmd burst
    - "0": normal operation
  - *ControllerCoreEnableRefPostpone* (boolean)
    - "1": enables the postpone refresh feature
    - "0": normal operation
  - *ControllerCoreEnableRefPullIn* (boolean)
    - "1": enables the pull-in refresh feature
    - "0": normal operation
  - *ControllerCoreMaxPostponedARCmd* (unsigned int)
    - Max AR commands to be postponed.
  - *ControllerCoreMaxPulledInARCmd* (unsigned int)
    - Max AR commands to be pulled-in.

For further details on the Flexible Refresh feature, please refer to [this](DRAMSys/docs/flexible-refresh.pdf) document.

- **Trace Setups**
  - *clkMhz* (unsigned int)
    - Speed of the trace player
  - *trace file*
    - A pre-recorded file containing memory transactions to be executed by a
      trace player.

Some attributes are self-explanatory while others require some previous
knowhow of memory technologies.

Resources of the simulator are available inside of the **resources** directory
and its sub-directories.

``` bash
$ cd DRAMSys/library/resources
```

A description of the content each directory follows.

- **resources**
  - **configs**: XML files that specify details of the simulation.
    - amconfigs: address mapping configs.
    - mcconfigs: memory controller configs.
    - memspecs: memory specification files (technology dependent).
    - simulator: simulator configs.
  - **scripts**: useful tools.
  - **simulations**: main configuration files.
  - **traces**: pre-recorded trace files that may be used as stimuli in simulations.
 
#### Log Collector Script

Users can profit of running multiple simulations automatically with
[DRAMSylva](DRAMSys/library/resources/scripts/DRAMSylva/DRAMSylva.sh).

Every time you run the script you get a new folder with the name containing the execution time: dram.vp.system\_YYYY\_MM\_DD-HH.MM.SS.

Example on how to run the script:

``` bash
$ cd DRAMSys/library/resources/scripts/DRAMSylva
$ ./DRAMSylva.sh
```

To see the generated plots and CSV files:

``` bash
$ nautilus dram.vp.system_YYYY_MM_DD-HH.MM.SS/build/simulator
```

In that folder you will find plots as PDF files and CSV files with the output data used to generate the plots. The CSV files are:

- **output.csv** (energy, average power, bandwidth, etc.)

- **metrics.csv** (DRAMSys metrics like average response latency, memory utilization and many others)

The generated CSV files can be open in a spreadsheet program for further manipulation.

Additionally, the database files (\*.tdb) generated will be available and can be open with the traceAnalyzer tool.

A DRAMSys simulation is defined by the main configuration file passed to the simulator. The main configuration file includes other files which contain specifc configs.
You can change what is going to be simulated by the script by editing it. There is a list of main configuration files on the top of the script:

``` bash
sim_files="
../../DRAMSys/library/resources/simulations/ddr3-example.xml
../../DRAMSys/library/resources/simulations/ddr3-single-device.xml
../../DRAMSys/library/resources/simulations/wideio-example.xml
"
```

The script runs one instance of DRAMSys for each of the files in the list. **The multiple instances run in parallel**.

For more information check the documentation in [DRAMSylva folder](DRAMSys/library/resources/scripts/DRAMSylva).

#### DRAMsys Diagrams

- **TLM Approximately Timed (AT)**

  The figure below shows a cheat sheet with the possibilities that the TLM AT protocol
  offers. The annotated references [X,Y] are placed into the source code for a better
  orientation.

  ![TLM AT Cheat Sheet](DRAMSys/docs/images/tlmATCheatSheet.png)


- **Payload Extension information**

  GenerationExtension is added in TracePlayer and DramExtension is added in Arbiter.

  DramExtension indicates the decoded address (channel, bank, colums, row) and
  the socket id (thread) of a payload. It is added in the Arbiter and is sent
  to the Controller.
  ![Payload Extension information](DRAMSys/docs/images/PayloadExtension.png)
  
- **Transaction object with Memory Manager**
  
  The TracePlayer allocates the memory for the transaction object by calling allocatePayload method.
 
  The acquire method is called before passing the transaction object in TracePlayer, Arbiter and Controller.

  The release method is called after each component is done with the transaction object. After the final call of release method, the free method of the memory manager is called to free the transaction object.
  
  ![Payload Memory Manager](DRAMSys/docs/images/PayloadMemoryManager.png)

- **Architecture of the backend TLM model**

  The below figure shows our custom TLM protocol between the Controller and the Dram. A new transaction enters the Controller with the BEGIN_REQ phase is stored in frontendPEQ. The callback function of the frontendPEQ is called and send the payload to the Scheduler.
 
  The Scheduler checks the address of payload and the current state to determine proper command (Active, Precharge, Read or Write). Then the ControllerCore sends the payload with the corresponding phase (BEGIN_ACT, BEGIN_PRE, BEGIN_RD or BEGIN_WR) to the Dram by calling nb_transport_fw method.
  
  The Dram receives the transaction then send back to the Controller by calling nb_transport_bw with appropriate END phase (END_ACT, END_PRE, END_RD or END_WR). 

  ![Architecture backend TLM](DRAMSys/docs/images/TransactionPhase.png)

### DRAMSys Thermal Simulation 

The thermal simulation is performed by a **3D-ICE** [8] server accessed
through the network. Therefore users interested in thermal simulation during
their DRAMSys simulations need to make sure they have a 3D-ICE server up and
running before starting. For more information about 3D-ICE visit the [official website](http://esl.epfl.ch/3D-ICE).

#### Installing the lastest 3D-ICE version

[Download](http://esl.epfl.ch/3d-ice/download.html) the lastest version. Make sure you got version 2.2.6 or greater:

``` bash
$ wget http://esl.epfl.ch/files/content/sites/esl/files/3dice/releases/3d-ice-latest.zip
$ tar -xvzf 3d-ice-latest.zip
```

Install [SuperLU](http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_5.2.1.tar.gz) dependencies:

``` bash
$ sudo apt-get install build-essential git bison flex libblas-dev
```

Download and install SuperLU:

``` bash
$ wget http://crd.lbl.gov/~xiaoye/SuperLU/superlu_4.3.tar.gz
$ tar xvfz superlu_4.3.tar.gz
$ cd SuperLU_4.3/
$ cp MAKE_INC/make.linux make.inc
```

Make sure the SuperLUroot variable in ./make.inc is properly set (in my case $(HOME)/repos/).

``` bash
SuperLUroot = $(HOME)/repos/SuperLU_4.3
```

Compile the library:

``` bash
$ make superlulib
```

Download and install bison-2.4.1:

``` bash
$ wget http://ftp.gnu.org/gnu/bison/bison-2.4.1.tar.gz
$ tar xvzf bison-2.4.1.tar.gz
$ cd bison-2.4.1
$ ./configure --program-suffix=-2.4.1
$ make
$ sudo make install
```

Go to the 3d-ice directory:

``` bash
$ cd 3d-ice-2.2.6
```

Open the file makefile.def and set some variables. Set the correct path to the
SuperLU library you just compiled (in my case $(HOME)/repos/):

``` bash
SLU_MAIN = $(HOME)/repos/SuperLU_$(SLU_VERSION)
```

Set the YACC variable to bison-2.4.1:

``` bash
YACC = bison-2.4.1
```

Set the systemC architecture and main folder variables:

``` bash
SYSTEMC_ARCH = [linux,linux64]
SYSTEMC_MAIN = $(HOME)/repos/systemc-$(SYSTEMC_VERSION)
```

Compile 3D-ICE with SystemC/TLM2.0 support:

``` bash
$ make clean
$ make SYSTEMC_WRAPPER=y
```

Users interested in thermal simulation can also add some extra environment
variables:

```bash
# Necessary for thermal simulation
export LIBTHREED_ICE_HOME=<path>
export LIBSUPERLU_HOME=<path>
```

#### Running DRAMSys with Thermal Simulation

Before starting make sure you have a **clean repository** without any previous
automatic generated Makefiles. One way to ensure this is by running the
command below inside your DRAMSys repository, but keep in mind that
**untracked files and directories will be removed** from the repository.

``` bash
$ git clean -fdx
```

This feature can be enabled via an environment variable.

``` bash
$ export THERMALSIM=true
$ qtcreator &
```

or

``` bash
$ mkdir build
$ cd build
$ export THERMALSIM=true
$ qmake ../DRAMSys/DRAMSys.pro
$ make
```

Before starting DRAMSys it is necessary to run the 3D-ICE server passing to it
two arguments: a suitable configuration file and an Internet socket port
number. And then wait until the server is ready to receive requests.

``` bash
$ 3D-ICE-Server <stack file> <port>
Preparing stk data ... done !
Preparing thermal data ... done !
Creating socket ... done !
Waiting for client ... done !
```

The IP address and the port number related to the server shall be informed in
DRAMSys' configuration to subsequent use by DRAMSys to access the thermal
simulation server.

#### Usage Example with Thermal Simulation

The DRAMSys' main configuration file is presented below.

``` xml
<simulation>
    <!-- Configuration for the DRAMSys Simulator -->
    <simconfig src="wideio_thermal.xml" />
    <!-- Temperature Simulator Configuration -->
    <thermalconfig src="confi.xml" />
    <!-- Memory Device Specification: Which Device is on the DDR3 DIMM -->
    <memspec src="WideIO.xml"></memspec>
    <!-- Addressmapping Configuration of the Memory Controller -->
    <addressmapping src="am_wideio.xml"></addressmapping>
    <!-- Memory Controller Configuration: -->
    <mcconfig src="fr_fcfs.xml"/>
    <!--
    The following trace setup is only used in standalone mode.
    In library mode e.g. in Platform Architect the trace setup is ignored.
    -->
    <tracesetup>
        <!--
        This device mimics an image processing application
        running on an FPGA with 200 Mhz.
        -->
        <device clkMhz="1000">test_error.stl</device>
    </tracesetup>
</simulation>
```

Enable the error model in fr_fcfs.xml.

``` xml
<memconfig>
    <BankwiseLogic value="0"/>
    <OpenPagePolicy value="1" />
    <MaxNrOfTransactions value="8" />
    <Scheduler value="FR_FCFS" />
    <Capsize value="5" />
    <PowerDownMode value="NoPowerDown"/>
    <PowerDownTimeout value="100" />
    <!-- Error Model: -->
    <ErrorChipSeed value="42" />
    <ErrorCSVFile value="../../DRAMSys/library/src/error/error.csv" />
    <!-- Modes: NoStorage, Store (store data without errormodel), ErrorModel (store data with errormodel) -->
    <StoreMode value="ErrorModel" />
</memconfig>
```

Generate the input trace file for DRAMSys.

``` bash
$ cd DRAMSys/tests/error/
$ ./generateErrorTest.pl > test_error.stl
```

Start the 3D-ICE server providing the stack file and the port number.

``` bash
$ cd DRAMSys/library/resources/configs/thermalsim
$ 3D-ICE-Server stack.stk 11880
```

In another terminal or terminal tab start DRAMSys. Here the program's output
is redirected to a file.

``` bash
$ cd build/simulator/
$ ./DRAMSys > output
```

## DRAMSys with gem5

Install gem5 by following the instructions on the gem5 wiki
http://gem5.org/Documentation#Getting_Started. In order to understand the
SystemC coupling with gem5 it is recommended to study util/tlm/README and [11].
The main steps for building gem5 with SystemC are:

``` bash
scons build/ARM/gem5.opt
scons --with-cxx-config --without-python build/ARM/libgem5_opt.so
#for mac: scons --with-cxx-config --without-python build/ARM/libgem5_opt.dylib
cd util/tlm
```

In order to use gem5 with DRAMSys set the GEM5 environment variable to the path
to gem5, for example in the QtCreator under Projects > Build & Run > Build
Environment:

```
GEM5=/path/to/gem5/
```

DRAMSys will detect gem5 and configures automatically the according compile
target *gem5* for QtCreator.

In order to run gem5 with DRAMSys its mandatory to run gem5 first without
DRAMSys in order to generate an ini file which will be read by the DRAMSys_gem5
binary. In the following several examples for DRAMSys gem5 bindings are shown.

### DRAMSys with gem5 traffic generator

In the following we will run a simple example with a gem5 traffic generator:

```
Base System Architecture:
+-------------+ +------+    ^
| System Port | | TGEN |    |
+-------+-----+ +--+---+    |
        |           |       | gem5 World
        |      +----+       | (see this file)
        |      |            |
+-------v------v-------+    |
|        Membus        |    v
+---------------+------+    External Port (see sc_slave_port.*)
                |           ^
           +----v----+      | TLM World
           | DRAMSys |      | (see sc_target.*)
           +---------+      v

```

As mentioned before we first need to create a config.ini
that represents the gem5 configuration. We do so by starting gem5 with the
desired python configuration script.

``` bash
cd gem5/utils/tlm/
../../build/ARM/gem5.opt conf/tlm_slave.py
```

The message
```
"fatal: Can't find port handler type 'tlm_slave'"
```
is totally okay. The configuration file config.ini will be stored in the m5out/
directory.  Copy this configuration file to the building directory of DRAMSys:

```
dram.vp.system/build-DRAMSys-Desktop_Qt_5_7_0_clang_64bit-Debug/gem5
```
Also the traffic generatior configuration file (conf/tgen.cfg) must be stored
in a conf directory of this building directory.

Then the simulation can be started with:

``` bash
./DRAMSys_gem5 /path/to/dram.vp.system/DRAMSys/library/resources/simulations/ddr3-example.xml config.ini
```

Lets run the simulation for some seconds and then stop the simulation with
ctrl+c.  Observe the output of the simulation in the trace analyzer. The trace
database file is stored in the gem5 directory in the building directory.

### Boot Linux with gem5 and DRAMSys

The procedure is very similar to the traffic generator example above.

First we have to generate the config.ini file by starting gem5 with the following configuration:

``` bash
build/ARM/gem5.opt configs/example/fs.py \
  --tlm-memory=transactor --cpu-type=TimingSimpleCPU --num-cpu=1 \
  --mem-type=SimpleMemory --mem-size=512MB --mem-channels=1 --caches \
  --l2cache --machine-type=VExpress_EMM \
  --dtb-filename=vexpress.aarch32.ll_20131205.0-gem5.1cpu.dtb \
  --kernel=vmlinux.aarch32.ll_20131205.0-gem5 \
  --disk-image=linux-aarch32-ael.img
```

The config.ini should be copied again to the DRAMSys_gem5 build folder.

The simconfig should be changed in order to support storage and address offsets:

``` xml
<simconfig>
    <SimulationName value="ddr3" />
    <Debug value="0" />
    <DatabaseRecording value="1" />
    <PowerAnalysis value="1" />
    <EnableWindowing value = "1" />
    <WindowSize value="1000" />
    <ThermalSimulation value="0"/>
    <SimulationProgressBar value="1"/>
    <NumberOfMemChannels value="1"/>
    <NumberOfDevicesOnDIMM value = "8" />
    <CheckTLM2Protocol value = "0" />
    <AddressOffset value = "0" />
    <ECCControllerMode value = "Disabled" />
    <ErrorChipSeed value="42" />
    <ErrorCSVFile value="" />
    <!-- Modes:
	- NoStorage,
	- Store (store data without errormodel),
	- ErrorModel (store data with errormodel)
    -->

    <!-- Gem5 Related Configuration:
    In the memory controller file the storage mode should be set to Store
    E.g. the DRAM is located at 0x80000000 for gem5
    -->
    <StoreMode value="Store" />
    <AddressOffset value = "2147483648" />
</simconfig>
```

Then start DRAMSys_gem5 with the following command:

``` bash
./DRAMSys_gem5 /Users/myzinsky/EMS/Programming/dram.vp.system/DRAMSys/library/resources/simulations/ddr3-example.xml config.ini
```

For further sophisticated address mappings or scenarios checkout the file DRAMSys/gem5/main.cpp

### DRAMSys with gem5 Elastic Traces

For understanding elastic traces and their generation, study the gem5 wiki
(http://gem5.org/TraceCPU) and the paper [13].  Some predefined configs are
stored in dram.vp.system/DRAMSys/gem5/configs and the related python files are
stored here: dram.vp.system/DRAMSys/gem5/examples

This is an example for running an elastic trace:

``` bash
./DRAMSys_gem5 /path/to/dram.vp.system/DRAMSys/library/resources/simulations/ddr3-example.xml ../../DRAMSys/gem5/configs/singleElasticTraceReplay.ini
```

Note that the address offset is usually zero for elastic traces.

If two elastic traces should be used the main.cpp must be modified:

``` c++
//#define CHOICE1
#define CHOICE2
//#define CHOICE3
```

Run the simulation with the following example:

```
./DRAMSys_gem5 /path/to/dram.vp.system/DRAMSys/library/resources/simulations/ddr3-example.xml ../../DRAMSys/gem5/configs/dualElasticTraceReplay.ini
```

For more spophisticated setups, even with l2 caches the proper ini file should be created.
If you need help please contact Matthias Jung.

## References

[1] TLM Modelling of 3D Stacked Wide I/O DRAM Subsystems, A Virtual Platform for Memory Controller Design Space Exploration
M. Jung, C. Weis, N. Wehn, K. Chandrasekar. International Conference on High-Performance and Embedded Architectures and Compilers 2013 (HiPEAC), Workshop on: Rapid Simulation and Performance Evaluation: Methods and Tools (RAPIDO), January, 2013, Berlin.

[2] DRAMPower: Open-source DRAM Power & Energy Estimation Tool
Karthik Chandrasekar, Christian Weis, Yonghui Li, Sven Goossens, Matthias Jung, Omar Naji, Benny Akesson, Norbert Wehn, and Kees Goossens
URL: http://www.drampower.info

[3] Energy Optimization in 3D MPSoCs with Wide-I/O DRAM
M. Sadri, M. Jung, C. Weis, N. Wehn, L. Benini. Conference Design, Automation and Test in Europe (DATE), March, 2014, Dresden, Germany.

[4] DRAMSys: A flexible DRAM Subsystem Design Space Exploration Framework
M. Jung, C. Weis, N. Wehn. Accepted for publication, IPSJ Transactions on System LSI Design Methodology (T-SLDM), October, 2015.

[5] Optimized Active and Power-Down Mode Refresh Control in 3D-DRAMs
M. Jung, M. Sadri, C. Weis, N. Wehn, L. Benini., VLSI-SoC, October, 2014, Playa del Carmen, Mexico.

[6] Retention Time Measurements and Modelling of Bit Error Rates of WIDE-I/O DRAM in MPSoCs
C. Weis, M. Jung, P. Ehses, C. Santos, P. Vivet, S. Goossens, M. Koedam, N. Wehn. Accepted for publication, IEEE Conference Design, Automation and Test in Europe (DATE), March, 2015, Grenoble, France

[7] http://www.uni-kl.de/3d-dram/publications/

[8] A Sridhar, A Vincenzi, D Atienza, T Brunschwiler, 3D-ICE: a compact
thermal model for early-stage design of liquid-cooled ICs, IEEE Transactions
on Computers (TC 2013, accepted for publication).

[9] A Sridhar, A Vincenzi, M Ruggiero, T Brunschwiler, D Atienza, 3D-ICE: Fast
compact transient thermal modeling for 3D-ICs with inter-tier liquid cooling,
Proceedings of the 2010 International Conference on Computer-Aided Design
(ICCAD 2010), San Jose, CA, USA, November 7-11 2010.

[10] A Sridhar, A Vincenzi, M Ruggiero, T Brunschwiler, D Atienza, Compact
transient thermal model for 3D ICs with liquid cooling via enhanced heat
transfer cavity geometries, Proceedings of the 16th International Workshop on
Thermal Investigations of ICs and Systems (THERMINIC'10), Barcelona, Spain,
6-8 October, 2010.

[11] http://esl.epfl.ch/3D-ICE

[12] System Simulation with gem5 and SystemC: The Keystone for Full
Interoperability C. Menard, M. Jung, J. Castrillon, N. Wehn. IEEE International
Conference on Embedded Computer Systems Architectures Modeling and Simulation
(SAMOS), July, 2017, Samos Island, Greece.

[13] Exploring System Performance using Elastic Traces: Fast, Accurate and
Portable Radhika Jagtap, Stephan Diestelhorst, Andreas Hansson, Matthias Jung
and Norbert Wehn, IEEE International Conference on Embedded Computer Systems
Architectures Modeling and Simulation (SAMOS), 2016, Samos Island, Greece.