DRAMSys 4.0
===========

**DRAMSys** [1] is a flexible DRAM subsystem design space exploration
framework that consists of models reflecting the DRAM functionality, power
consumption, temperature behaviour and retention time errors.

Pipeline Status: [![pipeline status](https://git.eit.uni-kl.de/ems/astdm/dram.sys/badges/master/pipeline.svg)](https://git.eit.uni-kl.de/ems/astdm/dram.sys/commits/master)
[![Coverage report](https://git.eit.uni-kl.de/ems/astdm/dram.sys/badges/master/coverage.svg?job=coverage)](https://git.eit.uni-kl.de/ems/astdm/dram.sys/commits/master)
## Basic Setup

Start using DRAMSys by cloning the current stable master branch:

```bash
$ git clone --recursive https://git.eit.uni-kl.de/ems/astdm/dram.sys.git
```

The *--recursive* flag tells git to initialize all submodules within the
repository, namely **DRAMPower** [2], **SystemC** and **tinyxml**.

Now you can implement, test, commit and push features into a **branch**.

When you consider your work stable enough to be merged into the master branch
it is time to open a **merge request** using the web interface.

Your changes will be reviewed and might be integrated into the master branch.

After cloning go to the project directory.

```bash
$ cd dram.sys
```

### Dependencies

Make sure you have properly installed all the required libraries and
tools in your system.

- **General dependencies**

You may want to have a look on the convenience scripts that are located in the
[utils](./utils) folder.

```bash
$ cd utils
$ ls
```

You can use [utils/install_deb.sh](./utils/install_deb.sh) in order to install
dependencies. Type your password if required.

```bash
$ ./install_deb.sh
```

### Coding Style

Please read the [coding-style document](coding-style.md) before starting to
code.

A script is provided to apply the coding style.
```bash
$ cd util
$ ./make_pretty.sh
```

### Buiding with QTCreator
Execute the *QTCreator*.

```bash
$ qtcreator &
```

Use the menu bar and open the DRAMSys project.

**File -> Open Project -> dram.sys/DRAMSys/CMakeLists.txt**

When you open the project for the first time a configuration window pops-up.
Then click in **Configure Project** and after that **Build** the project.

To speedup the building process one can use the additional **make** option
**-j[jobs]**. The command line below returns the number of CPUs on a Debian
Linux to be passed to make as the number of jobs that can run simultaneously to
improve the building time.

```bash
$ cat /proc/cpuinfo | grep processor | wc -l
```

In the left bar go to **Projects -> Build & Run -> Build Steps -> Make**.
Click in **Details** then **Make arguments** and add **-j** followed by the
number you got.

In case you face a problem related to the **Qt version** double check the **Qt
version** configuration shown in the image below.

![Qt Creator Configuration](DRAMSys/docs/images/QtCreatorConfig.png)

### Building without QTCreator

In case you prefer a command line interface to the QTCreator GUI you can also
use **cmake** to generate a Makefile and then compile the project.

```bash
$ cd dram.sys
$ mkdir build
$ cd build
$ cmake ../DRAMSys/
$ make -j4
```

The compilation generates executable binary files **DRAMSys** and
**TraceAnalyzer** that can be found inside sub-directories.

From the build directory use the commands below to execute DRAMSys.

```bash
$ cd simulator
$ ./DRAMSys
```

To run DRAMSys with a specific config:
```bash
$ ./DRAMSys ../../DRAMSys/library/resources/simulations/ddr3-example.json
```

To run DRAMSys with a specific config and a resource folder somewhere else to the standard:
```bash
$ ./DRAMSys ../../DRAMSys/tests/example_ddr3/simulations/ddr3-example.json ../../DRAMSys/tests/example_ddr3/
```

From the build directory use the commands below to execute the Trace Analyzer.

```bash
$ cd traceAnalyzer
$ export QT_QPA_PLATFORMTHEME=qgnomeplatform
$ ./TraceAnalyzer
```

### Building on MacOS (Formerly OSX)
- Install XCode as a Compiler (or any other via Homebrew)
- Install the required python3 over homebrew:

```bash
$ brew install python3
```

- Install the QtCreator using offical setup file from [link](https://www.qt.io/download-open-source/#section-2)

**Note:** You have later setup PATH for Qt5 and its tool if you install QtCreator manually, e.g:

```bash
# Setting PATH for Qt5 and its tools
PATH="/Users/<username>/Qt5.7.0/5.7/clang_64/bin:${PATH}"
export PATH
```

- Install the QWT manually to /opt/qwt, then do:

```bash
$ cd /Library/Frameworks
$ sudo ln -s /opt/qwt-6.1.2/lib/qwt.framework/ .
```

You compile DRAMSys also with QtCreator or CMAKE as described in the Linux
section.

### Building on Windows 10

- DRAMSys can also run on Windows 10 with the *Windows Subsystem for Linux* (WSL) feature. You can install a linux distribution like Debian over the windows app store.
  We refer to the following example [website](https://docs.microsoft.com/en-us/windows/wsl/install-win10).
  Then DRAMSys can be installed as described above for Linux.

- Native Windows 10 Implementation is currently under investigation

### DRAMSys Configuration

The **DRAMSys** executable supports one argument which is a JSON file that
contains certain arguments and the path of other configuration files for the
desired simulation. If no argument is passed through the command line a default
configuration file will be loaded.

The JSON code below shows a typic configuration:

```json
{
	"simulation": {
		"simulationid": "ddr3-example",
		"simconfig": "ddr3.json",
		"thermalconfig": "config.json",
		"memspec": "MICRON_1Gb_DDR3-1600_8bit_G.json",
		"addressmapping": "am_ddr3_8x1Gbx8_dimm_p1KB_brc.json",
		"mcconfig":"fifoStrict.json",
		"tracesetup": [{
				"clkMhz": 300,
				"name": "ddr3_example.stl"},
                  {
				"clkMhz": 400,
				"name": "ddr3_example.stl"}
                 ]	
	}
}
```
Fields Description
		"simulationid": Simulation file identifier
		"simconfig": Configuration file for the DRAMSys Simulator
		"thermalconfig": Temperature Simulator Configuration File
		"memspec": Memory Device Specification File
		"addressmapping": Addressmapping Configuration of the Memory Controller File.
		"mcconfig": Memory Controller Configuration File.
		"tracesetup": The trace setup is only used in standalone mode.
    In library mode e.g. in Platform Architect the trace setup is ignored.
    Each device should be addes as a json object inside the "tracesetup" array. 

Each **tracesetup** device configuration consists of two parameters - clkMhz
(operation frequency for this device) - and a **trace file**.    
                 
Some configuration fields reference other JSON files which contain more
specialized chunks of the configuration like memory specification, address
mapping and memory configurations

The JSON configuration files are parsed by the program and the configuration
details extracted are assigned to the correspondent attributes of the internal
configuration structure.


#### Trace files

A **trace file** is a pre-recorded file containing memory transactions. Each
memory transaction has a timestamp that tells the simulator when it shall
happen, a transaction type (read or write) and a memory address given in
hexadecimal.

There are two different kinds of trace files. They differ in their timing behaviour and are distingushed by their file extension.

##### STL Trace (.stl)

The timestamp corresponds to the time the request is to be issued and it is
given in cycles of the bus master device. Example: the device is a FPGA with
frequency 200 MHz (clock period of 5 ns). If the timestamp is 10 it means that
the request is to be issued when time is 50 ns.

Here is an example syntax:

```
# Comment lines begin with #
# [clock-cyle]: [write|read] [hex-address]
31:	read	0x400140
33:	read	0x400160
56:	write	0x7fff8000
81:	read	0x400180
```

##### Relative STL Traces (.rstl)

The timestamp corresponds to the time the request is to be issued relative to the end of the transaction before or the beginning of the trace. This results in a simulation in which the **trace player** is able to react to possible delays due to DRAM bottlenecks.

Here is an example syntax:

```
# Comment lines begin with #
# [clock-cyle]: [write|read] [hex-address]
31:	read	0x400140
2:	read	0x400160
23:	write	0x7fff8000
25:	read	0x400180
```

#### Trace player

A **trace player** is **equivalent** to a bus master **device**
(processor, FPGA, etc.). It reads an input trace file and translates each line into
a new memory request. By adding a new device element into the trace setup section
one can specify a new trace player, its operating frequency and the trace file
for that trace player.

#### Configuration File Sections

The main configuration file is divided into self-contained sections. Each of
these sections refers to sub-configuration files.

Below, the sub-configurations are listed and explained.

- **Simulator Configuration**

  The content of [ddr3.json](DRAMSys/library/resources/configs/simulator/ddr3.json) is presented below as an example.

```json
{
    "simconfig": {
        "AddressOffset": 0,
        "CheckTLM2Protocol": false,
        "DatabaseRecording": true,
        "Debug": false,
        "ECCControllerMode": "Disabled",
        "EnableWindowing": false,
        "ErrorCSVFile": "",
        "ErrorChipSeed": 42,
        "NumberOfDevicesOnDIMM": 8,
        "NumberOfMemChannels": 1,
        "PowerAnalysis": false,
        "SimulationName": "ddr3",
        "SimulationProgressBar": true,
        "StoreMode": "NoStorage",
        "ThermalSimulation": false,
        "UseMalloc": false,
        "WindowSize": 1000
    }
}
```

  - *SimulationName* (boolean)
    
    - Give the name of the simulation for distingushing from other simulations.
  - *Debug* (boolean)
    - true: enables debug output on console
    - false: disables debug output
  - *DatabaseRecording* (boolean)
    - true: enables trace file recording for the trace analyser tool
    - false: disables trace file recording
  - *PowerAnalysis* (boolean)
    - true: enables live power analysis with the DRAMPower tool
    - false: disables power analysis
  - *EnableWindowing* (boolean)
    - true: enables temporal windowing
    - false: disables temporal windowing
  - *WindowSize* (unisgned int)
    
    - Size of the window in clock cycles used to evaluate average bandwidth and average power consumption
  - *NumberOfMemChannels* (unsigned int)
    
    - Number of memory channels
  - *ControllerCoreRefDisable* (boolean)
    - true: disables refreshes
    - false: normal operation (refreshes enabled)
  - *ControllerCoreRGR* (boolean)
    - true: enable row granular refresh
    - false: normal operation
  - *ThermalSimulation* (boolean)
    - true: enables thermal simulation
    - false: static temperature during simulation
  - *SimulationProgressBar* (boolean)
    - true: enables the simulation progress bar
    - false: disables the simulation progress bar
  - *NumberOfDevicesOnDIMM* (unsigned int)
    
    - Number of devices on dual inline memory module
  - *CheckTLM2Protocol* (boolean)
    - true: enables the TLM-2.0 Protocol Checking
    - false: disables the TLM-2.0 Protocol Checking
  - *ECCControllerMode* (string)
    - "Disabled": No ECC Controller is used
    - "Hamming": Enables an ECC Controller with classic SECDED implementation using Hamming Code
  - *UseMalloc* (boolean)
    - false: model storage using mmap() (DEFAULT)
    - true: allocate memory for modeling storage using malloc()

- **Temperature Simulator Configuration**

  The content of [config.json](DRAMSys/library/resources/configs/thermalsim/config.json) is presented below as an example.

```json
{
	"thermalsimconfig": {
		"TemperatureScale": "Celsius",
		"StaticTemperatureDefaultValue": 89,
		"ThermalSimPeriod":100,
		"ThermalSimUnit":"us",
		"PowerInfoFile": "powerInfo.json",
		"IceServerIp": "127.0.0.1",
		"IceServerPort": 11880,
		"SimPeriodAdjustFactor" : 10,
		"NPowStableCyclesToIncreasePeriod": 5,
		"GenerateTemperatureMap": true,
		"GeneratePowerMap": true
	}
}
```
  - *TemperatureScale* (string)
    - "Celsius"
    - "Fahrenheit"
    - "Kelvin"
  - *StaticTemperatureDefaultValue* (int)
    
    - Temperature value for simulations with static temperature
  - *ThermalSimPeriod* (double)
    
    - Period of the thermal simulation
  - *ThermalSimUnit* (string)
    - "s": seconds
    - "ms": millisecond
    - "us": microseconds
    - "ns": nanoseconds
    - "ps": picoseconds
    - "fs": femtoseconds
  - *PowerInfoFile* (string)
    
    - File containing power related information: devices identifiers, initial power values and power thresholds.
  - *IceServerIp* (string)
    
    - 3D-Ice server IP address
  - *IceServerPort* (unsigned int)
    
    - 3D-Ice server port
  - *SimPeriodAdjustFactor* (unsigned int)
    
    - When substantial changes in power occur (i.e., changes that exceed the thresholds), then the simulation period will be divided by this number causing the thermal simulation to be executed more often.
  - *NPowStableCyclesToIncreasePeriod* (unsigned int)
    
    - Wait this number of thermal simulation cycles with power stability (i.e., changes that do not exceed the thresholds) to start increasing the simulation period back to its configured value.
  - *GenerateTemperatureMap* (boolean)
    - true: generate temperature map files during thermal simulation
    - false: do not generate temperature map files during thermal simulation
  - *GeneratePowerMap* (boolean)
    - true: generate power map files during thermal simulation
    - false: do not generate power map files during thermal simulation

- **Memory Specification**

  A file with memory specifications. This information comes from datasheets and
  measurements, and usually does not change.

  The fields inside "mempowerspec" can be written directly as a **double** type. "memoryId" and "memoryType" are **string**. The others are **unsigned int** as it can be checked in the files at <DRAMSys/library/resources/configs/memspecs>.


- **Address Mapping**

  Currently the CONGEN format is supported. It provides bit-wise granularity. It also provides the possibility of XOR address bits in order to map page misses to different banks and reduce latencies. 
  There is an optional field called **SOLUTION**. If added it will look for the solution with the field "ID" equals to 0.
  Example with "SOLUTION" field at: <DRAMSys/library/resources/configs/amconfigs/congen_extended_solution.json>
  Example without "SOLUTION" field at: <DRAMSys/library/resources/configs/amconfigs/congen_extended.json>


- **ConGen XML file format**

  This file format is generated by ConGen.

  The format delivers more information than needed for an address mapping.
  Optional data (unused):
  
  - "NAME": Name of the trace file which was used by ConGen
- "COSTS": Number of row misses which this configuration produces while playing the trace.
  
  - "CONFIG": Gives you information about the ConGen configuration
  
  Used data:
  
  - "SOLUTION": (OBS.:Different solutions should be added as json objects inside the "SOLUTION" array)
    - "ID": Unique identifier for this solution. It is used to specify a certain solution.
    - "XOR": Defines an xor connection of a bank and row bit
    - "BYTE_BIT": Address bits that are connected to the byte bits in ascending order
    - "COLUMN_BIT": Address bits that are connected to the column bits in ascending order
    - "ROW_BIT": Address bits that are connected to the row bits in ascending order
    - "BANK_BIT": Address bits that are connected to the bank bits in ascending order
    - "BANKGROUP_BIT": Address bits that are connected to the bankgroup bits in ascending order
    - "RANK_BIT": Address bits that are connected to the rank bits in ascending order
    - "CHANNEL_BIT": Address bits that are connected to the channel bits in ascending order

```json
{
    "CONGEN": {
        "SOLUTION": [
            {
                "ID": 0,
                "XOR": [
                    {
                        "FIRST": 13,
                        "SECOND": 16
                    }
                ],
                "BYTE_BIT": [0,1,2],
                "COLUMN_BIT": [3,4,5,6,7,8,9,10,11,12],
                "BANK_BIT": [13,14,15],
                "ROW_BIT": [16,17,18,19,20,21,22,23,24,25,26,27,28,29]
            }
        ]
	}	
}

```


- **Memory Controller Configuration**

  An example follows.

```json
{
    "mcconfig": {
        "PagePolicy": "Open", 
        "Scheduler": "Fifo", 
        "RequestBufferSize": 8, 
        "CmdMux": "Oldest", 
        "RespQueue": "Fifo", 
        "RefreshPolicy": "Rankwise", 
        "RefreshMode": 1, 
        "RefreshMaxPostponed": 8, 
        "RefreshMaxPulledin": 8, 
        "PowerDownPolicy": "NoPowerDown", 
        "PowerDownTimeout": 100
    }
}
```

  - *BankwiseLogic* (boolean)
    - true: perform bankwise-refresh [3] and bankwise-powerdown [4]
    - false: do not perform bankwise operations
  - *OpenPagePolicy* (boolean)
    - true: use open page precharge policy
    - false: do not use open page precharge policy
  - *MaxNrOfTransactions* (unsigned int)
    - Maximum number of transactions.
  - *Scheduler* (string)
    - "Fifo": first in, first out
    - "FifoStrict": out-of-order treatment of queue elements not allowed
    - "FrFcfs": first ready first-come-first-served
    - "FrFcfsRp": first ready first-come-first-served read priority
    - "FrFcfsGrp": first ready first-come-first-served grouper
    - "Grp": grouper
    - "SMS": will be removed
  - *Capsize* (unsigned int)
    - Capacitor cell size.
  - *PowerDownMode* (enum EPowerDownMode)
    - "NoPowerDown": no power down mode (active idle)
    - "Staggered": staggered power down policy [5]
    - "TimeoutPDN": precharge idle
    - "TimeoutSREF": self refresh
  - *ReadWriteGrouping* (boolean)
    - true: enable read writing grouping
    - false: disable read writing grouping
  - *ReorderBuffer* (boolean)
    - true: use reordering buffer
    - false: do not use reordering buffer
  - *ErrorChipSeed* (unsigned int)
    - Seed to initialize the random error generator.
  - *ErrorCSVFile* (string)
    - CSV file with error injection information.
  - *StoreMode* (enum StorageMode)
    - "NoStorage": no storage
    - "Store": store data without error model
    - "ErrorModel": store data with error model [6]
  - *ControllerCoreRefDisable* (boolean)
    - true: disables refreshes
    - false: normal operation (refreshes enabled)
  - ControllerCoreRefMode (unsigned int)
    - Refresh mode. 1: 1X, 2: 2X, 4: 4X. Refresh period is tREFI, tREFI/2,
      tREFI/4, respectively. Number of rows per refresh is affected. Maximum
      values for pull-in and postpone are affected. There are different values
      of tRFC for each mode that come from memory specifications.
  - *ControllerCoreRefForceMaxPostponeBurst* (boolean)
    - true: always postpone, resulting in a ControllerCoreRefMaxPostponed burst
    - false: normal operation
  - *ControllerCoreRefEnablePostpone* (boolean)
    - true: enables the postpone refresh feature
    - false: normal operation
  - *ControllerCoreRefEnablePullIn* (boolean)
    - true: enables the pull-in refresh feature
    - false: normal operation
  - *ControllerCoreRefMaxPostponed* (unsigned int)
    - Max AR commands to be postponed. Refresh mode affects this config.
  - *ControllerCoreRefMaxPulledIn* (unsigned int)
    - Max AR commands to be pulled-in. Refresh mode affects this config.
  - *ControllerCoreRGR* (boolean)
    - true: enables row granular refresh feature (RGR)
    - false: normal operation
  - *ControllerCoreRefNumARCmdsIntREFI* (unsigned int)
    - Number of AR commands to to be issued in a refresh period tREFI in 1X
      mode
  - *ControllerCoreRGRRowInc* (unsigned int)
    - Row increment for each AR command (selective refresh)
  - *ControllerCoreRGRB0* (boolean)
    - true: RGR this bank
    - false: skip this bank
  - *ControllerCoreRGRB1* (boolean)
    - true: RGR this bank
    - false: skip this bank
  - *ControllerCoreRGRB2* (boolean)
    - true: RGR this bank
    - false: skip this bank
  - *ControllerCoreRGRB3* (boolean)
    - true: RGR this bank
    - false: skip this bank
  - *ControllerCoreRGRB4* (boolean)
    - true: RGR this bank
    - false: skip this bank
  - *ControllerCoreRGRB5* (boolean)
    - true: RGR this bank
    - false: skip this bank
  - *ControllerCoreRGRB6* (boolean)
    - true: RGR this bank
    - false: skip this bank
  - *ControllerCoreRGRB7* (boolean)
    - true: RGR this bank
    - false: skip this bank
  - *ControllerCoreRGRB8* (boolean)
    - true: RGR this bank
    - false: skip this bank
  - *ControllerCoreRGRB9* (boolean)
    - true: RGR this bank
    - false: skip this bank
  - *ControllerCoreRGRB10* (boolean)
    - true: RGR this bank
    - false: skip this bank
  - *ControllerCoreRGRB11* (boolean)
    - true: RGR this bank
    - false: skip this bank
  - *ControllerCoreRGRB12* (boolean)
    - true: RGR this bank
    - false: skip this bank
  - *ControllerCoreRGRB13* (boolean)
    - true: RGR this bank
    - false: skip this bank
  - *ControllerCoreRGRB14* (boolean)
    - true: RGR this bank
    - false: skip this bank
  - *ControllerCoreRGRB15* (boolean)
    - true: RGR this bank
    - false: skip this bank
  - *ControllerCoreRGRtRASBInClkCycles*  (unsigned int)
    - Timing can be changed to explore optimum row granular refresh (ORGR)
  - *ControllerCoreRGRtRRDB_LInClkCycles* (unsigned int)
    - Timing can be changed to explore optimum row granular refresh (ORGR)
  - *ControllerCoreRGRtRRDB_SInClkCycles* (unsigned int)
    - Timing can be changed to explore optimum row granular refresh (ORGR)
  - *ControllerCoreRGRtRPBInClkCycles* (unsigned int)
    - Timing can be changed to explore optimum row granular refresh (ORGR)
  - *ControllerCoreRGRtRCBInClkCycles* (unsigned int)
    - Timing can be changed to explore optimum row granular refresh (ORGR)
  - *ControllerCoreRGRtFAWBInClkCycles* (unsigned int)
    - Timing can be changed to explore optimum row granular refresh (ORGR)


**Refresh modes**

The default refresh mode is fixed 1X mode where refresh commands should be
issued with the normal rate, i.e., tREFI. The duration of each refresh command
is the normal refresh cycle time tRFC. In 2X mode Refresh commands are issued
to the DRAM at the double frequency (tREFI/2). In 4X mode Refresh commands are
issued to the DRAM at the quadruple frequency (tREFI/4). There are different
values of tRFC for each mode that come from memory specifications.

The number of refresh commands in a tREFI is multiplied by two in 2X mode and
by four in 4X mode. The maximum number of refresh commands that can be
postponed or pulled-in is affected in the same manner. The number of rows per
refresh command is divided by two and by four in 2X and 4X mode respectively.

The nomenclature tREFIx is used to denote the refresh interval which value
changes accordingly to the operation mode, e.g., in 2X mode tREFIx corresponds
to tREFI/2. Similarly tRFCx denotes the refresh cycle time which value changes
accordingly to the operation mode. Nevertheless, the values of tRFCx must be
obtained from memory specifications, estimated, measured, etc.

**Flexible Refresh**

The feature can be used together with regular refresh, bankwise refresh and
also with row granular refresh (RGR) non-bankwise and bankwise. Combinations
with all refresh modes are possible.

**Pull-In Refresh**

A pull-in starts when a refresh is triggered (in a multiple of tREFIx) and
there are no pending requests in the memory controller's buffer. This can be
done in order to prepare for possible accesses that might happen in the
future. When a burst of REF commands is initiated a REF command is issued (due
to the current tREFIx) followed by one or more REF commands separated in time
by tRFCx. The burst is interrupted if requests arrive, meaning that the
maximum additional delay for a request (considering the worst case scenario in
which a request arrives at the same time a REF is issued) is a refresh cycle
time (tRFCx).

The advantage of pulling-in refreshes is that they will not be issued in the
near future, i.e., in their actual times multiples of tREFIx, allowing for
more efficient accesses to the memory.

**Postpone Refresh**

Similarly, the decision to postpone a refresh is done if by the time of a
refresh due (multiple of tREFIx) there are pending requests on the memory
controller's buffer. Buffered requests may generate row-hits, so postponing
refreshes may be beneficial for it avoids breaking row-hit sequences what
reduces the number of commands (e.g., ACT, PRE) to carry out the memory
accesses and improves the overall system preformance because accesses that are
row-hits consume less time. After postponing refreshes, if there are no
pending requests in the next refresh interval (tREFIx) a burst is issued for
the same number of REF commands postponed plus the actual refresh for that
tREFIx. When the maximum number of postponed refreshes is reached a burst is
issued in the next tREFIx despite the state of the memory controller's buffer
(empty or not). A burst of postponed refreshes cannot be interrupted.

**The Flexible Refresh FSM**

![Flexible](DRAMSys/docs/images/flexreffsm.png)


- **Trace Setups**
  - *clkMhz* (unsigned int)
    - Speed of the trace player
  - *trace file*
    - A pre-recorded file containing memory transactions to be executed by a
      trace player.

Some attributes are self-explanatory while others require some previous
knowhow of memory technologies.

Resources of the simulator are available inside of the **resources** directory
and its sub-directories.

```bash
$ cd DRAMSys/library/resources
```

A description of the content each directory follows.

- **resources**
  - **configs**: JSON files that specify details of the simulation.
    - amconfigs: address mapping configs.
    - mcconfigs: memory controller configs.
    - memspecs: memory specification files (technology dependent).
    - simulator: simulator configs.
  - **scripts**: useful tools.
  - **simulations**: main configuration files.
  - **traces**: pre-recorded trace files that may be used as stimuli in simulations.


### Log Collector Script

Users can profit of running multiple simulations automatically with
[DRAMSylva.sh].

Every time you run the script you get a new folder with the name containing
the execution time: dram.sys\_YYYY\_MM\_DD-HH.MM.SS.

Example on how to run the script:

```bash
$ cd DRAMSys/library/resources/scripts/DRAMSylva
$ bash DRAMSylva.sh
```

To see the generated plots and CSV files:

```bash
$ nautilus dram.sys_YYYY_MM_DD-HH.MM.SS/build/simulator
```

In that folder you will find plots as PDF files and CSV files with the output
data used to generate the plots. The CSV files are:

- **out.csv** (energy, average power, bandwidth, etc.)

- **metrics.csv** (DRAMSys metrics like average response latency, memory
  utilization and many others)

Use the command below to find all generated CSV files:

```bash
$ ls -l dram.sys_YYYY_MM_DD-HH.MM.SS/build/simulator/*.csv
```

The generated CSV files can be open in a spreadsheet program for further
manipulation.

Set the variable **create_comparison_plots** to **yes** in order to get plots
for quick comparison from the CSV files generated.

```bash
create_comparison_plots="yes"
```

Additionally, the database files (\*.tdb) generated will be available and can
be open with the traceAnalyzer tool for debugging, plot generation, etc.

Set the variable **create_analyzer_plots** to **yes** in order to get plots
generated from the trace databases (the same plots generated by the trace
analyzer tool). **Note**: enabling this option may incur extra time for
database manipulation and plot generation. Depending on the size and amount of
database files it may take long to finish.

```bash
create_analyzer_plots="yes"
```

A DRAMSys simulation is defined by the main configuration file passed to the
simulator. The main configuration file includes other files which contain
specifc configs.
You can change what is going to be simulated by the script by editing it.
There is a list of main configuration files on the top of the script:

```bash
sim_files="
ddr3-example.json
ddr3-single-device.json
wideio-example.json
"
```

Simulation files are expected to be available (already commited and pushed to
be available after cloning) in the [simulation folder](DRAMSys/library/resources/simulations).

Set the variable **use_trace_list** to **yes** in order to use all traces in
the **trace list** with all simulation files. Each pair generates a new simulation
with the original trace specified in the simulation file replaced by a trace
from the list. Otherwise it runs a simulation per simulation file using the
trace specified in the simulation file. Files are expected to be available
(already commited and pushed to be available after cloning) in the
[traces folder](DRAMSys/library/resources/traces).

```bash
use_trace_list="yes"
```

```bash
trace_list="
chstone-bf_32.stl
chstone-jpeg_32.stl
chstone-adpcm_32.stl
mediabench-unepic_32.stl
"
```

The script runs one instance of DRAMSys for each of the files in the list.
**The multiple instances run in parallel**.

If some traces in trace_list are compressed in a tar.gz and require
decompression before execution the option **tgz_traces** can be set to
**yes**. The tarball is specified by the variable **tgz_file** and it is
expected to be available (already commited and pushed to be available after
cloning) in the [trace folder](DRAMSys/library/resources/traces).
[DRAMSylva.sh] will uncompress the tarball extracting the traces before using
them.

```bash
tgz_traces="yes"
tgz_file="rgr_traces_flauer_ddr4_8b.tar.gz"
```

Set the variable **use_json_cfg** to **yes** in order to override sim_files
with new simulation files generated from a JSON description. Otherwise the
simulation files are the ones specified by sim_files. Files are expected to be
available (already commited and pushed to be available after cloning) in
[configs_json].

```bash
use_json_cfg="yes"
```

All the essential simuation files are auto generated accordingly to each of
the JSON descriptions provided in **json_cfg_list**. Several examples of JSON
configuration files are provided in [configs_json].

+ Insert the desired simulation data in one or multiple JSON files following
  any of the examples provided, e.g.,
  [**configs.json**](DRAMSys/library/resources/scripts/DRAMSylva/configs_json/configs.json).
  Multiple arrays are allowed and encouraged. Each array corresponds to a full
  simulation setup.

+ Add your JSON files to **json_cfg_list** in [DRAMSylva.sh].

```bash
json_cfg_list="
ref.json
ref_bw.json
"
```

+ Commit and push your changes.

+ Run **[DRAMSylva.sh]** as previously described. All generated files will be
  inside the output folder, so it will be possible to keep a perfect track of
  all simulations.

For more information check the documentation inside [DRAMSylva folder].

### Trace Generator Script

The [trace_gen](DRAMSys/library/resources/scripts/trace_gen.py) script for
generating input traces for simple tests is provided.

Example on how to run the script:

```bash
$ cd DRAMSys/library/resources/scripts
$ ./trace_gen.py > trace.stl
```

Now change your configuration file to use the new generated trace file and run
your simulation.

The script can be easily changed and provides a way to quickly generate
accesses to all channels, all bank groups, all banks, all rows and all columns
of a memory.
    
**Be aware that a trace which covers all rows and all columns may be huge
(several gigabytes) depending on your memory.**

The defaul values in the script serve as an example. They consider the address
mapping that follows.

```
DDR3-SDRAM DIMM Characteristics:
Byte Offset (Y): 8    [0:2]   (8-byte-wide memory module, i.e., 64-bit-wide data bus) -> 3 bit
Cols        (C): 1K   [3:12]  (A0 - A9)   -> 10 bit
Rows        (R): 128K [13:29] (A0 - A16)  -> 17 bit
Bank        (B): 8    [30:32] (BA0 - BA2) -> 3  bit

3 3 3 | 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 | 1 1 1
2 1 0 | 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 | 2 1 0 9 8 7 6 5 4 3 | 2 1 0
B B B | R R R R R R R R R R R R R R R R R | C C C C C C C C C C | Y Y Y
```

The parameters for the address mapping just described are presented below. 

```
# Channel information.
num_ch = 1              # Number of channels
ch_shift = 34           # Shift to reach the frist bit reserved for channels in the address
ch_mask = 0x1           # Mask for all channel bits in the address

# Bank group information.
num_bank_groups = 1     # Number of bank groups
bgroup_shift = 33   	# Shift to reach the frist bit reserved for bank groups in the address
bgroup_mask = 0x1   	# Mask for all bits in the address related to bank groups

# Bank information
num_banks = 8           # Number of banks
bank_shift = 30         # Shift to reach the frist bit reserved for banks in the address
bank_mask = 0x7         # Mask for all bank bits in the address

# Row information
num_rows = 128 * 1024   # Number of rows
row_shift = 13          # Shift to reach the frist bit reserved for rows in the address
row_mask = 0x1ffff      # Mask for all row bits in the address

# Column information
num_col = 1 * 1024      # Number of columns
col_shift = 3           # Shift to reach the frist bit reserved for columns in the address
col_mask = 0x3ff        # Mask for all column bits in the address

# Burst length
burst_len = 8
```

Open the script with a text editor and change some parameters to fit your
needs.

#### DRAMsys Diagrams

- **TLM Approximately Timed (AT)**

  The figure below shows a cheat sheet with the possibilities that the TLM AT protocol
  offers. The annotated references [X,Y] are placed into the source code for a better
  orientation.

  ![TLM AT Cheat Sheet](DRAMSys/docs/images/tlmATCheatSheet.png)


- **Payload Extension information**

  GenerationExtension is added in TracePlayer and DramExtension is added in Arbiter.

  DramExtension indicates the decoded address (channel, bank, colums, row) and
  the socket id (thread) of a payload. It is added in the Arbiter and is sent
  to the Controller.
  ![Payload Extension information](DRAMSys/docs/images/PayloadExtension.png)

- **Transaction object with Memory Manager**

  The TracePlayer allocates the memory for the transaction object by calling allocatePayload method.

  The acquire method is called before passing the transaction object in TracePlayer, Arbiter and Controller.

  The release method is called after each component is done with the transaction object. After the final call of release method, the free method of the memory manager is called to free the transaction object.

  ![Payload Memory Manager](DRAMSys/docs/images/PayloadMemoryManager.png)

- **Architecture of the backend TLM model**

  The below figure shows our custom TLM protocol between the Controller and the Dram. A new transaction enters the Controller with the BEGIN_REQ phase is stored in frontendPEQ. The callback function of the frontendPEQ is called and send the payload to the Scheduler.

  The Scheduler checks the address of payload and the current state to determine proper command (Active, Precharge, Read or Write). Then the ControllerCore sends the payload with the corresponding phase (BEGIN_ACT, BEGIN_PRE, BEGIN_RD or BEGIN_WR) to the Dram by calling nb_transport_fw method.

  The Dram receives the transaction then send back to the Controller by calling nb_transport_bw with appropriate END phase (END_ACT, END_PRE, END_RD or END_WR).

  ![Architecture backend TLM](DRAMSys/docs/images/TransactionPhase.png)

### DRAMSys Thermal Simulation

The thermal simulation is performed by a **3D-ICE** [8] server accessed
through the network. Therefore users interested in thermal simulation during
their DRAMSys simulations need to make sure they have a 3D-ICE server up and
running before starting. For more information about 3D-ICE visit the [official website](http://esl.epfl.ch/3D-ICE).

#### Installing the lastest 3D-ICE version

[Download](https://www.epfl.ch/labs/esl/open-source-software-projects/3d-ice/3d-ice-download/) the lastest version. Make sure you got version 2.2.6 or greater:

```bash
$ wget https://www.epfl.ch/labs/esl/wp-content/uploads/2018/12/3d-ice-latest.zip
$ unzip 3d-ice-latest.zip
```

Install [SuperLU](http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_5.2.1.tar.gz) dependencies:

```bash
$ sudo apt-get install build-essential git bison flex libblas-dev
```

Download and install SuperLU:

```bash
$ wget http://crd.lbl.gov/~xiaoye/SuperLU/superlu_4.3.tar.gz
$ tar xvfz superlu_4.3.tar.gz
$ cd SuperLU_4.3/
$ cp MAKE_INC/make.linux make.inc
```

Make sure the SuperLUroot variable in ./make.inc is properly set. For example,
if you downloaded it to your home folder set as follows.

```bash
SuperLUroot = $(HOME)/SuperLU_4.3
```

Compile the library:

```bash
$ make superlulib
```

Download and install bison-2.4.1:

```bash
$ wget http://ftp.gnu.org/gnu/bison/bison-2.4.1.tar.gz
$ tar xvzf bison-2.4.1.tar.gz
$ cd bison-2.4.1
$ ./configure --program-suffix=-2.4.1
$ make
$ sudo make install
```

Go to the 3d-ice directory:

```bash
$ cd 3d-ice-2.2.6
```

Open the file makefile.def and set some variables. Set the correct path to the
SuperLU library you just compiled.

```bash
SLU_MAIN = $(HOME)/SuperLU_$(SLU_VERSION)
```

Set the YACC variable to bison-2.4.1:

```bash
YACC = bison-2.4.1
```

Set the following variables with proper values.

```bash
SYSTEMC_ARCH = linux64
SYSTEMC_MAIN = $(HOME)/systemc-2.3.1a
```

Compile 3D-ICE with SystemC TLM-2.0 support:

```bash
$ make clean
$ make SYSTEMC_WRAPPER=y
```

Users interested in thermal simulation can also add some extra environment
variables:

```bash
# Necessary for thermal simulation
export LIBTHREED_ICE_HOME=<path>
export LIBSUPERLU_HOME=<path>
```

#### Running DRAMSys with Thermal Simulation

Before starting make sure you have a **clean repository** without any previous
automatic generated Makefiles. One way to ensure this is by running the
command below inside your DRAMSys repository, but keep in mind that
**untracked files and directories will be removed** from the repository.

```bash
$ git clean -fdx
```

This feature can be enabled via an environment variable.

```bash
$ export THERMALSIM=true
$ qtcreator &
```

or

```bash
$ mkdir build
$ cd build
$ export THERMALSIM=true
$ qmake ../DRAMSys/DRAMSys.pro
$ make
```

Before starting DRAMSys it is necessary to run the 3D-ICE server passing to it
two arguments: a suitable configuration file and an Internet socket port
number. And then wait until the server is ready to receive requests.

```bash
$ 3D-ICE-Server <stack file> <port>
Preparing stk data ... done !
Preparing thermal data ... done !
Creating socket ... done !
Waiting for client ... done !
```

The IP address and the port number related to the server shall be informed in
DRAMSys' configuration to subsequent use by DRAMSys to access the thermal
simulation server.

#### Usage Example with Thermal Simulation

Generate the input trace file for DRAMSys.

```bash
$ cd DRAMSys/tests/error/
$ ./generateErrorTest.pl > test_error.stl
```

Start the 3D-ICE server providing the stack file and the port number.

```bash
$ cd DRAMSys/library/resources/configs/thermalsim
$ 3D-ICE-Server stack.stk 11880
```

In another terminal or terminal tab start DRAMSys. Here the program's output
is redirected to a file.

```bash
$ cd build/simulator/
$ ./DRAMSys > output
```

## DRAMSys with gem5

Install gem5 by following the instructions on the [gem5 wiki](http://gem5.org/Documentation#Getting_Started).
Optionally, use the scripts from [gem5.TnT] to install gem5, build it, get some benchmark programs and learn more about gem5.

In order to understand the SystemC coupling with gem5 it is recommended to
read the documentation in the gem5 repository *util/tlm/README* and [12].

The main steps for building gem5 and libgem5 follow:

```bash
scons build/ARM/gem5.opt
```

```bash
scons --with-cxx-config --without-python --without-tcmalloc build/ARM/libgem5_opt.so
```

For MacOS:

```bash
scons --with-cxx-config --without-python --without-tcmalloc build/ARM/libgem5_opt.dylib
```

In order to use gem5 with DRAMSys set the **GEM5** environment variable to the
path to gem5, for example in the *QtCreator under Projects > Build
& Run > Build Environment*:

```
GEM5=/path/to/gem5/
```

Example:

```
GEM5=$HOME/gem5_tnt/gem5
```

Optionally, export environment variables in your **~/.bashrc** file or
equivalent and open a new terminal:

```bash
# In this example gem5 is located at $HOME/gem5_tnt/gem5.
export GEM5=$HOME/gem5_tnt/gem5

# Add the folder containing libgem5_opt.so to the list where libraries should
# be searched for.
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${GEM5}/build/ARM
```

The project file [DRAMSys.pro](DRAMSys/DRAMSys.pro) checks whether the
environment variable **GEM5** is defined or not and configures automatically
the corresponding build target **gem5** for QtCreator.

In order to run gem5 with DRAMSys it is mandatory to run gem5 first without
DRAMSys and generate a configuration file **config.ini** which will be the
value of the second parameter passed to DRAMSys_gem5.

### DRAMSys with gem5 traffic generator

In the following we will run a simple example with a gem5 traffic generator:

```
Base System Architecture:
+-------------+ +------+    ^
| System Port | | TGEN |    |
+-------+-----+ +--+---+    |
        |           |       | gem5 World
        |      +----+       |
        |      |            |
+-------v------v-------+    |
|        Membus        |    v
+---------------+------+    External Port (see sc_slave_port.*)
                |           ^
           +----v----+      | TLM World
           | DRAMSys |      | (see sc_target.*)
           +---------+      v

```

As mentioned before we first need to create a config.ini
that represents the gem5 configuration. We do so by starting gem5 with the
desired python configuration script.

```bash
cd gem5/utils/tlm/
../../build/ARM/gem5.opt conf/tlm_slave.py
```

**Ignore the message below.**
```
"fatal: Can't find port handler type 'tlm_slave'"
```

The configuration file config.ini will be stored in the **m5out** directory.
Copy this configuration file to the building directory of DRAMSys where the
executable **DRAMSys_gem5** is located:

```
dram.sys/build-DRAMSys-Desktop_Qt_5_7_0_clang_64bit-Debug/gem5
```

Also the traffic generatior configuration file (conf/tgen.cfg) must be stored
in a conf directory of this building directory.

Then the simulation can be started with:

```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-example.json config.ini 1
```

Let the simulation run for some seconds and then stop it with **CTRL-C**.
Observe the output of the simulation in the trace analyzer. The trace database
can be found inside the gem5 directory in the building directory.

### Gem5 SE mode and DRAMSys

All essential files for some functional examples are provided.

Execute a hello world application:

```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/configs/hello.ini 1
```

A **Hello world!** message should be printed to the standard output.

Execute applications:

```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/Oscar/config.ini 1
```

```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/Bubblesort/config.ini 1
```

Wait some minutes for the application to finish.

The hello application binary was copied from gem5 repository.

Other applications were obtained with [gem5.TnT].

Command template for generating **.ini** configuration files follows:

```bash
build/ARM/gem5.opt configs/example/se.py \
  -c <application> --mem-size=512MB --mem-channels=1 \
  --caches --l2cache --mem-type=SimpleMemory \
  --cpu-type=TimingSimpleCPU --num-cpu=1 \
  --tlm-memory=transactor
```

An overview of the architcture being simulated is presented below:

![arch](DRAMSys/docs/images/gem5_se_mode_arch.png)

**Note**: this is a gem5 generated file, therefore DRAMSys is omitted. DRAMSys is
direct connected as external tlm slave.

**Note**: workaround in se.py required:

```python
...
if options.tlm_memory:
    system.physmem = SimpleMemory()
MemConfig.config_mem(options, system)
...
```

A convenience script to execute several applications automatically
[**run.sh**](DRAMSys/gem5/gem5_se/run.sh) is provided . Take a look and learn
from it.

### [PARSEC] FS Mode

Full system simulation files for ARM available in [DRAMSys/gem5/gem5_fs/parsec_arm_minor_2c_8GB](DRAMSys/gem5/gem5_fs/parsec_arm_minor_2c_8GB).

Choose the benchmark in [parsec_arm_minor_2c_8GB.rcS](DRAMSys/gem5/gem5_fs/parsec_arm_minor_2c_8GB/parsec_arm_minor_2c_8GB.rcS).

Edit the paths in [config.ini](DRAMSys/gem5/gem5_fs/parsec_arm_minor_2c_8GB/config.ini).

All files required to build DRAMSys_gem5 and execute the simulation (gem5
library, benchmarks, disk image, etc.) can be obtained with [gem5.TnT].

Start a simulation. Example:

```bash
dram.sys/build/gem5$ ./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/rgrsim-gem5-fs.json ../../DRAMSys/gem5/gem5_fs/parsec_arm_minor_2c_8GB/config.ini 1
```

Optionally, open another terminal or tab and connect to gem5.

```bash
$ telnet localhost 3456
```

Note: the port may vary, gem5 prints it during initialization. Example:

```
system.terminal: Listening for connections on port 3456
```

### [PARSEC] SE Mode


Binaries and gem5 SE configuration files for ARM available in [DRAMSys/gem5/gem5_se/parsec-arm](DRAMSys/gem5/gem5_se/parsec-arm).

Use [gem5.TnT] to download parsec. Example:

Go to your **gem5.TnT** folder. Then go to **arch/arm** folder. Execute the
script *build-parsec-serial.sh*.

```bash
gem5.TnT/arch/arm$ ./build-parsec-serial.sh 
```

Extract inputs files. Example:

```bash
cd $HOME/gem5_tnt/benchmarks/parsec-3.0/pkgs/kernels/canneal/inputs
tar -xf input_simdev.tar
tar -xf input_test.tar
tar -xf input_simmedium.tar
tar -xf input_simsmall.tar
tar -xf input_native.tar
tar -xf input_simlarge.tar

cd $HOME/gem5_tnt/benchmarks/parsec-3.0/pkgs/apps/fluidanimate/inputs
tar -xf input_simdev.tar
tar -xf input_test.tar
tar -xf input_native.tar
tar -xf input_simlarge.tar
tar -xf input_simmedium.tar
tar -xf input_simsmall.tar

cd $HOME/gem5_tnt/benchmarks/parsec-3.0/pkgs/apps/blackscholes/inputs
tar -xf input_simdev.tar
tar -xf input_test.tar
tar -xf input_native.tar
tar -xf input_simlarge.tar
tar -xf input_simmedium.tar
tar -xf input_simsmall.tar
```

Open [DRAMSys/gem5/gem5_se/parsec-arm/config.ini](DRAMSys/gem5/gem5_se/parsec-arm/config.ini)

Edit **cmd=**.

Edit **executable=**.

Examples (**Replace USER. Use the correct path in your computer.**):

```
-- canneal --

cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/canneal/canneal 1 5 100 /home/USER/gem5_tnt/benchmarks/parsec-3.0/pkgs/kernels/canneal/inputs/10.nets 1
cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/canneal/canneal 1 100 300 /home/USER/gem5_tnt/benchmarks/parsec-3.0/pkgs/kernels/canneal/inputs/100.nets 2
cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/canneal/canneal 1 10000 2000 /home/USER/gem5_tnt/benchmarks/parsec-3.0/pkgs/kernels/canneal/inputs/100000.nets 32
cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/canneal/canneal 1 15000 2000 /home/USER/gem5_tnt/benchmarks/parsec-3.0/pkgs/kernels/canneal/inputs/200000.nets 64
cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/canneal/canneal 1 15000 2000 /home/USER/gem5_tnt/benchmarks/parsec-3.0/pkgs/kernels/canneal/inputs/400000.nets 128

executable=../../DRAMSys/gem5/gem5_se/parsec-arm/canneal/canneal

-- streamcluster --

cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/streamcluster/streamcluster 2 5 1 10 10 5 none output.txt 1
cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/streamcluster/streamcluster 3 10 3 16 16 10 none output.txt 1
cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/streamcluster/streamcluster 10 20 32 4096 4096 1000 none output.txt 1
cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/streamcluster/streamcluster 10 20 64 8192 8192 1000 none output.txt 1
cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/streamcluster/streamcluster 10 20 128 16384 16384 1000 none output.txt 1
cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/streamcluster/streamcluster 10 20 128 1000000 200000 5000 none output.txt 1

executable=../../DRAMSys/gem5/gem5_se/parsec-arm/streamcluster/streamcluster

-- swaptions --

cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/swaptions/swaptions -ns 1 -sm 5 -nt 1
cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/swaptions/swaptions -ns 3 -sm 50 -nt 1
cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/swaptions/swaptions -ns 16 -sm 5000 -nt 1
cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/swaptions/swaptions -ns 32 -sm 10000 -nt 1
cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/swaptions/swaptions -ns 64 -sm 20000 -nt 1

executable=../../DRAMSys/gem5/gem5_se/parsec-arm/swaptions/swaptions

-- fluidanimate --

cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/fluidanimate/fluidanimate 1 1 /home/USER/gem5_tnt/benchmarks/parsec-3.0/pkgs/apps/fluidanimate/inputs/in_5K.fluid out.fluid
cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/fluidanimate/fluidanimate 1 3 /home/USER/gem5_tnt/benchmarks/parsec-3.0/pkgs/apps/fluidanimate/inputs/in_15K.fluid out.fluid
cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/fluidanimate/fluidanimate 1 5 /home/USER/gem5_tnt/benchmarks/parsec-3.0/pkgs/apps/fluidanimate/inputs/in_35K.fluid out.fluid
cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/fluidanimate/fluidanimate 1 5 /home/USER/gem5_tnt/benchmarks/parsec-3.0/pkgs/apps/fluidanimate/inputs/in_100K.fluid out.fluid
cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/fluidanimate/fluidanimate 1 5 /home/USER/gem5_tnt/benchmarks/parsec-3.0/pkgs/apps/fluidanimate/inputs/in_300K.fluid out.fluid

executable=../../DRAMSys/gem5/gem5_se/parsec-arm/fluidanimate/fluidanimate

-- blackscholes --

cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/blackscholes/blackscholes 1 /home/USER/gem5_tnt/benchmarks/parsec-3.0/pkgs/apps/blackscholes/inputs/in_4.txt prices.txt
cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/blackscholes/blackscholes 1 /home/USER/gem5_tnt/benchmarks/parsec-3.0/pkgs/apps/blackscholes/inputs/in_16.txt prices.txt
cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/blackscholes/blackscholes 1 /home/USER/gem5_tnt/benchmarks/parsec-3.0/pkgs/apps/blackscholes/inputs/in_4K.txt prices.txt
cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/blackscholes/blackscholes 1 /home/USER/gem5_tnt/benchmarks/parsec-3.0/pkgs/apps/blackscholes/inputs/in_16K.txt prices.txt
cmd=../../DRAMSys/gem5/gem5_se/parsec-arm/blackscholes/blackscholes 1 /home/USER/gem5_tnt/benchmarks/parsec-3.0/pkgs/apps/blackscholes/inputs/in_64K.txt prices.txt

executable=../../DRAMSys/gem5/gem5_se/parsec-arm/blackscholes/blackscholes

```

Start a simulation. Example:

```bash
dram.sys/build/gem5$ ./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/rgrsim-gem5-se.json ../../DRAMSys/gem5/gem5_se/parsec-arm/config.ini 1
```

### Boot Linux with gem5 and DRAMSys

The procedure is very similar to the traffic generator example above.

First we have to generate the config.ini file by starting gem5 with the following configuration:

```bash
build/ARM/gem5.opt configs/example/fs.py \
  --tlm-memory=transactor --cpu-type=TimingSimpleCPU --num-cpu=1 \
  --mem-type=SimpleMemory --mem-size=512MB --mem-channels=1 --caches \
  --l2cache --machine-type=VExpress_EMM \
  --dtb-filename=vexpress.aarch32.ll_20131205.0-gem5.1cpu.dtb \
  --kernel=vmlinux.aarch32.ll_20131205.0-gem5 \
  --disk-image=linux-aarch32-ael.img
```

The config.ini should be copied again to the DRAMSys_gem5 build folder.

The simconfig should be changed in order to support storage and address offsets:

``` json
{
    "simconfig": {
        "CheckTLM2Protocol": false,
        "DatabaseRecording": true,
        "Debug": false,
        "ECCControllerMode": "Disabled",
        "EnableWindowing": false,
        "ErrorCSVFile": "",
        "ErrorChipSeed": 42,
        "NumberOfDevicesOnDIMM": 8,
        "NumberOfMemChannels": 1,
        "PowerAnalysis": false,
        "SimulationName": "ddr3",
        "SimulationProgressBar": true,
        "ThermalSimulation": false,
        "WindowSize": 1000,

        "StoreMode": "Store",
        "AddressOffset": 2147483648,
        "UseMalloc": true
    }
}
```

Then start DRAMSys_gem5 with the following command:

```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-example.json config.ini 1
```

For further sophisticated address mappings or scenarios checkout the file DRAMSys/gem5/main.cpp

#### Boot Linux with gem5 and DRAMSys Example

**All essential files for a functional example are provided.**

Unzip the disk image:

```bash
tar -xaf DRAMSys/gem5/boot_linux/linux-aarch32-ael.img.tar.gz -C DRAMSys/gem5/boot_linux/
```

Execute the example:

```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-boot-linux.json ../../DRAMSys/gem5/configs/boot_linux.ini 1
```

Open a new terminal and connect to gem5:

```bash
telnet localhost 3456
```

Wait some minutes for the Linux boot process to complete then login. Username is
**root** no password required.


### DRAMSys with gem5 Elastic Traces

For understanding elastic traces and their generation, study the [gem5
wiki](http://gem5.org/TraceCPU) and the paper [13].
Some predefined configs are stored [here](DRAMSys/gem5/configs) and the related
python files are stored [here](DRAMSys/gem5/examples).

This is an example for running an elastic trace:

```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-example.json ../../DRAMSys/gem5/configs/singleElasticTraceReplay.ini 1
```

An overview of the architcture being simulated is presented below:

![arch](DRAMSys/docs/images/singleElasticTraceReplay.png)

Note that the address offset is usually zero for elastic traces.

Another example with L2 cache:

```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-example.json ../../DRAMSys/gem5/configs/singleElasticTraceReplayWithL2.ini 1
```

If two elastic traces should be used run the simulation with the following example:

```
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-example.json ../../DRAMSys/gem5/configs/dualElasticTraceReplay.ini 2
```

An overview of the architcture being simulated is presented below:

![arch](DRAMSys/docs/images/dualElasticTraceReplay.png)

For more spophisticated setups, even with l2 caches the proper ini file should be created.
If you need help please contact Matthias Jung.

### DRAMSys + GEM5 Log Collector Scripts

Users can profit of running multiple **DRAMSys + gem5** simulations
automatically with [gem5ilva.sh] for **gem5 syscall emulation (SE) mode** and
[gem5ilva_fs.sh] for **gem5 full system (FS) mode**.

Normally you will have to push your changes before running the scripts. This
approach makes it easier to track back what exactly was tested by the scripts.

The scripts provide variables that tell **git** where to get the source
code from (repository URL), user name to be used (your git account),
**branch** to checkout (your working branch), etc. They are:

```bash
# Git info.
git_user="$USER"
git_branch="master"
git_url="git.eit.uni-kl.de:ems/astdm/dram.sys.git"
git_url_https="git.eit.uni-kl.de/ems/astdm/dram.sys.git"
```

The default values of the variables presented above assume that your git
account uses the same name as your user name in your PC. If that is not the
case, replace the value of the **git_user** variable with your git account
name. Similarly, replace the value of the variable **git_branch** with your
working branch name. There (in your working branch) you can push your changes
and/or new files before executing the scripts.

Open the script in QtCreator or another text editor of your choice and set the
variables with values that fit your needs.

Nevertheless, for some cases, you may want to have gem5 essential files out of
the main repository (usually because they are too big to be added to the
repository).

For those cases uncomment and properly set the variable
**external_inifile_path** in [gem5ilva_fs.sh].

This allows you to use a gem5 **config.ini** file external to the repository.
Note, however, that in this case it is up to you to keep track of your
simulation setup.

**Hint:**
[gem5.TnT] provides convenience scripts
to create gem5 disk images with benchmarking programs embedded.

### Notes for [Elwetritsch] Users

Firstly, take a look at [High Performance Computing at the TU Kaiserslautern](https://elwe.rhrk.uni-kl.de/).

After that, please give yourself a change to learn a bit about [Batch Usage at
RHRK TU Kaiserslautern](https://elwe.rhrk.uni-kl.de/elwetritsch/batch.shtml).
This will probably save you some time later on.

When using DRAMSys + gem5 on the [Elwetritsch] gem5 can be installed with
convenience scripts provided by [gem5.TnT].

[gem5.TnT] also provides convenience scripts
to create gem5 disk images with benchmarking programs embedded. The creation
of disk images for gem5 requires superuser privilege. A solution is to copy
(e.g., using scp or mounting a folder, etc.) the locally created disk images
to [Elwetritsch]. Since there is no compilation involved, copying disk images
created in one machine to another machine should not incur in incompatibility
problems.

On [gem5.TnT] repository open a [gem5.TnT] config file.

```bash
$ vim common/defaults.in
```

Note the variable **ROOTDIR**. Its default value is *ROOTDIR=$HOME/gem5_tnt*.
That means that [gem5.TnT] will download to *$HOME/gem5_tnt*.

Currently the space one can use in its Elwetrich *$HOME* folder is limited to
a few tens of GiB. Nevertheless, a directory **/scratch/$USER** is provided
with less space restrictions.

One can create a symlink pointing to **/scratch/$USER/gem5_tnt**.

```bash
$ cd $SCRATCH
$ mkdir gem5_tnt
$ cd
$ ln -s /scratch/$USER/gem5_tnt
```

On [gem5.TnT] repository use the commands below to get files and build gem5: 

```bash
$ ./get_essential_fs.sh
$ ./get_benchmarks.sh
$ ./get_extra_fs.sh
$ ./build_gem5.sh
```

To get DRAMSys installed follow the traditional setup instructions described
in this document.

For building DRAMSys one can profit from using [DRAMSylva.sh] which loads the
modules that are necessary for building DRAMSys on [Elwetritsch].

Regarding dependencies for building DRAMSys and DRAMSys + gem5, the scripts
provided inside the [DRAMSylva folder], when running on [Elwetritsch], will
load the required modules automatically.

As usual, one may export environment variables from his/her **~/.bashrc** file
on Elwetritch. Some segments extracted from a functional ~/.bashrc file are
presented below to be used as reference. Note that you may have to adapt it,
for example, changing paths to point to the place you installed some of the
libraries.

```bash
# User specific aliases and functions
# SystemC home
export SYSTEMC_HOME=$HOME/systemc-2.3.1a
# SystemC target architecture
export SYSTEMC_TARGET_ARCH=linux64

# Qwt lib
export LIBQWT_HOME=$HOME/qwt-6.1/lib
export LIBQWT_HEADERS=$HOME/qwt-6.1/src
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH:+${LD_LIBRARY_PATH}:}$LIBQWT_HOME

# Python lib
export LIBPYTHON_VERSION="3.6m"
export PYTHON_HOME=/usr/lib64
export PYTHON_HEADERS=/usr/include/python3.6m

# Gem5 + DRAMsys
export GEM5=$HOME/gem5_tnt/gem5

# Gem5 SystemC TLM-2.0 coupling (see also: $HOME/gem5_tnt/gem5/util/tlm/README)
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${GEM5}/build/ARM
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${SYSTEMC_HOME}/lib-$SYSTEMC_TARGET_ARCH
export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:${SYSTEMC_HOME}/lib-$SYSTEMC_TARGET_ARCH/pkgconfig

# M5_PATH for gem5
export M5_PATH=$HOME/gem5_tnt/full_system/arm/aarch-system-20180409

# Do not close my terminal when inactive after a timeout
unset TMOUT
```

[SLURM](https://slurm.schedmd.com/overview.html) **job scripts** are available
inside the [DRAMSylva folder]. They can be used directly without changes or as
examples on how to start jobs using nodes of the [Elwetritsch] cluster. Of
course, one can create his/her own job scripts.


### Coverage Check

Coverage check is enabled by default and can be disabled with an environment
variable.

```bash
export DRAMSYS_DISABLE_COVERAGE_CHECK=1
```

### DRAMSys + GEM5 x86

Make sure you have built **gem5/build/X86/libgem5_opt.so**. If you build with
[gem5.TnT] you can check if the library exists as follows.

```bash
$ ls $HOME/gem5_tnt/gem5/build/X86/libgem5_opt.so
```

Change your ~/.bashrc.

```bash
# In this example gem5 is located at $HOME/gem5_tnt/gem5.
export GEM5=$HOME/gem5_tnt/gem5

# Add the folder containing libgem5_opt.so to the list where libraries should
# be searched for.
#export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${GEM5}/build/ARM
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${GEM5}/build/X86
```

After that close QtCreator and all terminals.

Open a new terminal.

Change the architecture in [DRAMSys/gem5/gem5.pro](DRAMSys/gem5/gem5.pro).

```
gem5_arch = 'X86'
```

Delete the file **DRAMSys/DRAMSys.pro.user** from the repository.

```bash
$ rm DRAMSys/DRAMSys.pro.user
```

Open a new QtCreator.

Build DRAMSys as usual.

After building, go the the folder where *DRAMSys_gem5* is located.

Test with a hello world application for X86.

```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/hello-x86/config.ini 1
```

A **Hello world!** message should be printed to the standard output.

### [MiBench]

Applications for x86 and configuration files available in [DRAMSys/gem5/gem5_se/MiBench](DRAMSys/gem5/gem5_se/MiBench).

Examples:

**Automotive Applications**

**Basicmath**
```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/automotive/basicmath/small/config.ini 1
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/automotive/basicmath/large/config.ini 1
```

**Bitcount**
```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/automotive/bitcount/small/config.ini 1
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/automotive/bitcount/large/config.ini 1
```

**Qsort**
```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/automotive/qsort/small/config.ini 1
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/automotive/qsort/large/config.ini 1
```

**Susan**
```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/automotive/susan/small/corners/config.ini 1
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/automotive/susan/large/corners/config.ini 1

./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/automotive/susan/small/edges/config.ini 1
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/automotive/susan/large/edges/config.ini 1

./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/automotive/susan/small/smoothing/config.ini 1
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/automotive/susan/large/smoothing/config.ini 1
```

**Network Applications**

**Dijkstra**
```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/network/dijkstra/small/config.ini 1
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/network/dijkstra/large/config.ini 1
```

**Patricia**
```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/network/patricia/small/config.ini 1
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/network/patricia/large/config.ini 1
```

**Security Applications**

**Blowfish Encode**
```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/security/blowfish/encode/small/config.ini 1
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/security/blowfish/encode/large/config.ini 1
```

**Blowfish Decode**
```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/security/blowfish/decode/small/config.ini 1
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/security/blowfish/decode/large/config.ini 1
```

**SHA**
```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/security/sha/small/config.ini 1
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/security/sha/large/config.ini 1
```

**Telecom Applications**

**CRC32**
```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/telecomm/crc32/small/config.ini 1
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/telecomm/crc32/large/config.ini 1
```

**FFT**
```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/telecomm/fft/small/config.ini 1
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/telecomm/fft/large/config.ini 1
```

**FFT-INV**
```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/telecomm/fft-inv/small/config.ini 1
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/telecomm/fft-inv/large/config.ini 1
```

**GSM Encode**
```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/telecomm/gsm/encode/small/config.ini 1
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/telecomm/gsm/encode/large/config.ini 1
```

**GSM Decode**
```bash
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/telecomm/gsm/decode/small/config.ini 1
./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/ddr3-gem5-se.json ../../DRAMSys/gem5/gem5_se/MiBench/telecomm/gsm/decode/large/config.ini 1
```

Check the folder [DRAMSys/gem5/gem5_se/MiBench](DRAMSys/gem5/gem5_se/MiBench) for all applications and configuration files.

### More AARCH64 Apps

Full system simulation files for ARM available in [DRAMSys/gem5/gem5_fs/arm64](DRAMSys/gem5/gem5_fs/arm64).

You can edit [arm64.rcS](DRAMSys/gem5/gem5_fs/arm64/arm64.rcS) to start an application and call *m5 exit* when it finishes.

Edit the paths in [config.ini](DRAMSys/gem5/gem5_fs/arm64/config.ini).

All files required to build DRAMSys_gem5 and execute the simulation (gem5
library, benchmarks, disk image, etc.) can be obtained with [gem5.TnT].

Start a simulation. Example:

```bash
dram.sys/build/gem5$ ./DRAMSys_gem5 ../../DRAMSys/library/resources/simulations/rgrsim-gem5-fs.json ../../DRAMSys/gem5/gem5_fs/arm64/config.ini 1
```

Optionally, open another terminal or tab and connect to gem5.

```bash
$ telnet localhost 3456
```

Note: the port may vary, gem5 prints it during initialization. Example:

```
system.terminal: Listening for connections on port 3456
```

## References

[1] TLM Modelling of 3D Stacked Wide I/O DRAM Subsystems, A Virtual Platform for Memory Controller Design Space Exploration
M. Jung, C. Weis, N. Wehn, K. Chandrasekar. International Conference on High-Performance and Embedded Architectures and Compilers 2013 (HiPEAC), Workshop on: Rapid Simulation and Performance Evaluation: Methods and Tools (RAPIDO), January, 2013, Berlin.

[2] DRAMPower: Open-source DRAM Power & Energy Estimation Tool
Karthik Chandrasekar, Christian Weis, Yonghui Li, Sven Goossens, Matthias Jung, Omar Naji, Benny Akesson, Norbert Wehn, and Kees Goossens
URL: http://www.drampower.info

[3] Energy Optimization in 3D MPSoCs with Wide-I/O DRAM
M. Sadri, M. Jung, C. Weis, N. Wehn, L. Benini. Conference Design, Automation and Test in Europe (DATE), March, 2014, Dresden, Germany.

[4] DRAMSys: A flexible DRAM Subsystem Design Space Exploration Framework
M. Jung, C. Weis, N. Wehn. Accepted for publication, IPSJ Transactions on System LSI Design Methodology (T-SLDM), October, 2015.

[5] Optimized Active and Power-Down Mode Refresh Control in 3D-DRAMs
M. Jung, M. Sadri, C. Weis, N. Wehn, L. Benini., VLSI-SoC, October, 2014, Playa del Carmen, Mexico.

[6] Retention Time Measurements and Modelling of Bit Error Rates of WIDE-I/O DRAM in MPSoCs
C. Weis, M. Jung, P. Ehses, C. Santos, P. Vivet, S. Goossens, M. Koedam, N. Wehn. Accepted for publication, IEEE Conference Design, Automation and Test in Europe (DATE), March, 2015, Grenoble, France

[7] http://www.uni-kl.de/3d-dram/publications/

[8] A Sridhar, A Vincenzi, D Atienza, T Brunschwiler, 3D-ICE: a compact
thermal model for early-stage design of liquid-cooled ICs, IEEE Transactions
on Computers (TC 2013, accepted for publication).

[9] A Sridhar, A Vincenzi, M Ruggiero, T Brunschwiler, D Atienza, 3D-ICE: Fast
compact transient thermal modeling for 3D-ICs with inter-tier liquid cooling,
Proceedings of the 2010 International Conference on Computer-Aided Design
(ICCAD 2010), San Jose, CA, USA, November 7-11 2010.

[10] A Sridhar, A Vincenzi, M Ruggiero, T Brunschwiler, D Atienza, Compact
transient thermal model for 3D ICs with liquid cooling via enhanced heat
transfer cavity geometries, Proceedings of the 16th International Workshop on
Thermal Investigations of ICs and Systems (THERMINIC'10), Barcelona, Spain,
6-8 October, 2010.

[11] http://esl.epfl.ch/3D-ICE

[12] System Simulation with gem5 and SystemC: The Keystone for Full
Interoperability C. Menard, M. Jung, J. Castrillon, N. Wehn. IEEE International
Conference on Embedded Computer Systems Architectures Modeling and Simulation
(SAMOS), July, 2017, Samos Island, Greece.

[13] Exploring System Performance using Elastic Traces: Fast, Accurate and
Portable Radhika Jagtap, Stephan Diestelhorst, Andreas Hansson, Matthias Jung
and Norbert Wehn, IEEE International Conference on Embedded Computer Systems
Architectures Modeling and Simulation (SAMOS), 2016, Samos Island, Greece.

[gem5.TnT]: https://github.com/tukl-msd/gem5.TnT
[gem5ilva.sh]: DRAMSys/library/resources/scripts/DRAMSylva/gem5ilva.sh
[gem5ilva_fs.sh]: DRAMSys/library/resources/scripts/DRAMSylva/gem5ilva_fs.sh
[Elwetritsch]: https://elwe.rhrk.uni-kl.de/
[DRAMSylva.sh]: DRAMSys/library/resources/scripts/DRAMSylva/DRAMSylva.sh
[DRAMSylva folder]: DRAMSys/library/resources/scripts/DRAMSylva
[configs_json]: DRAMSys/library/resources/scripts/DRAMSylva/configs_json
[MiBench]: http://vhosts.eecs.umich.edu/mibench/
[PARSEC]: http://parsec.cs.princeton.edu/